tenderpilot/EMAIL_DIGEST.md

TenderRadar Email Digest System

Overview

The email digest system automatically sends matched tender alerts to subscribed users. The system matches new tenders published in the last 24 hours against each user's saved preferences (keywords, sectors, value ranges, locations, authority types) and delivers personalized HTML email digests daily.

Architecture

Components

1. Database Schema: Uses existing profiles and matches tables

   • profiles table stores user alert preferences
   • matches table tracks which tenders have been sent to which users

2. Email Digest Script: /scripts/send-digest.js

   • Daily script that finds matching tenders for each user
   • Sends professional HTML emails with tender details
   • Marks matches as sent to avoid duplicates
   • Supports dry-run mode for testing

3. API Endpoints: Alert preference management

   • GET /api/alerts/preferences - Retrieve user's alert settings
   • POST /api/alerts/preferences - Create/update alert settings

4. Cron Job: Scheduled daily execution at 7am UTC

   • Configured in /etc/cron.d/ (or user crontab)
   • Logs to /home/peter/tenderpilot/digest.log

Database Schema

Profiles Table

Used to store user alert preferences. Required columns:

• id: Primary key
• user_id: Foreign key to users table (unique constraint)
• keywords: TEXT[] - Array of keywords to match in tender title/description
• sectors: TEXT[] - Array of sector/CPV codes
• min_value: DECIMAL - Minimum tender value (GBP)
• max_value: DECIMAL - Maximum tender value (GBP)
• locations: TEXT[] - Array of location filters
• authority_types: TEXT[] - Array of authority type filters
• created_at: TIMESTAMP - Record creation time
• updated_at: TIMESTAMP - Last update time

Matches Table

Tracks which tenders have been sent to which users.

• id: Primary key
• user_id: Foreign key to users table
• tender_id: Foreign key to tenders table
• sent: BOOLEAN - Whether email was sent
• created_at: TIMESTAMP
• Unique constraint on (user_id, tender_id)

Configuration

Environment Variables

Add to .env:

# Email Digest Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=alerts@tenderradar.co.uk
SMTP_PASS=your-email-password
SMTP_FROM=TenderRadar Alerts <alerts@tenderradar.co.uk>

Note: Use placeholder values initially. Peter will update with production credentials.

SMTP Options

The script supports any SMTP provider. Common configurations:

Gmail:

• Host: smtp.gmail.com
• Port: 587 (TLS) or 465 (SSL)
• User: Your Gmail address
• Pass: App-specific password (generate in Google Account settings)

Office 365:

• Host: smtp.office365.com
• Port: 587 (TLS)
• User: Your Office 365 email
• Pass: Your Office 365 password

Generic SMTP:

• Update SMTP_HOST and SMTP_PORT accordingly

Usage

Running the Digest Script

Dry-run (test mode - no emails sent):

cd /home/peter/tenderpilot
node scripts/send-digest.js --dry-run

Output shows which users have matches and how many tenders match their preferences.

Production (sends emails):

cd /home/peter/tenderpilot
node scripts/send-digest.js

Automatic Execution

The cron job runs daily at 7am UTC:

0 7 * * * cd /home/peter/tenderpilot && node scripts/send-digest.js >> /home/peter/tenderpilot/digest.log 2>&1

View logs:

tail -f /home/peter/tenderpilot/digest.log

Manual Cron Management

View existing cron jobs:

crontab -l

Edit cron jobs:

crontab -e

Remove a cron job:

crontab -r

API Endpoints

GET /api/alerts/preferences

Retrieve the current user's alert preferences.

Request:

curl -H "Authorization: Bearer <token>" \
  http://localhost:3456/api/alerts/preferences

Response:

{
  "preferences": {
    "id": 1,
    "user_id": 5,
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"],
    "created_at": "2026-02-14T12:00:00Z",
    "updated_at": "2026-02-14T12:00:00Z"
  }
}

Or if no preferences exist:

{
  "preferences": null
}

POST /api/alerts/preferences

Create or update alert preferences for the current user.

Request:

curl -X POST -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000", "71000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"]
  }' \
  http://localhost:3456/api/alerts/preferences

Parameters:

• keywords (array, optional): Keywords to match in tender title/description
• sectors (array, optional): CPV codes or sector categories
• min_value (number, optional): Minimum tender value (GBP)
• max_value (number, optional): Maximum tender value (GBP)
• locations (array, optional): Geographic locations
• authority_types (array, optional): Types of procuring authority

Response:

{
  "preferences": {
    "id": 1,
    "user_id": 5,
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000", "71000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"],
    "created_at": "2026-02-14T12:00:00Z",
    "updated_at": "2026-02-14T12:30:00Z"
  },
  "message": "Alert preferences updated successfully"
}

Email Template

The digest sends professional HTML emails with:

• TenderRadar branding header
• Summary of matched tenders count
• Table showing:
  • Tender title and source
  • Tender deadline
  • Estimated value (£)
  • Link to tender details
• Call-to-action button to dashboard
• Link to manage preferences
• TenderRadar footer with unsubscribe link

Emails are fully HTML-formatted with responsive design suitable for desktop and mobile clients.

Matching Algorithm

For each user, the script:

1. Fetches all open tenders published in the last 24 hours
2. Filters out tenders already sent to the user (using matches table)
3. Applies user preference filters:
   • Keywords: Matches against tender title + description (case-insensitive)
   • Value Range: Filters by tender value_high >= min_value AND value_low <= max_value
   • Locations: Matches against tender location
   • Authority Types: Matches against tender authority_type
   • Sectors: Matches CPV codes
4. For matching tenders:
   • Generates personalized HTML email
   • Sends via SMTP
   • Creates match record with sent=true

Note: All filters are AND conditions. If keywords are specified AND location is specified, tender must match BOTH criteria.

Troubleshooting

No emails being sent

1. Check that users have verified emails: SELECT * FROM users WHERE verified = true;
2. Check that users have preferences: SELECT * FROM profiles;
3. Check that tenders exist: SELECT * FROM tenders WHERE status = 'open' ORDER BY published_date DESC LIMIT 5;
4. Run dry-run to see matching logic: node scripts/send-digest.js --dry-run
5. Check logs: tail -100 digest.log

SMTP connection errors

1. Verify credentials in .env
2. Test SMTP connection manually (can use tools like telnet or nc)
3. Check firewall/network: ensure port is open outbound to SMTP server
4. For Gmail: ensure "Less secure apps" is enabled or use App Password
5. Check SMTP logs on server

Emails stuck in queue

1. Check node process: ps aux | grep node
2. Check for zombie processes: ps aux | grep defunct
3. View recent logs: tail -50 digest.log
4. Run script manually to see real-time errors

Database connection issues

1. Verify DATABASE_URL in .env
2. Test connection: psql $DATABASE_URL -c "SELECT 1"
3. Check database is running: sudo systemctl status postgresql
4. Check user permissions: SELECT grantee, privilege_type FROM information_schema.role_table_grants WHERE table_name='profiles';

Monitoring

Key Metrics to Monitor

1. Digest runs: Check cron execution with grep send-digest digest.log
2. Email send rate: Count successful sends in logs
3. Match rate: Ratio of tenders matched vs. users with preferences
4. Error rate: Failed SMTP connections or database queries
5. Database size: As matches table grows, consider archival/cleanup

Maintenance

Weekly:

• Review logs for errors: grep -i "error\|failed" digest.log
• Check disk space for logs: du -sh digest.log

Monthly:

• Archive old logs: gzip digest.log.* && mv digest.log.*.gz /archive/
• Verify cron job is still scheduled: crontab -l
• Test dry-run to ensure system is functional

Quarterly:

• Review SMTP provider limits and usage
• Check for database performance issues with large matches table
• Consider implementing match archival for old records

Future Enhancements

Potential improvements to the system:

1. Frequency Options: Allow users to choose daily/weekly/instant digests
2. Digest Format: Support plain text alternative to HTML
3. Unsubscribe: Track unsubscribe preferences
4. Match Scoring: Rank matches by relevance to user preferences
5. Batch Sending: Use queue system (Bull, Bee-Queue) for high volume
6. Analytics: Track open rates, click rates, conversion
7. A/B Testing: Test different email templates
8. Timezone Support: Send at user's local time, not UTC
9. Webhook Delivery: Alternative to SMTP for certain providers
10. Digest Personalization: Include user name, company, custom messages

File Structure

/home/peter/tenderpilot/
├── scripts/
│   └── send-digest.js          # Main digest script
├── server.js                    # Updated with alert API endpoints
├── .env                         # Config (includes SMTP settings)
├── package.json                 # Updated with nodemailer dependency
├── digest.log                   # Output log (created on first run)
└── init-db.js                   # Database initialization

Dependencies

Required npm packages:

• nodemailer: ^6.x - SMTP email sending
• pg: ^8.x - PostgreSQL client
• dotenv: ^16.x - Environment variable loading

All dependencies are already in package.json and node_modules.

Support & Questions

For issues or questions about the email digest system:

1. Check logs: tail -100 /home/peter/tenderpilot/digest.log
2. Run dry-run: node scripts/send-digest.js --dry-run
3. Review this documentation
4. Check database: Verify profiles and tenders exist
5. Review API endpoint responses

---

Last Updated: 2026-02-14 System Version: 1.0