tenderpilot/EMAIL_DIGEST.md

# TenderRadar Email Digest System

## Overview

The email digest system automatically sends matched tender alerts to subscribed users. The system matches new tenders published in the last 24 hours against each user's saved preferences (keywords, sectors, value ranges, locations, authority types) and delivers personalized HTML email digests daily.

## Architecture

### Components

1. **Database Schema**: Uses existing `profiles` and `matches` tables
   - `profiles` table stores user alert preferences
   - `matches` table tracks which tenders have been sent to which users

2. **Email Digest Script**: `/scripts/send-digest.js`
   - Daily script that finds matching tenders for each user
   - Sends professional HTML emails with tender details
   - Marks matches as sent to avoid duplicates
   - Supports dry-run mode for testing

3. **API Endpoints**: Alert preference management
   - `GET /api/alerts/preferences` - Retrieve user's alert settings
   - `POST /api/alerts/preferences` - Create/update alert settings

4. **Cron Job**: Scheduled daily execution at 7am UTC
   - Configured in `/etc/cron.d/` (or user crontab)
   - Logs to `/home/peter/tenderpilot/digest.log`

## Database Schema

### Profiles Table

Used to store user alert preferences. Required columns:

- `id`: Primary key
- `user_id`: Foreign key to users table (unique constraint)
- `keywords`: TEXT[] - Array of keywords to match in tender title/description
- `sectors`: TEXT[] - Array of sector/CPV codes
- `min_value`: DECIMAL - Minimum tender value (GBP)
- `max_value`: DECIMAL - Maximum tender value (GBP)
- `locations`: TEXT[] - Array of location filters
- `authority_types`: TEXT[] - Array of authority type filters
- `created_at`: TIMESTAMP - Record creation time
- `updated_at`: TIMESTAMP - Last update time

### Matches Table

Tracks which tenders have been sent to which users.

- `id`: Primary key
- `user_id`: Foreign key to users table
- `tender_id`: Foreign key to tenders table
- `sent`: BOOLEAN - Whether email was sent
- `created_at`: TIMESTAMP
- Unique constraint on (user_id, tender_id)

## Configuration

### Environment Variables

Add to `.env`:

```
# Email Digest Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=alerts@tenderradar.co.uk
SMTP_PASS=your-email-password
SMTP_FROM=TenderRadar Alerts <alerts@tenderradar.co.uk>
```

**Note**: Use placeholder values initially. Peter will update with production credentials.

### SMTP Options

The script supports any SMTP provider. Common configurations:

**Gmail:**
- Host: `smtp.gmail.com`
- Port: `587` (TLS) or `465` (SSL)
- User: Your Gmail address
- Pass: App-specific password (generate in Google Account settings)

**Office 365:**
- Host: `smtp.office365.com`
- Port: `587` (TLS)
- User: Your Office 365 email
- Pass: Your Office 365 password

**Generic SMTP:**
- Update SMTP_HOST and SMTP_PORT accordingly

## Usage

### Running the Digest Script

**Dry-run (test mode - no emails sent):**
```bash
cd /home/peter/tenderpilot
node scripts/send-digest.js --dry-run
```

Output shows which users have matches and how many tenders match their preferences.

**Production (sends emails):**
```bash
cd /home/peter/tenderpilot
node scripts/send-digest.js
```

### Automatic Execution

The cron job runs daily at 7am UTC:

```
0 7 * * * cd /home/peter/tenderpilot && node scripts/send-digest.js >> /home/peter/tenderpilot/digest.log 2>&1
```

View logs:
```bash
tail -f /home/peter/tenderpilot/digest.log
```

### Manual Cron Management

View existing cron jobs:
```bash
crontab -l
```

Edit cron jobs:
```bash
crontab -e
```

Remove a cron job:
```bash
crontab -r
```

## API Endpoints

### GET /api/alerts/preferences

Retrieve the current user's alert preferences.

**Request:**
```bash
curl -H "Authorization: Bearer <token>" \
  http://localhost:3456/api/alerts/preferences
```

**Response:**
```json
{
  "preferences": {
    "id": 1,
    "user_id": 5,
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"],
    "created_at": "2026-02-14T12:00:00Z",
    "updated_at": "2026-02-14T12:00:00Z"
  }
}
```

Or if no preferences exist:
```json
{
  "preferences": null
}
```

### POST /api/alerts/preferences

Create or update alert preferences for the current user.

**Request:**
```bash
curl -X POST -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000", "71000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"]
  }' \
  http://localhost:3456/api/alerts/preferences
```

**Parameters:**

- `keywords` (array, optional): Keywords to match in tender title/description
- `sectors` (array, optional): CPV codes or sector categories
- `min_value` (number, optional): Minimum tender value (GBP)
- `max_value` (number, optional): Maximum tender value (GBP)
- `locations` (array, optional): Geographic locations
- `authority_types` (array, optional): Types of procuring authority

**Response:**
```json
{
  "preferences": {
    "id": 1,
    "user_id": 5,
    "keywords": ["infrastructure", "cleaning"],
    "sectors": ["45000000", "71000000"],
    "min_value": 10000,
    "max_value": 500000,
    "locations": ["London", "Scotland"],
    "authority_types": ["Local Authority", "NHS Trust"],
    "created_at": "2026-02-14T12:00:00Z",
    "updated_at": "2026-02-14T12:30:00Z"
  },
  "message": "Alert preferences updated successfully"
}
```

## Email Template

The digest sends professional HTML emails with:

- TenderRadar branding header
- Summary of matched tenders count
- Table showing:
  - Tender title and source
  - Tender deadline
  - Estimated value (£)
  - Link to tender details
- Call-to-action button to dashboard
- Link to manage preferences
- TenderRadar footer with unsubscribe link

Emails are fully HTML-formatted with responsive design suitable for desktop and mobile clients.

## Matching Algorithm

For each user, the script:

1. Fetches all open tenders published in the last 24 hours
2. Filters out tenders already sent to the user (using matches table)
3. Applies user preference filters:
   - **Keywords**: Matches against tender title + description (case-insensitive)
   - **Value Range**: Filters by tender value_high >= min_value AND value_low <= max_value
   - **Locations**: Matches against tender location
   - **Authority Types**: Matches against tender authority_type
   - **Sectors**: Matches CPV codes
4. For matching tenders:
   - Generates personalized HTML email
   - Sends via SMTP
   - Creates match record with sent=true

**Note**: All filters are AND conditions. If keywords are specified AND location is specified, tender must match BOTH criteria.

## Troubleshooting

### No emails being sent

1. Check that users have verified emails: `SELECT * FROM users WHERE verified = true;`
2. Check that users have preferences: `SELECT * FROM profiles;`
3. Check that tenders exist: `SELECT * FROM tenders WHERE status = 'open' ORDER BY published_date DESC LIMIT 5;`
4. Run dry-run to see matching logic: `node scripts/send-digest.js --dry-run`
5. Check logs: `tail -100 digest.log`

### SMTP connection errors

1. Verify credentials in `.env`
2. Test SMTP connection manually (can use tools like `telnet` or `nc`)
3. Check firewall/network: ensure port is open outbound to SMTP server
4. For Gmail: ensure "Less secure apps" is enabled or use App Password
5. Check SMTP logs on server

### Emails stuck in queue

1. Check node process: `ps aux | grep node`
2. Check for zombie processes: `ps aux | grep defunct`
3. View recent logs: `tail -50 digest.log`
4. Run script manually to see real-time errors

### Database connection issues

1. Verify DATABASE_URL in `.env`
2. Test connection: `psql $DATABASE_URL -c "SELECT 1"`
3. Check database is running: `sudo systemctl status postgresql`
4. Check user permissions: `SELECT grantee, privilege_type FROM information_schema.role_table_grants WHERE table_name='profiles';`

## Monitoring

### Key Metrics to Monitor

1. **Digest runs**: Check cron execution with `grep send-digest digest.log`
2. **Email send rate**: Count successful sends in logs
3. **Match rate**: Ratio of tenders matched vs. users with preferences
4. **Error rate**: Failed SMTP connections or database queries
5. **Database size**: As matches table grows, consider archival/cleanup

### Maintenance

**Weekly:**
- Review logs for errors: `grep -i "error\|failed" digest.log`
- Check disk space for logs: `du -sh digest.log`

**Monthly:**
- Archive old logs: `gzip digest.log.* && mv digest.log.*.gz /archive/`
- Verify cron job is still scheduled: `crontab -l`
- Test dry-run to ensure system is functional

**Quarterly:**
- Review SMTP provider limits and usage
- Check for database performance issues with large matches table
- Consider implementing match archival for old records

## Future Enhancements

Potential improvements to the system:

1. **Frequency Options**: Allow users to choose daily/weekly/instant digests
2. **Digest Format**: Support plain text alternative to HTML
3. **Unsubscribe**: Track unsubscribe preferences
4. **Match Scoring**: Rank matches by relevance to user preferences
5. **Batch Sending**: Use queue system (Bull, Bee-Queue) for high volume
6. **Analytics**: Track open rates, click rates, conversion
7. **A/B Testing**: Test different email templates
8. **Timezone Support**: Send at user's local time, not UTC
9. **Webhook Delivery**: Alternative to SMTP for certain providers
10. **Digest Personalization**: Include user name, company, custom messages

## File Structure

```
/home/peter/tenderpilot/
├── scripts/
│   └── send-digest.js          # Main digest script
├── server.js                    # Updated with alert API endpoints
├── .env                         # Config (includes SMTP settings)
├── package.json                 # Updated with nodemailer dependency
├── digest.log                   # Output log (created on first run)
└── init-db.js                   # Database initialization
```

## Dependencies

Required npm packages:

- `nodemailer`: ^6.x - SMTP email sending
- `pg`: ^8.x - PostgreSQL client
- `dotenv`: ^16.x - Environment variable loading

All dependencies are already in package.json and node_modules.

## Support & Questions

For issues or questions about the email digest system:

1. Check logs: `tail -100 /home/peter/tenderpilot/digest.log`
2. Run dry-run: `node scripts/send-digest.js --dry-run`
3. Review this documentation
4. Check database: Verify profiles and tenders exist
5. Review API endpoint responses

---

**Last Updated**: 2026-02-14
**System Version**: 1.0