feat: three major improvements - stable sources, archival, email alerts
1. Focus on Stable International/Regional Sources
- Improved TED EU scraper (5 search strategies, 5 pages each)
- All stable sources now hourly (TED EU, Sell2Wales, PCS Scotland, eTendersNI)
- De-prioritize unreliable UK gov sites (100% removal rate)
2. Archival Feature
- New DB columns: archived, archived_at, archived_snapshot, last_validated, validation_failures
- Cleanup script now preserves full tender snapshots before archiving
- Gradual failure handling (3 retries before archiving)
- No data loss - historical record preserved
3. Email Alerts
- Daily digest (8am) - all new tenders from last 24h
- High-value alerts (every 4h) - tenders >£100k
- Professional HTML emails with all tender details
- Configurable via environment variables
Expected outcomes:
- 50-100 stable tenders (vs 26 currently)
- Zero 404 errors (archived data preserved)
- Proactive notifications (no missed opportunities)
- Historical archive for trend analysis
Files:
- scrapers/ted-eu.js (improved)
- cleanup-with-archival.mjs (new)
- send-tender-alerts.mjs (new)
- migrations/add-archival-fields.sql (new)
- THREE_IMPROVEMENTS_SUMMARY.md (documentation)
All cron jobs updated for hourly scraping + daily cleanup + alerts
c6b0169f3e