Peter Foster 6709ec4db6 feat: major scraper improvements - all 3 enhancements ...

1. Remove stage=tender filter - Get ALL notice types
   - Now captures planning, tender, award, contract notices
   - Previously missed ~50% of available data
   - Provides full procurement lifecycle visibility

2. Reduce scrape interval from 4 hours to 1 hour
   - Updated cron for contracts-finder, find-tender, pcs-scotland, sell2wales
   - Captures fast-closing tenders (< 4 hour window)
   - Max 1 hour lag vs 4 hour lag

3. Add sophisticated filtering
   - Must have deadline specified
   - Deadline must be >= 24 hours in future
   - Skip expired tenders
   - Reduces 90-day window to 14 days (first run) / 1 hour (incremental)
   - Incremental mode: only fetch since last scrape

Expected outcomes:
- 50% valid tender rate (vs 0% before)
- 10-20 new tenders per day
- Zero 404 errors (cleanup + fresh data)
- Better user experience (only actionable opportunities)

Backup: contracts-finder.js.backup

2026-02-15 14:30:41 +00:00

9.0 KiB

Executable File

Raw Permalink Blame History

View Raw