fix: clean Apply Now URLs and disable TED demo scraper

- Strip tracking query params from find_tender URLs (?origin=SearchResults)
- Disable TED EU scraper (requires browser automation, was using demo data)
- Update 220 find_tender database records with clean URLs
- Delete 4 TED demo records from database
- Add URL_FIX_SUMMARY.md documentation

All 615 tenders now have direct links to tender detail pages.
Fixes Apply Now button UX issue.
This commit is contained in:
Peter Foster
2026-02-15 13:18:50 +00:00
parent bba8f97bbe
commit 6ca3e9c576
3 changed files with 82 additions and 186 deletions

57
URL_FIX_SUMMARY.md Normal file
View File

@@ -0,0 +1,57 @@
# TenderRadar Apply Now Button - Fix Complete
**Date:** 2026-02-15
**Status:** ✅ FIXED
## Summary
Fixed "Apply Now" button URLs for 615 tenders across 5 data sources. All URLs now link directly to tender detail pages instead of search portals.
## Changes Made
### 1. Find a Tender (220 tenders)
- **Fixed:** Removed tracking query parameters from URLs
- **Before:** `https://www.find-tender.service.gov.uk/Notice/013656-2026?origin=SearchResults&p=1`
- **After:** `https://www.find-tender.service.gov.uk/Notice/013656-2026`
- **Files:** `scrapers/find-tender.js`, database updated
### 2. TED EU (4 demo records)
- **Fixed:** Disabled scraper using hardcoded demo data
- **Status:** Requires browser automation (Playwright/Puppeteer)
- **Action:** Deleted 4 demo records, documented for future
- **Files:** `scrapers/ted-eu.js` rewritten with documentation
### 3. Other Sources (Already Working)
These were already correct - no changes needed:
- contracts_finder (364 tenders)
- etendersni (11 tenders)
- pcs_scotland (10 tenders)
- sell2wales (10 tenders)
## Database State After Fix
Total tenders: 615
All URLs validated: ✅
No broken URLs: ✅
No tracking parameters (except legitimate notice IDs): ✅
## Files Modified
1. `scrapers/find-tender.js` - Strip query params
2. `scrapers/ted-eu.js` - Disabled with documentation
3. Database - 220 find_tender records cleaned, 4 TED records deleted
## Backups Created
- `scrapers/find-tender.js.bak`
## Testing Completed
✅ Verified all source URL patterns
✅ Checked for invalid/broken URLs
✅ Confirmed query parameters removed from find_tender
✅ Validated notice IDs in other sources
## Result
Users clicking "Apply Now" will now land directly on tender detail pages for all 615 active tenders.