117 lines
3.6 KiB
Markdown
117 lines
3.6 KiB
Markdown
|
# TenderRadar Scrapers
|
||
|
|
|
||
|
|
This directory contains scrapers for UK public procurement tender sources.
|
||
|
|
|
||
|
|
## Scrapers
|
||
|
|
|
||
|
|
### 1. Contracts Finder (`contracts-finder.js`)
|
||
|
|
- **Source**: https://www.contractsfinder.service.gov.uk
|
||
|
|
- **Coverage**: England and non-devolved territories
|
||
|
|
- **Method**: JSON API
|
||
|
|
- **Frequency**: Every 4 hours (0, 4, 8, 12, 16, 20:00)
|
||
|
|
- **Data Range**: Last 30 days
|
||
|
|
- **Status**: ✅ Working
|
||
|
|
|
||
|
|
### 2. Find a Tender (`find-tender.js`)
|
||
|
|
- **Source**: https://www.find-tender.service.gov.uk
|
||
|
|
- **Coverage**: UK-wide above-threshold procurement notices
|
||
|
|
- **Method**: HTML scraping with pagination (5 pages)
|
||
|
|
- **Frequency**: Every 4 hours (0:10, 4:10, 8:10, 12:10, 16:10, 20:10)
|
||
|
|
- **Status**: ✅ Working
|
||
|
|
|
||
|
|
### 3. Public Contracts Scotland (`pcs-scotland.js`)
|
||
|
|
- **Source**: https://www.publiccontractsscotland.gov.uk
|
||
|
|
- **Coverage**: Scottish public sector tenders
|
||
|
|
- **Method**: HTML scraping
|
||
|
|
- **Frequency**: Every 4 hours (0:20, 4:20, 8:20, 12:20, 16:20, 20:20)
|
||
|
|
- **Status**: ✅ Working
|
||
|
|
|
||
|
|
### 4. Sell2Wales (`sell2wales.js`)
|
||
|
|
- **Source**: https://www.sell2wales.gov.wales
|
||
|
|
- **Coverage**: Welsh public sector tenders
|
||
|
|
- **Method**: HTML scraping
|
||
|
|
- **Frequency**: Every 4 hours (0:30, 4:30, 8:30, 12:30, 16:30, 20:30)
|
||
|
|
- **Status**: ✅ Working
|
||
|
|
|
||
|
|
## Database Schema
|
||
|
|
|
||
|
|
All scrapers insert into the `tenders` table with the following key fields:
|
||
|
|
|
||
|
|
- `source`: Identifier for the data source (contracts_finder, find_tender, pcs_scotland, sell2wales)
|
||
|
|
- `source_id`: Unique identifier from the source (used for deduplication via UNIQUE constraint)
|
||
|
|
- `title`: Tender title
|
||
|
|
- `description`: Full description
|
||
|
|
- `summary`: Shortened description
|
||
|
|
- `authority_name`: Publishing authority
|
||
|
|
- `location`: Geographic location
|
||
|
|
- `published_date`: When the tender was published
|
||
|
|
- `deadline`: Application deadline
|
||
|
|
- `notice_url`: Link to full notice
|
||
|
|
- `status`: open/closed based on deadline
|
||
|
|
|
||
|
|
## Running Scrapers
|
||
|
|
|
||
|
|
### Individual Scraper
|
||
|
|
```bash
|
||
|
|
cd /home/peter/tenderpilot
|
||
|
|
node scrapers/contracts-finder.js
|
||
|
|
node scrapers/find-tender.js
|
||
|
|
node scrapers/pcs-scotland.js
|
||
|
|
node scrapers/sell2wales.js
|
||
|
|
```
|
||
|
|
|
||
|
|
### All Scrapers
|
||
|
|
```bash
|
||
|
|
cd /home/peter/tenderpilot
|
||
|
|
./run-all-scrapers.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
## Cron Schedule
|
||
|
|
|
||
|
|
The scrapers run automatically every 4 hours, staggered by 10 minutes:
|
||
|
|
|
||
|
|
```cron
|
||
|
|
0 */4 * * * cd /home/peter/tenderpilot && node scrapers/contracts-finder.js >> /home/peter/tenderpilot/scraper.log 2>&1
|
||
|
|
10 */4 * * * cd /home/peter/tenderpilot && node scrapers/find-tender.js >> /home/peter/tenderpilot/scraper.log 2>&1
|
||
|
|
20 */4 * * * cd /home/peter/tenderpilot && node scrapers/pcs-scotland.js >> /home/peter/tenderpilot/scraper.log 2>&1
|
||
|
|
30 */4 * * * cd /home/peter/tenderpilot && node scrapers/sell2wales.js >> /home/peter/tenderpilot/scraper.log 2>&1
|
||
|
|
```
|
||
|
|
|
||
|
|
## Monitoring
|
||
|
|
|
||
|
|
Check logs:
|
||
|
|
```bash
|
||
|
|
tail -f /home/peter/tenderpilot/scraper.log
|
||
|
|
```
|
||
|
|
|
||
|
|
Check database:
|
||
|
|
```bash
|
||
|
|
PGPASSWORD=tenderpilot123 psql -h localhost -U tenderpilot -d tenderpilot -c "SELECT source, COUNT(*) FROM tenders GROUP BY source;"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Rate Limiting & Ethical Scraping
|
||
|
|
|
||
|
|
All scrapers implement:
|
||
|
|
- Proper User-Agent headers identifying the service
|
||
|
|
- Rate limiting (2-5 second delays between requests)
|
||
|
|
- Pagination limits where applicable
|
||
|
|
- Respectful request patterns
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
- axios: HTTP client
|
||
|
|
- cheerio: HTML parsing (for web scrapers)
|
||
|
|
- pg: PostgreSQL client
|
||
|
|
- dotenv: Environment variables
|
||
|
|
|
||
|
|
## Maintenance
|
||
|
|
|
||
|
|
- Scrapers use `ON CONFLICT (source_id) DO NOTHING` to avoid duplicates
|
||
|
|
- Old scrapers can update existing records if needed
|
||
|
|
- Monitor for HTML structure changes on scraped sites
|
||
|
|
- API endpoints (Contracts Finder) are more stable than HTML scraping
|
||
|
|
|
||
|
|
## Last Updated
|
||
|
|
|
||
|
|
2026-02-14 - Initial deployment with all four scrapers
|