peter/ukaiautomation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
35f34a91d6078bf635a9ce1db175998830cebcf0
ukaiautomation/robots.txt

62 lines
1.3 KiB
Plaintext
Raw Normal View History

Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# UK Data Services - robots.txt
# https://ukdataservices.co.uk
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
User-agent: *
Allow: /
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Block sensitive directories and files
Disallow: /includes/
Disallow: /assets/
Disallow: /admin/
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Disallow: /logs/
Disallow: /vendor/
Disallow: /config/
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Disallow: /database/
Disallow: /docker/
Disallow: /redis/
# Block configuration and handler files
Disallow: /*-handler.php
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Disallow: /*.log$
Disallow: /*.inc$
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Disallow: /*.sql$
Disallow: /*.sh$
Disallow: /*.bak$
Disallow: /db-config.php
Disallow: /.email-config.php
Disallow: /.recaptcha-config.php
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Block query string URLs to prevent duplicate content
Disallow: /*?*
# Allow important static assets for rendering
Allow: /assets/css/*.css
Allow: /assets/js/*.js
Allow: /assets/images/*.webp
Allow: /assets/images/*.png
Allow: /assets/images/*.jpg
Allow: /assets/images/*.svg
# Sitemaps
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Sitemap: https://ukdataservices.co.uk/sitemap.xml
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Sitemap: https://ukdataservices.co.uk/sitemap-index.xml
Backup database and code changes - 2025-06-08 20:51:14
2025-06-08 20:51:14 +00:00
Sitemap: https://ukdataservices.co.uk/sitemap-blog.xml
Sitemap: https://ukdataservices.co.uk/sitemap-services.xml
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Crawl-delay for respectful crawling
Crawl-delay: 1
# Specific instructions for major search engines
User-agent: Googlebot
Allow: /
Crawl-delay: 0
User-agent: Bingbot
Allow: /
Crawl-delay: 1
User-agent: Slurp
Allow: /
Crawl-delay: 2
Security hardening + new tools deployment - Hide Apache version (ServerTokens Prod) - Add Permissions-Policy header - Remove deprecated X-XSS-Protection - Consolidate security headers to .htaccess only (remove duplicates from PHP) - Deploy free tools: robots-analyzer, data-converter - Deploy tools announcement blog post - Update sitemap with new tools and blog post
2026-02-05 04:11:15 +00:00
Sitemap: https://ukdataservices.co.uk/sitemap-tools.xml
Reference in New Issue Copy Permalink