Implement comprehensive SEO infrastructure and new service pages
Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
61
robots.txt
61
robots.txt
@@ -1,27 +1,60 @@
|
||||
# UK Data Services - robots.txt
|
||||
# https://ukdataservices.co.uk
|
||||
|
||||
User-agent: *
|
||||
Allow: /
|
||||
|
||||
# Prioritize important pages
|
||||
Allow: /index.php
|
||||
Allow: /quote.php
|
||||
Allow: /assets/css/
|
||||
Allow: /assets/js/
|
||||
Allow: /assets/images/
|
||||
|
||||
# Block sensitive areas
|
||||
# Block sensitive directories and files
|
||||
Disallow: /includes/
|
||||
Disallow: /assets/
|
||||
Disallow: /admin/
|
||||
Disallow: /logs/
|
||||
Disallow: /vendor/
|
||||
Disallow: /config/
|
||||
Disallow: /database/
|
||||
Disallow: /docker/
|
||||
Disallow: /redis/
|
||||
|
||||
# Block configuration and handler files
|
||||
Disallow: /*-handler.php
|
||||
Disallow: /*.log$
|
||||
Disallow: /*.inc$
|
||||
Disallow: /contact-handler.php
|
||||
Disallow: /quote-handler.php
|
||||
Disallow: /*.sql$
|
||||
Disallow: /*.sh$
|
||||
Disallow: /*.bak$
|
||||
Disallow: /db-config.php
|
||||
Disallow: /.email-config.php
|
||||
Disallow: /.recaptcha-config.php
|
||||
|
||||
# Sitemap locations
|
||||
Sitemap: https://ukdataservices.co.uk/sitemap-index.xml
|
||||
# Block query string URLs to prevent duplicate content
|
||||
Disallow: /*?*
|
||||
|
||||
# Allow important static assets for rendering
|
||||
Allow: /assets/css/*.css
|
||||
Allow: /assets/js/*.js
|
||||
Allow: /assets/images/*.webp
|
||||
Allow: /assets/images/*.png
|
||||
Allow: /assets/images/*.jpg
|
||||
Allow: /assets/images/*.svg
|
||||
|
||||
# Sitemaps
|
||||
Sitemap: https://ukdataservices.co.uk/sitemap.xml
|
||||
Sitemap: https://ukdataservices.co.uk/sitemap-index.xml
|
||||
Sitemap: https://ukdataservices.co.uk/sitemap-blog.xml
|
||||
Sitemap: https://ukdataservices.co.uk/sitemap-services.xml
|
||||
|
||||
# Crawl delay to be respectful
|
||||
Crawl-delay: 1
|
||||
# Crawl-delay for respectful crawling
|
||||
Crawl-delay: 1
|
||||
|
||||
# Specific instructions for major search engines
|
||||
User-agent: Googlebot
|
||||
Allow: /
|
||||
Crawl-delay: 0
|
||||
|
||||
User-agent: Bingbot
|
||||
Allow: /
|
||||
Crawl-delay: 1
|
||||
|
||||
User-agent: Slurp
|
||||
Allow: /
|
||||
Crawl-delay: 2
|
||||
|
||||
Reference in New Issue
Block a user