peter/ukaiautomation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b5a7347f0cd993d196c76d4d21b17b28c81c5e7a
ukaiautomation/robots.txt

87 lines
1.4 KiB
Plaintext
Raw Normal View History

Replace web scraping content with AI automation brand - Remove all web scraping services, blog articles, locations, tools pages - Remove fake author profiles and old categories - Add 6 new AI automation blog articles targeting legal/consultancy firms - Rewrite blog index with new AI automation content - Update robots.txt with correct ukaiautomation.co.uk domain - Update sitemap.xml with current pages only
2026-03-21 10:04:47 +00:00
# UK AI Automation - robots.txt
# https://ukaiautomation.co.uk
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
User-agent: *
Allow: /
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Block sensitive directories and files
Disallow: /includes/
Disallow: /assets/
Disallow: /admin/
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Disallow: /logs/
Disallow: /vendor/
Disallow: /config/
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Disallow: /database/
Disallow: /docker/
# Block configuration and handler files
Disallow: /*-handler.php
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Disallow: /*.log$
Disallow: /*.inc$
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
Disallow: /*.sql$
Disallow: /*.sh$
Disallow: /*.bak$
Disallow: /db-config.php
Disallow: /.email-config.php
Disallow: /.recaptcha-config.php
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Block query string URLs to prevent duplicate content
Disallow: /*?*
# Allow important static assets for rendering
Allow: /assets/css/*.css
Allow: /assets/js/*.js
Allow: /assets/images/*.webp
Allow: /assets/images/*.png
Allow: /assets/images/*.jpg
Allow: /assets/images/*.svg
# Sitemaps
Replace web scraping content with AI automation brand - Remove all web scraping services, blog articles, locations, tools pages - Remove fake author profiles and old categories - Add 6 new AI automation blog articles targeting legal/consultancy firms - Rewrite blog index with new AI automation content - Update robots.txt with correct ukaiautomation.co.uk domain - Update sitemap.xml with current pages only
2026-03-21 10:04:47 +00:00
Sitemap: https://ukaiautomation.co.uk/sitemap.xml
Initial commit: UK Data Services website
2025-06-07 10:53:32 +01:00
Implement comprehensive SEO infrastructure and new service pages Phase 1 - Schema Markup: - Add reusable schema components in /includes/schema/ - organization-schema.php for site-wide Organization structured data - service-schema.php with generator function and predefined configs - local-business-schema.php for location pages with geo coordinates - faq-schema.php with FAQPage generator and common FAQ sets - article-schema.php for blog posts with BlogPosting schema - review-schema.php with AggregateRating and testimonials Phase 6 - Meta Tags & Helpers: - meta-tags.php: Complete meta tag generator (OG, Twitter, article) - canonical.php: URL normalization and canonical tag helper - url-config.php: Centralized URL mapping for internal linking Phase 7 - Internal Linking Components: - related-services.php: Cross-linking component for service pages - location-cta.php: Location links component for service pages Phase 3 - New Service Pages: - property-data-extraction.php: UK property data extraction service - financial-data-services.php: FCA-aware financial data services Phase 5 & 8 - Technical SEO: - Update robots.txt with improved blocking rules - Update sitemap.php with clean URLs and new pages - Update sitemap-services.xml with new service pages - Add new services to footer navigation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 09:07:07 +00:00
# Crawl-delay for respectful crawling
Crawl-delay: 1
# Specific instructions for major search engines
User-agent: Googlebot
Allow: /
Crawl-delay: 0
User-agent: Bingbot
Allow: /
Crawl-delay: 1
SEO audit fixes: schema, content, technical and AI readiness - Remove restricted FAQPage schema and duplicate Organization schema - Remove incomplete LocalBusiness schema - Fix Organization: founder Peter Foster, foundingDate 2015 - Rewrite H1 to include target keywords - Fix BreadcrumbList with correct page URLs - Remove SearchAction pointing to 404 endpoint - Simplify verbose content across 13 key paragraphs - Promote service card headings from H3 to H2 with keyword-rich names - Add width/height attributes to all images (CLS fix) - Create 1200x630 OG social card image - Update og:image, og:image:width, og:image:height meta tags - Fix broken LinkedIn (www prefix) and Twitter (x.com) social links - Add Free Tools footer section linking to individual tools - Remove google-oauth-callback and /index URLs from sitemap.xml - Add AI crawler rules to robots.txt (GPTBot, ClaudeBot, Perplexity, etc.) - Add Disallow for oauth callback endpoints in robots.txt - Create llms.txt for AI search engine readiness
2026-03-06 06:42:24 +00:00
# AI crawlers - explicitly allowed for citation
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
SEO audit fixes: H1 keyword, schema, robots.txt, llms.txt, sitemap, alt texts, logo - H1: add primary keyword (AI Automation for UK Law Firms & Consultancies) - Hero subtitle: replace old scraping copy with accurate consultancy copy - WebPage schema name aligned with title tag - Organization schema: add telephone, fix sameAs (remove broken social URLs) - robots.txt: add 5 missing AI crawlers (Applebot-Extended, Bytespider, CCBot, FacebookBot, Amazonbot) - llms.txt: add Key Pages section with 7 linked pages - sitemap.xml: add 4 legal pages (privacy, terms, cookie, gdpr) - Icon alt texts: simplify 6 keyword-stuffed verbose alts - Nav logo: add width/height dimensions, switch to loading=eager
2026-03-22 03:14:52 +00:00
User-agent: Applebot-Extended
Allow: /
User-agent: Bytespider
Allow: /
User-agent: CCBot
Allow: /
User-agent: FacebookBot
Allow: /
User-agent: Amazonbot
Allow: /
Reference in New Issue Copy Permalink