2026-03-21 10:04:47 +00:00
|
|
|
# UK AI Automation - robots.txt
|
|
|
|
|
# https://ukaiautomation.co.uk
|
2025-12-09 09:07:07 +00:00
|
|
|
|
2025-06-07 10:53:32 +01:00
|
|
|
User-agent: *
|
|
|
|
|
Allow: /
|
|
|
|
|
|
2025-12-09 09:07:07 +00:00
|
|
|
# Block sensitive directories and files
|
|
|
|
|
Disallow: /includes/
|
|
|
|
|
Disallow: /assets/
|
|
|
|
|
Disallow: /admin/
|
2025-06-07 10:53:32 +01:00
|
|
|
Disallow: /logs/
|
|
|
|
|
Disallow: /vendor/
|
|
|
|
|
Disallow: /config/
|
2025-12-09 09:07:07 +00:00
|
|
|
Disallow: /database/
|
|
|
|
|
Disallow: /docker/
|
|
|
|
|
|
|
|
|
|
# Block configuration and handler files
|
|
|
|
|
Disallow: /*-handler.php
|
2025-06-07 10:53:32 +01:00
|
|
|
Disallow: /*.log$
|
|
|
|
|
Disallow: /*.inc$
|
2025-12-09 09:07:07 +00:00
|
|
|
Disallow: /*.sql$
|
|
|
|
|
Disallow: /*.sh$
|
|
|
|
|
Disallow: /*.bak$
|
2025-06-07 10:53:32 +01:00
|
|
|
|
2025-12-09 09:07:07 +00:00
|
|
|
# Block query string URLs to prevent duplicate content
|
|
|
|
|
Disallow: /*?*
|
|
|
|
|
|
|
|
|
|
# Allow important static assets for rendering
|
|
|
|
|
Allow: /assets/css/*.css
|
|
|
|
|
Allow: /assets/js/*.js
|
|
|
|
|
Allow: /assets/images/*.webp
|
|
|
|
|
Allow: /assets/images/*.png
|
|
|
|
|
Allow: /assets/images/*.jpg
|
|
|
|
|
Allow: /assets/images/*.svg
|
|
|
|
|
|
|
|
|
|
# Sitemaps
|
2026-03-21 10:04:47 +00:00
|
|
|
Sitemap: https://ukaiautomation.co.uk/sitemap.xml
|
2025-06-07 10:53:32 +01:00
|
|
|
|
2025-12-09 09:07:07 +00:00
|
|
|
# Crawl-delay for respectful crawling
|
|
|
|
|
Crawl-delay: 1
|
|
|
|
|
|
|
|
|
|
# Specific instructions for major search engines
|
|
|
|
|
User-agent: Googlebot
|
|
|
|
|
Allow: /
|
|
|
|
|
Crawl-delay: 0
|
|
|
|
|
|
|
|
|
|
User-agent: Bingbot
|
|
|
|
|
Allow: /
|
|
|
|
|
Crawl-delay: 1
|
|
|
|
|
|
2026-03-06 06:42:24 +00:00
|
|
|
# AI crawlers - explicitly allowed for citation
|
|
|
|
|
User-agent: GPTBot
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: ChatGPT-User
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: ClaudeBot
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: anthropic-ai
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: PerplexityBot
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: Google-Extended
|
|
|
|
|
Allow: /
|
SEO audit fixes: H1 keyword, schema, robots.txt, llms.txt, sitemap, alt texts, logo
- H1: add primary keyword (AI Automation for UK Law Firms & Consultancies)
- Hero subtitle: replace old scraping copy with accurate consultancy copy
- WebPage schema name aligned with title tag
- Organization schema: add telephone, fix sameAs (remove broken social URLs)
- robots.txt: add 5 missing AI crawlers (Applebot-Extended, Bytespider, CCBot, FacebookBot, Amazonbot)
- llms.txt: add Key Pages section with 7 linked pages
- sitemap.xml: add 4 legal pages (privacy, terms, cookie, gdpr)
- Icon alt texts: simplify 6 keyword-stuffed verbose alts
- Nav logo: add width/height dimensions, switch to loading=eager
2026-03-22 03:14:52 +00:00
|
|
|
|
|
|
|
|
User-agent: Applebot-Extended
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: Bytespider
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: CCBot
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: FacebookBot
|
|
|
|
|
Allow: /
|
|
|
|
|
|
|
|
|
|
User-agent: Amazonbot
|
|
|
|
|
Allow: /
|
