Make sure AI crawlers can actually read your site
Many sites block AI crawlers without knowing it — through a robots.txt line, a firewall rule or content that only appears after JavaScript runs. We fix access first, then make every page easy to parse.
“If a crawler can't fetch the page, the answer will be written without you.”
Why we always start here
The technical layer between you and AI
Access, rendering, speed and freshness — for both search engines and AI crawlers.
AI crawler access policy
A deliberate decision for each bot — GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and others — separating training access from search access.
Firewall and CDN review
Cloudflare and hosting rules that block AI bots by default, found and adjusted to match your policy.
Server-side content
Key content and links delivered in the initial HTML. Many AI crawlers don't run JavaScript the way Googlebot does.
llms.txt and freshness signals
An llms.txt file, clean XML sitemaps with accurate lastmod dates, and IndexNow for engines that support it.
Which crawler does what
Blocking the wrong user agent can remove you from AI search while doing nothing about training. These are the main ones we configure.
- GPTBot
- OpenAI's crawler for model training. Blocking it does not remove you from ChatGPT search.
- OAI-SearchBot
- Used to surface websites in ChatGPT search results. Block this and ChatGPT search can't show your pages.
- ChatGPT-User
- Fetches a page when a user's conversation asks for it.
- PerplexityBot
- Indexes pages for Perplexity's answers.
- ClaudeBot
- Anthropic's crawler; Anthropic also uses separate agents for search and user-requested fetches.
- Google-Extended
- A robots.txt token that controls use of your content for Gemini models. It doesn't affect Google Search or AI Overviews, which use Googlebot.
How the work runs
Access test
We request your pages as each AI crawler and record what comes back.
Policy decision
Together we decide what to allow for training, search and user fetches.
Fixes
robots.txt, firewall rules, rendering and sitemaps updated in staging first.
Verification
Server logs checked for 30 days to confirm AI crawlers are reaching the right pages.
What you receive
Everything is yours to keep: documents, data and access.
- AI crawler access test report
- Documented robots.txt and firewall policy
- Rendering fixes for JavaScript-dependent content
- llms.txt and segmented XML sitemaps
- 30-day log-file verification
Questions about Technical GEO
Should we block AI crawlers?
It depends on what you sell. Most businesses that want to be recommended should allow search-related AI crawlers. Publishers who license their content may block training crawlers. The point is to choose deliberately — most blocks we find are accidental.
Does llms.txt work?
llms.txt is a proposed standard, and major AI companies haven't confirmed they use it for search. It takes an hour to create, so we add it as low-cost housekeeping — but we don't sell it as a ranking factor.
Is this different from technical SEO?
It overlaps a lot. The additions are the AI-specific crawlers, the training-versus-search distinction, and the fact that many AI crawlers are less forgiving of JavaScript than Googlebot.
Check whether AI can reach you
Free test: we'll request your homepage as five AI crawlers and send you what each one received.