Technical AEO that lets answer engines reach, read and quote your site

What does technical AEO cover that content work does not?

It covers everything between an answer engine's crawler and your words: permissions, delivery, markup and discovery. Content restructuring decides what a page says; this layer decides whether a machine ever receives it.

We treat it as a separate workstream because the failures are silent. A site can rank well in classic search while a single disallow line, a bot-fight rule at the CDN or an empty pre-render shell keeps one or more answer engines out. Our AEO audit flags these problems; this service fixes them and keeps them fixed after every redesign, migration or security change.

• robots.txt rules for every search, user-fetch and training token.
• CDN and web application firewall rules that sit in front of robots.txt.
• Server-rendered HTML for the main answer on each important page.
• Organization, Product, Article and FAQ schema that matches what visitors see.
• XML sitemaps, last-updated dates and IndexNow pings for fresh content.

Which AI crawlers should you allow in robots.txt?

Allow the crawlers that put pages into answers, then make a separate business decision about training crawlers. Mixing the two up is the most common mistake we see in AI crawler access.

OpenAI's crawler documentation says OAI-SearchBot surfaces sites in ChatGPT search, follows robots.txt and picks up changes in about 24 hours, while sites that opt out are not shown in search answers. GPTBot collects training data, and blocking it only opts you out of training. Perplexity's bot guide describes PerplexityBot as a search crawler that is not used for training. Anthropic lists Claude-SearchBot for search quality and ClaudeBot for training. Google's crawler documentation explains that AI Overviews and AI Mode use Googlebot, and that Google-Extended is a control token, not a crawler, covering Gemini training and Gemini Apps grounding without affecting Google Search. Bingbot crawls for Copilot; there is no separate Copilot crawler.

• Keep allowed for visibility: Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot.
• User-triggered fetchers: ChatGPT-User, Perplexity-User, Claude-User.
• Training-only controls you may block: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.

Is Cloudflare blocking AI crawlers on your site?

It may be, even if your robots.txt looks open. Cloudflare changed its default on 1 July 2025 to block AI crawlers unless they pay, and its June 2026 bot report says AI training crawlers are blocked by default for all new domains, noting that training makes up 52% of crawler requests.

Cloudflare's managed robots.txt can add a content signal line such as search=yes, ai-train=no and disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. That is a sensible setup for many brands, but we check that the search crawlers stay allowed in the dashboard rules too. Pay per crawl can answer a crawler with HTTP 402 Payment Required, which is worth knowing if your logs show 402s to bots you meant to welcome. We also note that Cloudflare reported Perplexity using undeclared crawlers in August 2025 and delisted it as a verified bot, so rules that only admit verified bots need a deliberate decision about PerplexityBot.

Does schema markup for AI help you get cited?

Not directly, and we will not sell it as a citation lever. Google's AI optimization guide says structured data is not required for generative AI search and there is no special schema.org markup to add, while recommending you keep using it for general SEO.

We still implement schema carefully because it states your entity facts in one consistent place. Organization markup with sameAs links to your LinkedIn, Wikidata, G2 or Google Business Profile listings helps keep your name, category and locations consistent wherever you appear. Google's guidance on succeeding in AI search adds that structured data must match visible content, so we remove FAQ or review markup that describes text the visitor cannot see.

How should pages be built so answer engines can read them?

Put the answer in the HTML the server sends, near the top, under a heading that matches the question. We do not assume every crawler executes JavaScript, so we test each key template with scripts turned off and fix anything that disappears.

Beyond rendering, we check canonical tags, status codes, redirect chains and duplicate paths, because a crawler that wastes fetches on parameters or soft 404s may never reach the page you want quoted. Each important page gets a visible last-updated date, a clean heading outline and tables in real table markup rather than images. Google states no special files, markup or Markdown copies are needed, and it does not require breaking content into tiny pieces, so we keep pages whole and readable.

How do you confirm the fixes actually worked?

We read the server logs. A robots.txt change proves intent; a 200 response to OAI-SearchBot, PerplexityBot or Claude-SearchBot on the right URLs proves access.

We match user agents against published IP ranges where vendors provide them, as Perplexity does, so spoofed bots do not distort the picture. Then we submit updated sitemaps, ping IndexNow for Bing, and watch Search Console's generative AI performance report and Bing Webmaster Tools AI Performance for cited pages. Ongoing citation reporting sits with our AI visibility tracking service.

Do you need an llms.txt file?

You can add one cheaply, but do not expect it to change citations. Jeremy Howard of Answer.AI proposed llms.txt in September 2024, and Google's John Mueller said none of the AI services have said they are using it, comparing it to the meta keywords tag.

It is genuinely useful for developer documentation that coding agents read, which is why AI companies publish one for their own docs. For a marketing site, we treat it as an optional tidy-up after the real access work is done.