We audited 48 sites for AI-citability. The tools selling AI visibility block AI the most.
An AI engine can only cite what it can crawl and extract. We ran page_audit across 48 sites in seven categories to see who is actually eligible to be cited, and who has quietly locked themselves out. The results are not what you would guess.
Original study · measured 2026-08-10
Before any of the usual GEO advice matters, one thing has to be true: the AI engine has to be able to fetch your page and pull something out of it. A brilliant page that returns a 403 to the crawlers that build the answer index (OAI-SearchBot for ChatGPT, Googlebot for Google), or renders client-side so the crawler sees an empty shell, is invisible to those engines no matter what it says.
So we started there. We ran page_audit across 48 sites in seven categories (AI labs, developer tools, scraping APIs, GEO and SEO tools, SaaS, media, and docs) and recorded, for each one, whether every major AI crawler is allowed, whether the site publishes an llms.txt and a dated sitemap, and what structure the page actually carries. No opinions, just the fetch.
Four sites (OpenAI, Perplexity, Stack Overflow, and Oxylabs) returned a 403 or timed out on a plain request and are excluded from the on-page numbers. That is finding zero: four large sites block an ordinary fetch outright. The other 44 are below.
Finding 1: the worst offenders sell AI visibility
The single most useful thing page_audit surfaces is not a missing tag. It is the sites that have told the AI engines not to read them at all.
- orvio.ai sells "become the top recommendation across AI search engines." Its robots.txt blocks every crawler we test, including the ones that actually gate citation: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot, and Googlebot. It cannot be cited by a single engine it promises to win.
- provioai.com sells "track AI traffic and revenue." Its homepage returns 7 words and zero headings to a crawler. It renders entirely client-side, so there is nothing to extract.
Two GEO tools, two fatal own-goals. Then the big brands:
- figma.com blocks all seven AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot) while leaving Googlebot allowed. A deliberate choice to be in Google but out of AI answers.
- techcrunch.com is the instructive one. It blocks the training crawlers (GPTBot, Google-Extended, CCBot) and the live ChatGPT-User fetch, but leaves OAI-SearchBot open, so it stays in ChatGPT's search index while refusing training and live retrieval. A publisher threading the needle, not a blanket block.
Of 44 fetchable sites, 5 block at least one AI crawler, but the crawler is what matters. Only 3 block a crawler that actually gates citation (a search bot: OAI-SearchBot, ChatGPT-User, or PerplexityBot). The other 2, together.ai and clerk.com, block only a training crawler (Google-Extended, CCBot) and keep every citation path open, which is the right move, not a mistake. GPTBot is a training crawler: blocking it does not remove you from ChatGPT, which indexes through OAI-SearchBot.
Finding 2: sites cut the training crawlers first, and it costs them nothing
When a site blocks, it does not block at random. Google-Extended and CCBot are the most-blocked crawlers, four sites each, ahead of GPTBot, ClaudeBot, and ChatGPT-User at three each. The most-cut crawlers are the training ones: Google-Extended feeds Gemini's model training, and CCBot is Common Crawl, which seeds a large share of every open model. Sites that want the traffic but not the training cut those first, and by our own crawler-trap data that costs them nothing in citations. AI citation runs off the live search index, not the training corpus: a page blocked to GPTBot and CCBot still gets cited once it is in the index.
The one real caveat is scope, and it runs opposite to the usual advice. Google-Extended also gates grounding inside the Gemini apps and Vertex, so blocking it can cost you those grounded citations. It does not touch Google Search or AI Overviews, which are governed by Googlebot, not Google-Extended. So "block Google-Extended" means "opt out of Gemini-app grounding and training," not "disappear from Google's AI answers."
Finding 3: llms.txt has crossed 70%, but adoption is lopsided
llms.txt, the proposed index that points an LLM at your key pages, is no longer niche. 31 of 44 sites, 70%, publish one. But the split by category is stark:
| category | llms.txt adoption |
|---|---|
| developer tools | 10 of 10 (100%) |
| scraping APIs | 5 of 5 (100%) |
| GEO / SEO tools | 4 of 8 (50%) |
| media | 1 of 4 (25%) |
Developer and API companies have adopted it wholesale. Media sites, the same ones already blocking AI crawlers, have not. And notably, Anthropic, an AI lab, has no llms.txt at all.
But adoption is not effect, and those are different claims. Our own crawler trap separates them: across a full day of live AI-bot traffic on sites that publish llms.txt, not one AI crawler fetched it. Read this as an observed convention in developer niches, not a demonstrated citation factor. Google states plainly that it does not use llms.txt for Search or its AI features. page_audit measures whether you publish it; it does not claim it wins you anything.
Finding 4: schema is not the ticket in
The advice to "add schema" is everywhere. The data does not support treating it as the price of admission. Only 28 of 44 sites, 64%, carry any structured data, and the ones without it include Anthropic, Notion, Linear, Firecrawl, Supabase, MDN, and the govconapi guide that Gemini cites roughly 85 times a day. Schema helps a machine parse you cleanly. It is not what earns the citation, and shipping it as a checkbox while ignoring crawlability is backwards.
Finding 5: the GEO and SEO category has the worst hygiene relative to what it sells
Grouped by category, the tools that sell search and AI visibility are, on average, the least well configured for it. Of eight GEO/SEO tools, two commit fatal own-goals: orvio blocks the citation crawlers outright, and provioai ships an empty client-rendered page. Those are the failures that actually cost citations. The developer-tools category, which sells none of this, is cleaner where it counts: all ten are fully crawlable by the citation bots, and eight of ten carry schema.
What good actually looks like
A handful of sites carry the full set. Not because a checklist told them to, but because in their niche the cited pages look like this:
| site | schema | llms.txt | sitemap lastmod | headings h1/h2/h3 | words |
|---|---|---|---|---|---|
| scrapingbee.com | AggregateRating, FAQPage, Organization | yes | yes | 1 / 7 / 14 | 2882 |
| brightdata.com | Organization, Product | yes | yes | 1 / 7 / 16 | 1522 |
| twilio.com | Organization, VideoObject, WebSite | yes | yes | 1 / 11 / 1 | 4100 |
| htsapi.dev | FAQPage, SoftwareApplication | yes | yes | 1 / 10 / 8 | 1076 |
Crawlable by every bot, structured, indexed, fresh, and deep enough to answer a question.
What this means for your site
The lesson from 48 sites is that the expensive, interesting GEO work (which queries, what content) is downstream of three boring checks most teams never run:
- Can the citation crawlers fetch you? Read your robots.txt per bot, and know which bot is which. A blanket Disallow that catches OAI-SearchBot or Googlebot can zero out your AI citations. Blocking only the training crawlers (GPTBot, Google-Extended, CCBot) does not, and is a legitimate choice.
- Does the crawler see content, or a shell? If your page renders client-side, the bot may get an empty body while your users see a full page.
- Have you published the basics? A dated sitemap is worth it, crawlers do fetch it. llms.txt is a developer-niche convention (100% of the dev tools here publish one), but no crawler we have logged actually reads it, so treat its presence as hygiene, not a citation lever.
page_audit runs all three in one call: page_audit(url="https://yoursite.com/page"). It returns the per-crawler allow or block, the site files, the schema, and the on-page structure as raw measurements, so you or your agent can catch the own-goals before spending a dollar on content.
Method
48 URLs across seven categories, one page_audit call each, run 2026-08-10. For each we recorded the robots.txt verdict per crawler, grouped by function: the citation crawlers that gate whether an engine can surface you (OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot for Perplexity, Googlebot and Bingbot for Search and AI Overviews) versus the training crawlers (GPTBot, Google-Extended, CCBot, ClaudeBot). We also recorded llms.txt, llms-full.txt, ai.txt, and a dated sitemap, every ld+json type on the page, the H1/H2/H3 counts, the current-year flag, outbound authority links, and word count. Four sites returned 403 or timed out and are excluded from on-page percentages but counted in the crawlability discussion. page_audit is deterministic: no model, no score, same inputs give the same output.
Corrected 2026-08-11: an earlier version treated GPTBot as citation-critical. GPTBot is a training crawler; blocking it does not remove a page from ChatGPT, which indexes through OAI-SearchBot. Our own crawler-trap data (citation is decoupled from training) is the basis for the correction.
A point-in-time measurement. Search results and AI answers change, and grounded models vary between runs, so your own numbers will differ. Verithia measures. The interpretation is yours.