Auditing a site for AI visibility, not just SEO

Every agency has a story about the client who ranked well on Google and still lost enquiries. There is a new version of that story now: the business that ranks fine and never gets cited by an assistant. When somebody asks ChatGPT or Perplexity for "a good emergency plumber in Leeds", the answer is assembled from what those systems can fetch, parse and trust, and that is a measurably different set of signals from the ones a classic SEO audit looks at.

Access comes first, and it is where most sites fail

Before any of the content questions matter, an answer engine has to be allowed in. A surprising number of sites block the very crawlers they would benefit from, sometimes because a security plugin added a blanket rule, sometimes because somebody pasted a robots.txt from a site that had made a deliberate choice to opt out.

So the first thing we check is plain: is GPTBot allowed? ClaudeBot? PerplexityBot? Google-Extended? A site can be immaculate in every other respect and be invisible because of four lines in a text file. That is also the easiest finding in the world to convert into a conversation, because it is unambiguous and it takes five minutes to fix.

Search crawlers and training crawlers are different decisions

Most AI companies now run more than one crawler, and they do different jobs. OpenAI's GPTBot collects training data, while OAI-SearchBot is what lets a site appear in ChatGPT's search results. Google-Extended controls whether content is used to train and ground Gemini, and has no effect on ordinary Google Search. Anthropic and Perplexity also separate the crawler that builds an index from the agent that fetches a page because a user asked.

That gives a business a real choice: it can decline training and still be findable in AI search. A robots.txt that blocks everything with "bot" in its name makes that choice badly, by accident. Google is clear that its own AI features draw on the normal search index, so a page that is not indexable in Google Search will not appear in an AI Overview either.

llms.txt is the least interesting part

llms.txt gets the attention because it is new and easy to write about. We do check for it, but honestly, it is a weak signal. It is a proposal, adoption is uneven, and no assistant currently treats it as load-bearing. Selling a client on an llms.txt file as an AI-visibility strategy is selling them a deliverable, not an outcome.

What actually moves the needle is duller: structured data that is present and valid, an entity the model can resolve (this business, this location, this service, stated unambiguously in text), content that answers questions in the form people ask them, and consistency between what the site claims and what third parties say about it.

Answerability is a content property, not a markup property

Assistants quote passages. A page that buries "we open at 7am on Saturdays" inside a paragraph about the team's values is harder to quote than a page with a heading that asks the question and a sentence that answers it. This is why FAQ content keeps outperforming its reputation, not because FAQPage schema is magic, but because the format forces a question-and-answer shape that is trivially extractable.

A ten-minute check you can run by hand

1. Open /robots.txt. Look for Disallow: / under GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended, and for a catch-all rule that blocks every agent.

2. Check the page is indexable. No noindex in the page source, and a canonical that points at itself.

3. Read the home page as a stranger. Does one sentence near the top say what the business is, where it is and what it does? If you have to infer it, so does a model.

4. Look for the facts people ask about. Opening hours, prices or price ranges, the areas served, how to book. Each should be stated in text, not only in an image or a PDF.

5. View the structured data. An Organization or LocalBusiness node with the same name, address and phone number as the Google listing.

Anything that fails here is a finding you can show the owner on their own site in under a minute.

What we score, and what we deliberately do not

Our AI-visibility category combines crawler access, structured-data presence and validity, entity definition, answerable content patterns, and citation signals. What it does not do is claim to measure your actual share of AI answers. Nobody can measure that reliably at the moment: the outputs are non-deterministic, personalised and unlogged, and a score that pretends otherwise would be the kind of number that falls apart the first time a client checks it.

That distinction matters commercially. "Your site blocks three of the four major AI crawlers, here are the lines that do it" is a claim you can defend in a meeting. "Your AI visibility score is 42" is not, unless you can say exactly what produced the 42, which is why every finding in our reports carries its measurement, and why the methodology is public.

To see where a site stands, the free AI robots.txt generator shows which AI crawlers a site allows, the AI visibility page explains how the category is scored, and a free website audit runs the fast checks with no account.

Questions

What is an AI visibility audit?

A check of whether answer engines such as ChatGPT, Perplexity and Google's AI features can reach a site, understand what the business is, and find content in a form they can quote. It covers crawler access, structured data, entity definition and answerable content.

Does blocking GPTBot remove a site from ChatGPT?

Not on its own. GPTBot collects training data; OAI-SearchBot is the crawler behind ChatGPT's search results. A site can block the first and allow the second.

Is llms.txt required?

No. It is a proposal that some sites publish, and no major assistant treats it as a requirement. It does no harm, but crawler access and clear content matter far more.

Can anyone measure a site's share of AI answers?

Not reliably. Answers vary between runs, users and locations, and the systems do not publish logs. Treat any single "AI visibility score" as a readiness measure, not a measure of how often a business is cited.