Being found by buyers who research through AI assistants

Why does it matter that buyers ask an assistant before they ask a supplier?

A buyer who needs a supplier can now put the whole question to an assistant, in full sentences, and read a composed answer that names businesses and links to the pages it drew on. Google describes the same shift inside its own results: its AI Overviews and AI Mode produce a generated response with links to supporting websites. For any one business, an answer does one of these things. It names the business accurately, it names a competitor, or it says nothing about the business at all. Exhibit 1 shows where in the buyer's route that last outcome is decided.

What does the evidence say about buyers and the answer?

Here is where the loss happens. The assistant reads what it can reach, composes an answer and names suppliers. A business that isn't in the answer isn't in the shortlist the buyer forms from it, so the buyer never visits, never fills in a form and never appears in a lost-deal note. Every usual measure of demand starts with a contact, so each one reports what it reported before, and the shortfall is put down to the market or to timing.

One measured study shows why the answer itself carries weight. Pew Research Center followed the Google searches of a panel of US adults who agreed to share their browsing activity, for the month of March 2025, and recorded what each person did after a search page loaded. It compared visits to pages that showed an AI-generated summary with visits to pages that didn't. Exhibit 2 shows the actions the study reports. When a summary appeared, visits ended in a click on an ordinary result less often, ended the browsing session more often, and rarely included a click on the links inside the summary.

What do we make of it, and who owns the problem?

What follows is our reading, not a statement of the study. When the answer sits on the page and fewer visits continue to a website, attention goes to the businesses the answer names. The study covers Google only, a panel of adults in one country and one month. It doesn't measure business buyers or any assistant other than Google's. It does report that longer searches and searches phrased as questions produced a summary more often, and a buyer weighing suppliers tends to write that way, which is another inference of ours.

You should care now because the failure sits between teams. Marketing writes the pages. The web team owns the robots file and the network rules. Legal or finance owns the registered name. Communications owns the profiles and listings. Whether an assistant names the business correctly belongs to none of them. This report takes that question apart by mechanism, keeps what the providers document apart from what we infer, and ends with a self-check a marketing or digital lead can run in an afternoon.

What can an assistant read about a business, and what can it not?

An assistant can learn about a business in separate ways, and the providers' own pages treat them as separate jobs. The first is training. A training crawler collects content that may contribute to future models: OpenAI names GPTBot for this and Anthropic names ClaudeBot. Google-Extended is a control token that says whether content Google crawls may train future Gemini models or ground answers in its Gemini apps and Vertex AI. The second is search. A search crawler builds an index ahead of time: Googlebot for Google Search, OAI-SearchBot for ChatGPT search, Claude-SearchBot for Anthropic and PerplexityBot for Perplexity. The third is a fetch at the moment of the question. ChatGPT-User, Claude-User and Perplexity-User each visit a page when a person asks something.

That distinction decides what a business can change. None of the documents we read describes a way to read back what a model learned about a business, so we infer that an answer composed from memory can be out of date. What a business can act on is what an assistant finds and fetches when the question is asked, which is why this report is about pages, rules and names.

Exhibit 3 sets out what an assistant reads as layers: access, pages, structure, identity and other sources. Each depends on the one above it. A page that is perfectly written but refused at the door contributes nothing, and a consistent name that no page states plainly gives the assistant nothing to repeat.

What does an assistant fail to read?

Some things an assistant can't read follow from how the web works. Google's guidance for AI features lists, among the basics, that important content should be available in textual form and that pages should be findable through internal links. Its JavaScript guidance says some sites ship an empty shell and need a script to run before the content exists, that Google renders such pages in a later step, and that server-side or pre-rendering remains a good idea because not all bots can run JavaScript. The OpenAI, Anthropic and Perplexity pages we read don't say whether their crawlers run scripts, so don't assume they do.

Pages behind a login or a form are a second case. Google's JavaScript guidance says to return a meaningful status code, such as 401, for a page behind a login, so the crawler knows it can't crawl or index it. Anthropic states that its bots respect anti-circumvention technology, with the example that it won't try to get past a CAPTCHA. Text inside an image is a third case. In our experience these share one cause: the substance of the business sits in a place designed for a person with a browser, and the assistant meets only the frame around it.

Which crawlers can reach your pages, and who sets the rules?

Access is governed by separate mechanisms that firms often treat as one. The first is the robots file. The Robots Exclusion Protocol, published by the IETF as RFC 9309 in September 2022, lets a site owner state which paths automatic clients may fetch, and says plainly that these rules are not a form of access authorization. It works only on a crawler that chooses to honor it. Google states that its common crawlers always obey robots.txt rules when crawling automatically, and Anthropic states that its bots honor the directives. The second is the network in front of the site: a content delivery network, a firewall or a bot-management rule. Google's own list of basics includes allowing crawling in the robots file and also at the CDN or hosting layer. A rule at that layer refuses a crawler whatever the robots file says.

For Google's own AI features there is a third place. The Search generative AI control, in the Search Console settings, decides whether a site's links and content can appear in AI Overviews, AI Mode and generative features in Discover. Google's help page says the control reached all websites on 31 August 2026 and that sites are included by default. Excluding a site removes its links and content from those features. Google says the control isn't a ranking signal and doesn't affect AI training, which Google-Extended handles.

Exhibit 4 sets out the crawlers that Google, OpenAI, Anthropic and Perplexity document, and what each provider says each one is for. The wording follows the providers' pages, and the table records only what they state.

How do the crawler rules differ, and what does a block cost?

Some patterns in the table are easy to miss. First, each provider separates the jobs, so a business can allow search and refuse training. OpenAI says each of its settings is independent and gives that very example, and adds that when its search and training crawlers are both permitted it may draw on a single crawl for both purposes, so a business that permits both shouldn't suppose it has agreed to separate visits. Google's position differs in kind. Google-Extended is a token with no user agent string of its own, it covers training and grounding for Gemini, and Google says it doesn't affect inclusion in Search. Inclusion in Search stays with the Googlebot rules, which Google's AI features page names as the control, and the Search Console setting described above covers its AI features.

Second, user-initiated fetches follow different rules. OpenAI says that because a person starts a ChatGPT-User visit, robots rules may not apply, and Perplexity says its user fetcher generally ignores them. Anthropic says its bots honor robots directives. So the robots file governs the automatic crawlers, and in practice the network layer decides whether a user-triggered visit is welcome. Perplexity's page gives firewall notes, saying a site behind a web application firewall may need to allow its bots explicitly, and OpenAI recommends allowing its published address ranges as well as the robots entry.

Third, a block costs something the provider names. OpenAI says a site that opts out of its search crawler won't be shown in ChatGPT search answers, though it can still appear as a plain link. Anthropic says that disabling Claude-User or Claude-SearchBot may reduce a site's visibility in search, and for the search bot its accuracy in results too. It adds that blocking by address is no reliable way to opt out, because it stops the crawler reading the robots file at all. Those statements are the reason to treat the training question and the search question as separate decisions.

How does access go wrong, and why does nobody notice?

In our experience access goes wrong in mundane ways, and rarely through a decision anyone remembers making. These patterns recur. A robots file written when fewer crawlers mattered allows those by name and refuses everything else, and the standard explains why that now works against the business: when no group matches a crawler's name, the crawler follows the group for all agents, so a new retrieval crawler inherits the refusal. A template or plugin adds a blanket refusal of AI crawlers when the intent was to refuse training only. A bot-management rule on the content delivery network, switched on to stop scraping, challenges every unfamiliar agent. And a staging setting that disallowed everything is carried into production at launch.

The standard is also unforgiving about errors. If the robots file can't be reached because of a server or network error, the crawler must assume complete disallow. If the server says the file doesn't exist, the crawler may fetch anything. A robots file that errors during a deployment, or on just one of your host names, reads as a closed door. The file sits at the root of each host name, and Anthropic tells owners to repeat any instruction for every subdomain.

The cost is silence. None of the documents we read describes a notice to the site owner when a crawler is refused, and the buyer sees only an answer without the business in it. The delay hides the cause as well. OpenAI says a change to the robots file can take about a day to be reflected, Perplexity says up to a day, and Google says that after a change to preview controls a recrawl can take anywhere from days to months. A business that fixes a block on Monday and tests on Tuesday may conclude that nothing changed.

How do the engines choose and cite their sources?

The providers say more about who can be used than about who is chosen. Google states that to be linked from AI Overviews or AI Mode, a page must be indexed and eligible to appear with a snippet, and its AI features page adds no further technical requirement. Its newer guide to generative AI search adds one condition: the site must be included in Search generative AI features in Search Console, which is how sites start. Google says no special markup, machine-readable file or AI text file is needed, and that meeting every requirement doesn't make crawling, indexing or serving certain. OpenAI says OAI-SearchBot surfaces websites in ChatGPT's search features. Anthropic says Claude-SearchBot analyzes content to improve the relevance and accuracy of search responses. Perplexity describes PerplexityBot as built to surface and link sites in its results, and Perplexity-User as a visit made to answer a question, with a link to the page in the response.

Google also describes how answers are built. AI Overviews and AI Mode may use what it calls query fan-out: issuing multiple related searches across subtopics and data sources, and finding more supporting pages while the response is generated. It says this lets it show a wider and more diverse set of links than a classic search, that it shows an AI Overview only where its systems judge it an addition to classic results, and that the features may use different models, so their responses and links vary. Sites appearing in these features are counted in the Web search type of the Performance report in Search Console, and the newer guide points to a Generative AI performance report there too. A Google-side signal already exists in a tool a marketing lead can open.

None of the documents states a rule for choosing between eligible pages, the number of sources an answer uses, or the weight given to a page against a third party's description of the business. Anyone who gives you a formula for those is working beyond the documents. We can reason from the mechanism, and we mark it as inference. A retrieval step has to find passages that match a question. A page whose opening lines state the answer in plain words offers a passage that stands on its own, for the same reason it would in an ordinary search. A page that says nothing a reader could quote offers nothing, however well the site ranks.

Google's helpful-content guidance is the nearest published statement of what its systems are built to reward, and it is written for Search, not for the other providers. It asks whether content provides original information, reporting, research or analysis, and it asks who created a page, how, and why. The newer generative AI guide leans the same way, recommending a unique point of view over a summary that restates what exists elsewhere. We read that as a statement of what is worth writing, and we don't claim the other assistants follow it.

The llms.txt proposal belongs here because it tries to tell assistants what to read. Version 2 proposes a markdown file at /llms.txt with a name, a short summary and links to pages written for machines, and suggests markdown copies of pages. It says the file is used on demand, when an agent needs information, and mainly for inference, not training. It is a proposal by one author. Google says no AI text file is needed for its features, that Google Search ignores such files, and that keeping one for other services neither helps nor harms visibility there. The OpenAI, Anthropic and Perplexity crawler pages we read don't describe it as an input. If the file is cheap to publish and kept in step with the site, it does no harm. It doesn't replace pages a person can read, and we don't count its absence as a cause of disappearance.

Why do some reachable pages still go unused?

A reason we meet often when a business is reachable and still not named is that its pages contain no sentence worth repeating. As we read the mechanism, an assistant answering "who helps a clinic group replace its booking system?" needs text that connects a kind of business, a problem and a capability. Pages written in the language of a brand campaign, with a mission and a list of values, connect none of them. Pages that list service names with no statement of what each does, for whom, or what problem it removes give the assistant a noun and nothing to say about it. We infer that a competitor whose page says what it builds, for whom, and why is easier to use, for the same reason it is easier for a person to understand.

A second pattern is the claim with no source. A page says the firm is trusted, leading or experienced, and offers no method, no named document and no evidence a reader could follow. A person discounts such a claim, and we would expect a retrieval system to have the same problem in another form: nothing in the page can be checked or attributed, so it gives no support for a sentence an answer might want to make. Google's helpful-content guidance, which asks about original information, clear sourcing and background on the author, points the same way. Evidence beside the claim turns a statement into something that can be cited.

A third is the page that answers no question. Sites grow by department and by campaign, and the result is pages organized around the firm's structure instead of the buyer's problem. A buyer asks what a failed data migration costs to repair, and the site has a page called Solutions and a page called Team. A page can also fail when its title and description say something other than its content: Google says it builds snippets mainly from the page content and uses the description element only when that describes the page better. Eligibility for Google's AI features is tied to being shown with a snippet, so the text that produces the snippet matters.

The fourth is content behind scripts and forms. The usual version in firms that sell expertise is the gated report: the substance sits behind a form, and the public page carries a title, a button and a line of promotion. A person fills in the form. A crawler can't, so the assistant sees that a document exists and nothing of what it says. You don't have to ungate everything. A public page that states what the document concludes and how it was made, with only the full asset gated, does the job. Without one, the assistant never sees the firm's best thinking.

Does the assistant know which business you are?

Naming is the mechanism people underrate, because a human reader fills gaps without noticing. A firm trades under a short name, is registered under a longer one, calls its services by internal product names, was renamed after an acquisition, and appears in directories under whatever a past employee typed. An assistant composing a sentence about the business has to decide that these mentions are one entity. In our experience, when they aren't obviously one, the results are poor: the business is confused with a namesake, the answer is built only from the mentions that match the question's wording, or the description is taken from a directory and not from the firm.

Google documents some help here. It says organization markup on the home page can help it understand a business's administrative details and disambiguate the organization in search results. The Schema.org Organization type carries properties for this: a legal name, an alternate name for an alias, and a sameAs property for the address of a reference page that unambiguously indicates identity, such as a Wikipedia page, a Wikidata entry or an official website. Google adds that structured data must match the visible text, and that no special schema is required for its AI features. We treat markup as a way to say who the business is. We don't count it as a lever for being chosen.

Profile data outside the site matters too. Google's AI features guidance lists keeping Merchant Center and Business Profile information up to date among the basics, and its newer guide says generative AI responses can include information about local businesses. For clinics, hospitality and retail, found through map listings and review sites as much as through their own pages, an out-of-date listing may be the first thing an assistant finds. For legal and real estate firms the equivalent is the directory and the trade body's member page.

Other people's pages matter too, and the Pew study gives the only published evidence we use. It reports that the sources Google listed more often than any others, in its AI summaries and in ordinary results, were an encyclopedia, a video site and a discussion forum: sources the businesses in an answer don't control. The study covers Google results for one panel's searches in a single month, and says nothing about how any other assistant weighs sources. It does show that an answer isn't built only from the websites of the businesses it names. In our experience the symptom is an answer that describes a firm in a directory's wording, with a service the firm dropped or never offered, and the fix lies outside the website.

Why does a business disappear from answers, and how do you find out which reason applies?

Put side by side, the mechanisms give the reasons a business goes missing, set out in Exhibit 5. Access rules refuse the crawlers: bot rules, network rules and, for Google's AI features, the Search Console setting. Pages can't be read because their text lives in scripts, forms or images. Pages answer no question a buyer asks. Claims carry no source. The name varies, so mentions don't merge. And no one else says the thing the business wants to be known for, so nothing corroborates it. The first is drawn in blue because a firm can settle it with certainty, from documents the providers publish.

We can't say how likely each reason is, because we hold no count, and the reasons aren't independent. A firm refused at the door gives no evidence about its pages, and a firm with sound pages can still be mistaken for a namesake. The order of diagnosis therefore matters more than the list. Exhibit 6 sets it out as a sequence of questions: whether the crawlers can fetch, index and show the pages; whether the pages answer the buyer's question in plain text; and whether the business is named the same way wherever it appears. Only when all of those are answered yes does the work move to other sources, and to the next test.

What changes the outcome?

These changes alter the outcome, and we are surer of some than of others. The first is documented: decide access deliberately, as separate decisions, whether search and retrieval crawlers may reach the pages that answer buyers' questions and, separately, whether training crawlers may. Keep the key pages indexable and eligible for a snippet, and for Google check the Search Console setting too. The second follows from the mechanism: give each page that should be found an opening that states what the business does, for whom and for what problem, in a buyer's words, with the evidence beside the claim. The third is consistency: one name, one short description and one list of services, on the site, in markup, in listings and in profiles. The fourth lies outside the site: the directories, trade bodies and publications that describe the business should say the same thing, and you can ask for corrections. The fifth is measurement: ask the buyer's questions on a schedule, keep the answers, and change one thing at a time where you can.

Exhibit 7 arranges these as a ladder, and its break sits in the middle. The first rungs are technical and belong to the web team. From the third the work is words and agreement, and it stalls unless a named person owns it. In our experience the technical fixes get done because they arrive as tickets with an owner, while the rungs above wait because their owner is "marketing" or "the website", which means nobody. The change that makes the rest possible is organizational: one person who owns the question of whether an assistant names the business correctly, with standing to ask the web team for a rule change and the communications team for a corrected profile.

What do these changes not do?

These changes have limits. Fixing access removes a documented reason for absence and doesn't make a name appear. Clear pages and a consistent identity remove reasons for being skipped or mistaken, but the providers don't document that they cause selection, and no one can promise a citation. Results arrive late, because a recrawl takes time, and a single answer on a single day is a sample of one. Exhibit 8 shows the order of a fix in time and the point at which results lag the work.

Nor is there a trick to add. Google states that no special optimization is needed for its AI features beyond the basics of Search, and its newer guide says you can ignore tactics such as chunking content, writing AI text files and chasing inauthentic mentions. Nothing in the other providers' documents describes a technique that works. The work is access, plain pages, one name and a record of what the assistants say.

What is the step-by-step response?

1. 01

   Name an owner and write the questions

   The marketing or digital lead takes the question, with a sales lead who knows what buyers actually ask. Together they write the questions in buyers' words: category questions (which firms help a business like this with this problem), comparison questions, questions that name your business, questions for each service and sector you want to be known for, and the question asked after a supplier has failed. Include questions you expect to lose. At the end there is one set of questions, each tied to the page that should answer it.

2. 02

   Ask, and record the baseline

   The owner, or an analyst, puts each question to more than one assistant, in a fresh session without a signed-in history. The log holds the date, the assistant, whether the business is named, how it is described, which pages are linked and who is named instead. At the end there is a dated baseline. It is the only measurement available, and every later step is judged against it.

3. 03

   Read access as each crawler would

   The web or infrastructure lead fetches the robots file on every host name, confirms that it returns a normal page and not an error, and reads each group of rules. For every crawler in Exhibit 4 they record whether it is allowed or blocked and who decided. They check the content delivery network, the firewall, any bot-management rule and the server log for the crawlers' user agents and published address ranges, and they confirm in Search Console that the site is included in Search generative AI features. At the end there is a one-page access record: crawler, decision, reason, date.

4. 04

   Check eligibility and the plain text of the key pages

   The web lead and the content owner take the pages that should answer the questions. In Search Console they inspect each as Google received it, and confirm none carries a noindex or nosnippet instruction nobody remembers adding. They then read each page with scripts off and without signing in: does the substance remain? At the end there is a list of key pages with their indexing, snippet and plain-text status.

5. 05

   Rewrite the pages that answer nothing

   The content owner and a subject expert revise the pages the list marks. Each opens with a sentence saying what the business does, for whom and for what problem. The home page carries one statement of what the business is. Brand language gives way to plain statement, evidence sits beside each claim, and every gated asset has a public page stating what it concludes. At the end there are revised pages that each contain a sentence worth quoting.

6. 06

   Fix the name where you control it, then ask others to fix it

   The communications or brand owner, with legal, agrees one name, one short description and one list of services, and applies them to the site, titles, organization markup (legal name, alternate names, and sameAs links to official profiles, all matching the visible text), map and business listings, directories, trade-body pages and the press boilerplate. They then write to the owners of pages that are not theirs and ask for corrections. At the end there is a naming sheet and a set of corrected profiles.

7. 07

   Wait, ask again, and keep the change log

   The owner waits for the engines to refetch, then repeats the baseline step with the same questions and assistants and compares the answers with the baseline. A change log records what changed, when, and what moved. The check repeats on a schedule and after any change to the robots file, the network rules or a key page, because the products and their documents change. At the end there is a comparison and a note of what to change next.

What can you check in an afternoon?

• We have opened our robots file on every host name we use, and it returns a normal page and not an error.
• For each crawler in Exhibit 4 we can say whether it is allowed or blocked, who decided, and why.
• Our content delivery network, firewall and bot rules do not refuse the search and retrieval crawlers by user agent or by address range.
• We decided in writing, and separately, whether training crawlers may fetch our content.
• Our key pages are indexed in Google Search and shown with a snippet, none carries a noindex or nosnippet instruction nobody remembers adding, and our site is included in Search generative AI features in Search Console.
• With scripts off and without signing in, each key page still states what we do, for whom and for what problem.
• Each service page opens with a sentence that could be quoted on its own and still be correct.
• Every claim of capability on a key page has its evidence beside it: a method, a document or a named source.
• Anything we gate behind a form has a public page that says what it concludes and how it was made.
• We use one business name, one short description and one list of services on our site, in our markup, in our listings and in our profiles.
• Our map listings, directory entries and trade-body pages show our current name, address and services.
• We hold a written set of buyer questions, in buyers' words, for each service and sector, and have put them to more than one assistant.
• For each answer we recorded whether we were named, how we were described and who was named instead.
• One named person owns the question and the log, and can ask the web and communications teams for changes.
• We repeat the check on a schedule and after any change to the robots file, the network rules or a key page.

What do the answers tell you to fix?

When the answers are in, read each one for these things: whether the business is named, whether it is described correctly and in its own terms, which pages are linked, and who is named instead. The table maps what you see to the first place to look. It is a set of starting points drawn from the mechanisms above, not a diagnosis, and one answer is one sample.