Page one for this question is a Reddit thread and nine agency blogs arguing both sides without a number. Two numbers settle it for a local business: what the sites that already win the local map do, and what an AI answer engine actually sends. I read the first from 40 robots.txt files on 16 September 2026 and the second from our own analytics.
No — not if the site exists to be found. On 16 September 2026 I read robots.txt for the 40 top-ranked Grand Rapids plumbers, roofers, HVAC and electrical companies: 36 have one, 4 name an AI crawler, all 4 allow it, and 0 block any of the 22 AI user agents I checked. On our own sites, read in Plausible on 15 September 2026 for the previous 30 days, ChatGPT sent 57 of crispvideo.app’s 94 visitors (61%) and 14 of outlier.host’s 242. This site got none, because it was barely in any index yet.
The vendors split their bots, in their own words read 16 September 2026. OpenAI: GPTBot is for training and OAI-SearchBot for search, and sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”. Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User when a person asks. Google-Extended controls Gemini training only and “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Perplexity-User “generally ignores robots.txt rules” because a person requested the fetch.
If you object to training, block the training bots by name and leave the search bots alone. If you want the phone to ring, allow all of them — which is what this site’s robots.txt does, by name. And check the default you did not choose: Cloudflare’s new defaults from 15 September 2026 block Training and Agent bots on pages that display ads for new domains, with Search still allowed. Our SEO work starts by reading that file.
The case for blocking is real and it is about training: a crawler reads your pages and a model learns from them, and nobody pays you. The case against is also real and it is about being found: the same companies run the crawlers that decide whether ChatGPT, Claude, Perplexity or Gemini can name your business when someone asks for a plumber. A local trade has almost nothing a model wants to learn and everything to gain from being named, which is why the question has an answer rather than a debate. This page gives the answer with the two measurements and the vendors’ own documentation, and then the one case where a block is the right call and how to write it without breaking the rest of the file.
On 16 September 2026 I fetched /robots.txt for each of the 40 businesses in our 15 September map-pack sweep — the top ten local results for plumber, roofer, HVAC and electrician in Grand Rapids. A file counts when the server answers 200 with at least one User-agent line. A bot counts as blocked when its own named group contains Disallow: /; I also checked whether the wildcard group blocks everything. The 22 agents checked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot, omgili, YouBot, DuckAssistBot, MistralAI-User and Ai2Bot.
| Trade (top ten map results, Grand Rapids, read 16 Sep 2026) | Sites | Have a robots.txt | Name an AI crawler | Block an AI crawler |
|---|---|---|---|---|
| Plumbers | 10 | 8 | 1 | 0 |
| Roofers | 10 | 10 | 1 | 0 |
| HVAC | 10 | 9 | 2 | 0 |
| Electricians | 10 | 9 | 0 | 0 |
| All | 40 | 36 | 4 | 0 |
Of the four without a file, three (Eyk Plumbing, Godwin Plumbing & Hardware, Westshore Mechanical) answer 200 with an empty or non-robots body, and one (All Aspects Electric) did not respond at all. No site’s wildcard group blocks everything either. So the top forty are unanimous by omission: 36 files, and not one of them tells any AI crawler to stay out.
The four that mention AI crawlers do so to let them in. Mr. Rooter’s franchise site names 13 of them — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, CCBot, cohere-ai, YouBot, DuckAssistBot — each with Allow: /. Above Roofing names six (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended), each with Allow: /. Chris Climate and Little’s Heating and Cooling are both on Squarespace, whose standard file lists 14 AI agents and gives each the same path rules as everyone else — /config, /search, /account and so on — which is housekeeping, not a block. Two of the four are explicit invitations; the other two are a platform default that happens to name the bots.
The confusion in the page-one blogs comes from treating “AI crawlers” as one thing. The vendors publish separate agents for separate jobs, and what a block costs you depends entirely on which one you name. All quotes below were read on 16 September 2026 from the pages in the receipts.
| Agent | Owner | What it is for (owner’s words) | What blocking it does (owner’s words) |
|---|---|---|---|
| GPTBot | OpenAI | “used to make our generative AI foundation models more useful and safe” | “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Search is a separate setting. |
| OAI-SearchBot | OpenAI | “used to surface websites in search results in ChatGPT’s search features” | “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” About 24 hours to take effect. |
| ChatGPT-User | OpenAI | “When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” “Not used for crawling the web in an automatic fashion.” | “Because these actions are initiated by a user, robots.txt rules may not apply.” |
| ClaudeBot | Anthropic | “collecting web content that could potentially contribute to their training” | “signals that the site’s future materials should be excluded from our AI model training datasets” |
| Claude-SearchBot | Anthropic | “navigates the web to improve search result quality for users” | “may reduce your site’s visibility and accuracy in user search results” |
| Claude-User | Anthropic | “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” | “may reduce your site’s visibility for user-directed web search” |
| Google-Extended | “manage whether content Google crawls from their sites may be used for training future generations of Gemini models” | “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” | |
| PerplexityBot | Perplexity | “designed to surface and link websites in search results on Perplexity” | Perplexity “recommend[s] allowing PerplexityBot in your site’s robots.txt file” to appear in its results. |
| Perplexity-User | Perplexity | “supports user actions within Perplexity” | “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” |
Put those together and the shape is clear. There are training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot), there are search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and there are user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) that fetch a page because a person asked and mostly do not read robots.txt at all. Blocking the first group costs a local business nothing it can measure. Blocking the second group removes it from the answer. Blocking the third group is mostly a wish.
Kerr & Company runs Plausible analytics on this site and on the studio’s products, so the referral question has a first-hand answer. Read on 15 September 2026 for the previous 30 days:
| Site (30 days to 15 Sep 2026, Plausible) | Visitors | Referred by ChatGPT | Share |
|---|---|---|---|
| crispvideo.app (a Mac video app) | 94 | 57 | 61% |
| outlier.host (a local-AI guide) | 242 | 14 | 6% |
| docketseo.app | 8 | 0 | 0% |
| builtbykerr.com (this site) | 27 | 0 | 0% |
Two readings. First, when an engine names you, it sends people: 54 of Crisp’s 57 ChatGPT visitors landed on the home page, which means ChatGPT is answering “what app does X” with the product’s name and the person clicks through to buy or download. That is exactly the shape of a “who is a good plumber near me” answer. Second, this site got nothing, and the reason is not robots.txt — ours allows every one of these agents by name — it is that on the same day only 2 of its 47 pages were in Bing’s index (Bing Webmaster API, read 15 September 2026). An engine cannot cite a page it has never seen. That is the order of operations for a local business: get indexed, then get named, and never put a wall in front of either. It is also the argument of the page-one reading: what matters is being the answer, not being on the page.
A local trade has almost nothing a model wants to learn and everything to gain from being named.
If you object to your writing and photographs training a model, the vendors give you a switch that does only that. Anthropic’s page shows the syntax — “To block a Bot from your entire website, add this to the robots.txt file in your top-level directory: User-agent: ClaudeBot / Disallow: /” — and the same two lines work for the other training agents:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot untouched and you stay in the AI search results while opting out of training everywhere the opt-out is honoured. Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”; OpenAI says a search opt-out takes about a day to apply.
The trap in writing it: a named User-agent group replaces the wildcard group for that agent — directives do not merge. If your * group disallows /admin/ or a draft page, every named group must repeat that line or the named bot is allowed exactly what everyone else is denied. This site’s own robots.txt carries that warning in its comments because copying another site’s AI block verbatim would have exposed a page the wildcard rule hides. And a block in robots.txt is a request, not a wall: the user-triggered fetchers say they may ignore it, and anyone who wants a wall needs it at the server or the CDN, which is where the next surprise lives.
Cloudflare’s documentation, read 16 September 2026: “On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.” It adds that “mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training”. A trade site rarely displays ads, so the default should not bite; but if your site sits behind Cloudflare and was set up this week, the setting exists whether or not you chose it, and it is worth reading once.
| Option | What happens, per the owners’ documentation | For a local trade |
|---|---|---|
| Allow everything — what 36 of 36 top-ranked Grand Rapids files do | Eligible for ChatGPT search, Claude, Perplexity and Gemini answers; content may also be used for training | Yes. Being named is the point of the site. |
| Block the training agents only (GPTBot, ClaudeBot, Google-Extended, CCBot) | Still in AI search; opted out of training where the opt-out is honoured | Fine if you object to training. Costs nothing measurable. |
| Block every AI agent | “Will not be shown in ChatGPT search answers”; “may reduce your site’s visibility” in Claude; not in Perplexity’s results; Google Search itself unaffected | No. You keep Google and lose the rest for no gain. |
| Leave it to a host or CDN default | Cloudflare, from 15 September 2026: Training and Agent blocked on pages with ads for new domains, Search allowed | Read it once. Defaults change without asking you. |
This site’s robots.txt begins “allow all crawlers, including AI search bots” and then names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and Applebot-Extended, each with Allow: /, each repeating the one page the wildcard rule hides. Every site we build ships with the same posture, and the file is yours: you own the domain, the code and the hosting, so nobody can quietly change it later. The pages themselves are written to be quoted — the answer in the first lines, dated receipts, FAQ markup — because that is what an engine extracts. If you want to know what your file says today, the SEO page describes the technical audit that reads it, and a care plan is where a change like the Cloudflare default gets caught before it costs a month of answers. What a site needs to be found at all — speed, a site at all — is measured on the other pages.
The robots.txt reading. /robots.txt fetched with curl on 16 September 2026 for the 40 businesses in the 15 September 2026 map-pack sweep; 22 agent names checked per named group; wildcard group checked for a full disallow. Raw rows: ops/search/measurements/bbk-robots-ai-2026-09-16.json. This site’s own file: builtbykerr.com/robots.txt, read the same day.
Vendor documentation quoted. OpenAI, “Overview of OpenAI crawlers” (platform.openai.com/docs/bots); Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” (support.anthropic.com); Google, “Google’s common crawlers”, Google-Extended section (developers.google.com); Perplexity, “Perplexity crawlers” (docs.perplexity.ai/guides/bots); Cloudflare, “Block AI bots” (developers.cloudflare.com). All read 16 September 2026.
Our referral numbers. Plausible, sources report, last 30 days, read 15 September 2026 for crispvideo.app, outlier.host, docketseo.app and builtbykerr.com. Bing index coverage for builtbykerr.com (2 of 47 pages) from the Bing Webmaster API, read 15 September 2026.
Only if you object to your pages training OpenAI’s models, and even then block GPTBot alone. OpenAI’s documentation, read 16 September 2026, says GPTBot is for training and OAI-SearchBot is for search, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Of the 36 top-ranked Grand Rapids trade sites with a robots.txt, read the same day, none blocks GPTBot or any other AI crawler; the four that name AI crawlers allow them.
Not Google’s own ranking. Google’s crawler documentation, read 16 September 2026, says Google-Extended controls whether content is used to train Gemini models and does not impact a site’s inclusion in Google Search nor is it used as a ranking signal. What a block does cost is the AI answers: OpenAI says an OAI-SearchBot opt-out removes you from ChatGPT search answers, Anthropic says disabling Claude-SearchBot may reduce your visibility in Claude’s search results, and Perplexity recommends allowing PerplexityBot to appear in its results.
Add a named group for each training agent with a full disallow — User-agent: GPTBot, Disallow: /; the same for ClaudeBot, Google-Extended and CCBot — and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot out of the file so the wildcard rule applies to them. Repeat any path your wildcard group already disallows inside each named group, because a named group replaces the wildcard for that agent rather than adding to it. OpenAI says a robots.txt change takes about 24 hours to reach its search systems.
Send me your web address. I'll read the file and your host's bot settings and tell you in plain words who is allowed in, who is kept out, and whether any of it is costing you an answer.
Works whether or not you end up working with me.