What do you lose by blocking GPTBot, ClaudeBot or Google-Extended?

A furniture company sees odd lines like GPTBot and ClaudeBot in its server log. The owner wants to block everything, while marketing wants customers who ask ChatGPT to still see the company name. This is an illustrative scenario.

This article draws on documents from OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, RFC 9309 and an IETF draft, checked on November 4, 2026.

The short answer: each company runs several bots with different jobs, so blocking a training bot usually leaves that company's AI search alone. Google-Extended is the exception: blocking it also costs a page its place in Gemini Apps answers.

The article gives 3 tested robots.txt samples and a 5-step framework.

Key points

• OpenAI and Anthropic each run training bots, search bots and bots that run when a user asks AI to read 1 page, each with its own name in robots.txt; Perplexity has 2 kinds, and Google adds the Google-Extended token.
• Per OpenAI, GPTBot and OAI-SearchBot are set independently. Blocking OAI-SearchBot keeps a page out of ChatGPT search answers.
• Google-Extended is only a control token, not a bot, and does not affect Google Search. Google says a site that explicitly allows it is eligible to appear in Gemini Apps; blocking it keeps the content from being used there.
• robots.txt is a request, not a lock. OpenAI, Perplexity and Google say user-triggered bots may ignore it; Anthropic says its bots honor it.
• On October 30, 2026 Google and Microsoft announced plans to support a shared token, __usage_ai-training, now an IETF draft.

How many kinds of bots does an AI company run

OpenAI and Anthropic split the work among 3 kinds of bots (Click Star's summary of their documents).

A training bot collects content that may be used to train AI models. A search bot indexes pages so they can appear in answers. A user-requested bot runs when someone asks AI and AI opens that page.

What each company's bots do and whether they follow robots.txt

OpenAI

According to "Overview of OpenAI Crawlers" (no date shown), GPTBot collects content that may be used for training; blocking it signals the content should not be used that way. OAI-SearchBot puts pages into ChatGPT search.

A page blocking OAI-SearchBot does not appear in ChatGPT search answers, though it can still show as a navigational link.

ChatGPT-User runs when a user asks ChatGPT to open a page. OpenAI writes that because the user starts the action, robots.txt rules may not apply.

Anthropic

The article "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (updated April 7, 2026) lists 3 bots. ClaudeBot collects content that may be used for training, Claude-User visits when a user asks Claude, and Claude-SearchBot improves search results.

Anthropic says its bots honor "do not crawl" signals in robots.txt. Blocking Claude-User or Claude-SearchBot may reduce visibility in Claude's search.

Google

Per "List of Google's common crawlers" (updated October 30, 2026), Google-Extended has no user agent string of its own; it is only a control token.

The token governs whether content may be used to train Gemini and for grounding (feeding content from the Search index to the model at answer time) in Gemini Apps.

Google writes that it does not affect presence in Search. Google's post of October 30, 2026 says a site that explicitly allows Google-Extended is eligible to appear in answers and get clicks through grounding in Gemini Apps.

For AI Overviews and AI Mode, "AI features and your website" (updated December 10, 2025) says a page only needs to be indexed and snippet-eligible. Controls are robots.txt for Googlebot, plus snippet limits and noindex.

Per "List of Google user-triggered fetchers" (updated August 19, 2026), user-triggered fetchers such as Google-Agent generally ignore robots.txt.

Perplexity and Apple

"Perplexity Crawlers" (no date shown) says PerplexityBot serves search and is not used to collect content for foundation-model training. Perplexity-User generally ignores robots.txt.

"About Applebot" (published September 4, 2026) says Applebot data powers Apple's search and may help train Apple's models. To opt out of training, block Applebot-Extended (a token that does not crawl).

How to verify real bots and count them in logs

A user agent string in a log can be faked, so check the IP against each company's published ranges and reload the files regularly.

• OpenAI and Perplexity: an IP-range file per bot, such as gptbot.json, perplexitybot.json. Anthropic: 1 shared file for all its bots, claude.com/crawling/bots.json.
• Google: reverse DNS (looking up the host name from the IP) crawl-***.googlebot.com or the common-crawlers.json file, per "Verify requests from Google crawlers and fetchers" (updated March 20, 2026).
• Apple: reverse DNS *.applebot.apple.com. Bing: the bingbot.json file (last modified November 1, 2026; not checked against Bing's guidance page).

This Python snippet checks whether 1 IPv4 belongs to a range file:

import ipaddress, json
ds = json.load(open("gptbot.json"))["prefixes"]
ip = ipaddress.ip_address("20.171.206.77")
print(any(ip in ipaddress.ip_network(p["ipv4Prefix"]) for p in ds if "ipv4Prefix" in p))

Count hits per bot with grep -c "GPTBot" access.log. Click Star ran it with IP files downloaded on November 4, 2026 and a fake 5-line log: the count gave 2 GPTBot lines. IP 20.171.206.77 against gptbot.json returned True, while 8.8.8.8 returned False.

3 robots.txt samples for 3 goals

Sample A opens everything: no bot-specific group, only User-agent: * with Disallow: /admin/. Sample B blocks 4 training tokens and keeps search bots. Consecutive User-agent lines share 1 rule group:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /admin/

Sample B keeps the search bots of OpenAI, Anthropic and Perplexity, and Google Search. But Google-Extended also controls grounding in Gemini Apps, so sample B loses the place and clicks that grounding brings there (Click Star's inference from 2 Google pages).

Sample C is sample B plus 6 more User-agent lines in the first group (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), blocking all 10 AI tokens while Googlebot, Bingbot and Applebot stay in.

Click Star checked the 3 samples on 14 user agents and 3 paths (/, /blog/bai-1, /admin/x) with Python 3.9.6's urllib.robotparser and Protego 0.5.0. 2 parsers agreed on all 126 cells: sample B blocks 4 tokens on every path, sample C blocks 10.

Click Star also tried 2 cases where results differ. First, a bot group does not merge with the * group: with User-agent: * blocking /admin/ and an OAI-SearchBot group containing only Allow: /, OAI-SearchBot can read /admin/x in both parsers.

Second, line order. For a GPTBot group with Disallow: / then Allow: /blog/, Protego allows /blog/bai-1 while urllib.robotparser blocks it. With Allow first, both allow it. RFC 9309 and Google use the most specific rule (longest path), so do not rely on a single parser.

The limits of robots.txt

RFC 9309 (Robots Exclusion Protocol, IETF, September 2022) says these rules are not a form of access authorization.

To keep content private, use login, permissions or keep it off the web.

The shared signal announced on October 30, 2026

On October 30, 2026, Google and Microsoft announced plans to use 1 shared token instead of listing every bot.

The IETF draft "Machine-Readable Training Preference Signal" (October 30, 2026, expires May 3, 2027, author listed with Anthropic) defines the token __usage_ai-training, written like a User-agent: User-agent: __usage_ai-training then Disallow: /. The IETF has not adopted this draft.

Per the draft, a User-agent: * group must not be treated as matching this token. Yet with a file holding only User-agent: * blocking /, both urllib.robotparser and Protego still block the token in Click Star's test, so do not test the shared token with today's parsers.

Google's post "Supporting a shared AI training control for websites" (October 30, 2026) says that if a site opts out of training with the shared token without naming Google-Extended, Google plans to treat it as blocking Google-Extended, and aims to support it in 2027 together with other AI providers (Google's crawler page says early 2027).

Microsoft Bing's post the same day splits 2 choices. "AI Training" goes in robots.txt or on each page. "AI Grounding" goes on each page only, for example <meta name="Bingbot" content="ai-grounding:disallow">. Both default to allow.

Microsoft says blocking grounding reduces brand visibility in AI experiences, does not affect Bing results.

What this means for Vietnamese businesses

Click Star has found no public measurement of AI bot traffic to Vietnamese websites, so count your own logs.

Click Star's view. For a business that wants customers to find it, sample B is usually reasonable: refuse training and keep the search of OpenAI, Anthropic, Perplexity and Google Search, but lose its place in Gemini Apps through the Google-Extended line. This is Click Star's judgment, as no company has published how blocking affects citations.

Google-Extended covers both training and grounding, so blocking it loses the Gemini Apps place. To keep it, drop the Google-Extended line from sample B, which accepts Gemini training. Sample C fits only when the content is an asset sold directly.

Click Star offers SEO-ready, fast website design and AI integration, so it has an interest here.

What to do: a 5-step framework for choosing

Click Star's reference framework, not from an outside source.

Step 1. Fix the goal: be found in AI, or keep content private. Step 2. Sort the pages: promotional pages, price pages, paid documents, member areas.

Step 3. Pick sample A, B or C for each group; keep private content behind login.

Step 4. Write it and check it with at least 2 parsers on 3 or more paths, publish and wait (OpenAI and Perplexity cite about 24 hours). Step 5. Review logs monthly and revisit when a company changes, especially Microsoft's February 1, 2027 date.

Common mistakes

1. Blocking GPTBot and believing ChatGPT will never mention the site, although ChatGPT search uses OAI-SearchBot.
2. Blocking a search bot without realizing it lowers the chance of appearing in that company's search.
3. Adding a group for 1 bot and forgetting it does not merge with the * group.
4. Putting Disallow before Allow and trusting every parser to read it the same way.
5. Trusting the user agent in logs without checking the IP.
6. Using robots.txt instead of login, or blocking a bot's IP so it cannot read robots.txt.

What is not known

There is no public Vietnamese measurement of AI bot traffic or of how many businesses block. No company has published how much blocking each bot changes citations, only that a blocked page loses or may lose visibility.

The article reports each company's own claims about robots.txt, not measurements. OpenAI's post on the shared signal was blocked to Click Star's tools, so OpenAI's position is unconfirmed.

It is unclear when OpenAI and Anthropic will read the shared token: Google aims for 2027 together with other AI providers it does not name, and Microsoft starts February 1, 2027. Google also does not say how a site with no Google-Extended rule is treated in Gemini Apps.

The samples were not tried with real bots. Click Star has not read Vietnamese law on using web content for AI, so this article is not legal advice.