robots.txt for AI crawlers, bot by bot

robots.txt lets you treat AI bots by purpose: training, search indexing and fetches a person asked for. Most brands that want to appear in AI answers should allow the search bots, then decide on training separately. Blocking GPTBot doesn’t remove you from ChatGPT search, while blocking OAI-SearchBot does. Some user-triggered fetchers may ignore the file altogether.

robots.txt is also a request, not a lock, and allowing a bot doesn’t mean any assistant will name you. Bot names and purposes below come from each operator’s own documentation, opened on 6 November 2026. Tokens and policies change often, so recheck before you rely on them.

Which AI bots should you know about?

The table lists the main bots by what each operator says they do. “Follows” means the operator says or shows that robots.txt rules apply, and “may ignore” means it says they may not.

OpenAI: OAI-SearchBot, GPTBot and ChatGPT-User

OpenAI’s crawler page separates search from training, and says each robots.txt setting is independent of the other. If you allow both, OpenAI may use one crawl for both purposes. Changes can take about 24 hours to reach search results.

• OAI-SearchBot. Blocking it keeps your site out of ChatGPT’s search answers. It may still appear as a navigational link. OpenAI also says to allow its published IP ranges, not just the robots.txt rule.
• GPTBot. Blocking it signals that your content shouldn’t be used to train OpenAI’s foundation models. It doesn’t change ChatGPT search eligibility.
• ChatGPT-User. This one visits pages for certain user actions in ChatGPT and Custom GPTs. OpenAI says robots.txt may not apply, and that it doesn’t decide search appearance.
• OAI-AdsBot. This one only visits pages submitted as ads on ChatGPT. Its data isn’t used for model training. The page doesn’t say whether robots.txt applies.

Google: Googlebot and Google-Extended

Google has no separate robots.txt token for AI Overviews or AI Mode, so Googlebot is the crawl control for both. Google’s crawler documentation says Googlebot rules cover Search, including all its features, plus Images, Video mode, News and Discover. Blocking it to avoid AI Overviews would therefore cost you ordinary search too.

Google-Extended is a token with no crawler of its own, so you’ll never see it in your logs. It controls whether content Google has crawled may train future Gemini models, and whether it may ground answers in Gemini Apps and on Vertex AI. Grounding means giving the model Search index content when someone asks. Google says the token doesn’t affect Search inclusion or ranking. Its AI features page describes it as a control for Google’s other systems and points to Googlebot for Search. Google also notes that allowing it keeps you eligible for grounding in Gemini Apps.

To leave AI Overviews and AI Mode without leaving Search, robots.txt isn’t the tool. Search Console’s Search generative AI control is a site-wide setting, which Google says reached all websites by 31 August 2026. Excluding your site removes its links and content from AI Overviews, AI Mode and generative AI features in Discover. Google says it isn’t a ranking signal elsewhere in Search and that it doesn’t affect training. Content from other sites stays available, and may look similar to yours. For control over less than a whole site, Google lists nosnippet, data-nosnippet, max-snippet and noindex.

Three other Google names come up. Google-CloudVertexBot crawls for Vertex AI Agents when a site owner requests it, with no effect on Search. GoogleOther is a generic crawler for product teams, and its rules don’t affect any specific product. User-triggered fetchers, including Google-Agent and Google-GeminiNotebook, generally ignore robots.txt.

Anthropic: ClaudeBot and its others

Anthropic’s page describes three bots and says they honour robots.txt. Rules apply per subdomain, so add them to each subdomain you want to opt out.

• ClaudeBot. It gathers content that may be used for training. Blocking it should exclude your future content from training datasets, which says nothing about content already collected.
• Claude-SearchBot. It indexes pages to improve search results. Anthropic says blocking it may reduce your visibility and accuracy in user search results.
• Claude-User. It fetches pages when someone’s question needs them. Blocking it stops those fetches, and may reduce your visibility in user-directed search.
• Claude-Agent. A separate article covers Anthropic-hosted browsers doing tasks for users. They use a normal browser user agent with this token added, and publish their IPs at claude.com/.well-known/sail. The article doesn’t say how they treat robots.txt.

Anthropic supports the non-standard Crawl-delay line, and says robots.txt is the documented opt-out, because IP blocking may not work reliably.

Perplexity: PerplexityBot and Perplexity-User

Perplexity’s page says PerplexityBot surfaces and links sites in its search results and isn’t used to crawl content for AI foundation models. Blocking it should therefore cost you those results, not stop training. Perplexity-User makes fetches a person asked for, and generally ignores robots.txt. Changes may take up to 24 hours.

Whether Perplexity always keeps to this is disputed. In August 2025 Cloudflare reported, from test domains it set up, that Perplexity fetched blocked sites using an undeclared browser user agent. Cloudflare sells bot management. Perplexity denied it. We haven’t tested this ourselves.

Microsoft: Bingbot and new AI preferences

Microsoft’s crawler page lists Bingbot as its standard crawler, alongside AdIdxBot for ads, BingPreview, MicrosoftPreview and BingVideoPreview for snapshots. The page lists no separate token for Copilot. Blocking Bingbot affects Bing search, and it also stops Microsoft finding the page-level preferences described next.

From 1 February 2027, Microsoft says it will honour two new preferences, which control use and not crawling. AI Training can be set in robots.txt or on a page. AI Grounding, which covers Bing, Copilot and Microsoft 365 answers, can only be set on a page, using a meta tag or header. Neither changes Bing search eligibility. After 1 March 2027, NoArchive and NoCache stop being read as training or grounding preferences.

Other bots: Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl

• Apple. Applebot powers search in Spotlight, Siri and Safari, and its data may help train models. Disallowing Applebot-Extended opts out of training and leaves search results alone. If you have no Applebot rule but do have a Googlebot rule, Applebot follows the Googlebot one.
• Meta. Allowing meta-webindexer helps Meta cite and link you in Meta AI. meta-externalagent covers training and direct indexing. meta-externalfetcher may bypass robots.txt.
• Amazon. Amzn-SearchBot makes content eligible for experiences such as Alexa and doesn’t crawl for training. If it isn’t named, it follows the rules you give other search bots. Amazonbot may train models. Amzn-User may not follow every rule. Amazon uses a cached copy of robots.txt from the last 30 days, while saying settings changes may take about 24 hours.
• Mistral. MistralAI-Training builds training data. MistralAI-Index feeds search and isn’t used for training. MistralAI-User handles requests in its Vibe product.
• DuckDuckGo. DuckAssistBot fetches sources for AI-assisted answers, and the data isn’t used for training. Opting out takes about 72 hours and doesn’t affect organic rankings.
• Common Crawl. CCBot builds an open dataset that anyone can analyse. Blocking it affects your future presence in that dataset, whoever uses it.

We couldn’t find ByteDance’s own documentation for Bytespider, so we don’t describe it here.

Copyable robots.txt examples

Each example is a complete, valid file. Merge it with your existing rules, and keep your own Disallow paths and Sitemap line.

Allow AI search, opt out of training. This suits most brands that want to be found but don’t want their content used for training. Blocking Google-Extended also limits grounding in Gemini Apps, so remove that line if you’d rather stay eligible there.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Amazonbot and meta-externalagent do more than training by their operators’ accounts, so blocking them may cost other uses.

Allow everything, by naming nothing. This is the simple option. A bot with no matching group follows User-agent: *.

User-agent: *
Allow: /

Stay in ordinary search, block the AI bots that respect rules. Search engines stay open because Googlebot, Bingbot and Applebot also run ordinary search. This won’t stop fetchers that may ignore the file, and it won’t remove you from AI Overviews or Copilot. For AI Overviews, use the Search Console control.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

The shared training token. This new line states a preference against AI training across providers that support it. It’s from an individual IETF draft dated 30 October 2026, which the IETF hasn’t endorsed.

User-agent: __usage_ai-training
Disallow: /

Support for it is uneven. Anthropic says it’s rolling out recognition of this kind of signal now, and a rule naming ClaudeBot takes priority. Microsoft plans to honour it from 1 February 2027. Google aims to support it in 2027, and would read it as a Google-Extended block unless you name Google-Extended yourself. OpenAI’s crawler page doesn’t mention it. The draft says User-agent: * groups don’t count as this token, so a blanket rule isn’t read as a training preference.

A bot with its own group ignores your general rules

A bot follows the group that names it and ignores your User-agent: * rules. RFC 9309, the robots.txt standard, makes the * group a fallback only. So in this file, GPTBot may crawl /private/:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/

To keep it out of both, repeat the * paths in the GPTBot group. Rules apply per host too, so docs.example.com needs its own file. Under the RFC, a server error on robots.txt means crawlers must assume everything is disallowed, while a 404 lets them access anything.

What robots.txt can’t do

robots.txt asks well-behaved crawlers to stay away from paths. It can’t do most of what people want from it:

• It isn’t access control. The RFC says the rules aren’t a form of access authorisation, and the file itself is public, so it lists the paths you’d rather hide. Use authentication for anything private.
• It doesn’t keep a URL out of search. Google says a blocked URL can still be indexed if other pages link to it. To keep a page out, use noindex, and let the crawler reach the page so it can read that.
• It doesn’t reach content already collected. Anthropic’s wording covers future content, and we found no operator page promising to remove what it already holds.
• It doesn’t reach other websites. Reviews, directories and articles about you are on other sites, and assistants can still read those.
• It doesn’t bind fetchers that may ignore it. OpenAI, Google, Perplexity, Meta and Amazon each describe user-triggered fetchers that may not follow it. Block those at your firewall if you need to, knowing the person who asked won’t get your page.
• It doesn’t verify anyone. User agents can be faked, as Microsoft and Common Crawl both warn. Check against the operator’s published IP list before trusting a request.
• It doesn’t override your CDN. A firewall can refuse a bot your file allows, and teams often don’t notice. OpenAI and Google both tell site owners to allow their crawlers at that level too.

It also can’t get you named. Allowing every search bot makes you eligible to be read, and we know of no evidence that it raises your mentions by itself. That’s a limit on our own tools too. Our AI crawler checker reads your rules, not your server, and “allowed” doesn’t mean “crawled”.

How to check and change your rules

You can do this by hand:

1. Open yourcompany.com/robots.txt, and the same path on each subdomain.
2. Ask who wrote any AI bot rules, and whether they were copied from somewhere.
3. Choose a position on search, user fetches and training separately.
4. Test specific paths with the robots.txt tester, or generate rules with the AI robots.txt generator.
5. Check your CDN or firewall settings for the same bots.
6. A month later, look at what actually arrived, using the AI bot log analyser or your own logs, and verify the visits before you trust them.

Our free tools read pages as QuotableBot, and you can name that in robots.txt too.

What to tell security and legal

A note you can adapt: “We treat AI bots by purpose. We allow search crawlers so that assistants can read our public pages. We state a position on training through each operator’s documented control. We’re aware that some user-triggered fetchers may ignore robots.txt, and that robots.txt isn’t access control, so anything private stays behind authentication. We review this list quarterly, because tokens change.”

For why crawler access is only one part of being named, see our guide to AEO. Choices about llms.txt and schema are separate, and matter less on the evidence so far.