AI robots.txt generator

Decide separately whether AI may search, fetch or train on your site, and copy the rules.

Runs in your browser. What you enter isn’t sent or saved.

AI searchChatGPT search, Perplexity and others. Blocking can remove you from their answers.
Fetches a person asks forWhen someone asks an assistant to read your page.
AI trainingUsing your content to train AI models.
Your robots.txt
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /

User-agent: ChatGPT-User
User-agent: Google-Agent
User-agent: Claude-User
User-agent: Claude-Agent
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: MistralAI-User
Allow: /

User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Allow: /

User-agent: *
Allow: /

Save it as robots.txt at the top of your site.

OpenAI says that if GPTBot and OAI-SearchBot are both allowed, it may use one crawl for both.

Why it matters

Blocking every AI bot also takes you out of AI search. Choosing by purpose lets you stop training and stay in answers.

Good to know

Googlebot, Bingbot and Applebot stay open, because they also run ordinary search. Google’s AI Overviews and AI Mode use Googlebot’s crawl, so to leave them and stay in Search, use the generative AI control in Search Console.

Related tools

robots.txt testerTest any path against AI bots using a pasted robots.txt.AI crawler checkerSee which AI bots your robots.txt allows or blocks.llms.txt generatorFill in a form and get a ready-to-publish llms.txt.

See what AI says about you, every week

Quotable asks ChatGPT, Gemini and Google AI Overviews for you, then shows what they say about your brand and what to fix.

Start free trial