robots.txt tester

Test a robots.txt before you publish it, and see which rule decides each crawler.

Runs in your browser. What you enter isn’t sent or saved.

Why it matters

One wrong line can shut out a crawler or open a private path. A group naming a crawler replaces the * group, so rules under * don’t carry over.

Good to know

robots.txt is a request, not a lock. Well-behaved crawlers follow it, some user fetches may not, and it doesn’t protect private pages.

Related tools

AI robots.txt generatorChoose what AI bots may do, then copy the rules.AI crawler checkerSee which AI bots your robots.txt allows or blocks.AI bot log analyserCount AI crawler visits in your access log, in your browser.

See what AI says about you, every week

Quotable asks ChatGPT, Gemini and Google AI Overviews for you, then shows what they say about your brand and what to fix.

Start free trial