AI crawler checker

See which AI crawlers your robots.txt allows, so you don’t block AI search by accident.

We read the public page once. Results aren’t published.

Why it matters

A copied rule can shut out AI search without anyone noticing. Search, training and user fetches are separate crawlers, so you can block one and keep the others.

Good to know

This reads your rules, not your server: a firewall can still refuse a crawler. Blocking Google-Extended doesn’t remove you from AI Overviews.

Related tools

robots.txt testerTest any path against AI bots using a pasted robots.txt.AI robots.txt generatorChoose what AI bots may do, then copy the rules.AI bot log analyserCount AI crawler visits in your access log, in your browser.

See what AI says about you, every week

Quotable asks ChatGPT, Gemini and Google AI Overviews for you, then shows what they say about your brand and what to fix.

Start free trial