How to identify and verify AI crawlers in your server logs

To verify an AI crawler, check the request’s IP address against the list its operator publishes. The user agent alone proves nothing, because anyone can send one. Find requests by token in your access log, confirm the IP, then count pages by what each bot does. Most large operators publish a list, but not all of them do.

This guide covers who visited and how to prove it. For what to allow or block, see our robots.txt guide. We opened the operators’ pages and downloaded nine of their IP lists on 7 November 2026. Tokens and addresses change, so read the live lists and don’t copy addresses from any article, including this one.

How do you find AI crawlers in a server log?

To find AI crawlers in a server log, search the user agent field for each bot’s token, ignoring case. Don’t match the whole string, because version numbers change. OpenAI’s search crawler identifies itself with compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot, and OpenAI’s page says version numbers may change. Searching for OAI-SearchBot keeps working after an update.

Some bots look like browsers. Apple’s search crawler sends a full Safari string with Applebot/0.1 near the end. Bingbot sends a Chrome string with bingbot/2.0 inside it. Short bot names are safer than a pattern for the whole line.

Control tokens won’t appear at all. Google-Extended and Applebot-Extended are robots.txt settings with no crawler behind them, so finding none in your logs is normal.

Which AI crawlers publish a way to verify them?

Most operators we checked publish a list of IP addresses, and Google, Apple and Common Crawl also describe a reverse DNS check. Meta’s crawler page gives no list for its AI bots, and Mistral’s gives none for its training bot. For those, a user agent is all you have, so report those visits as unverified.

How do you check an IP against a published list?

Download the operator’s current list, then test each IP from your log against its ranges. The script below does that for any list in the format these operators share. We ran it on 7 November 2026 against a small made-up log and the live OpenAI list. An address inside a listed range passed, and a faked request from 203.0.113.9 failed.

Save this as verify_ips.py:

import ipaddress, json, sys

prefixes = json.load(open(sys.argv[1]))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]

for line in sys.stdin:
    ip = ipaddress.ip_address(line.strip())
    listed = any(ip.version == n.version and ip in n for n in nets)
    print(ip, "listed" if listed else "NOT LISTED")

Then run it for one bot. This assumes the client IP is the first field in each line of your log:

curl -s https://openai.com/searchbot.json -o searchbot.json
grep -i "OAI-SearchBot" access.log | awk '{print $1}' | sort -u | python3 verify_ips.py searchbot.json

We also ran the script against the Anthropic, Apple, Bing, Google and Perplexity lists, which it read without changes. Amazon’s pages and the Mistral, DuckDuckGo and Common Crawl lists weren’t part of that test.

If your site sits behind a CDN, the first field may be the CDN’s address, not the visitor’s. Use the client IP field your CDN provides, and check that nobody else can set it.

How often do these IP lists change?

Some of these IP lists change often, so fetch them each time you check and don’t store them once. Each file we downloaded states when it was created, and the dates ranged from January 2024 to 6 November 2026, the day before our check. The lists also differ widely in size.

One date caveat applies to any log review. A request from last month is best checked against last month’s list, which you may not have. An old visit from an address that has since left a list can look fake when it wasn’t.

What does each kind of AI crawler visit tell you?

Each kind of crawler visit tells you something different, so don’t add them into one “AI traffic” number. A search crawler means your page can enter that index. A training crawler means your content may be collected for a model. A user fetcher means a person’s request led the assistant to your page. None of them means an assistant named you.

Google doesn’t separate these jobs for Search. Googlebot crawls for ordinary results and for AI features, so your logs can’t show AI Overviews use. Search Console’s generative AI reports are the place for that, and they currently show impressions only.

Why can request counts mislead you?

Request counts can mislead you because one rendered page may set off many extra requests. A developer’s logs from one Next.js site, checked against OpenAI’s IP lists, show GPTBot going from 83 requests on 22 September 2026 to 152,009 on 28 September. On the 28th, 97% of those requests were Next.js prefetches, and they covered 4,395 pages. This is one operator’s data, and OpenAI hasn’t commented on it.

Report distinct pages next to requests, and look at the paths. A rise in search crawler visits to pages you care about means more than a large total. Errors matter too, because a verified bot getting a 403 from your firewall can’t read that page, whatever your robots.txt says. For what these rendering requests may mean, see whether AI crawlers can read JavaScript.

What can’t your logs show?

Your logs can’t show whether an assistant named you, cited you or described you correctly. An assistant can answer from what it stored earlier, or from what it learned in training, without any new visit. A visit can also lead nowhere. So a quiet log doesn’t prove you’re absent from answers, and a busy one doesn’t prove you’re present.

That’s a limit on our free tools too. The AI bot log analyser counts requests by user agent in your browser. It doesn’t check IPs, so use the script above before you trust its totals. The AI crawler checker reads your rules, not your traffic. To see what assistants actually say, you have to ask them, which is what Quotable’s scans do by asking each question several times.

What to tell your security team

A note you can adapt: “We treat a bot’s user agent as a claim. We count a visit as verified only when its IP matches the operator’s current published list or passes its reverse DNS check. Bots without a published method are reported as unverified. We use the client IP from our CDN, and we review failed requests from verified bots each month, so that firewall rules don’t block crawlers we’ve chosen to allow.”