What AI crawlers fetch from a small open-source site: 31 days of logs

How much AI crawler traffic really comes from AI crawlers?

Across the twelve AI user agents whose operators publish IP lists, relic.space saw 9100 requests in 31 days. 5609 came from inside the operator’s published ranges. 3491, or 38 percent, did not. Three names (Claude-SearchBot, Perplexity-User and Google-Extended) had no genuine requests at all. Googlebot’s spoofed share over the same window was 5 percent.

Which AI crawlers visit a small site?

The window runs from 2026-08-14 to 2026-09-14. Each request got one of three verdicts. Verified means the client IP was inside the ranges the operator publishes. Spoofed means the operator publishes ranges and the IP was outside them. Unchecked means the operator publishes nothing, so there was nothing to test against. Table 1 is the twelve AI crawler names that can be checked.

Claude-User was spoofed 94 percent of the time. Claude-SearchBot, Perplexity-User and Google-Extended were spoofed every single time. Table 2 runs the same check on the search engine and social crawlers, as a control.

Googlebot was spoofed 5 percent of the time and AdsBot-Google once in 3176 requests. The facebookexternalhit row is a different thing. Its 205 spoofed requests came from 30 different addresses, almost all on residential or mobile networks, each fetching / or /og.png two or four times. That is the shape of link previews from messaging apps that reuse Meta’s user agent string, and there is nothing hostile in it. Meta’s own crawler, meta-externalagent, verified 108 of 250.

A third group cannot be checked because the operator publishes no address list: YouBot (2638 requests), SemrushBot (1154), YandexBot (973), YandexRender (968), AhrefsBot (919), LumiaBot (832), OraBot (546), StackScopeBot (470), ExaSearchBot (354), GrokBot (343), MJ12bot (281), Bytespider (184), CCBot (48) and Brave (22). They are left out of every percentage above.

Why do scanners pretend to be AI crawlers?

68 distinct IP addresses sent at least one spoofed request, 4167 spoofed requests in all. Most of that came from a handful. 11 addresses used five or more different crawler names each, and those 11 sent 6001 requests. Their status mix says what they were doing: 404 for 2637 requests, 401 for 1396, 301 for 1034, 200 for 488, 307 for 400 and 403 for 29. Real crawlers get 200s. These got errors, because they were asking for things a static site does not have.

The paths they asked for most were /, /fetch, /api, /openapi.json, /api/.env, /api/v1, /.docker/config.json, /@fs/proc/self/environ, /v1/graphql, /.env, /graphql, /admin, /v1 and /proxy. Of the 6001 requests, 2914 asked for a path that names a credential, a secret, a key, an environment file, a cloud config or an AI tool config. That is a vulnerability scanner.

Each of the big ones was active for one day, then never again. The busiest sent 2425 requests on 2026-09-12 and wore 21 different crawler names while doing it. The next sent 1917 on 2026-09-03, also under 21 names. Another sent 502 on 2026-08-19 under 8 names. A smaller one sent 79 on 2026-08-21 under 9 names and went straight for /.git/config and /.aws/credentials. The most borrowed names were GPTBot and PerplexityBot (11 addresses each), ChatGPT-User (10), ClaudeBot (9), OAI-SearchBot and Applebot (7 each), Google-Extended (6) and Amazonbot (5).

The reason is simple. Site owners want to show up in AI answers, so they allow-list the AI crawlers by name, in robots.txt or in a firewall rule. A scanner that calls itself GPTBot gets through those rules more often than one that calls itself python-requests. Changing the string costs nothing, and the names on the allow-lists are public.

What do AI crawlers actually read?

From here on, verified requests only. Table 3 is every AI crawler with at least one verified request. Per-day rates divide by the 31 days of the window.

ChatGPT-User is the fetcher that visits a page when a ChatGPT user asks. It came on 31 of 31 days and fetched the homepage 455 times out of 735. Its other big section was the blog at 245, and the pages it read most are troubleshooting posts: universal clipboard not working (38), copy and paste in the terminal (35), copy and paste not working in Excel (25). It never fetched robots.txt or sitemap.xml.

OAI-SearchBot is the crawler that surfaces sites in ChatGPT search. It read blog pages 275 times and robots.txt 267 times, more robots.txt fetches than any other AI crawler. Its top pages were how-to posts: how to copy and paste (17), copy and paste not working on Windows 11 (15), how to copy and paste on a laptop (15). It was the only OpenAI crawler to touch the markdown twins or llms.txt, 36 times.

GPTBot is the training crawler. It came every day, 31 of 31, and spent most of its 410 fetches on build assets: 178 requests for JavaScript and font files under /_next. Pages were a minority: /tools 86, /blog 51, the homepage 37, sitemap.xml 32. No single page got more than six fetches.

ClaudeBot read robots.txt (232) and sitemap.xml (229) more than any page. Then /tools 138, build assets 121, /blog 113 and /help 78. It fetched the homepage 3 times and was active on 22 of the 31 days. Its most-read pages were tools, three fetches each. It fetched the markdown twins or llms.txt 10 times.

Claude-User made 28 verified requests on 7 days, and 15 of those were robots.txt. Verified Claude-SearchBot traffic was zero.

PerplexityBot was the one crawler that read the help wiki most: /help 166, then /blog 141, other pages 128 and /tools 117. It came on 16 days. Its top pages were unusual for a bot: the universal clipboard troubleshooting post (12), the support page (9), the self-host page (8) and the privacy policy (7). It fetched markdown twins or llms.txt 10 times. Verified Perplexity-User traffic was zero.

Amazonbot was the busiest of all: 1385 verified fetches, 44.7 a day, on every day of the window. It read /blog 471 times, /tools 450 and /help 177. It also collected the most errors, 144 404s and 74 307s, and it fetched the markdown twins or llms.txt 67 times, more than any other crawler.

Applebot mostly fetched build assets: 284 of its 472 requests were under /_next and 123 more were images and other files. It read other pages 18 times and was active on 14 days. meta-externalagent came on 3 days and fetched images and other files 58 times, build assets 33 times and other pages 12 times.

For scale, verified Googlebot made 2133 fetches, 68.8 a day, of which 1317 were images and other files. Verified Bingbot made 564 and DuckDuckBot 365.

How do you verify an AI crawler?

Every operator in Table 1 gives you something to check against. OpenAI’s crawler page says OAI-SearchBot surfaces sites in ChatGPT search, GPTBot crawls for training, and ChatGPT-User visits a page when a user asks. It publishes IP lists at openai.com/gptbot.json, chatgpt-user.json and searchbot.json. Anthropic’s page says a crawler whose source IP is on the list at claude.com/crawling/bots.json is coming from Anthropic. Perplexity publishes its ranges too. Apple’s page says Applebot traffic is identified by reverse DNS in the *.applebot.apple.com domain, or by matching the IP against search.developer.apple.com/applebot.json. Amazon publishes its addresses at developer.amazon.com/amazonbot/ip-addresses/. Google and Bingdocument how to verify their crawlers. Meta publishes no list; for meta-externalagent we used the routes registered to Meta’s AS32934 in the RADB routing database.

The second thing to do is block or challenge the scanner pattern rather than the name. A request for /.env, /.git/config, /.aws/credentials or /@fs/proc/self/environ is never a crawler doing its job, whatever the user agent says. A rule on those paths catches the scanners and leaves a real crawler alone. Blocking the AI crawler names instead would have blocked the 5609 genuine requests and let the 11 scanner addresses carry on under their next name.

How this was measured

The logs come from the Cloudflare account GraphQL API, filtered by user agent substring and grouped by path, status and client IP. Cloudflare’s docs say the API almost always uses adaptive sampling and estimates totals from the sampled counts. The average sample interval over this window for the whole zone was 1.16, so the true counts are slightly higher than the ones here, within roughly 15 percent, and the ratios are unchanged. The zone served 1,555,461 sampled requests in the window, humans included.

The window covers 32 calendar dates with the first and last partial, counted as 31 days for the per-day rates. That is why a few bots show up on 32 of 31 days. Crawlers whose operators publish nothing are listed but left out of every percentage, because a name with nothing to check against can be neither confirmed nor denied.

One more number for context. Bing Webmaster Tools has an AI Performance report, and on 2026-09-11 it showed 4.9K citations of this site’s pages in Copilot and Bing AI answers over the previous 3 months, across 15 cited pages. It is the only citation count any engine gives a site owner.

The data

The full dataset is a CSV at /data/ai-crawlers-2026-09-14.csv. The columns are date, user_agent_name, path, status, verdict and requests_sampled. It holds every sampled request whose user agent named a known crawler, including the unchecked ones. There are no IP addresses in it. It is licensed CC BY 4.0, so use it however you like with a link back. We will refresh it at 90 days from the site Worker’s own counter, which at this volume does not sample.

Sources

• Overview of OpenAI crawlers, OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
• Perplexity crawlers, Perplexity
• Verify requests from Google crawlers and fetchers, Google
• How to verify Bingbot, Microsoft Bing
• About Applebot, Apple
• About Amazonbot, Amazon
• Sampling, Cloudflare Docs

Frequently asked questions