Skip to the page

Get in touch

Which AI crawlers visit a small business website?

In 12 days, AI crawlers asked for our website 335 times, under 15 different names. Nine in ten were for two small housekeeping files that no person ever reads. Most knocked, checked the house rules and the floor plan, and left without coming in.

Journal · 8 minute read · Updated 17 November 2026 · Sources in the notes

The short answer

Yes, AI crawlers read small business websites. From 5 to 16 November 2026, our own counter logged 335 requests to our site from 15 named AI crawlers, on every day of the count, led by Anthropic’s ClaudeBot (206) and OpenAI’s OAI-SearchBot (57). But 301 of those requests were for robots.txt and the sitemap, and only 34 were for pages. A crawler visit shows that a bot can reach your site; it doesn’t mean an AI will cite or recommend you.

The counter at a glance

What it isA small visit counter we built ourselves and run in front of our own website.
What it countsPage views and loading speed from people, and requests from 69 named crawlers (40 AI, 13 search, 16 other) for pages and for robots.txt, llms.txt and the sitemap.
What it’s built not to collectCookies, anything stored in your browser, IP addresses, the browser’s name tag (its user agent), or the full address of the page. What we keep is on our privacy page.
Where it runsOn our own site, in front of every page. The script people load is 1,980 bytes, and the build refuses anything over 2 KB.

What we counted

Every AI crawler our counter recognised from the evening of 5 November to the end of 16 November 2026 (UTC). Requests that weren’t for pages were for robots.txt (the file that tells bots what they may read) or the sitemap (the list of your pages).

Requests and pages read, by AI crawler, 5 to 16 November 2026. ClaudeBot: 206 requests, 0 pages. OAI-SearchBot: 57, 0. Amazonbot: 27, 25. GPTBot: 20, 7. Meta’s three crawlers: 8, 1. Bytespider: 5, 0. Claude-User and Claude-SearchBot: 3, 1. CCBot: 3, 0. YouBot, PerplexityBot, DuckAssistBot and GrokBot: 6, 0.

Requests and pages read, by AI crawler, 5 to 16 November 2026. Each pale line is every request; the deep teal part is pages read, the rest robots.txt and the sitemap. Amazonbot, which read the most, is drawn heavier. Our own counter, on this site.
CrawlerWho runs itWhat it’s for, per the operatorRequestsOf which pages
ClaudeBotAnthropicTraining AI models12060
OAI-SearchBotOpenAIChatGPT search2570
AmazonbotAmazonAmazon’s products; may train its AI models32725
GPTBotOpenAITraining AI models2207
Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcherMetaMeta AI search, training, and fetches for a person48 (4, 2, 2)1
BytespiderByteDanceWe found no operator page to quote50
Claude-User, Claude-SearchBotAnthropicFetches for a person; Claude’s search13 (2, 1)1
CCBotCommon CrawlAn open archive of the web530
YouBot, PerplexityBot, DuckAssistBot, GrokBotYou.com, Perplexity, DuckDuckGo, and a name linked to xAIToo few requests to read anything into6 (3, 1, 1, 1)0
All AI crawlers10 operators33534

Over the same days, classic search crawlers made 118 requests and read 55 pages: Googlebot 55 requests (23 pages), Applebot 45 (30 pages), Bingbot 15 (no pages) and two others 3 (2 pages).

AI crawlers against search crawlers, 5 to 16 November 2026. AI crawlers: 335 requests, 34 pages read. Search crawlers: 118 requests, 55 pages read.

AI crawlers against search crawlers, over the same 12 days: more knocking, less reading. The deep teal part of each line is pages read. Our own counter, on this site.

They come often, mostly to knock. AI crawlers made 335 requests to Googlebot and Bingbot’s 70, and nearly all of the difference was robots.txt and the sitemap. The two busiest read no pages at all: ClaudeBot fetched none in 206 requests, and OAI-SearchBot read nothing but robots.txt. That may be less odd than it looks. OpenAI says that when a site allows both its bots, “we may use the results from just one crawl for both use cases,”2 and GPTBot did read seven pages.

The reader, as it turned out, was Amazon. Amazonbot fetched 25 of the 34 pages, including all 10 reads of a journal article. Across all AI crawlers, compare pages were read 6 times, the home page 5, the journal’s front page 3 and the FAQ 3. GPTBot read no articles at all.

Nobody asked for llms.txt. Ours is live, and no named crawler requested it in the whole period. The fetchers that load a page because someone asked a question just then showed up four times: Claude-User twice, Meta-ExternalFetcher twice.

Now the limits, which are real: one small site, 12 days, and each crawler identified by the name it gives. Seventeen of the 335 requests arrived in the same second as one under another company’s crawler name, which could be chance or a testing tool, and Common Crawl warns of “crawlers falsely identifying themselves as CCBot.”5 On 17 November, after the count closed, requests naming ClaudeBot, OAI-SearchBot and Google-Extended arrived within a tenth of a second, though Google says Google-Extended “doesn’t have a separate HTTP request user agent string.”6 A name is a claim. Assistants that browse under an ordinary browser’s name aren’t counted at all.

And the one thing a visit doesn’t mean: being fetched isn’t being cited. A crawler that reads robots.txt has only checked whether it may come in, and each AI company decides for itself what its answers show.

Who are these crawlers?

They do three different jobs. Training crawlers collect pages that may be used to train AI models. Search crawlers build the index an AI product searches when it answers, which is how your pages end up linked in an answer. User fetchers load one page because a person asked a question just then. The table puts each name in its job, in its operator’s own words.

NameJobIn the operator’s words
GPTBotTraining“It is used to crawl content that may be used in training our generative AI foundation models.”2
OAI-SearchBotSearch“OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features.”2
ChatGPT-UserUser fetcher“When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.”2
ClaudeBotTrainingCollects “web content that could potentially contribute to their training.”1
Claude-SearchBotSearch“Claude-SearchBot navigates the web to improve search result quality for users.”1
Claude-UserUser fetcher“When individuals ask questions to Claude, it may access websites using a Claude-User agent.”1
PerplexityBotSearch“PerplexityBot is designed to surface and link websites in search results on Perplexity.”7
Perplexity-UserUser fetcher“When users ask Perplexity a question, it might visit a web page to help provide an accurate answer.”7
GooglebotSearch“Googlebot is the generic name for two types of web crawlers used by Google Search.”8
Google-ExtendedTraining switch, not a crawlerIt “does not impact a site’s inclusion in Google Search.”6
ApplebotSearch, and may train“The data crawled by Applebot may also be used to help train Apple foundation models.”9
Applebot-ExtendedTraining switch, not a crawler“Applebot-Extended does not crawl webpages.”9
BingbotSearch“Bingbot is Bing’s crawler.”10
CCBotOpen archiveKeeps “an open repository of web crawl data that is universally accessible and analyzable by anyone.”5
Meta-ExternalAgentTraining, and moreCrawls “for use cases such as training foundation AI models.”4
Meta-WebIndexerSearchWorks “to improve Meta AI search result quality for users.”4
Meta-ExternalFetcherUser fetcherIt “fetches individual links at a user’s request.”4
AmazonbotProducts, and may trainIts data “may be used to train Amazon AI models.”3

How do you check and control your own site?

There are three ways in, from free and easy to technical.

  1. Your host’s report. If your site runs through Cloudflare, its AI Crawl Control is “available on all plans” and “gives you visibility into which AI services are accessing your content.”11 Other hosts may offer something similar; ask whoever runs your site.
  2. Your server logs. Ask your host for the raw access log and search it for the names above. OpenAI, Perplexity and Common Crawl publish the addresses their crawlers use, so a technical person can check that a request is real.
  3. Your robots.txt. Type your address followed by /robots.txt into a browser. “User-agent: GPTBot” then “Disallow: /” asks GPTBot to stay out; “User-agent: *” with “Disallow: /” asks every bot to. Your host can also block AI bots on its own; our guide to being recommended by ChatGPT covers that.
A plain text file: User-agent: *, Allow: /, Disallow: /api/, Disallow: /underwater-v2.html, then the address of our sitemap.
Our own robots.txt, as a browser shows it, 17 November 2026.

To welcome search crawlers and turn training away, a robots.txt can group several names over one rule, as the standard allows.12

Lines in robots.txtWhat they ask
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Search crawlers for AI answers: come in.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Training crawlers and switches: please don’t use our pages for training.
User-agent: *
Allow: /
Everyone else, Googlebot and Bingbot included: come in.

If you do block training, you don’t lose search, by the operators’ own account. OpenAI says you can allow OAI-SearchBot “while disallowing GPTBot,”2 and Google-Extended “does not impact a site’s inclusion in Google Search.”6 Blocking a search crawler does cost you: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers.”2 User fetchers may skip robots.txt altogether; Perplexity says its fetcher “generally ignores robots.txt rules.”7

robots.txt is a request, not a lock. The standard says plainly, “These rules are not a form of access authorization.”12 Google adds that it’s “up to the crawler to obey them.”13 To actually stop a bot, block it at your host.

Why we built our own counter

Because the visitors we most wanted to see don’t run scripts. Ordinary analytics is a script that runs in the visitor’s browser. A 2024 analysis by Vercel and MERJ found “none of the major AI crawlers currently render JavaScript,”14 and Google Analytics leaves out the bots it knows: “traffic from known bots and spiders is automatically excluded.”15

We count crawlers on the server, as each page goes out. A small program in front of our site notes when a crawler on our list fetches a page or one of four crawler files. The catch is that it knows a bot only by the name it gives, and we keep the list of names by hand, last checked on 5 November against each company’s own page where there is one.

People get a tiny script and no cookies. It sends the page’s path, the name of the site you came from and up to three campaign tags, nothing else from the address, and it sends nothing at all when the browser has Global Privacy Control turned on. So the same person on two different days counts as two visitors, and we can’t report returning visitors.

If it breaks, the site doesn’t. Each request passes straight through, the counting happens after the page is on its way, and any error sends the page out untouched. On a bad day we lose counts, not visitors.

Under the hood. The counter is a Cloudflare Worker, a small program that runs in front of the site. When the site answers, it checks that the answer succeeded, that it’s a web page or one of robots.txt, llms.txt, llms-full.txt and sitemap.xml, and that the user agent contains a name on the list (longest name first, so Googlebot-Image beats Googlebot). Then it writes one row: path, crawler name, kind and country, with no visitor attached. Google-Extended and Applebot-Extended are listed though they should never appear in a request, so nobody adds them again thinking they’re missing.

What we’d do differently. We’d check the big crawlers against the addresses their operators publish from the first day, so a name would be more than a claim. We’d count pages a crawler re-checks and finds unchanged, which may be slipping past us. And we’d have started counting well before writing this page.

Where we come in

A counter like this is something we can build into a client’s site when it helps, and it’s how we check our own work: whether search engines and AI crawlers can reach what we make, and which pages they read. It can’t tell you whether an AI recommends you, and we won’t pretend it can. The rest of our work is websites, apps and custom software, which are yours outright and which we look after once they’re live. If you want one, write to us; we write back within a business day.

Questions

Should I block AI crawlers?

Only if you have a reason to. To be found and linked by AI tools, keep their search crawlers, Googlebot and Bingbot open. Turning training crawlers away is a fair choice, and OpenAI and Google both say it doesn’t take you out of their search.

Does llms.txt matter?

Not for Google, which says the file “will neither harm nor help your site’s visibility or rankings in Google Search.”16 Its proposal says it’s read “on demand, when an agent needs information about a topic while assisting a user.”17 No named crawler asked for ours in 12 days; it’s cheap, so we keep one.

Can I tell a real crawler from a fake one?

Yes, with help. OpenAI publishes the addresses each of its crawlers uses, and Common Crawl runs CCBot “on dedicated IP address ranges with reverse DNS,”5 so a developer can check a logged request against them.

Where these numbers come from

We counted requests in our counter’s records for our own site only, from its first crawler record at 21:27 UTC on 5 November 2026 to the end of 16 November 2026 (UTC), grouped by crawler name and by page. We didn’t look at anyone’s visit. Each operator’s description comes from its own page.

  • OpenAI, “Overview of OpenAI Crawlers.” developers.openai.com, read 17 November 2026.
  • Claude Help Center, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” support.claude.com, read 17 November 2026.
  • Perplexity, “Perplexity Crawlers.” docs.perplexity.ai, read 17 November 2026.
  • Google, “Google’s common crawlers,” updated 30 October 2026. developers.google.com, read 17 November 2026.
  • Google Search Central, “What Is Googlebot,” updated 3 February 2026. developers.google.com, read 17 November 2026.
  • Apple, “About Applebot,” published 4 September 2026. support.apple.com, read 17 November 2026.
  • Bing Webmaster Blog, “bingbot Series: Maximizing Crawl Efficiency,” October 2018. blogs.bing.com, read 17 November 2026.
  • Common Crawl, “CCBot.” commoncrawl.org, read 17 November 2026.
  • Meta, “Meta Web Crawlers.” developers.facebook.com, read 17 November 2026.
  • Amazon, “Amazon Web Search crawlers.” developer.amazon.com, read 17 November 2026.
  • Cloudflare Docs, “AI Crawl Control,” updated 14 August 2026. developers.cloudflare.com, read 17 November 2026.
  • IETF, RFC 9309, “Robots Exclusion Protocol,” September 2022. rfc-editor.org, read 17 November 2026.
  • Google Search Central, “Robots.txt Introduction and Guide,” updated 10 December 2025. developers.google.com, read 17 November 2026.
  • Vercel, “The rise of the AI crawler,” 17 December 2024. vercel.com, read 17 November 2026.
  • Analytics Help, “Known bot-traffic exclusion.” support.google.com, read 17 November 2026.
  • Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search,” updated 10 July 2026. developers.google.com, read 17 November 2026.
  • Jeremy Howard, “The /llms.txt file.” llmstxt.org, read 17 November 2026.