supereo_

guide // how to // 2026

Which AI crawlers to allow in 2026: every bot, what it feeds and what a block costs

Allow the AI crawlers that fill a search index and the fetchers that open a page when someone asks: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User, plus Googlebot, Bingbot and Applebot. Opt out of OAI-SearchBot, for one, and ChatGPT's search answers won't show your site[1]. Treat the training crawlers apart (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot): disallowing GPTBot says your content shouldn't train OpenAI's models[2], a choice about future models rather than today's answers. Then check your CDN, because in the run below Canva's firewall challenged GPTBot and served OAI-SearchBot. supereo is our pick, and we make it: it reads your rules for each AI crawler, fixes what keeps you out of AI answers and proves each move against control prompts. In the real run below, the client called two of supereo's tools: check_ai_access and fetch_as_bot.

Updated . Prices checked 19 November 2026.

The verdict

  1. AI crawlers checked, fixed and proven

    supereoour pick, and we make it

    • best for teams that want AEO done, not a dashboard: supereo does everything it takes to get you recommended, from your site and pages to the roundups, placements and videos AI reads;
    • it proves each move against control prompts, with sample sizes and ranges on every number;
    • run it from Claude or ChatGPT: an MCP server with the same tools supereo's own agent uses.

    From $149 a month

    Run the free auditFree, no card, every answer shown.

The steps

  1. Sort every AI bot by what it feeds

    Put each one in one of three groups: search crawlers, fetchers that open a page for a person, and training crawlers or tokens. The sections below sort every company's bots that way, in the company's own words.

  2. Let the search crawlers and fetchers in

    Give each bot you want its own group, or leave it under your general rules. A crawler follows the group that names it and falls back to the rules for every bot only when no group names it[37], so a group written for GPTBot changes nothing for OAI-SearchBot.

  3. Decide on training crawler by crawler

    To keep future pages out of training, disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent by name. You can still open some pages to them, as Canva does with its create pages[38]. Remember that Google-Extended also covers grounding in the Gemini app[15].

  4. Read each rule the way the crawler does

    For every bot, find the group it matches, then check the paths that matter, not only the home page. supereo's check_ai_access reads it per crawler, as the run below shows, and the free audit shows what five AI engines answer about you today.

  5. Check what your CDN serves each bot

    Request a page with each bot's user agent and see what comes back: the page, a challenge or a block. Then open your CDN's bot settings. Cloudflare's AI rules act by behavior and apply to verified bots and to unverified requests that match them[34].

  6. Trust addresses, not names

    A user agent is a name anyone can send. OpenAI publishes the IP ranges its search crawler uses[1], and Common Crawl warns that other crawlers falsely call themselves CCBot[27]. supereo counts an AI read only when it came from the AI company's own addresses[39].

  7. Keep robots.txt quick to load

    Serve the file fast, from every host you want covered. When Google can't refresh it, after timeouts or 5xx errors, it keeps its cached copy longer[20], so an outage can freeze an old rule in place.

  8. Wait, then measure the answers

    Give a change a few days, then check that the bots you allowed read your pages and that answers move. supereo measures each change against control prompts it couldn't move, so you see whether the access you opened turned into answers. LLM SEO covers what each engine does with a page it can read.

Real runs

a real run // 19 November 2026

One real site's rules for each AI crawler, read by supereo's MCP server

Recorded from supereo's MCP server on its staging environment, https://staging.supereo.app/api/mcp. 3 calls.

check_ai_access · 377 ms ·

Sent

{
  "url": "https://www.canva.com/"
}

Received

{
  "origin": "https://www.canva.com",
  "path": "/",
  "robotsTxt": {
    "status": "ok",
    "detail": "robots.txt read."
  },
  "crawlers": [
    {
      "bot": "GPTBot",
      "aiCrawler": true,
      "allowed": false,
      "rule": "Disallow: /",
      "group": "GPTBot"
    },
    {
      "bot": "OAI-SearchBot",
      "aiCrawler": true,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "ChatGPT-User",
      "aiCrawler": true,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "ClaudeBot",
      "aiCrawler": true,
      "allowed": false,
      "rule": "Disallow: /",
      "group": "ClaudeBot"
    },
    {
      "bot": "Claude-SearchBot",
      "aiCrawler": true,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "Claude-User",
      "aiCrawler": true,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "PerplexityBot",
      "aiCrawler": true,
      "allowed": true,
      "rule": "Allow: /",
      "group": "PerplexityBot"
    },
    {
      "bot": "Google-Extended",
      "aiCrawler": true,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "Googlebot",
      "aiCrawler": false,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    },
    {
      "bot": "Bingbot",
      "aiCrawler": false,
      "allowed": true,
      "rule": "no rule matches the path",
      "group": "*"
    }
  ],
  "llmsTxt": {
    "served": false,
    "status": 403,
    "chars": 0,
    "excerpt": null
  },
  "edge": {
    "connected": false,
    "days": 7,
    "blockedFromOperatorAddresses": [],
    "spooferDenials": 0,
    "note": "Blocks at the edge show for your own connected sites only: canva.com isn't one."
  },
  "note": "From robots.txt and llms.txt alone. A block at the site's CDN or firewall, or an answer that depends on the crawler's address, shows only in its traffic."
}

fetch_as_bot · 274 ms ·

Sent

{
  "url": "https://www.canva.com/",
  "bot": "GPTBot",
  "max_html_chars": 1000
}

Received

{
  "url": "https://www.canva.com/",
  "finalUrl": "https://www.canva.com/",
  "bot": "GPTBot",
  "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
  "status": 403,
  "headers": {
    "date": "Thu, 19 Nov 2026 00:36:38 GMT",
    "content-type": "text/html; charset=UTF-8",
    "transfer-encoding": "chunked",
    "connection": "close",
    "accept-ch": "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA",
    "cf-mitigated": "challenge",
    "x-frame-options": "SAMEORIGIN",
    "server": "cloudflare",
    "critical-ch": "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA",
    "cross-origin-embedder-policy": "require-corp",
    "cross-origin-opener-policy": "same-origin",
    "cross-origin-resource-policy": "same-origin",
    "origin-agent-cluster": "?1",
    "permissions-policy": "accelerometer=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=(),xr-spatial-tracking=*",
    "referrer-policy": "same-origin",
    "server-timing": "chlray;desc=\"a5cb244e1e0c8f7d\"",
    "x-content-type-options": "nosniff",
    "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
    "vary": "accept-encoding",
    "report-to": "{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=hgmukxYmlAN1uGxJNZwrAaX1kv9IoNwdmFwSGl63fQGE2tL7%2F36qXyBpOKfW4CvyXFtdGo2fVb9n63F6%2F0k1iVYEDhvPU3A%2B4T5IxFVKX4LfUJfHtX4gtqmTxyTMMrw%3D\"}]}",
    "nel": "{\"report_to\":\"cf-nel\",\"success_fraction\":0.01,\"max_age\":604800}",
    "content-encoding": "br",
    "cf-ray": "a5cb244e1e0c8f7d-IAD"
  },
  "html": "<!DOCTYPE html>\n<html lang=\"en\">\n  <head>\n    <title>Canva</title>\n    <link rel=\"shortcut icon\" href=\"favicon.ico\" />\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <meta name=\"description\" content=\"\" />\n    <meta name=\"keywords\" content=\"\" />\n    <script>window.__STATUS_PAGE_CONFIG__ = {\"STATIC_ASSETS_BASE\":\"https://status.canva.com/\",\"STATUS_REDIRECT_URL\":\"https://www.canvastatus.com/\"};</script>\n    <style>\n      :root{--grid-cols: 1;--grid-rows: 1}.desktop-home-view{width:100%;height:100vh;position:relative}.desktop-home-view #canvas-container{position:fixed;top:0;left:0;width:100%;height:100vh;z-index:0;background:#fff}.desktop-home-view .text{position:absolute;top:12vh;left:50%;transform:translate(-50%);z-index:1;text-align:center;font-size:24px;font-weight:700;color:#000;width:80%;font-family:canva-sans}.desktop-home-view .text .logo-light{display:block;margi",
  "htmlChars": 755073,
  "truncated": true,
  "ms": 46,
  "redirects": [],
  "note": "What the site served this user agent from supereo's address. A site that checks crawler IP addresses may answer the real crawler differently."
}

Trimmed: the cookies the site set on this request, its set-cookie header.

fetch_as_bot · 590 ms ·

Sent

{
  "url": "https://www.canva.com/",
  "bot": "OAI-SearchBot",
  "max_html_chars": 1000
}

Received

{
  "url": "https://www.canva.com/",
  "finalUrl": "https://www.canva.com/",
  "bot": "OAI-SearchBot",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot",
  "status": 200,
  "headers": {
    "date": "Thu, 19 Nov 2026 00:36:39 GMT",
    "content-type": "text/html;charset=utf-8",
    "transfer-encoding": "chunked",
    "connection": "keep-alive",
    "cf-ray": "a5cb24500aeb4a55-IAD",
    "cf-cache-status": "DYNAMIC",
    "cache-control": "no-cache,no-store",
    "nel": "{\"report_to\":\"cf-nel\",\"success_fraction\":0.01,\"max_age\":604800}",
    "expires": "Thu, 1 Jan 1970 00:00:00 GMT",
    "server": "cloudflare",
    "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
    "vary": "Accept-Encoding",
    "pragma": "No-cache",
    "accept-ch": "Sec-CH-UA-Platform-Version",
    "content-security-policy": "frame-ancestors https://canvateam.zendesk.com https://phoenix.canva.com https://www.canva-iris.com https://enterprise.canva.com https://www.affinity.studio;",
    "document-policy": "include-js-call-stacks-in-crash-reports",
    "referrer-policy": "strict-origin-when-cross-origin",
    "reporting-endpoints": "default=\"https://www.canva.com/_spi/browserreporting/default?page_load_id=133115a5-15df-4248-a3be-a12b7094e216&page_load_trace_id=fa78cf6a113c881cc1c61c7432fcfd83&page=anon_home&version=7f668b9a68d2b5a692c226c609a3f0437a6b0398&release_name=20261118-02-7f668b9&deployment_group=STABLE\", cross_origin=\"https://www.canva.com/_spi/browserreporting/cross_origin\"",
    "server-timing": "gateway_region;desc=\"us\", cfCacheStatus;desc=\"DYNAMIC\", cfEdge;dur=4,cfOrigin;dur=0,cfWorker;dur=284",
    "x-content-type-options": "nosniff",
    "x-frame-options": "deny",
    "x-request-id": "a5cb24500aeb4a55",
    "report-to": "{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=kTwgbE%2Foo1WgsKLdpqHbiMskAn6vMfdEvncBXDrDq3ThxJ%2F%2Bh%2F7kJmvP%2B6FUiC1VJCtpQCczHM0X0TU5yed9%2BXY6QWlFrJjEvr5X3Ottaw5eL%2F3bzCIrAZH%2F0nvOOPY%3D\"}]}",
    "content-encoding": "br"
  },
  "html": "<!DOCTYPE html><html dir=\"ltr\" lang=\"en\" class=\"x-default-marker theme light classic x6brqfx x1yowkxt x19aimcq\"><head><meta charset=\"utf-8\"><title>Canva: Visual Suite for Everyone</title><meta name=\"app-name\" content=\"anon_home\"/><meta name=\"build-version\" content=\"7f668b9a68d2b5a692c226c609a3f0437a6b0398\"/><meta name=\"build-release-name\" content=\"20261118-02-7f668b9\"/><meta name=\"build-deployment-group\" content=\"STABLE\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1, interactive-widget=resizes-content\"><script>(function pushPageBuildIdentityToNativeShell(identity) {\n  if (typeof window === 'undefined') {\n    return;\n  }\n  if (Object.values(identity).every(value => value == null)) {\n    return;\n  }\n  const json = JSON.stringify(identity);\n  if (window.Android?.onPageBuildIdentityReady) {\n    // Android-shaped device\n    window.Android.onPageBuildIdentityReady(json);\n  } else if (window.Harmony?.onPageBuildIdentityReady) {\n    // HarmonyOS-shaped device\n    window.H",
  "htmlChars": 512027,
  "truncated": true,
  "ms": 314,
  "redirects": [],
  "note": "What the site served this user agent from supereo's address. A site that checks crawler IP addresses may answer the real crawler differently."
}

Trimmed: the cookies the site set on this request, its set-cookie header.

check_ai_access read canva.com's robots.txt as it stood: it shuts out GPTBot and ClaudeBot, the crawlers that collect training data, and leaves the search crawlers and user agents of OpenAI and Anthropic free to read. fetch_as_bot then asked for the home page as each of OpenAI's two crawlers: the site's firewall answered GPTBot with a challenge and served OAI-SearchBot the page.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot

OpenAI gives each job its own agent and its own line in robots.txt. OAI-SearchBot is the one that decides ChatGPT search: OpenAI says sites opted out of it won't be shown in ChatGPT's search answers, though they can still appear as navigational links, and it asks sites to let in requests from the IP ranges it publishes too[1]. How to get cited by ChatGPT covers what happens once it reads you.

GPTBot collects content that may be used to train OpenAI's foundation models, and disallowing it tells OpenAI your content shouldn't be used that way[2]. The two settings stand apart, so a site can welcome OAI-SearchBot and turn GPTBot away[3].

ChatGPT-User opens a page when someone asks ChatGPT a question. It doesn't crawl on its own or decide what appears in search, and OpenAI says robots.txt rules may not apply to it, since a person started the request[4]. OAI-AdsBot visits only pages submitted as ads on ChatGPT, to check them against OpenAI's policies, and what it collects isn't used for training[5]. Unless you advertise on ChatGPT, it won't come.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User and a training signal

Anthropic draws the same line. Restricting ClaudeBot tells Anthropic to leave your future pages out of its training data, while disabling Claude-User stops Claude fetching your page when someone asks it something, which may cost you visibility in searches people direct[7]. Claude-SearchBot indexes pages for Claude's search, and blocking it may reduce how visible and accurate you are there[8]. How to get recommended by Claude covers the index that search draws on.

There's now a second way to say no to training. Since October 30, 2026, Anthropic honors a general machine-readable signal for a site's preference on AI training[10], and Google says it intends to support an AIPREF training preference signal, written as a robots.txt group for the user agent __usage_ai-training, in early 2027[16]. One group can then state the preference without listing every training crawler by name.

Anthropic's bots also won't try to get past a CAPTCHA[9]. A firewall that challenges them keeps them out whatever your robots.txt invites, which is the gap the run above exposes on a real site.

Perplexity: PerplexityBot, Perplexity-User, and what a block leaves behind

PerplexityBot is the crawler behind Perplexity's search results, and Perplexity says it isn't used to crawl content for AI foundation models[11]. Blocking it is a search decision, with no training to weigh against it, and how to get cited by Perplexity covers what comes after.

A block doesn't make you vanish, though. Perplexity won't index the text of a site that disallows PerplexityBot, but it may still index the domain, the headline and a brief factual summary[13]. So a block is partial: you can still appear, thinly, with none of your own words to quote. Perplexity-User, its fetcher for a person's question, isn't used for crawling or training, and Perplexity says it generally ignores robots.txt rules because a user requested the fetch[12].

Google: Googlebot, Google-Extended and the switch for AI Overviews

For its AI search features, Google's crawler is Googlebot itself. Google says robots.txt rules for Googlebot are the control for how a site is crawled for Search, AI built in, and that nosnippet, data-nosnippet, max-snippet and noindex limit what it shows from your pages[14]. Blocking Googlebot to stay out of AI Overviews takes you out of Google Search with them, and how to show up in AI Overviews and AI Mode covers getting in.

Google-Extended is a token, not a crawler. It decides whether what Google crawls may train future Gemini models and ground answers in the Gemini app, and it doesn't affect whether you appear in Google Search[15]. Disallowing it costs more than training: your pages stop grounding the Gemini app's answers, while AI Overviews carry on as before.

For AI Overviews and AI Mode themselves, the switch sits in Search Console: since August 31, 2026, every site can manage its inclusion in AI Overviews, AI Mode and the generative AI features in Discover[17]. That switch leaves training alone, which is Google-Extended's job[18]. So Google gives you three separate controls: Googlebot for crawling, the Search Console switch for its AI features, and Google-Extended for Gemini.

Microsoft, Apple and Meta: Bingbot, Applebot and Meta's bots

For Copilot, the crawler that counts is Bingbot: Microsoft says Copilot and its other experiences are powered by Bing's search index[21], and Bing says it respects the preferences you set in robots.txt and its other controls[22]. A Bingbot block is a Copilot block.

Apple splits its rules in two. Applebot's crawl powers search in Spotlight, Siri and Safari, and allowing it lets your pages show there, while disallowing Applebot-Extended opts your content out of training Apple's foundation models[23]. If your robots.txt names Googlebot but not Applebot, Applebot follows your Googlebot rules[24], so a rule you wrote for Google also decides Siri.

Meta runs separate bots for training, fetching and search. As PPC Land reports Meta's crawler page, Meta-ExternalAgent collects data for training and for indexing, Meta-ExternalFetcher fetches links at a person's request and may bypass robots.txt, and Meta-WebIndexer feeds the search index[25]. Allowing Meta-WebIndexer is what helps Meta cite and link to your pages in Meta AI's answers[26]. Meta's own page refused our reader, so the source here is a report of it.

Common Crawl, Amazon, Mistral, DuckDuckGo and ByteDance

CCBot is Common Crawl's crawler, and Vercel notes that Common Crawl is often used as a training dataset for large language models[28]. Blocking it is a training decision. Common Crawl gives the two lines that block it, and warns that other crawlers falsely call themselves CCBot[27].

Amazonbot improves Amazon's products and may be used to train its AI models[29]. Amzn-SearchBot is the one for answers: it makes your pages eligible for search experiences such as Alexa, isn't used for training, and when robots.txt doesn't name it, it follows the rules you give other search bots[30].

Mistral's MistralAI-Training builds its training datasets and is used neither for its search index nor for live answers[32]. DuckDuckGo's DuckAssistBot crawls pages in real time for its AI-assisted answers, isn't used to train models, and a block on it takes effect after 72 hours[33]. ByteDance publishes no page we could find on its Bytespider, so this guide makes no claim about what it feeds.

What a firewall does that a robots.txt check misses

robots.txt asks; the CDN in front of a site decides. In the run above, Canva's robots.txt keeps GPTBot, ClaudeBot and CCBot out of everything but the pages it opens to them, such as its create pages[38], while OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User fall under rules that let them read the home page. That is the training-versus-search split, written down.

Then the firewall enforced it. Asked for the home page as GPTBot, Canva's Cloudflare edge answered with a challenge instead of the page, and it served the same page to OAI-SearchBot. In Cloudflare's own terms, a Training rule covers crawlers that train models, mixed-purpose crawlers used for search too among them, and applies to verified bots and to unverified requests that match them[34]. Canva's settings aren't public; what its edge answered is.

Two lessons follow. A firewall can turn away a crawler your robots.txt invites, and Cloudflare itself calls robots.txt directives voluntary[35], so a robots.txt check tells you what you asked for, not what a crawler gets. And since an edge rule against training can take mixed-purpose crawlers with it, a setting meant to guard your content can cost you answers too.

One caution runs both ways: supereo's requests came from supereo's own address, not OpenAI's, so an edge that checks addresses may treat the real crawlers differently. Read what your edge serves each bot by name, then confirm it in your CDN's logs, where the real crawlers' requests show; how to track ChatGPT traffic in GA4 shows how to tell crawlers from visitors there.

Bots that may skip robots.txt, and how long a change takes

Fetchers that act for a person may not follow robots.txt at all. OpenAI says its rules may not apply to ChatGPT-User[4], Perplexity says its user fetcher generally ignores them[12], Google says the same of its user-triggered fetchers[19], and PPC Land reports that Meta says Meta-ExternalFetcher may bypass them[25]. If you want those out, robots.txt is the wrong lever; only your firewall meets them.

Changes take time, too. OpenAI says a robots.txt change can take about 24 hours to reach its search results[3], Google caches robots.txt for up to 24 hours and longer when it can't refresh it[20], DuckDuckGo's block takes effect after 72 hours[33], and Amazon's crawlers may use a cached copy from the last 30 days[31]. Give a change a few days before you judge it, and don't read one day's logs as a crawler ignoring you.

Have it done: supereo

01supereoour pick, and we make it

  • best for teams that want AEO done, not a dashboard: supereo does everything it takes to get you recommended, from your site and pages to the roundups, placements and videos AI reads;
  • it proves each move against control prompts, with sample sizes and ranges on every number;
  • run it from Claude or ChatGPT: an MCP server with the same tools supereo's own agent uses.

From $149 a month

  • Does the worksupereo writes those pages for you, aimed at the questions where AI skips you. supereo.app
  • Proves it against controlsEach change has a few target questions, and we compare them with your other questions, the ones no change targets. supereo.app
  • Runs from Claude or ChatGPTRun the same loop from Claude, Cursor or Codex. Your agent gets every tool supereo's own agent uses: your buyers' questions, every answer, the pages AI cites, your brand facts. supereo.app
  • Free auditfree audit // about 2 minutes // no card supereo.app
PlanA monthPromptsAI engines
Autopilot$14940ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity, Claude

Prices from supereo.app, checked 19 November 2026. What it does, from its own pages, checked 16 November 2026.

How the proof works, with every number it runs on: How supereo measures.

Questions

Which AI crawlers should I allow?

Every search crawler and user fetcher of the engines whose answers you want, from OAI-SearchBot and ChatGPT-User to Googlebot, Bingbot and Applebot. Decide on the training crawlers on their own merits, and make sure your firewall serves the bots you allowed. supereo checks all of it and proves each change against control prompts.

What is GPTBot?

OpenAI's training crawler. It collects content that may be used to train OpenAI's foundation models, and a robots.txt rule that disallows it says your content shouldn't be used for that[2]. It doesn't decide whether ChatGPT search shows you: OAI-SearchBot does[1].

How do I block GPTBot in robots.txt?

Add a group with the line User-agent: GPTBot followed by Disallow: / to keep it off the whole site, or Disallow lines for single folders. Disallowing GPTBot tells OpenAI your content shouldn't be used for training[2], and it leaves OAI-SearchBot and ChatGPT search as they were[3].

What is OAI-SearchBot?

The crawler that decides ChatGPT search: a site opted out of it won't be shown in ChatGPT's search answers, except as a navigational link, and OpenAI asks sites to let in its published IP ranges too[1].

Is the ChatGPT bot free?

If you mean GPTBot, there's nothing to buy: it's the crawler OpenAI uses to collect training content[2], and you only decide whether to let it in. If you mean ChatGPT itself, it has a free plan, and web search is part of it[6].

What are the top AI bots?

For a site that wants to be in AI answers, start with three pairs: OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, and PerplexityBot with Perplexity-User. Google's AI features run on Googlebot's crawl rather than a bot of their own[14].

What is a crawler?

An automated client, the kind search engines run to follow links and build an index[36]. An AI crawler does the same for an AI company, to fill the index its assistant searches or to gather pages for training.

Is there such a thing as an AI bot?

Yes, several kinds. OpenAI alone documents a crawler for training[2], one for search[1], a fetcher that opens a page when someone asks[4] and a bot that checks ad landing pages[5].

What are the top web crawlers?

For AI answers, the ones that matter are Googlebot and Bingbot, since Google's AI features and Copilot answer from their indexes[14][21], then Applebot for Siri and the AI companies' own crawlers above. CCBot matters for training data more than for answers[28].

What does GPT stand for?

Generative pre-trained transformer, a type of large language model[40]. GPTBot is the crawler OpenAI uses to collect pages that may be used to train its models[2].

How we compiled this

Where this page's words say how an AI engine finds, reads or cites pages, the sentence links the engine's own documentation, or another source, and the day it was read.

The runs are real calls to supereo's MCP server on its staging environment, made as a house account, recorded as they happened: the arguments as sent, each answer as the client received it, its JSON indented and trimmed only where it says so, the time each call took and when it ran.

We make supereo, so it is the one tool this page names as a pick.

How supereo measures its own results, with every number it runs on: How supereo measures.

supereo (supereo.app) is an AI agent that does everything it takes to get your brand recommended by ChatGPT, Gemini, Perplexity, Claude and Google, on your site and off it: site fixes, new pages, roundups, placements and video. It proves what worked against control prompts. Made by superagnt.

Sources

  1. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below. developers.openai.com, read 19 November 2026.
  2. GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models. developers.openai.com, read 19 November 2026.
  3. for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models. If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling. For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust. developers.openai.com, read 19 November 2026.
  4. When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent. ChatGPT users may also interact with external applications via GPT Actions . ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search. developers.openai.com, read 19 November 2026.
  5. OAI-AdsBot is used to validate the safety of web pages submitted as ads on ChatGPT. When you submit an ad, OpenAI may visit the landing page to ensure it complies with our policies. We may also use content from the landing page to determine when it’s most relevant to show the ad to users. OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. developers.openai.com, read 19 November 2026.
  6. Web search is available on ChatGPT Free, Go, Plus, Pro, Business, Enterprise, and Edu. People who are not signed in can also use web search. help.openai.com, read 19 November 2026.
  7. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets. Claude-User Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. support.claude.com, read 19 November 2026.
  8. Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results. support.claude.com, read 19 November 2026.
  9. respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.) support.claude.com, read 19 November 2026.
  10. As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets website owners express a preference about whether their content may be used to train AI models. This announcement is a continued commitment to respecting robot.txt where ClaudeBot is explicitly named. We’re now confirming that we will honor a new preference signal that lets website owners say “no AI training”. support.claude.com, read 19 November 2026.
  11. PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models. docs.perplexity.ai, read 19 November 2026.
  12. Perplexity-User controls which sites these user requests can access. It is not used for web crawling or to collect content for training AI foundation models. Example user-agent string (the version number may change): Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Published IP addresses: https://www.perplexity.ai/perplexity-user.json Since a user requested the fetch, this fetcher generally ignores robots.txt rules. docs.perplexity.ai, read 19 November 2026.
  13. Perplexity respects robots.txt directives. Our crawler, PerplexityBot, will not index the full or partial text content of any site that disallows it via robots.txt. However, if a page is blocked, we may still index the domain, headline, and a brief factual summary. perplexity.ai, read 19 November 2026.
  14. AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search. To limit the information shown from your pages in Search, use nosnippet , data-nosnippet , max-snippet , or noindex controls. developers.google.com, read 19 November 2026.
  15. Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI . Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. developers.google.com, read 19 November 2026.
  16. Google intends to support the AIPREF shared machine-readable training preference signal (with " user-agent: __usage_ai-training ") in early 2027. developers.google.com, read 19 November 2026.
  17. Note : As of August 31, 2026, we’ve rolled out this control to all websites worldwide. Use the Search generative AI control to manage your site's inclusion in the following generative AI features on Google Search. We expect to update this list over time as we develop Google Search. AI Overviews AI Mode Generative AI features in Google Discover support.google.com, read 19 November 2026.
  18. This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended . support.google.com, read 19 November 2026.
  19. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. developers.google.com, read 19 November 2026.
  20. Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn't possible (for example, due to timeouts or 5xx errors). developers.google.com, read 19 November 2026.
  21. Powered by Bing’s search index, experiences like Microsoft Copilot, Microsoft Start, and others handle billions of queries each month, embedding Bing deeply into how people search, shop, and explore online. about.ads.microsoft.com, read 19 November 2026.
  22. Bing respects all content owner preferences expressed through robots.txt and other supported control mechanisms. blogs.bing.com, read 19 November 2026.
  23. including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products. The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file. support.apple.com, read 19 November 2026.
  24. If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions. support.apple.com, read 19 November 2026.
  25. Meta-ExternalAgent collects data for training foundation models and for indexing content directly. Meta-ExternalAds crawls for advertising and other business products. Meta-ExternalFetcher retrieves individual links at a user's request to support agentic functions, and Meta says it may bypass robots.txt for that reason. Meta-WebIndexer feeds the search index. ppc.land, read 19 November 2026.
  26. The company's crawler page states that Meta-WebIndexer navigates the web to improve Meta AI search result quality, and that Meta analyzes online content to enhance the relevance and accuracy of the assistant. It then adds the sentence that makes the crawler commercially interesting: allowing Meta-WebIndexer in a robots.txt file helps Meta cite and link to that content in Meta AI's responses. ppc.land, read 19 November 2026.
  27. To prevent Common Crawl from crawling your website, include the following in your robots.txt : User-agent: CCBot Disallow: / Please note that we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity. commoncrawl.org, read 19 November 2026.
  28. Common Crawl (CCBot), which is often used as a training dataset for Large Language Models (LLMs) does not render pages. vercel.com, read 19 November 2026.
  29. Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models. developer.amazon.com, read 19 November 2026.
  30. Amzn-SearchBot is used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa. If robots.txt files don’t mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots. Amzn-SearchBot does not crawl content for generative AI model training. developer.amazon.com, read 19 November 2026.
  31. They will fetch host-level robots.txt files or use a cached copy from the last 30 days. developer.amazon.com, read 19 November 2026.
  32. MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file. This crawler is not used for search indexing or to answer live user queries in Vibe. docs.mistral.ai, read 19 November 2026.
  33. DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers , which prominently cite their sources. This data is not used in any way to train AI models. If you’re a publisher, you may opt out of being a potential source for AI-assisted answers by modifying the robots.txt file for your domain(s). Once you’ve updated the file to disallow the DuckAssistBot user agent, the change will take effect after 72 hours and DuckAssistBot will stop crawling your site. duckduckgo.com, read 19 November 2026.
  34. Training : crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers that are used both for Training and for Search. Each policy applies to Verified bots with that behavior and matching unverified bots. developers.cloudflare.com, read 19 November 2026.
  35. robots.txt directives are voluntary. Crawlers can ignore them. When Bot Preference Sync is enabled, Disallow also blocks AI training crawlers at Cloudflare's edge. Mixed-purpose crawlers used for both Search and Training are excluded from this edge block. developers.cloudflare.com, read 19 November 2026.
  36. Crawlers are automated clients. Search engines, for instance, have crawlers to recursively traverse links for indexing rfc-editor.org, read 19 November 2026.
  37. If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present. rfc-editor.org, read 19 November 2026.
  38. User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Applebot-Extended User-agent: anthropic-ai User-agent: Claude-Web Disallow: / Allow: /create/ canva.com, read 19 November 2026.
  39. A read counts when the request came from the AI company's own addresses and your Cloudflare or Vercel edge answered it with the page; a read for a person is one made while someone asked that AI a question, never one supereo's own prompts caused. supereo.app, read 19 November 2026.
  40. A generative pre-trained transformer ( GPT ) is a type of large language model (LLM) en.wikipedia.org, read 19 November 2026.

install with one prompt // any AI agent

Install supereo with one prompt.

Paste one line into the agent you already use. Coding agents set supereo up themselves, and Claude and ChatGPT walk you through adding it. You sign in once, then ask it how AI sees your site.

setup prompt

Install supereo for me. It's the supereo MCP server at https://mcp.supereo.app/mcp (on Claude it's a connector I add in my browser; elsewhere I sign in through my browser). The setup guide for each kind of agent is at https://supereo.app/install
  1. 01copy the prompt
  2. 02paste it into your agent
  3. 03sign in once, in your browser

or wire it by hand

https://mcp.supereo.app/mcp
claude · ~/dev/acme
Claude Code v2.1.206

Welcome back Alex!

Opus 5.5 · Claude Max · ~/dev/acme

⏸ manual mode on · ? for shortcuts

An example session. The same prompt works in every client.