guide // how to // 2026
How to get your brand recommended by Claude in 2026
To get your brand recommended by Claude, let its search crawlers read your site, make sure the search index it uses holds your pages, and state plainly, on your own pages and on the pages Claude reads about your market, what you sell, who it's for and what it costs. Claude answers from what it learned in training and, when a question needs current facts, from a web search whose results it cites[1]. supereo is our pick, and we make it: it does everything it takes to get your brand recommended by Claude and the other AI engines, and proves each move against control questions. In the supereo Index's run of 19 October 2026, lists, comparisons and reviews were 31 of Claude's 50 top cited pages, and articles and other pages 18, ten per topic across five topics (196 answers).
Updated . Prices checked 12 November 2026.
- The verdict
- The steps
- What gets cited
- Real runs
- How Claude finds the pages it cites
- What each of Claude's bots does, and what blocking it costs
- Brave's index follows your rules for Googlebot
- What won't get you recommended by Claude
- How Claude differs from ChatGPT and Perplexity
- Getting recommended by Claude Code
- How to know it worked
- Have it done: supereo
- Questions
- How we compiled this
- Sources
The verdict
Recommended by Claude, done for you
supereoour pick, and we make it
- best for teams that want AEO done, not a dashboard: supereo does everything it takes to get you recommended, from your site and pages to the roundups, placements and videos AI reads;
- it proves each move against control questions, with sample sizes and ranges on every number;
- run it from Claude or ChatGPT: an MCP server with the same tools supereo's own agent uses.
From $149 a month
Run the free auditFree, no card, every answer shown.
The steps
Let Claude-SearchBot and Claude-User in
Check robots.txt for both bots by name, and for a wildcard rule that catches them. Blocking either may reduce your visibility in Claude's answers[4]. The real run below checks every AI crawler's rule for one page.
Decide on ClaudeBot on its own
ClaudeBot is for training, so blocking it is a choice about your future pages in Anthropic's training data[6], not about Claude's search.
Don't block Googlebot from pages you want in Claude
Brave's crawler skips any page Googlebot may not crawl[8], and Brave's index is where Claude's search looks.
Check what Brave holds
Search Brave for site: and your domain. A page missing there can't come up when Claude searches, however good it is.
Answer the question on your own page
Put the answer to your buyer's question first, in plain sentences: what you sell, who it's for, what it costs and how it compares, with names, prices and dates Claude can quote back.
Get named on the pages Claude reads
Claude reads other sites about your market as well as yours. The table below counts the kinds of pages Claude cited in supereo's Index run; get your brand named, accurately, on the lists, comparisons and reviews of that kind.
Keep your facts the same everywhere
Say your name, category, price and audience the same way on your site, your profiles and the pages about you, and correct the ones that are wrong. Claude puts its answer together from all of them.
Measure it, again and again
Ask Claude your buyers' questions with web search on, many times, before and after each change, against questions the change couldn't move. That is what supereo does with every change it makes.
What gets cited
What Claude cites
Its top cited pages in the supereo Index run of 19 October 2026: ten per topic across 5 topics, 196 answers.
| Page type | Pages | Times cited |
|---|---|---|
| Lists, comparisons and reviews | 31 / 50 | 48 |
| Articles and other pages | 18 / 50 | 22 |
| Home pages | 1 / 50 | 1 |
For example:
- getinboxzero.com/blog/post/best-ai-email-assistants (cited in 4 of 39 answers on AI email assistants)
- clay.com/guides/best/email-finder-tools (cited in 2 of 33 answers on AI lead generation and SDRs)
- eesel.ai/blog/best-lindy-ai-alternatives (cited in 2 of 33 answers on AI lead generation and SDRs)
- marketbetter.ai/blog/best-ai-sdr-tools (cited in 2 of 33 answers on AI lead generation and SDRs)
- oneaway.io/blog/best-clay-alternatives-2026 (cited in 2 of 33 answers on AI lead generation and SDRs)
Real runs
a real run // 7 November 2026
supereo's MCP server checks a page for Claude's crawlers
Recorded from supereo's MCP server on its staging environment, https://staging.supereo.app/api/mcp. 3 calls.
check_ai_access · 323 ms ·
Sent
{
"url": "https://supereo.app/guides"
}Received
{
"origin": "https://supereo.app",
"path": "/guides",
"robotsTxt": {
"status": "ok",
"detail": "robots.txt read."
},
"crawlers": [
{
"bot": "GPTBot",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "*"
},
{
"bot": "OAI-SearchBot",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "OAI-SearchBot"
},
{
"bot": "ChatGPT-User",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "*"
},
{
"bot": "ClaudeBot",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "*"
},
{
"bot": "Claude-SearchBot",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "Claude-SearchBot"
},
{
"bot": "Claude-User",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "*"
},
{
"bot": "PerplexityBot",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "PerplexityBot"
},
{
"bot": "Google-Extended",
"aiCrawler": true,
"allowed": true,
"rule": "Allow: /",
"group": "*"
},
{
"bot": "Googlebot",
"aiCrawler": false,
"allowed": true,
"rule": "Allow: /",
"group": "Googlebot"
},
{
"bot": "Bingbot",
"aiCrawler": false,
"allowed": true,
"rule": "Allow: /",
"group": "Bingbot"
}
],
"llmsTxt": {
"served": true,
"status": 200,
"chars": 3138,
"excerpt": "# supereo\n\n> supereo (supereo.app) is an AI agent that does everything it takes to get your brand recommended by ChatGPT, Gemini, Perplexity, Claude and Google, on your site and off it: site fixes, new pages, roundups, placements and video. It proves what worked against control questions. Made by superagnt.\n\n- best for teams that want AEO done, not a dashboard: supereo does everything it takes to get you recommended, from your site and pages to the roundups, placements and videos AI reads;\n- it proves each move against control questions, with sample sizes and ranges on every number;\n- run it from Claude or ChatGPT: an MCP server with the same tools supereo's own agent uses.\n\nCopilot is $149 a month and Autopilot is $499 a month, per brand, or two months free when you pay for a year. Every plan starts with a 7-day free trial. It takes your card up front and charges nothing until the week ends, and it gets you your same-day plan and your first fix shipped.\n\n## Product\n\n- [supereo](https://supereo.app/)\n- [Free AI visibility checker](https://supereo.app/ai-visibility-checker)\n- [About supereo](https://supereo.app/about)\n- [supereo for AI assistants](https://supereo.app/ai-instructions)\n\n## Guides\n\n- [AEO tool guides](https://supereo.app/guides)\n- [Best AEO tools (2026): the trackers vs the tools that do the work](https://supereo.app/guides/best-aeo-tools)\n- [AEO tool pricing (2026): every plan, prompt cap and AI engine](https://supereo.app/guides/aeo-tool-pricing)\n- [Best Profound alternatives (2026), after its move off self-serve](https://supereo.app/guides/profound-alternatives)\n- [Best Peec AI alternatives (2026): tools that do the work, not just track](https://supereo.app/guides/peec-ai-alternatives)\n- [Best Otterly alternatives (2026): beyond AI search monitoring](https://supereo.app/guides/otterly-alternatives)\n- [Best MCP servers for AEO (2026): AI visibility from Claude and ChatGPT](https://supereo.app/guides/best-mcp-servers-for-aeo)\n- [Can Claude Code do GEO?"
},
"edge": {
"connected": false,
"days": 7,
"blockedFromOperatorAddresses": [],
"spooferDenials": 0,
"note": "Not connected yet. This needs supereo to read the site's own traffic at Cloudflare or Vercel: connect it at /dashboard/reads/connect in supereo."
},
"note": "From robots.txt and llms.txt alone. A block at the site's CDN or firewall, or an answer that depends on the crawler's address, shows only in its traffic."
}fetch_as_bot · 161 ms ·
Sent
{
"url": "https://supereo.app/guides",
"bot": "Claude-SearchBot",
"max_html_chars": 1000
}Received
{
"url": "https://supereo.app/guides",
"finalUrl": "https://supereo.app/guides",
"bot": "Claude-SearchBot",
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)",
"status": 200,
"headers": {
"access-control-allow-origin": "*",
"age": "5987",
"cache-control": "public, max-age=0, must-revalidate",
"content-disposition": "inline",
"content-encoding": "br",
"content-type": "text/html; charset=utf-8",
"date": "Sat, 07 Nov 2026 11:58:11 GMT",
"etag": "W/\"89898f1ff3c385a9a4a7069c5c094ba5\"",
"server": "Vercel",
"strict-transport-security": "max-age=63072000",
"vary": "rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch",
"x-matched-path": "/guides",
"x-nextjs-prerender": "1",
"x-nextjs-stale-time": "300",
"x-vercel-cache": "HIT",
"x-vercel-id": "sfo1:iad1::zcnd5-1794052691660-4512e57413e7",
"transfer-encoding": "chunked"
},
"html": "<!DOCTYPE html><html data-dpl-id=\"dpl_jH6cmntAqQtpBRbWGLC2qZkfFZGN\" lang=\"en\" class=\"PoSCYa_variable bZoIEG_variable\" data-scroll-behavior=\"smooth\"><head><meta charSet=\"utf-8\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1, viewport-fit=cover\"/><link rel=\"preload\" as=\"image\" imageSrcSet=\"/_next/static/immutable/media/supereo-mark-48.352lohjq8it1r.webp 48w, /_next/static/immutable/media/supereo-mark-96.3dmd7y-7xqqyx.webp 96w, /_next/static/immutable/media/supereo-mark-256.2rytqxeadnl81.webp 256w\" imageSizes=\"26px\"/><link rel=\"preload\" as=\"image\" imageSrcSet=\"/_next/static/immutable/media/supereo-mark-48.352lohjq8it1r.webp 48w, /_next/static/immutable/media/supereo-mark-96.3dmd7y-7xqqyx.webp 96w, /_next/static/immutable/media/supereo-mark-256.2rytqxeadnl81.webp 256w\" imageSizes=\"22px\"/><link rel=\"stylesheet\" href=\"/_next/static/immutable/chunks/05geyyv5c4iq9.css\" data-precedence=\"next\"/><link rel=\"stylesheet\" href=\"/_next/static/immutable/chunks/43csb7ux0taf0.css\" da",
"htmlChars": 31281,
"truncated": true,
"ms": 32,
"redirects": [],
"note": "What the site served this user agent from supereo's address. A site that checks crawler IP addresses may answer the real crawler differently."
}fetch_as_bot · 163 ms ·
Sent
{
"url": "https://supereo.app/guides",
"bot": "Claude-User",
"max_html_chars": 1000
}Received
{
"url": "https://supereo.app/guides",
"finalUrl": "https://supereo.app/guides",
"bot": "Claude-User",
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)",
"status": 200,
"headers": {
"access-control-allow-origin": "*",
"age": "5987",
"cache-control": "public, max-age=0, must-revalidate",
"content-disposition": "inline",
"content-encoding": "br",
"content-type": "text/html; charset=utf-8",
"date": "Sat, 07 Nov 2026 11:58:11 GMT",
"etag": "W/\"89898f1ff3c385a9a4a7069c5c094ba5\"",
"server": "Vercel",
"strict-transport-security": "max-age=63072000",
"vary": "rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch",
"x-matched-path": "/guides",
"x-nextjs-prerender": "1",
"x-nextjs-stale-time": "300",
"x-vercel-cache": "HIT",
"x-vercel-id": "sfo1:iad1::gdxkq-1794052691836-eebed452cfbc",
"transfer-encoding": "chunked"
},
"html": "<!DOCTYPE html><html data-dpl-id=\"dpl_jH6cmntAqQtpBRbWGLC2qZkfFZGN\" lang=\"en\" class=\"PoSCYa_variable bZoIEG_variable\" data-scroll-behavior=\"smooth\"><head><meta charSet=\"utf-8\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1, viewport-fit=cover\"/><link rel=\"preload\" as=\"image\" imageSrcSet=\"/_next/static/immutable/media/supereo-mark-48.352lohjq8it1r.webp 48w, /_next/static/immutable/media/supereo-mark-96.3dmd7y-7xqqyx.webp 96w, /_next/static/immutable/media/supereo-mark-256.2rytqxeadnl81.webp 256w\" imageSizes=\"26px\"/><link rel=\"preload\" as=\"image\" imageSrcSet=\"/_next/static/immutable/media/supereo-mark-48.352lohjq8it1r.webp 48w, /_next/static/immutable/media/supereo-mark-96.3dmd7y-7xqqyx.webp 96w, /_next/static/immutable/media/supereo-mark-256.2rytqxeadnl81.webp 256w\" imageSizes=\"22px\"/><link rel=\"stylesheet\" href=\"/_next/static/immutable/chunks/05geyyv5c4iq9.css\" data-precedence=\"next\"/><link rel=\"stylesheet\" href=\"/_next/static/immutable/chunks/43csb7ux0taf0.css\" da",
"htmlChars": 31281,
"truncated": true,
"ms": 20,
"redirects": [],
"note": "What the site served this user agent from supereo's address. A site that checks crawler IP addresses may answer the real crawler differently."
}Each call reads the open web as it stood: robots.txt's rule for every AI crawler, Claude's three among them, then the same page fetched as Claude-SearchBot and as Claude-User. Both were served the page, so Claude's search can index it and Claude can read it when someone asks.
How Claude finds the pages it cites
Claude has two ways to know about your brand. The first is training: what it read before its cutoff. The second is search: when a question depends on something current or changing, such as an organization or a product that might have changed, Claude searches the web first[1], then answers with the sources it used[2].
Buyers' questions about tools, prices and vendors are exactly the questions that change, so a brand's best route into Claude's answers is its search. That search has a supplier: Anthropic added Brave Search to the list of companies that process data for Claude in March 2025, the month it launched web search[3].
So getting recommended by Claude comes down to three things you can work on: Claude's own crawlers can read you, Brave's index holds you, and the pages both of them find say the right things about you.
What each of Claude's bots does, and what blocking it costs
Anthropic runs three robots, and each one is a separate line in robots.txt. Claude-SearchBot indexes pages for Claude's search: blocking it stops Anthropic indexing your content for search, which may reduce your visibility in Claude's search results[4].
Claude-User fetches a page when someone asks Claude a question that needs it: blocking it stops Claude retrieving your content for that person, which may reduce your visibility for searches people direct[5].
ClaudeBot is the training crawler: restricting it tells Anthropic to leave your future pages out of its training data[6]. That is a choice about training, not about search, so you can block ClaudeBot and still let the other two in.
Use robots.txt for all of it. Anthropic says its bots honor robots.txt, and that blocking its IP addresses may not work, because it stops the bots reading your robots.txt at all[7].
Brave's index follows your rules for Googlebot
Brave's crawler doesn't announce itself with a user agent of its own, and it follows the rules you set for Google: if Googlebot may not crawl a page, Brave's crawler won't either[8]. So a rule written to block Googlebot from a section of your site also keeps that section out of the index Claude searches.
Brave also finds pages through its own browser's users who opt in to share what they visit[10], so a page people actually visit is more likely to be found than an orphan.
To see what Brave holds, search Brave for your domain with the site: operator. To keep a page out of Brave, robots.txt is the wrong tool: Brave says a noindex tag delists a page, and robots.txt doesn't[9].
What won't get you recommended by Claude
You can't buy your way in. Anthropic has said Claude will remain ad-free: no sponsored links beside conversations, and no answers influenced by advertisers[13].
Blocking ClaudeBot won't hide you from Claude's search, either: it's the training crawler, and Claude-SearchBot and Claude-User have rules of their own[4]. Blocking Anthropic's IP addresses is the wrong lever too, since it can stop the bots reading the robots.txt rules you set[7].
And a page Claude can't read doesn't count, however good it is: one that needs JavaScript to show its words, or one behind a login, gives a crawler little to quote. Check what a crawler gets, as the real run above does for one page.
How Claude differs from ChatGPT and Perplexity
Each engine has its own way in. ChatGPT's search surfaces sites through OpenAI's own crawler, OAI-SearchBot[14]. Perplexity searches the web in real time for each question[15]. Claude searches through Brave's index, which follows your rules for Googlebot[8].
So one site can show up in one engine's answers and not another's. Check each engine's crawlers by name, and measure each engine's answers on their own. How to get cited by Perplexity covers Perplexity's route.
Google's AI Overviews and AI Mode work from Google's own index instead: a page must be indexed and eligible for a snippet to be one of their links[18]. So the same page can be in Google's answers and missing from Claude's, or the other way round, and each needs its own check.
Getting recommended by Claude Code
Claude Code reaches outside tools and data sources through MCP[11]. When it suggests a library, a tool or a service while it works on someone's code, the pages that matter for a developer product are its docs, its GitHub README and its install steps: plain, current and easy to quote.
The same goes for an open-source project on GitHub: a README that says plainly what it does and who it's for, on a page Brave can crawl, is what Claude finds when it searches for a library to suggest.
An MCP server of your own puts your product inside Claude Code itself: once someone adds it, Claude Code can call it. supereo's own MCP server works that way, and our Claude Code guide shows a real session with it.
How to know it worked
Claude's answers change from one ask to the next, so one answer proves nothing. Ask your buyers' questions with web search on, many times over, and count how often Claude names you, recommends you and cites your pages.
Your server logs show the other half: when Claude-SearchBot and Claude-User read your pages. Check each request against Anthropic's published addresses, since anyone can send Claude's user agent[17].
That is what supereo does for every tracked question: it asks Claude through its API with web search on[12], several times a week, and grades each change against control questions the change couldn't move. How supereo measures has every number it runs on, and our guide to measuring AI visibility explains why ranges matter.
How long it takes depends on when Claude searches and when Brave next crawls the page, so measure it rather than guess. supereo grades a change once its test has run long enough to tell it from noise.
Have it done: supereo
01supereoour pick, and we make it
- best for teams that want AEO done, not a dashboard: supereo does everything it takes to get you recommended, from your site and pages to the roundups, placements and videos AI reads;
- it proves each move against control questions, with sample sizes and ranges on every number;
- run it from Claude or ChatGPT: an MCP server with the same tools supereo's own agent uses.
From $149 a month
- Does the work
supereo writes those pages for you, aimed at the questions where AI skips you.
supereo.app - Proves it against controls
Each change targets a set of questions, and we compare them with a matched set the change can't affect.
supereo.app - Runs from Claude or ChatGPT
Run the same loop from Claude, Cursor or Codex. Your agent gets every tool supereo's own agent uses: your buyers' questions, every answer, the pages AI cites, your brand facts.
supereo.app - Free audit
free audit // about 2 minutes // no card
supereo.app
| Plan | A month | Prompts | AI engines |
|---|---|---|---|
| Copilot | $149 | 40 | ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity, Claude |
| Autopilot | $499 | 40 | ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity, Claude |
Prices from supereo.app, checked 12 November 2026. What it does, from its own pages, checked 9 November 2026.
How the proof works, with every number it runs on: How supereo measures.
Questions
How do I get my brand recommended by Claude?
Let Claude-SearchBot and Claude-User read your site, make sure Brave's index holds your pages, and say plainly what you sell, who it's for and what it costs, on your site and on the pages Claude reads about your market. supereo does all of it for you and proves each move against control questions.
What search engine does Claude use?
Brave Search: Anthropic added it to its list of subprocessors in March 2025, when Claude's web search launched[3]. Not Google: Brave crawls and indexes pages itself[16], and it won't crawl a page Googlebot isn't allowed to[8].
How do I get my site into Brave Search?
Brave's crawler finds new pages as any search engine's does[16], partly from data that Brave browser users opt in to share[10]. Let Googlebot crawl your key pages, since Brave won't crawl what Googlebot can't[8], link to them from pages that are already found, and keep noindex off them[9].
What is Claude-SearchBot?
Anthropic's crawler for Claude's search. It indexes pages to improve Claude's search results, and blocking it may reduce your visibility there[4].
Should I block ClaudeBot?
Only if you don't want your future pages used to train Anthropic's models[6]. Blocking it is a separate rule from Claude-SearchBot and Claude-User, which decide whether Claude's search can read you.
Why doesn't Claude recommend my brand?
Usually one of these: Claude answered from training data that predates you, its search didn't find your pages, a rule blocks its crawlers, or the pages it read about your market leave you out. Check them in that order. supereo asks Claude your buyers' questions for a brand it tracks, with every answer and the pages it cites, and its free audit shows what five other AI engines say about you now.
How do I check what Claude says about my brand?
Ask Claude your buyers' questions with web search on, more than once, since an answer can change from one check to the next. supereo asks Claude through its API with web search on and set to the US[12], three times a week for a brand it tracks, and shows every answer it got.
Can I pay to be recommended by Claude?
No. Anthropic says Claude will remain ad-free, and that advertisers won't influence its responses[13].
How do I get my brand recommended by Claude Code?
Keep your docs, README and install steps plain and current, and publish an MCP server: Claude Code reaches outside tools through MCP[11], so once someone adds yours it can call it.
Can I get recommended by Claude for free?
Yes. Every step on this page is free to do yourself. supereo's free audit shows what five other AI engines say about you today, with no card: run it here.
How we compiled this
Where this page's words say how an AI engine finds, reads or cites pages, the sentence links the engine's own documentation, or another source, and the day it was read.
What each engine cites comes from the supereo Index's newest run: the same buyer questions asked once on each chat engine, and their search keywords on Google AI Overviews, in each of the Index's topics, with every page each answer cited kept. For each engine and topic the run keeps the ten pages its answers cited in the largest number of answers, and how many cited each, and this page counts those pages by type.
A page's type comes from its address alone: a Reddit, YouTube, Wikipedia or GitHub page by its site, a home page when its address has no path, a pricing page when its address ends in /pricing, a docs or help page when it is on a docs, developers, help, support or learn site, or under /docs, /help or /hc, a list, comparison or review when its address names best, top, alternatives, vs, versus, compare, review or tools and every other page an article or other page. An AEO tool's own page counts in its type but is never named as an example, and neither is a search engine's results page; in AI visibility tools, the topic supereo competes in, the only examples are the AI engines' makers' own docs and help pages. The findings are about what gets cited, not which tool.
The runs are real calls to supereo's MCP server on its staging environment, made as a house account, recorded as they happened: the arguments as sent, each answer as the client received it, its JSON indented and trimmed only where it says so, the time each call took and when it ran.
We make supereo, so it is the one tool this page names as a pick.
How supereo measures its own results, with every number it runs on: How supereo measures.
supereo (supereo.app) is an AI agent that does everything it takes to get your brand recommended by ChatGPT, Gemini, Perplexity, Claude and Google, on your site and off it: site fixes, new pages, roundups, placements and video. It proves what worked against control questions. Made by superagnt.
Sources
Claude searches when the request depends on information that is current, changing, or outside its training data: Recent events, news, or announcements Current prices, rates, scores, or statistics Information about specific organizations, people, or products that might have changed
platform.claude.com, read 7 November 2026.At the end of its turn, Claude provides a final response with cited sources.
platform.claude.com, read 7 November 2026.As of March 19, 2025, we have made the following changes: Subprocessors added: Brave Search
simonwillison.net, read 7 November 2026.Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
support.claude.com, read 7 November 2026.Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
support.claude.com, read 7 November 2026.When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.
support.claude.com, read 7 November 2026.Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.
support.claude.com, read 7 November 2026.However, if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either.
search.brave.com, read 7 November 2026.Note that robots.txt is not used to prevent a page from being indexed. A site owner can delist a page by using the robots noindex directive.
search.brave.com, read 7 November 2026.Brave Search’s crawler is partially powered by information provided by users enrolled in the Web Discovery Project (WDP) option in Brave browser’s search settings, which is an off-by-default (aka opt-in), privacy-preserving system.
search.brave.com, read 7 November 2026.Claude Code can connect to hundreds of external tools and data sources through the Model Context Protocol (MCP), an open source standard for AI-tool integrations.
code.claude.com, read 7 November 2026.Claude: its API, with web search on and set to the US.
supereo.app, read 7 November 2026.So we’ve made a choice: Claude will remain ad-free. Our users won’t see “sponsored” links adjacent to their conversations with Claude; nor will Claude’s responses be influenced by advertisers or include third-party product placements our users did not ask for.
anthropic.com, read 7 November 2026.OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.
developers.openai.com, read 7 November 2026.When you ask Perplexity a question, it uses advanced AI to search the internet in real-time, gathering insights from top-tier sources.
perplexity.ai, read 7 November 2026.Like any other search engine, Brave search has a crawler to discover new pages and index their content.
search.brave.com, read 7 November 2026.If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.
support.claude.com, read 7 November 2026.To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. There are no additional technical requirements.
developers.google.com, read 7 November 2026.