How to Block AI Bots Without Losing Google and Bing

You block AI bots and keep your search visibility by treating them as separate categories, not one undifferentiated threat. Allow the crawlers that power search indexing and AI citations (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot). Disallow the ones that harvest content for model training (GPTBot, Google-Extended, ClaudeBot, CCBot). Then back that up with identity verification and network-level enforcement, because robots.txt alone is a polite request that a growing share of bots simply ignore.

That last part matters more than most robots.txt tutorials admit. Get the layering wrong and you either leak content to model trainers or, worse, accidentally block the crawlers that put you in search results and AI Overviews in the first place.

Why 'just block AI bots' breaks your search visibility

The instinct to write one blanket rule, Disallow: / for anything with "AI" or "bot" in its name, is understandable and wrong. A lot of AI companies route their search-answer crawlers and their training crawlers through overlapping infrastructure, and some search-adjacent bots share naming conventions with the crawlers you actually want blocked. Block too broadly and you knock Googlebot or Bingbot out along with GPTBot, and that tanks your organic rankings and your odds of showing up in AI-generated answers that cite live search results.

The fix is not less blocking. It is more precise blocking, aimed at the right target.

The bot landscape: training, search, and real-time agents

AI companies have largely split their crawlers into distinct user-agents for distinct jobs. Once you see the split, the strategy gets a lot simpler.

• Training crawlers. These scrape content to fine-tune or pretrain models. Examples: OpenAI's GPTBot, Google's Google-Extended, Anthropic's ClaudeBot, Common Crawl's CCBot.
• Search and citation crawlers. These index content so an AI assistant can cite or summarize it in an answer, functionally similar to how Googlebot feeds search results. Examples: OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot.¹²
• Real-time agent fetches. These happen when a user asks an AI assistant to pull a specific page right now, on their behalf. This traffic is often treated by AI providers as user-directed access rather than crawling, which means robots.txt rules may not apply to it the way site owners assume.²

Google-Extended is the clearest example of why this split matters. It is a separate, independent user-agent from Googlebot.³ Disallowing it stops your content from training Gemini and Vertex AI models, but has zero effect on Google Search rankings, indexing, or eligibility for AI Overviews, all of which draw from the standard Search index.¹² Block Google-Extended, keep Googlebot fully welcome, and take no search penalty.

Step 1: Split your robots.txt directives by user-agent

Start here. It's free, it's fast, and it's what search engines and well-behaved AI companies actually check first.

Write explicit, separate blocks for each user-agent rather than one generic rule. Allow the crawlers you want:

• Googlebot and Bingbot for standard search indexing.
• OAI-SearchBot and Claude-SearchBot for AI citation and answer traffic, if you want your content eligible to be quoted in ChatGPT or Claude answers.

Then explicitly disallow the training crawlers:

• GPTBot
• Google-Extended
• ClaudeBot
• CCBot

Do not rely on a single wildcard rule to cover "everything AI." Name each user-agent individually. This is also the moment to decide whether you want AI-answer citation traffic at all. Some publishers want the referral exposure that comes with being cited in an AI answer. Others have concluded the trade is bad, given how little traffic actually comes back.

Why isn't robots.txt enough to block AI bots?

Here's the uncomfortable part. Robots.txt is a voluntary, unauthenticated text file with no legal force behind it, formalized as RFC 9309 but enforced by nothing.² Compliance is a courtesy, and the data shows that courtesy is eroding.

A large-scale academic study of 3.9 million web requests across 36 sites found that compliance with robots.txt drops as directives get stricter, and that search engine crawlers and AI search crawlers were among the least compliant categories measured, even less compliant in places than the AI training bots.⁴ The same study found that just 20 named bots accounted for over 40% of all observed traffic in the dataset.⁴

TollBit's bot-tracking data shows the trend getting worse, not better. AI scrapes that bypassed robots.txt directives climbed from 3.3% of requests in Q4 2024 to 12.9% by Q1 2025, and TollBit logged 26 million bypassing scrapes in March 2025 alone.⁵ Over the course of 2025, AI bot traffic on monitored sites grew from roughly 1 bot visit per 200 human visits in Q1 to about 1 per 31 by year end.⁶

Some bots do not just ignore robots.txt. They actively evade it. Cloudflare found that Perplexity's crawlers, once blocked by declared user-agent and network rule, started modifying their user-agent string and rotating source ASNs to impersonate a generic Chrome browser on macOS, making millions of requests a day to sites that had explicitly denied them.⁷⁸ Cloudflare's security team put it plainly: "We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity."⁵

The economic backdrop explains why site owners bother fighting this at all. TollBit's scrape-to-referral ratios, how much content a company takes versus how much visitor traffic it sends back, are lopsided: Bing sits at roughly 11:1, OpenAI at 179:1, Perplexity at 369:1, and Anthropic at a striking 8,692:1.⁵ Training crawlers take content and give almost nothing back in return traffic. That is the real reason to separate them from search crawlers instead of blocking everything with an AI-sounding name.

Step 3: Verify identity instead of trusting headers

If a bot can lie about its user-agent, the fix is to stop trusting the header and start verifying the source.

Google publishes downloadable IP-range JSON files and reverse-DNS masks for Googlebot and its other crawlers, so you can programmatically confirm a request claiming to be Googlebot actually originates from Google's infrastructure, not a spoofed string. That closes one obvious hole: someone impersonating Googlebot to slip past your bot rules.

The more durable fix, especially for bots that rotate IPs and ASNs the way Perplexity's do, is Web Bot Auth, an IETF-chartered standard built on HTTP Message Signatures (RFC 9421).⁹¹⁰ Bots sign requests with an Ed25519 key, attach a Signature-Agent header, and publish a JWKS key directory. Your origin verifies identity cryptographically instead of guessing from an IP or a user-agent string. Cloudflare, Amazon, and Akamai back it, and OpenAI's ChatGPT Agent already uses it.⁹ The IETF's own use-case draft frames the goal directly: mitigate volumetric abuse, control bot access, and audit bot behavior with actual proof of identity.¹⁰

Step 4: Let your CDN or WAF enforce what robots.txt can't

Robots.txt asks nicely. Your CDN or WAF can actually say no.

Cloudflare's bot management now classifies AI crawlers into three behavioral buckets: Search (crawlers that index content to answer questions later), Agent (real-time automated activity on a person's behalf), and Training (crawlers harvesting content for model fine-tuning).¹¹ Left unmanaged, Agent traffic at volume starts to look like the agentic flooding that can hit your APIs and support channels harder than any human ever could. You can block Agent and Training traffic at the network layer while explicitly keeping Search traffic flowing, which is the practical version of the user-agent split from Step 1, enforced at a layer bots can't simply lie their way past. As of September 15, 2026, this becomes the default posture for new domains on Cloudflare: Training and Agent traffic blocked on ad-supported pages, Search left alone.¹¹

This layer also gives you managed challenges and rate limiting, which matter for the bots that don't announce themselves at all. A basic anti-bot escalation looks like this in practice, and it degrades in a known order:

1. IP blocking stops the laziest scrapers but is trivially bypassed by rotating addresses.¹²
2. ASN blocking catches datacenter-hosted crawlers but does nothing against traffic routed through residential proxies.¹²
3. User-agent filtering is defeated by simply rewriting a header, which is exactly what Perplexity's crawlers were caught doing.⁷⁵
4. TLS/JA4 fingerprinting and CAPTCHAs raise the cost of scraping further, but face increasingly capable AI-driven bypass tools, so treat them as friction, not a wall.¹²

No single control here is sufficient by itself. That is the whole argument for stacking them.

Step 5: Put internal tools behind Zero Trust, not robots.txt

Everything above assumes you want some crawler traffic, just not all of it. Internal tools are a different case entirely.

A dashboard an employee built to pull customer data, an internal API wrapper, a scraped report someone stood up on a subdomain, none of that has any legitimate reason to be publicly crawlable, by a search engine or an AI bot. These are often exactly the kind of shadow tools employees spin up fast with AI coding assistants, described in our look at the shadow AI tech stack, and they rarely get the same access review a procured SaaS tool would, the exact governance gap covered in governing the apps your employees build.

Don't rely on a disallow rule to keep them off the internet. Robots.txt is a courtesy note, not a lock, and legal scholarship has already flagged that AI companies' growing disregard for that courtesy threatens the web's long-standing crawling norms, which is pushing more publishers toward logins and paywalls by default. Internal tools should start there, not end there. Put them behind Zero Trust network access so they require authentication before a single request resolves, bot or human. If you are building internal tools from scratch anyway, platforms like Remy bake access control in at the start, instead of leaving a team to bolt on authentication after something has already been indexed by a crawler nobody invited.

How do you know the split is actually working?

Assume nothing. Confirm it.

• Check Search Console and Bing Webmaster Tools crawl stats to make sure Googlebot and Bingbot are still crawling and indexing at normal volume after you change robots.txt.
• Pull server logs and filter by user-agent and ASN to see who is actually hitting your site, not just who claims to be hitting it. This is the only way to catch a bot spoofing a browser string.
• Use bot analytics tooling, such as TollBit's bot tracking or Cloudflare's bot analytics, to see the split between allowed search traffic, blocked training traffic, and anything sneaking through disguised as something else.
• Re-test after every CDN or robots.txt change. A rule meant to block GPTBot can silently catch a legitimate crawler if it's written too broadly, and you often won't notice until rankings dip weeks later.

Quick reference: a layered blocking checklist

Use this as your decision tree, not a one-time setup:

1. Public marketing and content pages. Allow Googlebot, Bingbot, and any AI search/citation crawlers you want. Disallow named training crawlers explicitly in robots.txt.
2. Internal tools and dashboards. Skip robots.txt entirely as a control. Put these behind Zero Trust or equivalent authentication so they are never reachable by an unauthenticated request, crawler or not.
3. Anything with real proprietary value. Verify identity, don't trust headers. Use published IP ranges for known search engines and move toward Web Bot Auth verification for anything else claiming a privileged identity.
4. Network layer. Configure your CDN/WAF's AI bot classification (Search vs. Agent vs. Training) as your enforcement backstop, since robots.txt compliance is voluntary and declining.
5. Ongoing. Audit logs and crawl stats regularly. The bot landscape changes fast enough that a policy set six months ago is worth re-checking today.

None of this is a one-and-done fix. The bots that matter most, the ones actively evading blocks, will keep evolving. The site owners keeping up are the ones treating this as infrastructure to maintain, not a text file written once and forgotten.