Cloudflare Changes Its AI Bot Defaults on 15 September: What to Check in robots.txt
On 1 July 2026 Cloudflare announced a change to how AI traffic is handled by default, taking effect on 15 September 2026. It has been summarised online as “Cloudflare will block AI bots by default”, which is close enough to be dangerous and wrong in the two places that matter operationally.
Roughly a fifth of the web sits behind Cloudflare, so this is worth ten minutes even if you are certain it does not apply to you.
What actually changes on 15 September
Cloudflare now sorts AI traffic into three categories rather than one. From 15 September, for all new domains onboarding to Cloudflare, Training and Agent traffic will be blocked by default on the pages that display ads, while Search remains allowed by default. Three details in that sentence are routinely dropped:
- It is not sitewide. The default block is scoped to ad-carrying pages. If your site runs no ads, the new default block has very little to bite on.
- Search stays allowed. The category that governs whether AI systems can find and cite you is not the one being blocked.
- The trigger is onboarding a new domain. Cloudflare’s own blog and changelog say new domains onboarding to Cloudflare, and nothing more.
That last point needs care, because a widely repeated version of this story says the change also applies to the entire free tier. That claim comes from TechCrunch’s coverage on the same day, which attributes it to the company. We could not find it in Cloudflare’s published blog post or in the corresponding changelog entry, both of which say new domains. So: reported by TechCrunch, not documented by Cloudflare. If you are on the free tier, check your own settings rather than trusting either of us.
Everyone can opt out of the new defaults in their security settings at any point before 15 September, and the opt-out is the part of this that deserves a calendar entry.
Three categories, which means this is no longer one decision
The taxonomy is the actually useful part of the announcement, because it separates things that were previously lumped together as “AI bots”.
| Category | What it covers | What blocking it costs you |
|---|---|---|
| Search | Indexing your content so it can be used to answer questions later | Your visibility inside AI answers. This is the expensive one to block |
| Agent | Real-time fetching on a person’s behalf, including chat fetch bots and browser-driving agents | The ability of a user’s assistant to open your page during a live session |
| Training | Collecting content to train or fine-tune a model, where it is permanently absorbed | Almost nothing you can measure, which is why most publishers start here |
The trap in the middle: multi-purpose crawlers
This is the part with real downside and it is barely being discussed. Under the new rules, crawlers that combine behaviours are judged on all of them. Cloudflare states that Googlebot, Applebot and Bingbot will be blocked by customers who have chosen to block Training.
Read that again if you have ever ticked a “block AI training” box. The same toggle that costs you nothing measurable in AI answers can cost you your presence in Google, Bing and Apple search entirely. That is not a hypothetical trade-off; it is the difference between an ideological position and an outage.
This is also the one part of the change that reaches existing customers, since the reclassification applies to how those toggles are interpreted, not only to new signups. If you set an AI blocking preference at some point in the last two years and have not looked at it since, that setting now means something different from what it meant when you set it.
OAI-SearchBot and GPTBot are not the same decision
Most robots.txt files that mention AI at all were written in a rush in 2023 or 2024 and block by brand rather than by function. The vendors have since documented what each agent does, and the differences are large.
| Agent | Vendor’s stated purpose | What blocking it does |
|---|---|---|
| OAI-SearchBot | Surfacing sites in ChatGPT’s search features | You will not be shown in ChatGPT search answers, though you can still appear as a navigational link |
| GPTBot | Making OpenAI’s foundation models more useful and safe | Signals that your content should not be used in model training |
| ChatGPT-User | Certain user actions in ChatGPT and custom GPTs | Little. OpenAI states robots.txt rules may not apply, because a person initiated it |
| ClaudeBot | Collecting content that may contribute to model training | Excludes your material from future training datasets |
| Claude-SearchBot | Improving search result quality for users | Prevents indexing for search |
| PerplexityBot | Surfacing and linking sites in Perplexity results, explicitly not model training | Removes you from Perplexity results |
| Perplexity-User | User-initiated fetch | Little. Perplexity states this fetcher generally ignores robots.txt |
| Google-Extended | A robots.txt token governing Gemini and Vertex AI training | Nothing in Google Search. Google states it is not a ranking signal |
The exact strings and stated purposes are in OpenAI’s bot documentation, Perplexity’s, and Google’s crawler list. Read them before writing rules rather than after.
The practical shape that falls out of this for most businesses: allow the search agents, allow or ignore the user-initiated fetchers because you cannot reliably stop them anyway, and make training a deliberate decision rather than a default. Publishers whose product is the text itself will reasonably reach the opposite conclusion, and that is a licensing position, not a technical one.
The counter-fact: blocking does not reliably stop citation
Here is the finding that should temper any confidence in robots.txt as a control. BuzzStream examined citations against a set of news sites that block AI agents and reported that 70.6% of the sites blocking ChatGPT-User still appeared in AI citations.
Two caveats, one about the number and one about the mechanism.
The number is fragile. The blocking universe is drawn from a top-50 news site list, and every headline percentage in the study resolves cleanly to seventeenths, which means the denominator is almost certainly seventeen sites. One site moves the figure by six points. Treat it as an illustration, not a statistic.
The mechanism, though, is solid and is the part worth keeping. BuzzStream’s own explanation is that some AI retrieval works from search-result data alone: titles, URLs and snippets, without ever fetching the page or touching your origin server. OpenAI’s documentation says the same thing from the other side about user-initiated fetches. A robots.txt rule governs crawling. It does not govern what a system already knows about you, or what it can assemble from a results page.
So the honest framing of a block is: it is a statement of preference that most well-behaved crawlers respect, and it is not an access control. If your content genuinely must not be reachable, that is an authentication problem, not a robots.txt problem, which is the same distinction we draw in our vendor risk checklist.
Content signals: the part of robots.txt nobody reads
Alongside the blocking controls, Cloudflare has been pushing an extension that expresses intent rather than access. The Content Signals Policy, published in September 2025, adds a line to robots.txt declaring separately whether you permit search indexing, use as AI input for grounding, and use for training. Cloudflare applied a default of search allowed and training disallowed across several million managed domains, deliberately leaving the AI input signal unset. In 2026 it added a fourth field describing how the content may be used, defaulting existing managed files to a reference-style value.
It is not enforceable and nobody pretends otherwise. It is a clearer statement of intent than a blanket disallow, and it costs one line. The specification is short enough to read in full.
The checklist before 15 September
- Find out whether you are behind Cloudflare at all. Plenty of teams do not know, because the CDN was set up by an agency three years ago. Our note on choosing business hosting covers why that ownership question keeps mattering.
- Read your current robots.txt out loud. Not the one in your repository, the one being served. Managed rules can add lines you never wrote.
- Check any AI blocking toggle set in the last two years. Under the new classification, a Training block also blocks the multi-purpose search crawlers. This is the item that can cost you real traffic.
- Decide per category, not per vendor. Search, agent and training are three separate commercial questions and only one of them affects visibility.
- Confirm you are not blocking by accident. A firewall rule or bot-management setting can return a challenge to a crawler that robots.txt happily allows. The curl check we published separately catches this in one command.
- Diarise the opt-out date. The window closes on 15 September.
Why this is happening at all
The volumes explain the policy better than the press releases do. Cloudflare’s network-scale bot report, published the same day, states that 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025, and that mixed-use crawlers now account for over 36% of activity. More than half of all internet traffic is now non-human. Whatever you think of the compensation argument, that is a genuine infrastructure cost being borne by publishers.
It is also worth remembering that declared identity is not guaranteed. In August 2025 Cloudflare accused Perplexity of using undeclared crawlers to bypass no-crawl directives and removed it from the verified bot list. Perplexity disputed the characterisation. Whoever was right, the episode is the practical argument for treating robots.txt as a signal rather than a fence.
The short version
From 15 September 2026, new Cloudflare domains get Training and Agent traffic blocked by default on ad-carrying pages, while Search stays allowed. The free-tier extension of that is TechCrunch’s reporting rather than Cloudflare’s documentation. The change with real risk is that multi-purpose crawlers are now judged on all their behaviours, so blocking Training can also block Googlebot, Bingbot and Applebot. Block by function rather than by brand: OAI-SearchBot costs you ChatGPT visibility, GPTBot costs you almost nothing measurable, and the user-initiated fetchers ignore robots.txt by design. And blocking does not reliably prevent citation, because a lot of AI retrieval never touches your server. If you are moving hosts or CDNs anyway, the settings audit is nearly free at that moment, which is one of the checks we build into a website migration. The wider question of which factors actually earn a citation is covered in our ranked breakdown of the evidence.
