Afternoon BriefAI Search & Discovery

Cloudflare Just Put a Gatekeeper Between Your Earned Media and AI Search

Cloudflare will block AI training and agent bots from ad-supported pages by September 15, 2026. If your earned media lives on those sites, AI engines may never see it. Here is what that means for your brand visibility strategy.

Jaxon Parrott
Jaxon ParrottJul 1, 2026

Cloudflare just announced it will block AI training and agent bots from ad-supported web pages by default starting September 15, 2026. Cloudflare serves roughly 20% of the web. If your best earned media placement lives on one of those pages, the AI engines deciding who gets cited may never read it.

That is not a publisher problem. That is your problem.

The Layer Nobody in the Publisher Conversation Is Seeing

I have spent nearly a decade placing brands in publications that move markets. I have watched the entire value chain of a media placement shift: from human eyeballs to machine extraction to AI citation. Every step of that shift has increased what a single placement is worth, because a placement that gets extracted by ChatGPT or Perplexity compounds in a way a print mention never could.

Now a third party just inserted itself into that chain, and most founders have no idea it happened.

Cloudflare's agentic internet bot report crossed a threshold that should alarm every brand investing in earned authority: more than 50% of web traffic is now non-human. Of the crawler requests specifically, 52% are for AI training, up from 22% in Spring 2025. That is a more than doubling in 15 months. The bots now outnumber the humans. And Cloudflare just gave publishers a one-click option to lock them out.

Cloudflare's New Three-Tier System and Why It Traps Mixed-Use Bots

Here is the mechanism. Cloudflare now classifies every bot into three tiers: Search, Agent, and Training. Search bots index your content for traditional search results. Training bots crawl to feed AI models. Agent bots act on behalf of users in real time.

The catch is the middle tier. Over 36% of crawler traffic comes from mixed-use bots that blend search, training, and agent behavior into a single crawler. PerplexityBot, for example, both indexes for its search engine and feeds its AI answer engine. When a publisher enables Cloudflare's default block on training and agent bots, multi-purpose crawlers follow the most restrictive applicable rule.

That means a publisher who only intended to block AI training might also block the bot that delivers their content into Perplexity's AI answers. The brand that earned a placement on that site loses visibility in a place the publisher did not even intend to restrict.

Your Earned Media ROI Now Has a Dependency You Did Not Negotiate

Let me make this concrete. You paid for a placement in a major tech publication. The article is strong. It names your company, frames your value proposition, links to your site. A year ago, that article would have been crawled by ChatGPT-User, PerplexityBot, ClaudeBot, and OAI-SearchBot. Those bots would have extracted your claims, stored them, and cited you when a buyer asked the right question.

After September 15, if that publication is on Cloudflare and flips the default, those same bots hit a wall. Your placement still exists for humans who navigate to the page. But the machines deciding who shows up in AI answers never see it.

And this is not theoretical. Cloudflare's own data shows the scale: their developers.cloudflare.com received 4.8 million AI crawler visits in 30 days. In March 2026 alone, legacy documentation pages were crawled 46,000 times by OpenAI, 3,600 times by Anthropic, and 1,700 times by Meta. That is the volume of machine attention at stake. When it disappears from a page your brand sits on, you lose citation surface you cannot replace with ad spend.

The Real Question: Which of Your Placements Are AI-Accessible After September 15?

The move most founders will make is to watch this play out. That is the wrong move.

Here is what to do instead:

1. Audit your earned media portfolio for Cloudflare exposure. Every publication you have been placed in that uses Cloudflare is now a variable. Not a guarantee.

2. Ask your publishers directly. Will they block training bots? Agent bots? Both? The defaults change September 15. If your publisher has not opted out, your placement is blocked from AI crawlers on ad-supported pages automatically.

3. Weight AI-accessible placements higher in your Machine Relations strategy. A placement that an AI engine cannot read is still valuable for human credibility. But it will not compound in AI search the way an accessible placement does. Your citation surface depends on it.

4. Prioritize owned content as the floor. Your own site, your own blog, your own research pages are the placements no third-party gatekeeper can block. Earned media is still the highest-leverage way to build citation authority across AI engines. But it is no longer sufficient by itself if the infrastructure layer can cut the connection between the placement and the machine.

The click-through data makes the urgency clear. When AI summaries appear in search, users click traditional links only 8% of the time. Within summaries, the click rate is 1%. Traffic is already not coming from clicks. It is coming from extraction. And extraction requires the bot to read the page.

This Is the Infrastructure Layer of Machine Relations

I coined Machine Relations because the old PR model was built for a world where humans were the audience. Machines are now the first reader of every placement. That has been true for a while.

What changed today is that a third party now controls whether the machines can read at all.

Publishers have a legitimate concern. Their content is being consumed by AI models without compensation, and some heavily crawled categories saw human traffic declines of 40% in under a year. Cloudflare is offering a real tool for a real problem. I respect the move.

But if you are a brand that built its authority on earned media, you need to understand what this means for the other side of the equation. Every placement you earned is now split into two categories: AI-accessible and AI-invisible. The publisher controls the switch. You do not.

The brands that adjust now will audit their citation surfaces, rebalance toward AI-accessible placements and owned content, and build relationships with publishers who choose to stay open to AI crawlers. The brands that wait will discover in Q4 that their pipeline dried up and they cannot explain why.

The shift already happened. The infrastructure just caught up.

FAQ

Will Cloudflare's September 15 changes affect all AI search engines equally?

Not equally. Cloudflare's system classifies bots into Search, Agent, and Training tiers. Pure search bots remain allowed by default. But mixed-use bots that combine search and training, like many AI answer engines use, face the most restrictive rule. The impact depends on which bots your key publications choose to block and how each AI engine's crawler is classified.

How can I check if my earned media placements are at risk?

Run a simple audit. Identify every major publication where you have a placement. Check whether that site uses Cloudflare (a DNS lookup or site header check will tell you). Then contact the publication's digital team to ask whether they plan to adopt the default AI bot blocking settings before September 15. The placements on sites that block agent and training bots will be invisible to those AI engines after the deadline.

Does this mean earned media is less valuable now?

No. Earned media is still the highest-leverage asset for building brand authority that AI engines trust. But its value now depends on whether the AI engines can actually access it. A placement on a site that blocks AI crawlers is still valuable for human credibility and direct reader trust. It just will not compound in AI search. The fix is not to abandon earned media but to factor AI accessibility into every placement decision.

Additional source context

Continue Exploring

The most relevant next read from elsewhere on AuthorityTech.

BlogJun 30
AI Citation: What It Actually Means When an AI Engine Names Your BrandExecution playbook
BlogJun 29
AI Citation: What Actually Happens When an LLM Decides to Name Your BrandExecution playbook
Glossary
Citation EligibilityRelated glossary concept
ResultsJun 12
Firecrawl on SourceForge: The Extraction Infrastructure That Decides Whether AI Agents Work or HallucinateLive proof