From September, a fifth of the web starts turning your agent away at the door
Cloudflare has split bot traffic into three behaviours - Search (indexes you, sends referrals), Agent (acting in real time on a person's behalf, including chat fetch bots and browser-use agents driving Chrome), and Training (absorbing your content into a model). On 15 September 2026, new defaults land: for all new domains and all existing free customers, Training and Agent bots will be blocked by default on any page that displays ads. Search stays allowed. "Verified" no longer means "allowed" - a bot is now allowed only by its category. Cloudflare is also opening up verification and testing transitive trust, so agents that identify themselves honestly keep the door open. Roughly 20% of web domains sit behind Cloudflare.
What it actually means
If you're building an agent that browses the live web, its supply of pages is about to shrink - and the reason isn't hostility to AI, it's ads. Cloudflare's logic is blunt and hard to argue with: an ad means a human was supposed to land here. Your agent isn't a human, and it doesn't look at ads. Two things follow. First, quietly test what your agent does when a fetch returns a block, because "the page came back empty" is about to become, and an agent that hallucinates around a 403 is worse than one that stops. Second - and this is the bigger shift - anonymous scraping is on its way out and identity is on its way in. Cloudflare is explicitly building an incentive model where being honest about who you are and what you'll do with the content earns you access. If you run automation against other people's sites, "unidentified traffic that hopes nobody notices" has an expiry date. Start declaring yourself now, while declaring yourself still gets you in. I have no idea what this will mean for agent to agent commerce!
Anonymous scraping is ending; identity is the new access token.
Source: Cloudflare · Tuesday 14 July 2026