AI companies are backing one robots.txt line to opt out of training, but on Google it also decides whether Gemini cites you
On 30th Oct'26 Anthropic published a draft standard for a single robots.txt entry, the policy token __usage_ai-training, that tells AI companies whether a site's pages may be used to train their models. A group with that token and Disallow: / opts the whole site out, and Allow lines carve out exceptions such as a press page. The same day Google said it aims to support the token in 2027 and will treat an opt-out as if the site had blocked Google-Extended, its existing AI control, while sites that explicitly allow Google-Extended stay eligible to be cited and send clicks through Gemini's grounded answers. Reporting by The Verge says Anthropic already honours the token, with Microsoft planning to follow next year and OpenAI in the coming months. The draft is an individual submission with no formal IETF standing, and each company's own crawler controls keep working alongside it.
What it actually means
For anyone running a content site, this replaces a growing list of crawler names with one line, which is the useful part. The catch is that the line means different things to different companies, and Google will read it as blocking Google-Extended, the same switch that decides whether Gemini can cite your pages and send you its clicks. A marketer who adds it to keep content out of training could also drop out of the AI answers their buyers increasingly read.
So decide which matters more before you add it. As Google describes it, an explicit allow for Google-Extended keeps you eligible for Gemini's cited answers but takes precedence over the new opt-out, so on Google you are choosing one or the other. Nothing changes on Google until 2027, which gives you time to check how much of your traffic comes from AI answers first.
One robots.txt line now keeps your site out of AI training, and on Google it may keep you out of AI answers too.
Source: Google Search Central · Sunday, 1 November 2026