One line in robots.txt to say "don't train on this" — Google, Microsoft, OpenAI and Anthropic agreed on the same token, and Bing split "training" from "citing in answers." Three things our site decides this week

On October 30 four companies made the same promise on the same day. If a site owner writes one shared name, "__usage_ai-training," into robots.txt, Google, Microsoft, OpenAI and Anthropic will read it as "do not use this site's content to train generative AI." Until now you had to block each company's crawler by its own name. The same day Microsoft Bing went a step further, separating "used for training" from "cited when building an AI answer" so each can be chosen on its own, with enforcement starting February 1, 2027. But this one line is a declaration, not a lock, and it is not yet a standard. When we opened our own site's robots.txt, it expressed none of these preferences. We wrote down three things a small team running a website should decide this week. At the end we added a short note on Anthropic's amended IPO filing, published the same day.

This article was written with AI (Claude) and reviewed by an editor. The author of the standards draft in the first item works for Anthropic, the company that makes Claude, and Anthropic is one of the four companies that made the promise. The "In brief" section at the end concerns Anthropic's IPO filing. Our company is a paying customer of Anthropic and OpenAI. Wording from company documents is reported as "what the company document says," and where something comes from a single source we name that source. This is not an investment opinion about any company and not material for investment decisions.

For three years, teams running a website have had to look up, company by company, "where do I write what, to stop our articles being used for AI training." Google had Google-Extended, OpenAI had GPTBot, Anthropic had ClaudeBot, Apple had Applebot-Extended. Different names, different scopes. In NO.28 we covered a survey finding AI summaries on 88% of commercial searches; in NO.25, what web share figures fail to count. This week's news is the other side of that: for the first time, the way a site says "here is how AI may use my content" has been gathered into one line. But a line existing and a line being honored are different things, so we separate the facts from the limits.

1. Four companies agreed on one token — the IETF draft "__usage_ai-training" (facts)

On October 30, Kevin Kelley, Anthropic's technical lead for standards, submitted a draft to the IETF titled "Machine-Readable Training Preference Signal." Its document number is draft-kelley-aipref-training-preference-signal-00, and it is an individual draft that expires on May 3, 2027. Its header says it is "not endorsed by the IETF" and has "no formal standing." The draft defines one thing: a policy token, `__usage_ai-training`, written where a User-Agent group name goes in robots.txt. A Disallow under that group expresses a preference that "content at this path should not be used for AI training"; an Allow expresses a preference that it may be. "Training" follows the vocabulary draft of the IETF AI Preferences working group: "using an asset to modify the learned parameters of a generative AI model."

The same day, four companies said they would honor the token. We confirmed three of them directly. Microsoft wrote on the Bing blog that it is "joining industry peers in announcing a shared AI training preference in robots.txt," with enforcement from February 1, 2027. Google wrote on its Search Central blog that it "plans to voluntarily adopt" a proposal under IETF consideration, which "will help websites express their AI training preferences across multiple AI providers in a single step" (our fetch of the post showed only navigation, so we confirmed the wording through Search Engine Roundtable and PPC Land), and added a note to the Google-Extended entry in its crawler documentation that support is intended for "early 2027." Anthropic's draft author told the IETF mailing list that the company "has already started implementing support," without saying which crawlers or when, and with caveats: "it'll take us some time," "this complements our existing crawler controls rather than replacing them," and "how we interpret and apply these signals may change." OpenAI's official post was blocked for our fetch; Engadget reported OpenAI pledged to implement it "in the coming months," while PPC Land wrote that no timeline was given.

The draft has a few more rules an operator should know. First, the token is not a crawling directive. It says content-usage preferences and crawler traffic rules are "independent." Second, the `User-Agent: *` wildcard does not apply to this token. Only a group that names the token is read as a preference. Third, specific expressions take precedence over general ones, and "private agreements MAY override preferences expressed via this mechanism." Fourth, preferences apply based on the robots.txt "in effect when the asset was retrieved." Engadget put that as: the directive "cannot undo any past scraping," and it "doesn't prevent crawling altogether or prevent content from showing up on Google search."

Google's interpretation was explained by Google Search Advocate John Mueller (Search Engine Roundtable, PPC Land). If a site blocks only the shared token without mentioning Google-Extended, Google will "interpret that as if the publisher had disallowed Google-Extended." Google's crawler documentation says Google-Extended is not a crawler of its own but a token used "in a control capacity," governing both training of Gemini models and grounding in Gemini Apps and Vertex AI (providing content from the Search index to the model at prompt time), and that it does not affect inclusion in Google Search or ranking. Pushback from the Google side appeared in the same coverage. Google's Gary Illyes said robots.txt is a suggestion, not a security control — "anyone can impersonate Googlebot in about 30 seconds" (a podcast remark relayed by PPC Land) — and IETF participant Nate Hake said "it's not actually done at the ietf level" with "significant areas of dispute," and that the draft "basically exempts the preferences from applying to anything a user does" (Search Engine Roundtable).

2. Bing split "training" from "citing in answers" — three settings and four dates (facts)

On the same October 30, Microsoft Bing announced its own controls on top of the shared token (Krishna Madhavan, Bing blog). The core is separating two uses. In the company document's definitions, "training" is using content to modify the learned parameters of a generative AI model, and "grounding" is using content as input to help inform AI-generated responses. The document says traditional indexing and ranking are not grounding.

The settings work like this. Training has two values, Allow and Disallow: site-wide through the `__usage_ai-training` group in robots.txt, page-level through `<meta name="Bingbot" content="ai-training:disallow">` or an equivalent X-Robots-Tag header. Page-level overrides robots.txt. Grounding has three values, Allow / DisallowExtended / Disallow, and is set at page level only (meta or header). There is no site-wide robots.txt setting for it. DisallowExtended permits citation only in experiences Microsoft operates itself (AI Mode, AI Overview, Copilot, Microsoft 365) and refuses it in grounding services sold to third parties. If nothing is written, both default to Allow. The company document says these preferences "do not change eligibility for traditional Bing search results," while also saying that setting grounding to Disallow or DisallowExtended will "significantly reduce" visibility in AI-generated experiences. It adds that neither choice guarantees inclusion, traffic or payment.

There are four dates. December 15, 2026: temporary grounding settings open in Bing Webmaster Tools (only for sites using NoArchive or NoCache as of that day). February 1, 2027: the new preferences take effect as pages are crawled. After March 1, 2027, NoArchive and NoCache, used since 2023, are no longer read as training or grounding preferences. September 30, 2027: the temporary grounding settings expire. The document adds that IndexNow can request a recrawl but does not guarantee a date.

In brief — Anthropic's amended IPO filing (facts)

On November 2 Anthropic added third-quarter numbers to its S-1 in an amendment. The EDGAR original was blocked for our fetch this time, so we report figures that four outlets (Fortune, Business Insider, SiliconANGLE, Investing.com) cite from the filing. Third-quarter (July–September) revenue of about $17 billion (Q2 $11.6 billion, year-ago quarter $1.4 billion — Fortune), operating expenses $17.5 billion, an operating loss of $567 million (Q2 was an operating profit of $974 million; SiliconANGLE alone reports $536 million, so the outlets differ), and a net loss of $23.3 billion, mostly an accounting charge from fair-value changes in convertible notes. The main reasons the filing gives for the operating loss are "$5.0 billion of compute under short-term commitments" (Business Insider, SiliconANGLE) and a $2 billion non-cash charge from the employee donation-matching program. The sentence that price reductions cut revenue by about $600 million between Q2 and Q3 was reported by Fortune alone. Total purchase and lease commitments for compute and infrastructure are $895.4 billion (all four agree); the split of $447.5 billion through 2030 and $69.8 billion within twelve months is Investing.com's alone. In NO.30 we wrote "the price you pay halves, the cost of making it runs to billions a month"; this filing shows the two meeting inside one company in one quarter (interpretation). This is not material for investment decisions, and we do not report offering size, price or timing. We will return to it as a "price and cost" follow-up once we have read the original.

Reading the two together (interpretation)

First, one line is a declaration, not a lock. The draft itself says it is "not a directive for crawling behavior," and a Google employee says impersonation takes thirty seconds. What this line does is make four companies read "please don't train on our content" the same way. Whether they honor it depends on each company's promise, and a site currently has no way to verify that they did. Still, the past three years' chore of hunting each company's name shrinks. For a small team in particular, the real change is that "edit robots.txt every time a new AI company appears" becomes "check whether that company accepts one token" (interpretation).

Second, training and citing are different questions. Only Bing split the two; the shared token covers training only. Google reads the token as a Google-Extended block, and by Google's documentation Google-Extended governs not just training but also grounding in Gemini Apps and Vertex. So how a site tells Google "refuse training, allow citation in Gemini answers" with one shared token is a gap. We could not find this in Google's documentation and leave it as a question to be answered when Google starts support in 2027. Bing's line that blocking citation "significantly reduces" visibility in AI experiences is an honest one, and points the same way as the "AI summaries on 88% of commercial searches" we saw in NO.28. Refusing training and refusing citation carry different costs for a site (interpretation).

Third, "what a user asks for" sits outside. As Hake pointed out, the draft effectively exempts fetches a user directly requests, and Kelley's mail says interpretation "may change." Our articles being read when an AI chat answers someone's question continues regardless of this token. Whether to block that is a separate decision, and we are on the side of not blocking. AP News is written to be read (interpretation).

If it were us — three things to decide this week (practice)

Sort content into three groups and fill two columns, "training / citing." When we opened our site's robots.txt, it had a single `User-agent: *` group with Allow plus folder Disallows (source, scripts, backups, docs, and the four languages awaiting review) — and not one AI token. Under the draft's rules `*` does not apply to the token, so we currently express no preference at all, and by Bing's defaults both are Allow. So first we build a decision table. ①AP News, product pages and the company page: "training allowed, citing allowed" — they are written to be read. ②Source, script, backup and internal-document folders are already blocked from crawling, but that is a traffic rule, not a training preference, so we write the same paths as Disallow under the token group too. ③Pages where user content mixes in (reviews, boards, inquiries): "training refused," and citing set to DisallowExtended on Bing. The table sits next to the permissions table from NO.29 and is reviewed quarterly.
When writing it, add the token group separately and keep the company-specific names. The draft, Google and Anthropic all say existing controls keep working, so we add the `__usage_ai-training` group and do not delete existing groups such as Google-Extended, GPTBot and ClaudeBot. We do not expect anything written under `*` to apply to the token. Bing's grounding preference is not in robots.txt and works only as a page-level meta or header, so it goes only into the template for group ③ pages. On the calendar: February 1, 2027 (Bing enforcement) and March 1, 2027 (NoArchive/NoCache no longer read as preferences), and in the first week of February we reopen each company's documentation to see whether Google and OpenAI have actually started support. If an official document will not open, we cross-check with two outlets.
Keep "written" and "honored" in different columns. A site has no way to know whether this token was honored. So we do only what we can. Once a month we count visits by AI crawler names (GPTBot, ClaudeBot, Bingbot and so on) in server logs, and note in the table that Google-Extended has no crawler name of its own and will not appear in logs. The report that click data may not come to Google Search Console's AI performance report (an Illyes remark, Search Engine Roundtable's November 2 recap, one source) is kept as a flagged reference only. In three months we check each company's documentation again for changes to the definition of "training." We do not report "we wrote the preference" as "AI doesn't use our content."

What we did not choose this time

The report that Reddit's CEO confirmed declining Google Search traffic (November 2, Search Engine Roundtable) is deferred because we could not open the original remarks or the earnings call materials. SemiAnalysis's agent benchmark GDPx, published November 2 (11 domains, the best model at 12%, finance and legal all zero under binary grading), has an Anthropic model in first place, a large conflict of interest for us, and little cross-coverage yet, so it goes into a separate follow-up to NO.33 and NO.34 on "what a scoreboard measures." Alphabet's €6 billion bond and Oracle's $20 billion raise (TNW citing Reuters) belong to the same family as the commitment figure in "In brief," but could read as investment material, so we will look at them in the "price and cost" follow-up. The report that OpenAI wants data centers called "compute campuses" (Axios, Business Insider) we did not open. Model stories circulating as leaks or rumors, and a kernel-flaw exploit story, we did not use: no official confirmation, or offensive detail.

What we could not confirm

OpenAI's official post ("Helping websites express AI training preferences") was blocked for our fetch. "In the coming months" is Engadget's relay alone; PPC Land wrote there was no timeline. We also could not confirm which crawlers (GPTBot and others) will read the token.
The body of Google's Search Central post showed only navigation to our fetch, so we confirmed its sentences through two outlets. Whether blocking the shared token alone also blocks grounding in Gemini Apps we could not find in Google's documentation.
Anthropic's "already started implementing" is one mail from the author, with no crawler or date. We did not check whether Anthropic's official crawler documentation reflects it.
Bing's dates and settings we confirmed directly, but "does not change eligibility for traditional Bing search results" is the company's statement, with no independent verification.
All figures in "In brief" are outlets citing the filing. The operating loss differs between outlets ($567 million vs. $536 million), and the $600 million price reduction, $447.5 billion through 2030, $69.8 billion within twelve months, and $9.9 billion compute expense each come from one outlet. We will compare against the EDGAR original when it opens.

What we read alongside

We opened sources of different kinds and confirmed them directly. They are grouped into standards documents, official company documents and statements, and press coverage.

Standards documents:

datatracker.ietf.orgMachine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00 (IETF Datatracker)English; token definition, "not a crawling directive," wildcard rule, private-agreement clause

Official company documents and statements:

Press coverage:

Our earlier articles: