This article was written with AI (Claude) and reviewed by an editor. Anthropic, the subject of this article, is the company that makes the Claude we used to write it, so we have a conflict of interest. What Anthropic says about itself is marked as "what the company announced," and third-party interpretation is marked separately. This is not investment advice on any particular company.
On October 15, Anthropic posted an update on "independent evaluators" to its official blog (it is shown as having had some footnotes edited on October 16 after posting). The core of it is that a structure has begun in which 6 staff of the outside evaluation organization METR use Anthropic's internal equipment and tools to check for risks. This article separates what is confirmed in this announcement from what is open to interpretation.
What Anthropic said (what the company announced)
- METR: 6 staff of the nonprofit evaluation organization METR are onboarded as if they were joining the company, use company equipment and office space, and are said to get access to internal tools and systems at a level similar to Anthropic's internal risk assessment team. The staff are encouraged to share everything with them except customer data, attorney-client privileged information, and model architecture information.
- Publication terms: METR publishes its own analysis of Anthropic's risk levels, incidents, and practices. Anthropic wrote that it "cannot block the publication," but that it can require information in the three categories above to be removed. Anthropic decides which information falls in those categories, and METR has the right to give its opinion on the level of access it received.
- Separate operation: The collaboration is limited to a small team under an enforceable contract, and METR's other staff who work on different projects with Anthropic are kept separate from this team's work. METR is said to have reviewed Anthropic's risk reports several times over the past year.
- Three-tier framework: The first tier is a narrowly scoped check, where Anthropic sets the scope of access and the evaluator's publication requires Anthropic's approval. In the second tier, the evaluator can publish without approval and sees more information, but may receive removal requests for certain categories, and the evaluator can publicly give its opinion on whether that removal was justified. The third tier is METR's model, combining employee-level access with publication rights similar to the second tier. Anthropic said it wants to place multiple evaluators in the third tier over the long term.
- Others: SecureBio is reviewing the chemical and biological weapons chapter of the August risk report, and is said to have more limited information and publication rights than METR. Anthropic is pursuing broader collaboration with Accenture and Faculty covering model evaluation, red teaming, and alignment evaluation, and wrote only that it is in "early discussions" with several organizations including Hugging Face. The scope of discussions or level of access for Hugging Face is not in that post.
How far "no veto" goes (interpretation)
1. A veto and editorial control are different. Being unable to block publication itself is a different matter from being able to have information in categories Anthropic has defined removed. The three categories are relatively clear, but what falls inside them is Anthropic's call. There is, however, a safeguard in that METR can speak publicly about its level of access.
2. The terms are not the same for every evaluator. According to Benzinga and CellCog, SecureBio is in the first tier, and the first tier requires Anthropic's approval for publication. The sentence "evaluators publish whatever they like" holds only for the third tier (and the second tier).
3. There are no results yet. Crypto Briefing pointed out that METR's evaluation results from this access did not yet exist at the time of the announcement. Only when a first report appears and we can see how much was removed will we know how substantive this structure is.
4. We could not confirm who pays or how long it lasts. CellCog pointed out that Anthropic's post lists financial independence from the company being evaluated as a requirement but does not say who pays for the METR team's costs. Whether this collaboration is permanent, whether it applies to future model releases, and whether evaluators have the authority to delay a release could not be found in the official post (Tech-Insider also pointed out the same thing).
Background: the trend in the same week
Benzinga reported that the US FTC sent civil investigative demands to OpenAI and Anthropic in the same week, and summarized that Anthropic promised last month to strengthen external evaluation and signed the "White House compact on superintelligence." Tech-Insider interpreted this structure as connected to such pressure, but that connection and the outlook that "other companies will follow" are that outlet's speculation, not reported fact. It is part of the same trend as the FTC matter covered in the previous article (NO.23), but Anthropic did not say that this announcement was made because of it.
What we could not confirm in this news
- We could not find METR's own account. The METR blog listing only went up to the October 6 post ("AI systems could cover up misbehavior"), and no post explaining this collaboration was visible. So we know METR's position only through Anthropic's post.
- The figures and conditions above all come from Anthropic's own announcement and have not been independently verified. Most of the coverage repeats this announcement. Crypto Briefing described the tier assignments differently from the official post (showing Faculty as second tier), but no such assignment appears anywhere in the official post, so we wrote according to the official post.
- The Stanford HAI research we had been considering as a candidate in the previous check: in this check too, we could not find recent research directly related to this matter. The HAI piece that came up in search was a 2021 proposal on third-party audits, so we did not include it in this article.
Our view
Fact: Anthropic opened employee-level access to 6 METR staff, and said that instead of blocking publication, it can require removal of defined categories. The terms differ by evaluator.
Interpretation: The phrase "outsiders are looking" comes with questions attached: who sets the scope of access, who approves publication, and who pays. This announcement answered some of those questions, and some remain open until the first report comes out.
What we would do: We read the words "safe" and "verified" about the AI tools we use with the same questions. We write down who measured, whether the cases were chosen by the company, and what the conditions were, and for our own work we run it through once more by our own standards. This newsletter likewise discloses the parts where we have a stake at the start of the article, and says "could not be verified" for anything we could not open and check directly.
Articles we read together
We opened and checked sources of different kinds directly. Figures and conditions are as given in each source.
Official company material:
Reporting and analysis:
- benzinga.comAnthropic Gives AI Watchdog Unprecedented Internal Access (Benzinga)English, outlet (notes some use of AI tools)
- cryptobriefing.comAnthropic gives outside AI safety evaluators METR and SecureBio inside access (Crypto Briefing)English, outlet (tier assignments differ from the official post)
- cellcog.aiAnthropic Gives METR Employee-Like Access: What It Means (CellCog)English, company blog (includes promotion of its own AI service)
- tech-insider.orgAnthropic Gives METR 6 Staff AI System Access (Tech-Insider)English, outlet (outlook is the writer's speculation)