AI Auditors Need the Right to Publish, Not Just the Right to Look

<p><strong>Opinion:</strong> Independent AI evaluation is becoming fashionable, but the label will mean very little unless evaluators can both see the evidence and tell the public what they found. Access without publication rights produces informed advisers. Publication rights without meaningful access produce outside commentators. Credible oversight requires both.</p><p>That is why <a href="https://www.anthropic.com/news/an-update-on-independent-evaluators" target="_blank" rel="noopener noreferrer">Anthropic’s October 15 announcement</a> deserves attention beyond the company itself. Anthropic says it has given six employees from the nonprofit evaluator METR company hardware, office space and access to internal tools roughly comparable to its own risk teams, subject to exclusions for categories such as customer data, privileged information and model architecture. More important, Anthropic says it cannot veto METR’s publication. The company can require narrow, predefined redactions, but METR may publicly characterize those redactions and say whether it believes they were justified.</p><p>Those are company-reported terms, not proof that the arrangement works. METR has not yet published the results of this embedded engagement. Still, the design identifies the right governance problem: an evaluator’s independence is determined by its rights and incentives, not by the word “independent” in a press release.</p><h2>Independence has four parts</h2><p>A serious evaluation arrangement needs at least four things.</p><p><strong>First, relevant access.</strong> A polished model demo or a fixed benchmark is not enough. Evaluators may need training and deployment context, incident records, safeguard design, internal risk decisions and the ability to follow questions that emerge during the work. This is related to the problem described in <a href="/article/ai-agent-benchmarks-sandboxes-live-websites">my recent analysis of AI-agent benchmarks</a>: the test environment shapes what the result actually proves.</p><p><strong>Second, freedom to form conclusions.</strong> A vendor should be able to correct factual errors and protect legitimate secrets, but it should not decide whether an uncomfortable conclusion is publishable. If approval is required, the work is closer to consulting than public oversight.</p><p><strong>Third, a visible redaction process.</strong> Some confidentiality limits are necessary. Customer data, security details and privileged legal material should not be exposed merely to make an audit look tougher. But the evaluator should be able to disclose that a redaction occurred, describe its category and state whether the restriction impeded the assessment. Otherwise, readers cannot distinguish a clean result from a constrained one.</p><p><strong>Fourth, institutional independence.</strong> Contract language cannot solve every conflict. Who funds the evaluator? Does one developer supply most of its budget? Can the evaluator work for competing labs? What happens if the company ends the engagement after a critical report? Anthropic itself acknowledges that common standards and a settled funding model do not yet exist. Pooled industry funding, public funding or diversified fees could reduce dependence, but each model creates its own governance questions.</p><h2>Canada should put these rights into the standard</h2><p>The timing matters for Canada. On October 13, the governments of <a href="https://www.canada.ca/en/innovation-science-economic-development/news/2026/10/canada-and-the-united-kingdom-renew-their-partnership-on-artificial-intelligence-security.html" target="_blank" rel="noopener noreferrer">Canada and the United Kingdom renewed their AI security partnership</a>. Their statement prioritizes interoperable security standards, exchanges on evaluation methods, staff exchanges and coordinated evaluations between the Canadian and UK institutes.</p><p>That is useful infrastructure, but the public statement does not define what independent evaluators may publish. Canada and the UK should make publication rights part of any evaluation standard they develop. A common template could require disclosure of scope, access level, funding, excluded evidence, redaction categories, publication approval rights and whether the evaluator can report disagreements. That would let businesses, regulators and the public compare evaluations without pretending every “third-party review” is equivalent.</p><p>This would also help buyers. The practical question for a Canadian company is not simply, “Was this model independently evaluated?” Procurement teams should ask: Who selected and paid the evaluator? What systems and records could it inspect? Could it choose its own tests? Could it publish without vendor approval? Were findings tied to a specific model version and deployment context? What triggers a new review?</p><p>Those questions complement the architecture and data-flow checks in <a href="/article/canada-ai-vendor-privacy-guidance-architecture-checklist">my earlier analysis of Canadian AI vendor selection</a>. Safety claims, privacy claims and operational controls all weaken when evidence cannot move beyond the vendor’s boundary.</p><h2>What would change my assessment</h2><p>I would become more confident in embedded evaluation when METR publishes substantive findings from this arrangement, including limitations and material disagreements; when redaction disputes are reported consistently; and when similar rights are available across multiple frontier developers. I would become less confident if publication is delayed indefinitely, access exclusions expand, funding dependence goes undisclosed or critical evaluators lose access without explanation.</p><p>Anthropic’s proposal is therefore promising, but it is a test of governance rather than a completed success. The durable lesson is simple: an evaluator needs a protected path from evidence to public judgment. If AI oversight stops at the company door—or at the company’s approval desk—it is not yet independent.</p><h2>Sources</h2><ul><li><a href="https://www.anthropic.com/news/an-update-on-independent-evaluators" target="_blank" rel="noopener noreferrer">Anthropic — An update on independent evaluators</a> (October 15, 2026)</li><li><a href="https://metr.org/risk-assessment/" target="_blank" rel="noopener noreferrer">METR — Reviews of AI developers’ risk assessments</a> (accessed October 16, 2026)</li><li><a href="https://www.canada.ca/en/innovation-science-economic-development/news/2026/10/canada-and-the-united-kingdom-renew-their-partnership-on-artificial-intelligence-security.html" target="_blank" rel="noopener noreferrer">Innovation, Science and Economic Development Canada — Canada and the United Kingdom Renew Their Partnership on Artificial Intelligence Security</a> (October 13, 2026)</li></ul><p><em>Disclosure: Jason Ansell is the author of The AI Money Revolution and a co-founder of Vector Smart Chain. Neither interest is evaluated in this article, and this opinion makes no investment recommendation.</em></p>