AI’s quiet safety gatekeepers are stepping into the spotlight

Independent AI safety evaluators, mostly small nonprofits like METR, Apollo Research, and Transluce, have moved from an overlooked niche to a central role amid fierce debate over whether leading labs can safeguard advanced models while growing rapidly. Anthropic CEO Dario Amodei pledged to embed independent evaluators last month, a move OpenAI CEO Sam Altman quickly endorsed and President Donald Trump supported. However, major questions remain about funding, access, and reporting structures, with experts warning that a viable business model is needed and that financial dependence on AI labs could undermine true independence.

OpenAI recently fired three employees for policy violations related to sensitive information access; two of them believe the dismissals were tied to their communication with third-party evaluators, while OpenAI disputes this and says it is finalizing contracts with safety assessors. The evaluator ecosystem is maturing quickly, attracting new capital, but remains small compared with the leading labs. Without federal regulation, evaluators have taken on outsized importance, yet their power imbalance with well-funded AI companies raises conflict-of-interest concerns.

Policy efforts are advancing: Fathom proposed a marketplace framework for Independent Verification Organizations, and the FRONTIER Act includes IVO provisions. California Governor Gavin Newsom signed two related bills, with support from Anthropic and OpenAI. Trump’s recent voluntary “morally binding” accord encourages companies to work with independent auditors, though critics call it performative. Ultimately, the central challenge is whether embedded evaluators can be truly independent and effective while AI labs compete toward public markets and trillion-dollar valuations.

(Source：CNBC)