For decades, when traditional software systems were deployed across the economy, we had the tools to trace behavior back to a specific code path. That kind of mechanistic understanding is hard to come by in today’s superintelligent systems, even though the frontier models driving them are now more powerful than traditional software ever was.

We cannot attribute what a model does or says to any particular input in its training data, or to any particular configuration of its weights. And yet we are deploying these complex agentic systems — giving them access to our most sensitive data, and letting them take critical actions on our behalf.

That is why it is time to step back and assess the trust architecture of this new era. We simply cannot outsource responsibility for what intelligence does on our behalf. A model provider’s assurances do not relieve us of that responsibility.

So how do we build systems we can actually constrain? Nadella’s answer starts by treating frontier models — both closed and open-weight — as internal risk. Not because they are necessarily malicious, but because any actor that is capable enough and has access to important systems can make mistakes or be compromised. The architecture for constraining and controlling has to account for that.

None of this is new. We have spent decades learning how to handle powerful insiders inside an enterprise: establishing identity, limiting permissions, logging activity, creating containment boundaries. Those are best practices, and they have been refined over decades. We are only now beginning to apply the same principles to superintelligence inside the enterprise.

Transparency has to start with uncompromising chain-of-thought visibility. “Neuralese” — a model’s own internal phrasing — is not an acceptable excuse for opaque reasoning. But chain-of-thought transparency alone is not enough, and it is not reliable, because we still do not know how to make a model’s output itself consistently faithful or transparent.

You can, and should, use models to test and validate each other adversarially. But do that too much and you can end up with an opaque model, inside an opaque orchestration layer, watched by yet another opaque model. That is still a stack of nested black boxes.

That is why the controls that determine what a model can access and what actions it can take must live outside the model. This follows an information security principle that traces back to the 1970s: a program must not be able to bypass or tamper with the mechanism that enforces its permissions.

Practically, that means separating the model from the runtime framework that orchestrates it, and from the action space that defines what it is allowed to do. It also means externalizing controls and safeguards. Put it another way: separate the supply of intelligence from the authority given to that intelligence.

Setting aside the hard problem of alignment, constraint and governance need to be handled as an engineering discipline. The idea is to surround non-deterministic models with strongly deterministic system design, human control, and reliable operating procedures — and to establish industry standards where existing ones fall short.

The systems should be designed around observability.

Model diversity: no single model should be the sole dependency for an important outcome, and no model should be responsible for validating its own work.

Observe everything: every meaningful model action must leave tamper-evident, human-readable evidence. If it can’t be observed, it can’t be trusted — and the record must let you reproduce how a result was reached, without relying on the model to prove it.

Verifiability: you need to test the entire system continuously — failures, attacks, edge cases, system changes — not just the tasks that succeed.

Independent control: the organization should be able to decide, on its own, what a model can access and what actions it can take.

Independent auditability: verification must be independent of the intelligence being verified. A single model should not be able to both control system behavior and produce the evidence needed to judge whether that behavior matches the original intent.

Constraints: assume from the start that the model will be compromised, and constrain it accordingly. Think of it as an emergency brake — an authorized person should always be able to pause or shut the model down mid-task. More advanced models will require more advanced constraint techniques, and those techniques need to be standardized.

Incident disclosure: when these systems do fail or are compromised, disclose it promptly to the people affected. Establish a mechanism for sharing what went wrong, which controls failed, how to prevent it from happening again, and to circulate those lessons across the industry. That should include implementation details that changed agent behavior at runtime.

The result is a simple idea. The most trustworthy superintelligent system will not be the one built around the model we trust most. It will be the one that lets us trust the model the least.