Governance for AI agents is an architecture problem, not a prompt

· Dhiraj Kumar Sah

Most of the AI agents I see deployed in businesses are governed the same way: a long system prompt that says what the agent may and may not do, a few keyword filters on the way in and out, and a person somewhere who is supposed to be watching. This works until it doesn't. The failure is rarely dramatic. It is a quiet drift: the agent is asked something slightly outside its brief, it reasons its way to a helpful-sounding answer, and the rule in the prompt is simply not the strongest signal in the room any more.

I don't say this as a critic from the outside. I have built agents this way myself, and for a demo or an internal tool it is often fine. The problem starts when the agent is allowed to act — call tools, move records, send messages, spend money — on behalf of an organisation that is accountable for what it does. At that point "we told it not to" is not a control. It is a hope.

Where the rules live matters more than what they say

A prompt is an input to the model. The model weighs it against everything else it is given, including the user's message, the documents it retrieves, and the output of the tools it calls. Any of those can contain text that argues with the prompt, and the model has no principled way of knowing which voice is the one it is supposed to obey. Prompt injection is just the sharp edge of this; the dull edge is ordinary ambiguity.

Contrast that with how we govern every other piece of software that acts on our behalf. A database user has permissions that the query cannot talk its way around. A process has a sandbox. A payment has a limit that is enforced by a system the requester does not control. None of these are perfect, but they share a property that prompts do not: the policy is enforced by a layer that sits underneath the thing being governed, and that layer does not reason, it checks.

This is the thesis we are working on at Zerava Technologies Inc.: that governance for AI agents has to be a property of the runtime they execute in, not an instruction they are given. Boundaries on what an agent can touch, a record of what it did and why, and a way to stop it that does not depend on asking nicely. The research behind our approach is my co-founder Andrej Ristić's; my part is turning it into something an enterprise can actually deploy, on their own infrastructure, with models they control.

What this looks like in practice

Three things change when you move governance out of the prompt and into the runtime.

  • The model becomes replaceable. If the controls are in the runtime, you can swap the model underneath for a smaller, local, or newer one without re-deriving your safety story from scratch. For regulated organisations that cannot send data to a third-party API, this is not a nice-to-have; it is the whole question.
  • Audit becomes a by-product, not a project. When every action passes through a layer that checks it, you get a log of every action and every check for free. Compliance teams stop asking the engineers to reconstruct what happened.
  • Failure becomes boring. An agent that tries something outside its boundary is stopped, logged, and reported. Nobody has to read a transcript to find out whether a rule was followed.

The honest caveat

None of this makes an agent wise. A runtime boundary stops an agent from doing what it is not permitted to do; it does not make the agent good at what it is permitted to do. You still need the model to be competent, the task to be well-specified, and a human to own the outcome. What architecture-level governance buys you is the ability to say, with evidence, what the agent could and could not have done. In most of the conversations I have with businesses, that is the thing they actually need before they will let an agent anywhere near real work.

I'll write more about how we think about this as we build. If you are deploying agents inside an organisation that answers to a regulator, I would like to hear what has and hasn't worked for you.