Human in the Loop Is Not an Answer

“Human in the loop.” It is the most comforting phrase in AI governance.

It is also the emptiest.

Which human? Allowed to do what? And who answers when it goes wrong?

In operations, nothing critical runs without a named owner. AI decisions should not be the exception.

The numbers show the gap

Deloitte surveyed 3,235 business and IT leaders across 24 countries for its State of AI in the Enterprise report, The Untapped Edge.

66% report productivity or efficiency gains from AI.

20% say AI already grows their revenue.

74% hope it will.

34% are starting to use AI to deeply transform their business.

Now the number that matters. Only about one in five companies, 21%, say they have a mature model to govern autonomous AI agents.

McKinsey sees progress, but from a low base. Its AI Trust Maturity Survey of about 500 organisations puts the average responsible AI maturity at 2.3 on a four-level scale, up from 2.0 a year earlier. McKinsey also added a new dimension to its model: governance and controls for agentic AI.

My reading: capability is scaling. Decision rights are not.

The exact numbers will change. The gap between what AI can do and who answers for it will not.

Policy, risk, human in the loop: necessary, not enough

When companies talk about AI governance, three questions usually come up.

Do we have the right policy? Are the risks classified? Is there a human in the loop?

All three are necessary. None of them is enough.

A policy says what is allowed. It does not say who decides on a Tuesday afternoon when the model flags a customer, and the account manager disagrees.

A risk class ranks the system. But risk does not live in the system. It lives in the decision the system feeds.

And “human in the loop” names a seat. It does not say who sits in it, what they are allowed to do, or what happens when they are on holiday.

So the harder questions are these. Where may AI advise? Where may it recommend? Where may it act? Where must it stop? Who may override it? Who owns the consequence?

Govern the decision, not the tool

Here is the shift I recommend.

Stop governing AI tool by tool. Govern it decision by decision.

The same model can be harmless in one place and dangerous in another. A model that summarises documents is low risk when it drafts meeting notes. It is not low risk when its summary lands on a credit committee table, or in a hiring shortlist.

Same tool. Same vendor. Very different consequence.

So the unit of governance is the decision. Not the model or the platform.

Three ways decision rights blur

Nobody plans to blur authority. It happens in three ways.

Role creep. The AI starts as an adviser. It is right most of the time. People stop reading the reasons and start accepting the answer. Six months later the AI is acting in practice, and nobody ever decided that. The process document says “recommend”. The reality says “act”.

Ownership by committee. The company sets up an AI governance committee. Good. Then the committee becomes the owner of everything AI. A committee can approve a policy. It cannot answer for one rejected customer. When everybody owns it, nobody does.

Defaults decide. Somebody has to set the thresholds. What score triggers a review? What amount needs a second pair of eyes? If the business does not set them, someone else will. A default in the software, or a developer working to a deadline. Neither will answer to your customer or your regulator.

All three have the same cure. Write the decision rights down, per decision, before the AI goes live. And look at them again when things change.

The Decision Rights Card

For every decision AI touches, write one card. Five lines. Plain words.

1. Role. Advise, recommend or act? Advise means it gives information. Recommend means it proposes an answer, and a person decides. Act means it executes, inside limits you set.

2. Evidence. What must it show first? The main reasons in plain words. The data it used. A check that the inputs are complete.

3. Stop. When does it hand back to a person? Missing data. A case outside the agreed limits. Low confidence. A complaint. Anything unusual.

4. Override. Who can overrule it? A named role, not “the business”. And every override is logged with a one-line reason.

5. Owner. Who owns the consequence? One name. Not “the AI team”. Not “the vendor”. The person who owns the business outcome.

One rule sits on top of the card. No card, no AI on that decision.

Copy the card below. Use it in your next AI review.

Decision: ______

Role: advise/recommend / act

Evidence: ______

Stop: ______

Override: ______ (named role)

Owner: ______ (one name)

What a card looks like

Take a simple example. It is an illustration, not a client case.

A lender uses AI to assess small business loan applications.

Role: recommend. AI never rejects an application on its own.

Evidence: the top three reasons in plain words, the data sources used, and a check that the financial statements are complete.

Stop: missing financials, an application outside the policy limits, or a score close to the threshold. These go straight to an analyst.

Override: the credit analyst can overrule the recommendation, with a one-line reason in the log.

Owner: the head of small business lending. Not the data science team.

Five lines. Ten minutes in a meeting. Nobody argues later about who was supposed to check.

The model, the vendor, and the platform are not on the card. Switch technology and the card stays. Decision rights should outlive any tool you buy.

Watch the override rate

The override line is the one to watch.

If people overrule the AI all the time, the AI is not ready, or the stop rules are wrong.

If nobody ever overrules it, be careful. Either the AI is perfect, or nobody is really checking anymore. Both deserve a look.

So track the override rate for every card. It tells you whether the human in the loop is really deciding, or just clicking.

The law points the same way

In Europe, the AI Act asks that people who oversee high-risk AI systems can disregard, override, or reverse the output, and can stop the system with a stop button or a similar procedure.

Regulation sets the outer fence. It does not tell you who in your company holds the stop button for each decision. That is your job. The card is where you write it down.

The ops lesson

I spent more than 25 years in enterprise IT operations and automation, and helped lead an automation estate of more than 3,000 bots.

The rule in operations is simple: nothing critical runs without a named owner and a clear escalation path. When something breaks at night, “the team” does not answer the phone. One person does.

AI is not different. It is the same discipline, applied to decisions instead of servers.

Start this week

You do not need a new platform. You need a spreadsheet and one meeting per process.

List the decisions AI touches today. Decisions, not tools.

Write one card per decision. Five lines.

Put one name on Owner. A person, not a team.

Test the stop. Make sure the hand-back really works before you need it.

Review the card every time the AI’s role changes, for example from recommend to act.

Start with the decisions that touch money, people or customers. That is where blurred authority costs the most.

The bottom line

In the next phase of AI, the test is simple. For every decision it touches, can you say who decided, on what evidence, and who answers for the result?

A human in the loop is a seat. Name who sits in it, and what they may do.

Which decision in your company has AI on it today, and no name on Owner?

Ship AI. Deliver outcomes. Keep control.

OG