The road ahead: What evidence would justify more autonomy?

Imagine a release review where the team asks to remove an approval step. The assistant has handled its recent tasks well, the queue is growing, and reviewers say they rarely reject a proposal. That is a reasonable request to investigate. It is not yet evidence that removing review will preserve the outcomes that matter.

A low rejection rate has several possible explanations. The proposals may be good. The queue may contain only easy cases. Reviewers may be missing errors. Or earlier controls may be doing most of the work. The decision depends on which explanation is true.

I want the next phase of production AI work to make those distinctions measurable. This is an evidence agenda, not a prediction that autonomy will succeed or fail on a particular timetable.

What did the human actually contribute?

An approval record shows that someone clicked. It does not tell us whether the review changed the outcome. To assess a proposed removal, I would first examine a sample of accepted, rejected, and escalated cases. What evidence did the reviewer add? Which mistakes were caught elsewhere? Which decisions remained wrong after approval?

The difficult part is the counterfactual. A prevented action has no observed production outcome. Calling every rejection an “incident avoided” inflates the benefit of review. Calling it wasted effort because no incident occurred makes the opposite mistake.

Replay and simulation can help investigate rejected proposals without executing them against live systems. They also have limits: the reconstructed environment may omit the condition that made the original action dangerous. Report the result as an assessment under stated assumptions, not a counted production save.

For lower-risk cases, a controlled comparison may be appropriate after responsible owners approve its scope. Keep downstream protections intact and specify stopping criteria before the trial. High-consequence decisions may require other forms of assurance; a live experiment is not automatically justified because it would produce cleaner data.

The standard should be symmetric. An expensive approval process should have to demonstrate its contribution too. “A human is involved” is no more a complete safety argument than “the model passed”.

What would count as a failure?

Before the trial, name the outcome being measured. Tool completion, correct state change, durable recovery, and absence of collateral effects are different observations. A system can succeed at the first and fail at the others.

Define the eligible task population and the observation window. Include cases that time out, escalate, or lack enough telemetry to judge. If the assistant only receives familiar requests, a strong result should be described as performance on familiar requests. It does not license claims about everything routed elsewhere.

Rare severe failures create another difficulty. A short pilot with no observed catastrophe cannot establish that catastrophe is acceptably unlikely. The evidence package needs exposure counts and a discussion of what the trial could not reveal. Task outcomes may also be correlated, so a large count is not necessarily a large set of independent tests.

Google's canarying guidance is useful here because it treats rollout as an evaluated, limited exposure rather than a single release event.[1] But canaries do not answer every AI risk question. Some failures require unusual inputs, longer sequences, or dependencies absent from the initial group. Expanding the rollout should therefore change what is observed, not just how much traffic is admitted.

Can people still take over?

An escalation policy assumes that someone can do something useful when the assistant stops. That assumption deserves a test.

In a rehearsal, remove the assistant's recommendation while retaining the evidence and ordinary operational tools. Can the responder reconstruct the state, identify unresolved questions, and choose a defensible next action? Repeat with unfamiliar synthetic cases and record the limits of the exercise.

This does not require believing that AI inevitably erodes skills. The knowledge-worker survey by Lee and colleagues reports associations between confidence and self-reported critical thinking, not a causal demonstration of engineering skill loss.[2] The operational reason to test takeover is simpler: we are relying on that capability.

Takeover also has a workload dimension. A responder may handle one escalation well and fail when many arrive together. Rehearse unavailable staff and missing telemetry, not just the ideal handoff. Do not remove an approval queue by creating an exception queue no one can service.

Who may accept the remaining risk?

Even good evidence leaves a decision. Faster handling may come with more ambiguous cases, extra monitoring work, or a different distribution of harm. The team receiving the productivity benefit may not be the team or customer bearing that risk.

The NIST AI Risk Management Framework offers a voluntary structure for managing AI risks throughout a system's lifecycle.[3] It is a useful starting point for organising the work, not a certificate that an agent is safe and not a substitute for applicable legal obligations.

For the proposed approval removal, I would require a named decision owner, an account of who may be affected, and a record of the unresolved risks. Relevant operational, security, and legal specialists should assess the parts within their remit. The release team should not infer permission from silence.

Evidence also has a version. A decision based on one model and one tool set should name the changes that require reconsideration. An added write permission can invalidate an old assurance argument even if the model itself is unchanged.

Write the condition for saying no

The most useful addition to a release proposal may be a sentence describing what would stop it. For the hypothetical approval change, that could be an unexplained state mutation, failed outcome verification, or an escalation load beyond the team's agreed capacity. The thresholds need to be chosen for that service, not copied from a generic maturity model.

Make the fallback executable. Identify who can suspend the action path, how in-flight work is reconciled, and what happens if the owner is unavailable. Test that procedure before relying on it.

Then ask the release owner to finish this sentence: “We would restore the approval step if we observed...” If the team cannot name an observation that would change its decision, it is not yet running an evidence-led trial.