A release goes out. Requests begin failing. An assistant recommends rolling back. Before anyone acts, the response system has to answer a less glamorous question: which exact deployment is it allowed to change?
The example in this article is entirely hypothetical. It describes a fictional document-preview service and a proposed incident workflow, not an employer's infrastructure or a production deployment. There are no operational results behind it. Its purpose is to expose the decisions an architecture must make before an assistant's plausible recommendation becomes an authorised change.
Two public SRE sources provide the starting point. Google's incident-management chapter separates command, operations, and communication responsibilities, with a living incident record.[1] Its canarying chapter describes evaluating a partial, time-limited release before expanding it.[2] The AI-specific workflow below is my proposal, not a design those sources validate.
Open an incident, not a conversation with unlimited tools
In the fictional service, a new renderer is deployed to a canary group. Monitoring detects failed preview requests and opens an incident record. The record identifies the affected service, deployment version, observation window, and the source of the alert. It does not assert that the release caused the failures.
The assistant initially has read-only access to selected telemetry and release metadata. It cannot fetch arbitrary customer documents, change alert thresholds, or obtain broader credentials by asking another tool. Its evidence collector strips unnecessary sensitive fields and records where each observation came from.
An incident coordinator assigns ownership before diagnosis proceeds. Duplicate alerts attach to the existing record rather than starting competing remediation sessions. A responder can take over without depending on the assistant remaining available.
That last detail is easy to postpone. If the model service fails during the incident, responders still need their normal monitoring and change tools. The assistant should not become the only route to the system it is helping repair.
Keep the diagnosis separate from the observation
The assistant compares the canary with the unchanged group, checks the release timeline, and examines dependency health. It proposes that the renderer change explains the failures. The incident record stores that as a hypothesis alongside supporting and conflicting observations.
A canary comparison is useful but imperfect. Different requests may reach different groups. A shared dependency could be failing, or a telemetry delay could distort the apparent sequence. Google makes the limits of testing and the importance of canary evaluation explicit; a small rollout is not automatically a representative experiment.[2]
Here, the assistant proposes reverting only the canary to the previous renderer version. The proposal includes the exact current and target versions, the deployment identity, relevant evidence, and the expected recovery signal. It also states what it has not established.
If the evidence does not distinguish a renderer problem from a shared dependency failure, the workflow routes to a responder. It does not convert the assistant's uncertainty into a larger action scope.
Authorise one change against current state
A policy service outside the model checks whether the proposed operation exists in the approved action catalogue. For this example, it permits a rollback proposal only if the previous version is available and compatible, the change requires no data migration, and the target remains the identified canary. Missing evidence blocks the proposal.
The responder sees the concrete diff and the evidence, not just “approve remediation”. Approval binds a single operation to that deployment and those versions, with an expiry and a unique execution identifier. It cannot be reused for a wider rollback.
The executor checks authorisation and state again when it acts. If another responder has already changed the deployment, the proposal is stale and must be reconsidered. The deployment service must atomically check the expected version when applying the change. If coordination uses a lease, the service must also reject writes from expired lease holders. A check performed only by the assistant leaves a race between checking and writing.
Human approval does not bypass these checks. The responder could misunderstand the incident, and the state could change after a correct review. Policy and execution constraints still apply.
Suppose the executor times out. The workflow queries the recorded operation before attempting anything else. A timeout is an unknown outcome, not evidence that nothing happened. If the execution result cannot be recovered, automation pauses for reconciliation rather than sending the same mutation again.
Observe recovery without letting the assistant declare it
Assume the rollback completes in this fictional scenario. The executor records the resulting version, but the incident stays open. An independent observer checks preview requests, the affected group's health, and relevant dependency signals against recovery criteria selected before execution.
There is no universal observation period. The service owner must choose one that covers the service's workload and likely recurrence patterns. A quiet interval with no relevant requests cannot demonstrate recovery. The record should distinguish insufficient traffic from healthy observed traffic.
If failures recur, or telemetry is unavailable, the action remains completed but resolution remains unverified. The assistant cannot close the incident by generating a more confident summary. The owner decides the next response, with further writes suspended on the disputed path.
If recovery criteria are met, the responder can record mitigation while leaving root-cause work open. Recovery after rollback supports the release hypothesis, but it does not prove the full causal explanation. A post-incident investigation still needs to account for conflicting evidence.
The record links the original alert, evidence timestamps, proposal, approval, execution receipt, and later observations. Access and retention rules apply to that record. “Keep everything forever” would introduce a different operational and privacy problem.
What a pilot would have to demonstrate
Before production use, I would exercise this design against synthetic incidents with stale approvals, duplicate events, unavailable telemetry, and executor timeouts. Test the handoff to a responder as well as the successful path. A system that pauses correctly but loses its evidence at handoff is not ready.
A subsequent limited pilot would need separate measures for proposed actions, rejected actions, completed changes, and independently verified recoveries. State the eligible incident population and observation period. Count escalations and unresolved cases rather than quietly dropping them from the denominator.
Responder effort and customer impact belong beside those counts. So do delayed recurrences and actions that made the situation worse. This design supplies no evidence of faster resolution or lower cost; those would be findings to establish, not benefits to claim in advance.
The most revealing rehearsal may be the one where the assistant disappears halfway through. Can the next responder tell what was authorised, what actually happened, and what remains unknown? I would want that answer before giving the system its first write permission.