'Human in the Loop' Is a Nice Way of Saying You Still Fix It at 3 A.M.

Blog · AIOps

'Human in the Loop' Is a Nice Way of Saying You Still Fix It at 3 A.M.

By Mohak Soni9 min read

Short answer

The category sells human-in-the-loop as prudence — but it is an admission the platform can't be trusted to act, because its execution is ungoverned and lives outside the tool.

It is 3 a.m. Your incident stack has already done the impressive part — it caught an anomaly a static threshold would have slept through, collapsed the alert storm into one signal, and handed you a cited root-cause analysis before you found your laptop. Then it stopped, and paged you, because the last step — actually changing the thing that is broken — is yours. "Human in the loop" sounds like a principle. At 3 a.m. it is a job description.

Be precise about what today's incident tooling gets right, because a serious operator will stop reading the moment you are unfair to it. ML anomaly detection genuinely catches issues static thresholds miss — seasonality-aware baselines flag the slow drift no fixed number would. Event correlation genuinely collapses an alert storm into a single signal, real relief for a pager that used to fire dozens of times an hour. Automated causal RCA genuinely compresses investigation from tens of minutes to a few, and the newest autonomous investigation agents genuinely produce audit-ready, cited root-cause write-ups faster than a rested human could. Pre-authored runbooks genuinely automate the known fixes. None of that is marketing. It is real engineering, and it is good.

Notice, though, where every one of these products draws its line — and how proudly it draws it. One intelligent-observability vendor ships an assistive SRE agent and states plainly that it does not make changes to production. An incident-response vendor builds its whole thesis around the idea that the next wave of automation makes more room for people, not less. An agentic-ITOps vendor reassures buyers that it never removes the human from the loop. Read together, these are not three coincidences. They are the same architectural confession, said three ways: the platform will investigate, recommend, and draft — but when it is time to act on production, it hands you the controls.

So "human in the loop" gets sold as prudence — as respect for human judgment, as safety, as maturity. Sometimes it is. But ask the uncomfortable question: why won't the platform push the button? Not because acting is inherently reckless — a health-check-gated, reversible change is less reckless than a sleep-deprived human doing it by hand. The platform won't act because it structurally cannot act safely. Its restraint is not a virtue it chose. It is a ceiling it hit, and then marketed as a feature.

The reason is architectural, and it is the same reason across the category. Detection, correlation, and RCA all happen inside the observability platform, where the telemetry and the evidence live. The fix does not. The corrective action executes in a different system — a runbook engine, a separate automation tool, a cloud provider's API, a change ticket in the service queue. Between the diagnosis and the fix runs a tool boundary, and the platform's authority ends at that boundary.

That boundary is exactly where inline change-control and rollback should be, and exactly where they are not. When the platform dispatches a corrective action across it, the action lands in a system that does not hold the incident's evidence and does not share the platform's model of what "healthy" means. There is no gate on the far side that makes "this change class needs approval" part of the action itself; no health check wired to the same signals that detected the fault; no one-motion revert that returns the system to its prior state if the fix makes things worse. So the only safe place to put change-control and rollback is a person. That is the loop. The human is not supervising the automation — the human is the missing governance layer, hand-carrying the change through the seam the architecture left open. At 3 a.m.

This is why "augment, not replace" is both honest and a dead end. On an architecture where the fix runs across a boundary, augmentation is the most you can safely offer. You cannot govern what you hand off. You can only advise it.

THE FIX RUNS ACROSS A BOUNDARY diagnose here · execute over there · a human bridges the seam DETECT CORRELATE AUTO-RCA observability platform hands off the fix tool boundary RUNBOOK AUTOMATION CLOUD API TICKET QUEUE the human is the change-control layer no inline change-control · no rollback THE LOOP CLOSES IN ONE RUNTIME detect → RCA → governed remediation, rollback inline DETECT → AUTO-RCA DETERMINISTIC WORKFLOW change-control gate · HITL exception GOVERNED CONNECTOR live production state health-check- gated rollback every action → immutable per-action record · no boundary

Put it in the language of an operating model, because that is what actually changes. On the hand-off architecture, the on-call engineer is Responsible in the RACI sense — the executor. The tool is Consulted; it advises. The human runs the corrective action, watches the graphs, and manually reverts if it goes sideways. Every incident routes the same accountability to the same exhausted person. The tooling got smarter at the top of the funnel and left the riskiest, most time-pressured step — the change itself — exactly where it was.

When governance and rollback are native to the action, the human's role moves up the RACI ladder from Responsible to Accountable. The engineer stops being the executor and becomes the approver: setting policy up front — which classes of remediation execute automatically, which pause for a human, who is allowed to approve what — approving or denying an exception in the moment, and reviewing the immutable record after. The human is still in the loop, by design, on the decisions that need judgment. But not on the mechanical execution of every known fix. That is a different job, and critically it is a daytime job. Policy is set when you are rested. Exceptions are rare by construction. Review happens after the incident, not during it.

This is what the alert-noise arms race keeps missing. The category's answer to on-call burnout is fewer alerts — better correlation, smarter suppression, higher-signal pages. That helps, and the improvement is real. But it treats the symptom. The disease is that when a page does fire, a human still has to execute the fix under pressure, and no amount of noise reduction changes what happens after the human wakes up. Studies of on-call rotations keep surfacing the same pattern — chronic sleep disruption, and the attrition that follows it — reported by kind, not just by volume. The durable cure for on-call toil is not fewer alerts to humans. It is fewer incidents that require a human at all — and, for the ones that do, a human who approves rather than executes.

Entroid is built so the fix does not cross a boundary — because there isn't one. It models, executes, and governs every process as a composition of five primitives — Deterministic Workflows (with change-control and rollback inline), Intelligence Orchestration, Atomic Agents (human-in-the-loop as a first-class control), Functions, and Connectors (the only primitive licensed to touch external systems) — on a single Semantic Ontology, in one runtime. Its incident capabilities — Sentinel for AI-driven investigation and remediation, Sherlock for automated RCA and corrective-procedure execution, Command Center for closed-loop automation — sit on that fabric, not beside it.

So the loop closes where the evidence lives. An incident is detected, investigated to an automated root cause, and then remediated by a Deterministic Workflow that executes the corrective action through a governed Connector — with the change-control gate and a health-check-gated rollback inline, as steps in the same workflow, in the same runtime that holds the telemetry and the RCA. This is an architectural property of where execution sits, not a benchmarked outcome: because the gate and the revert are part of the action rather than a human bolted on across a seam, the platform can be trusted to act — which is precisely the trust the hand-off architecture cannot extend to itself.

That is what turns human-in-the-loop into an upgrade instead of an alibi. Because Atomic Agents treat HITL as a first-class, governed control, the approval is exception-based, not mandatory-manual: policy decides up front which remediations run unattended and which pause for a named approver, and the pause is a real gate with the authority to stop the action — not a notification you acknowledge after the fact. And because the action is explainable by design — every step, input, and gate carried in an immutable per-action record — the approver can actually trust and audit what they are approving. The human moves from running the runbook to governing the policy. Executor to approver, by construction.

Illustratively — and only illustratively, not a delivered result — picture a memory leak degrading a checkout service. Detection flags the anomaly against a learned baseline; automated RCA isolates the leaking dependency; a Deterministic Workflow proposes the recycle-and-rehydrate procedure. If policy has pre-authorized that remediation class, it executes through a Connector, health-checks against the same signals that detected the fault, and rolls itself back inline if the checkout error rate does not recover — with no one woken. If policy requires approval, one named approver gets an explainable request, and either way the whole sequence lands in the immutable record. Nobody hand-carried a change through a boundary at 3 a.m., because there was no boundary and nothing to hand-carry.

Three honest caveats, because a technical reader will supply them if you don't.

  • This is not a claim that ES needs no integration. It reaches your cloud, your clusters, your service-management and delivery systems through governed Connectors — the one primitive licensed to touch the outside world, with authentication, authorization, and audit. The difference from the hand-off model is not integration versus none. It is where change-control lives: inline in the workflow that executes the fix, versus absent at the boundary the fix is thrown across.
  • This is not a claim that the category's agents can't grow more autonomous. Some already dispatch corrective actions; the roadmaps push further. The line is structural and holds regardless of roadmap: dispatching an action to a system the platform does not own, or proposing a fix a human runs elsewhere, is a hand-off across a tool boundary. Remediation executed under inline change-control and rollback, in the runtime that holds the evidence, is not. A faster, smarter hand-off is still a hand-off.
  • Keeping a human in the loop is not the error. On an architecture where execution is ungoverned and lives outside the tool, it is the only responsible design, and the vendors who insist on it are right to. The error is calling a structural ceiling a strategic virtue. ES does not remove the human either — it promotes them, from the person who fixes it at 3 a.m. to the person who decided, in daylight, how it gets fixed.
"Human in the loop" is the right answer to the wrong architecture. Fix the architecture, and the human stops being the one who runs the fix at 3 a.m. and becomes the one who decided, in daylight, how it runs.

See what this looks like for your enterprise.

Not a demo. A strategic conversation about how your enterprise could operate
when every process runs on one governed fabric.

Start the Conversation