Human-in-the-Loop Is Not a Control: The Case for Human Resilience Engineering

We have developed an oddly comforting habit in AI governance. Whenever we encounter a problem we cannot quite solve technically, legally, or organizationally, we put a human in the loop.

The model might hallucinate, so a human should check it. The agent can take consequential action, so a human should approve it. The reasoning is opaque, so a human should interpret it. Accountability is murky, so a human should own the decision. There is residual risk somewhere in the process? Excellent. Put a human there too.

At some point, human-in-the-loop stops describing a thoughtfully designed control and starts describing the place we put everything we have not yet figured out.

There is a famous I Love Lucy scene in which Lucy and Ethel are working at a chocolate factory and the conveyor belt keeps getting faster. At first, the job is perfectly reasonable: inspect the chocolates, wrap them, keep the line moving. Then throughput starts exceeding human capacity and the entire control system degenerates spectacularly. Chocolates go into mouths, hats and uniforms—anywhere that keeps them from visibly piling up while the conveyor continues cheerfully on its way.

A great deal of enterprise AI oversight is in danger of becoming the digital version of that conveyor belt.

The human is technically still there. The approval may even be logged. But if we keep increasing the speed, volume, complexity and opacity of the work while handing every unresolved exception to the same finite human attention system, we should not be terribly surprised when “meaningful oversight” eventually becomes a very expensive way of recording who clicked Approve.

This is one of the reasons we believe Human Risk Management has to evolve toward Human Resilience Engineering: deliberately designing the human and organizational capacity required to keep complex digital and AI-enabled systems working safely when the world does not behave exactly as the process diagram expected.

The longer-term opportunity is something broader still: Human Resilience Systems that connect that capacity to technology, governance, culture, risk, work design and learning rather than leaving resilience to individual heroics.

HITL is useful. The problem is what we are asking it to carry.

There is nothing inherently wrong with human-in-the-loop design. Some AI systems absolutely should require human judgment before consequential actions are taken, and serious governance frameworks are considerably more nuanced about this than the casual shorthand sometimes suggests.

The EU AI Act, for example, does not simply say “insert person here.” Its requirements for human oversight of high-risk AI include enabling the people responsible for oversight to understand relevant system capabilities and limitations, remain alert to over-reliance, interpret outputs, disregard or override them where appropriate, and intervene in or stop the system. Deployers are also expected to assign oversight to people with the necessary competence, training, authority and support.

NIST approaches the problem from a similar direction. Its AI Risk Management Framework calls for organizations to define and differentiate roles in Human-AI configurations, establish relevant proficiency and assess the mechanisms used for human oversight. Its human-centered work also makes an important point that governance diagrams sometimes glide past: simply assigning a human expert to an AI-supported decision does not mean that person is equipped to perform an effective oversight function.

That is a much richer conception of human oversight than an approval button at the end of a workflow.

Humans are extraordinarily good at things machines still struggle with: contextual judgment, ambiguity, novel situations, competing priorities, tacit knowledge and noticing that something is strange even when it is difficult to articulate exactly why. Those capabilities are a legitimate part of the control environment.

The problem appears when we start treating human adaptability as an unmetered utility.

Human attention is not infinitely scalable infrastructure

The limits become especially obvious when we move from occasional AI-assisted decisions toward agentic work.

A person reviewing one unusual recommendation may be able to inspect the evidence, think carefully about the context, challenge the output and decide whether intervention is warranted. Give the same person several hundred routine agent actions to supervise and the nature of the job changes. They are no longer performing considered oversight of exceptional decisions; they are monitoring a high-throughput automated system and waiting for the rare moment when something important looks different.

Human-factors research has been warning us about versions of this problem for decades. Automation bias—excessive reliance on automated decision support—is not simply a character flaw in inattentive operators. Verification complexity, cognitive load, system reliability and the design of the task itself all influence whether people continue scrutinizing automated decisions effectively.

That should make us wary of designing agentic systems whose safety case can essentially be summarized as:

The AI will perform the work at machine speed, and the human will catch anything weird.

There are quite a few assumptions hiding inside that sentence. The human has to know what “weird” looks like, have enough information to distinguish unusual-but-correct from subtly dangerous, possess relevant domain expertise, understand enough about the AI's limitations and have a realistic way to intervene. After 199 perfectly good outputs, they also need to bring meaningful attention to number 200.

The technology may scale almost frictionlessly. Human vigilance does not.

This is precisely the kind of problem we are beginning to examine with clients as they move from general AI adoption into more consequential human-agent workflows. One of the most revealing questions is not “Is there human approval?” but “What, exactly, are you asking the human to do here?”

Once you unpack the word approval, several different jobs usually appear underneath it. The reviewer may be expected to verify factual accuracy, recognize a security problem, interpret uncertainty, exercise business judgment, identify a policy exception, retain accountability or recognize that the issue belongs somewhere else entirely.

Calling all of that a “human-in-the-loop control” does not make it one control. It makes it a bundle of assumptions about human performance.

Human Resilience Engineering changes the question

We are borrowing from a discipline with a much longer history than enterprise AI.

Resilience engineering emerged from safety science and the study of complex socio-technical systems, where researchers became increasingly uncomfortable with the idea that safety could be understood simply by counting failures and eliminating human variability.

Complex systems work, much of the time, precisely because people adapt. They make small adjustments around incomplete information, conflicting goals, resource constraints, unexpected events and the persistent difference between the work imagined in a process document and the work required to keep an organization functioning.

Resilience engineering studies and strengthens that adaptive capacity rather than treating every departure from an idealized process as an error to be stamped out. One common formulation looks at the ability of a system to anticipate, monitor, respond and learn.

That lineage matters because we are not claiming to have invented resilience engineering.

What Cybermaniacs is developing through Human Resilience Engineering (HRE) is an application of that systems logic to the human layer of cyber, AI and digital risk.

Our working definition is:

Human Resilience Engineering is the deliberate design, measurement and strengthening of the human and organizational capacities that allow people to anticipate, monitor, respond, adapt and learn as digital and AI-enabled conditions change.

That is subtly but importantly different from asking how we prevent people from making mistakes.

Human Risk Management remains essential because organizations still need to understand where human-related risk exists, what is driving it, which populations are exposed and which interventions are likely to change the outcome. But a purely deficit-oriented conception of risk eventually hits a ceiling. Complex organizations do not become resilient because every person follows every procedure perfectly.

They become resilient because people, technology and organizational systems can continue producing good outcomes under conditions that were not completely anticipated in advance.

That is the conceptual move from managing human risk toward engineering human resilience.

A credible human control has to survive contact with actual work

Once you look at HITL through that lens, the standard becomes considerably harder than “a person approves this step.”

What is the human actually there to detect or decide? Can they see enough of the evidence and system state to make that judgment? Do they possess the capability required for this particular oversight task, and can that capability survive as AI does more of the underlying work?

Then there is capacity. The volume, timing and complexity of the work have to be compatible with sustained human attention. The person needs usable authority when they recognize a problem, and uncertainty needs somewhere sensible to go when the reviewer cannot resolve it alone.

The organizational conditions around the control matter just as much. A technically perfect escalation path is of limited value if using it is treated as slowing down the transformation program. An override mechanism is not much of a control if the employee has learned that nobody wants it used. A review process that works beautifully at twenty transactions a day may become mostly ceremonial at two thousand.

And the system needs to learn.

Overrides, near misses, unusual outputs, successful adaptations, workarounds and repeated escalations should tell the organization something about how the Human-AI system is behaving. Otherwise, we continue asking people to compensate for weaknesses without ever redesigning the conditions producing them.

None of these ideas is particularly exotic. The unsettling part is how often the word human is allowed to stand in for all of them.

This is also where Cybermaniacs' work on Human Control Effectiveness becomes useful. The question is not whether a human control exists in the workflow, but whether it can actually perform the risk-reducing function assigned to it under realistic operating conditions.

The human should not become the universal exception handler

There is a deeper architectural problem here too.

If every risk that cannot be fully eliminated technically is handed to a person, the human layer becomes the universal exception handler for the entire AI stack. That is a terrible use of human capability.

Some risks should be constrained technically. Some actions should never reach an approval queue because permissions make them impossible in the first place. Some high-volume decisions should be sampled, monitored or routed according to consequence rather than individually reviewed. Some outputs need provenance and evidence designed into the interface so verification is fast rather than forensic.

Some tasks should remain human-led because the judgment involved is genuinely difficult to delegate. Others should become mostly autonomous because human approval adds ritual rather than safety.

This is why Human Resilience Engineering cannot become another name for “train the humans better.”

Sometimes the correct intervention is to remove unnecessary cognitive work from the human altogether.

That can sound slightly odd coming from a Human Risk Management company, but it is central to the discipline we think HRM has to become. If people repeatedly fail a control because the workflow requires an unrealistic level of vigilance, the answer is not automatically another learning module about the importance of vigilance. We need to be willing to say that the control itself is badly engineered.

Our work on AI reliance and cognitive offloading approaches the same problem from another direction. Offloading cognitive work can be enormously useful. It becomes risky when people lose the situational awareness, expertise or practiced judgment required to recognize when automated work has gone wrong.

The design problem is therefore not “human or AI?” It is deciding which cognitive work belongs where and how the combined system preserves the capacity to adapt when conditions change.

Work-as-imagined versus work-as-done is about to become an AI governance problem

One of resilience engineering's most useful distinctions is between work-as-imagined and work-as-done.

The former is how an organization believes a process works: the policy, workflow diagram, control matrix and responsibility model. The latter is how people actually make the thing work amid time pressure, awkward systems, incomplete information, competing goals and all the other inconvenient features of reality.

Resilience engineering treats the gap between the two as something worth understanding rather than automatically interpreting it as worker non-compliance.

AI is going to make that gap fascinating.

Work-as-imagined may say that an employee reviews every significant agent output before execution. Work-as-done may reveal that the agent is accurate enough that detailed review feels wasteful, the queue is growing faster than the employee can examine it, the explanations are not particularly useful and experienced employees have developed an informal pattern for deciding which outputs deserve real scrutiny.

Traditional compliance thinking might look at that and conclude that people need to be reminded to follow the process.

Human Resilience Engineering asks a more useful set of questions. What adaptation have people made, and why did they need to make it? Does it increase risk, reduce risk or perhaps do both under different conditions? What information are experienced workers using that the formal control failed to capture? Could the useful part of that adaptation become part of the operating model rather than remaining invisible tribal knowledge?

Working with clients on human risk has taught us repeatedly that what people actually do is often diagnostic information about the system. Workarounds, shortcuts, unusual verification habits, reluctance to adopt a tool or excessive confidence in one can all tell us something that a policy-compliance metric cannot.

That becomes even more valuable as human and non-human work become intertwined.

Resilient oversight has to be able to anticipate, monitor, respond and learn

The classic resilience capabilities give us a better way to think about AI oversight than the binary question of whether a human is present.

Anticipation means understanding where the Human-AI combination is likely to become brittle before an incident proves it. Which tasks are becoming more autonomous? Where is human expertise decaying because the system normally performs the work? Where could volume overwhelm meaningful review? Which roles are inheriting accountability without equivalent visibility or authority?

Monitoring means seeing whether the conditions assumed by the control remain true. Agent telemetry matters, but so do human signals: capability, confidence, verification behavior, intervention, escalation, culture, workload and the different ways populations are adapting as AI becomes normal.

That is part of what Cybermaniacs means by Human-AI Risk Intelligence. Technical performance tells us what the AI is doing. Human-AI Risk Intelligence helps the organization understand whether the surrounding human system is still capable of governing, interpreting and adapting to it.

Response requires real intervention capacity. Employees need to know what to do, possess the authority to do it and have a usable route for situations that exceed their expertise. Telling somebody to verify is easy; designing proportionate, role-specific verification into the actual handoff between human and machine is harder and far more useful.

And learning means allowing the control to evolve. If people regularly override a particular recommendation, that is evidence. If almost nobody ever overrides anything, that is evidence too—although perhaps not the evidence the dashboard initially suggests. Near misses, strange outputs, successful interventions, workarounds, questions and escalation patterns should all make the next version of the system better.

Otherwise, we are not operating a resilient control. We are repeatedly asking the same human to compensate for the same unexamined weakness.

Human capability can erode while the control diagram remains unchanged

There is another problem resilience engineering makes easier to see: the people inside the system are changing too.

Imagine an analyst who once performed a task directly and now supervises an agent performing most of it. The governance diagram still shows an experienced human reviewing the work. Sixteen months later, the diagram may be identical while the analyst has had far fewer opportunities to exercise the reasoning the review depends upon.

The control appears unchanged. The capability underneath it may not be.

Our work on Agentic Skill Atrophy examines precisely this problem. Automation can remove the practice through which people maintain the expertise they are still expected to use when the automated system encounters something unusual.

This matters for what Cybermaniacs calls Supervision Sufficiency. A supervisory model can have excellent visibility, clean escalation routes and clearly documented authority while gradually losing the human capability required to interpret what is being supervised.

That is the kind of risk conventional process assurance can miss because nothing on the process diagram has broken.

Human Resilience Engineering asks whether the adaptive capacity underneath the diagram is still there.

Human-Agent Accountability needs more than a named owner

Agentic AI introduces another familiar governance comfort: assign a human owner.

Again, that may be entirely appropriate. But ownership becomes meaningful only when the person has a plausible relationship to the work they supposedly own.

If an agent can perform long sequences of activity, make intermediate decisions and interact with other systems before a human sees the result, accountability needs to be designed around actual authority and visibility rather than inherited from the nearest job title.

Cybermaniacs' work on Human-Agent Accountability looks at this broader problem of delegated authority. Human Decision Ownership goes one level deeper by asking which consequential judgments should remain explicitly human-owned even as more execution moves to the agent.

The resilience question is what happens when that clean allocation meets an unusual condition.

Can the person recognize that the agent has moved outside the expected pattern? Do they know whether they own the decision or should escalate it? Can they intervene quickly enough to matter? Has the work preserved enough independent capability for their judgment to add something when it returns?

Accountability that works only while everything behaves normally is not much of a resilience mechanism.

The goal is not humans in every loop. It is resilience across the system.

This may be the most important distinction.

A Human Resilience System does not maximize human involvement. It puts human capability where it contributes meaningful adaptive capacity and supports that capability with the right technology, information, authority, culture and learning mechanisms.

Sometimes that means a consequential decision needs careful human judgment. Sometimes an agent should act autonomously inside tightly engineered boundaries while humans monitor higher-order conditions. Sometimes several people need to contribute different forms of expertise. Sometimes the correct design is to stop asking a human to approve something that no human could realistically verify at the speed and scale required.

The objective is not to preserve human activity for sentimental reasons, nor is it to automate away the irritating unpredictability of people.

Complex socio-technical systems need both reliable technical constraints and human adaptability. The engineering challenge is deciding how those capacities should interact.

That is also why we think Human Resilience Engineering eventually reaches beyond AI governance. Cybersecurity has spent years framing people predominantly in deficit terms: error rates, failure to comply, phishing susceptibility, risky behavior. Human Risk Management improved that picture by asking what conditions are producing risk and what interventions can change it.

Human Resilience Engineering adds another question: what capabilities must we deliberately preserve and strengthen so the organization can continue to function safely when the environment changes faster than its controls?

In an AI-enabled organization, that is not a philosophical distinction. It is an operating requirement.

Where Cybermaniacs is taking this

Our work with customers across Human Risk Management, AI workforce readiness, culture, learning and now agentic change is pushing us toward a much more integrated model of the human layer.

We already know that competency alone does not explain behavior. Psychology matters. Culture matters. Organizational conditions matter. Work design matters. A technically capable employee can still make a poor decision under pressure, while a less experienced employee can produce an excellent outcome because the surrounding system gives them strong cues, clear escalation, usable guardrails and enough psychological safety to stop when something looks wrong.

AI adds new variables to that system—reliance, cognitive offloading, delegation, intervention, override, changing skills and new forms of human-machine interdependence—but the underlying lesson is familiar: human performance is contextual.

That is why our AI Workforce Risk Management work focuses on the management layer around these changing conditions rather than reducing AI risk to literacy or misuse. It is also why Agentic Readiness & Change does not start and end with agent literacy. Roles, workflows, human oversight, governance, capability, behavior and the organizational conditions surrounding the work all have to move together.

Our broader direction is increasingly about building Human Resilience Systems rather than simply producing more interventions at the point where somebody already appears risky.

There is a satisfying irony here. AI was supposed to remove a great deal of mundane human work. We should probably resist designing its governance in a way that gives all of that time back as an endless queue of approval requests.

Lucy did not need another memo explaining the importance of wrapping chocolates correctly. Somebody needed to slow down the conveyor belt, redesign the job or change the system around it.

Our AI controls deserve at least that much thought.

Frequently Asked Questions

What does human-in-the-loop mean in AI?

Human-in-the-loop, or HITL, describes an AI or automated workflow in which a person participates at some point in the decision or action process. Depending on the use case, the person may review an output, approve an action, correct the system, provide feedback, intervene in an exception or retain final decision authority.

Why is human-in-the-loop not automatically an effective control?

Human presence does not guarantee meaningful oversight. The person also needs enough information, capability, attention, authority and time to perform the intended oversight task. High workload, difficult verification, over-reliance on accurate automation, weak escalation, poor interfaces or incentives that discourage intervention can all weaken the control.

What is Human Resilience Engineering?

Cybermaniacs uses Human Resilience Engineering (HRE) to describe the deliberate design, measurement and strengthening of the human and organizational capacities required to anticipate, monitor, respond, adapt and learn as digital and AI-enabled conditions change. It applies established resilience-engineering principles to cyber, AI and workforce risk.

What is a Human Resilience System?

A Human Resilience System is the broader socio-technical environment through which an organization can sense changes in human risk, maintain important human capabilities, support effective intervention and learn as technology, work and risk change. It connects the human layer with governance, technology, culture, work design, learning and assurance rather than treating them as unrelated programs.

How is Human Resilience Engineering different from Human Risk Management?

Human Risk Management focuses on identifying, understanding and reducing human-related risk. Human Resilience Engineering extends that perspective by asking what adaptive capacities the human and organizational system needs to preserve so it can continue producing safe outcomes under changing and unexpected conditions. The two are complementary rather than competing disciplines.

Does Human Resilience Engineering mean putting more humans into AI workflows?

No. Better resilience engineering may remove unnecessary human approval where technical controls or better automation provide more reliable protection. The objective is to place human capability where judgment and adaptation genuinely add value and design the surrounding system so people can perform that role effectively.

What makes human oversight of AI effective?

Effective oversight depends on more than human presence. The person needs a clear purpose, sufficient visibility into the work, appropriate capability, manageable supervisory demand, usable authority and somewhere to escalate genuine uncertainty. The organization also needs evidence that those conditions continue to hold as the AI system and the work around it change.

How does Human Resilience Engineering relate to Human Control Effectiveness?

Human Control Effectiveness asks whether a human control can actually perform the risk-reducing function assigned to it. Human Resilience Engineering takes a broader systems view, asking whether the surrounding human and organizational capacity can continue adapting as workloads, technology, authority, skills and operating conditions change.