People rarely make a single, durable decision about whether they trust AI.

They make smaller decisions throughout the work. They accept one output and question another. They verify a recommendation when the stakes are high and take a routine result at face value when the cost of checking feels disproportionate. As systems become familiar, people may give them more work, inspect fewer intermediate steps or allow them to act with less direct supervision.

Those behaviors are better understood through reliance.

Reliance has a long history in human-automation research, and it has become increasingly important in contemporary AI research because effective human-AI collaboration depends on people knowing when to use the system's output and when to withhold that reliance. For Human Risk Management, the concept becomes even more useful once reliance moves beyond accepting an answer and begins shaping how much work, judgment and authority people are willing to hand to AI.

What is reliance in human-AI work?

Reliance is the extent to which a person allows an AI system's output, recommendation, process or action to influence or substitute for their own work, judgment or control in a particular context.

Reliance is behavioral. A person may say they distrust an AI system and still routinely follow its recommendations. Another may report high trust while continuing to verify every consequential result. Their stated attitudes are useful evidence, but they do not tell us the whole story about how human and machine judgment are actually being combined.

This distinction has been central to human-automation research for decades. Raja Parasuraman and Victor Riley's influential 1997 paper, Humans and Automation: Use, Misuse, Disuse, Abuse, described misuse as over-reliance on automation that can produce monitoring failures or decision bias, while disuse describes the neglect or underutilization of automation. John Lee and Katrina See later developed the relationship between trust and reliance in Trust in Automation: Designing for Appropriate Reliance, arguing that trust is particularly important when system complexity prevents people from fully understanding how automation will behave in every situation. Sage Journals

The contemporary AI literature has inherited that central question: when should a person rely on the system, and by how much?

Appropriate reliance is the goal

The most useful concept in this literature is appropriate reliance.

Microsoft Research defines appropriate reliance on generative AI as accepting AI outputs when they are correct and rejecting them when they are wrong. Its research program deliberately treats both over-reliance and under-reliance as problems because either can reduce the performance of the combined human-AI system. (Microsoft Research: Appropriate Reliance on Generative AI) Microsoft

That framing is valuable because discussions of AI adoption can become overly focused on increasing trust. More trust is useful only when the system, task and conditions justify it.

A person who refuses useful AI assistance can create poor outcomes through unnecessary manual work, lost efficiency or workarounds. A person who accepts an incorrect recommendation because the system has performed well in the past can create a different class of failure.

The desired condition is calibration: human reliance should move appropriately with evidence about system capability, task difficulty, consequence, uncertainty and the person's own ability to judge the result.

This idea already appears in the regulatory environment. Article 14 of the EU AI Act requires human oversight arrangements for relevant high-risk systems to account for the tendency to automatically rely or over-rely on AI output, particularly where systems provide information or recommendations for human decisions. The inclusion of automation bias in the legislation is notable because it recognizes that effective oversight depends partly on how people behave around system recommendations, not simply on whether a human has been assigned to review them. Cambridge University Press

Trust and reliance are related, but they are not interchangeable

Trust is one of the factors that can shape reliance.

It is not a direct behavioral measure of reliance.

A person's trust may be influenced by the system's history, brand, interface, explanations, confidence cues, prior experience or their general beliefs about AI. Reliance appears in what happens next: whether the person accepts the answer, verifies it, delegates another step, allows the agent to proceed or chooses to retain control themselves.

That distinction matters for Human Risk Management because surveying employees about whether they “trust AI” can produce an interesting psychological measure without establishing how they behave during actual work.

A workforce may report moderate trust while routinely accepting AI-generated content. Another group may express enthusiasm for AI while verifying high-consequence work rigorously because local norms and professional standards demand it.

Understanding reliance therefore requires evidence about both belief and behavior, interpreted in the context of the work.

Our Guide to AI Workforce Risk Intelligence develops this distinction more broadly. AI usage, attitudes, verification behavior, capability and task consequence provide different kinds of evidence; none of them independently establishes whether human judgment is appropriately engaged.

Over-reliance is more complicated than “trusting AI too much”

The phrase over-reliance sounds straightforward, but operationalizing it is surprisingly difficult.

Microsoft's research defines over-reliance in terms of accepting incorrect AI outputs. That gives researchers a clear way to evaluate human-AI performance where the correct answer can be established. In enterprise knowledge work, however, many tasks do not have a simple objectively correct answer available after every interaction.

An analyst may accept a plausible synthesis. A manager may allow an agent to draft a recommendation. A procurement team may use AI-produced rankings as one input into a decision. The outcome may depend on context, judgment and future events rather than a binary correct/incorrect label.

That means enterprise risk teams should be careful about treating acceptance rate as a direct measure of over-reliance.

High acceptance may reflect excellent system performance and appropriate use. It may also reflect habit, excessive confidence, weak review, time pressure or declining human capability. Low acceptance can likewise reflect healthy skepticism, poor system performance or unnecessary under-reliance.

The behavior is evidence. Interpretation requires context.

That distinction is consistent with our broader Human Risk Management approach: a measure or event does not become a diagnosis simply because it is easy to count.

Agentic AI expands the meaning of reliance

Much of the classic reliance literature concerns whether a human accepts the recommendation of automation.

Generative AI widened the problem because people could rely on systems to produce complex content, analysis and reasoning.

Agentic AI widens it again.

A person may increasingly rely on an agent at several levels of the work:

These forms can coexist, and they need not develop at the same rate.

An employee might trust an agent's research process while retaining close control of the final decision. Another might scrutinize generated text closely while allowing the system considerable freedom to gather information and use tools.

For Human Risk Management, this is an important extension of the classic reliance problem. The question is increasingly broader than whether people accept AI output. It includes how much of the work they are willing to let the system perform before human judgment re-enters the process.

This is where reliance connects directly with Delegation Drift. As reliance grows, the practical scope of delegation can expand even when no one explicitly decides to redesign the workflow.

Reliance changes as familiarity grows

People do not encounter a system for the first time every day.

They accumulate experience.

A new AI tool may initially receive intense scrutiny because employees are uncertain about its capability. If it performs reliably, people gradually learn which tasks it handles well, verification becomes faster and interaction becomes routine. That learning is often exactly what successful adoption is supposed to achieve.

It also changes the human-risk condition.

Repeated good performance creates reasonable evidence that future performance may also be good. At the same time, familiarity can make exceptions less salient and encourage people to devote less attention to routine outputs.

Research into automation has long examined this relationship between reliability, trust and reliance. Lee and See's work emphasizes that trust should correspond to the system's actual capability in the relevant context rather than becoming a general belief detached from performance. Sage Journals

Generative AI complicates calibration because systems can produce fluent, credible-looking results even when the underlying content is incomplete or wrong. Microsoft Research's recent work on appropriate reliance notes that this makes errors harder to spot and can increase the cognitive effort required for verification. (Fostering Appropriate Reliance on GenAI) Microsoft

The resulting challenge is familiar to anyone who has worked with an excellent colleague for years: repeated competence naturally earns more autonomy. AI adds a machine-scale version of that relationship, often without the same visibility into why a particular result should suddenly deserve less confidence.

Confidence cues can change reliance even when the AI is wrong

People do not base reliance only on actual system performance.

They respond to signals.

A 2026 AAAI study of AI confidence and human reliance found that well-calibrated confidence information improved participant decision accuracy, while miscalibrated confidence cues increased susceptibility to automation and conservatism biases. Participants were more likely to accept recommendations accompanied by high confidence even when those recommendations were incorrect. (AAAI: Too Sure for Our Own Good) AAAI Publications

Other research has found that explanations can themselves increase reliance. A Microsoft-led CHI 2025 study found that providing explanations increased reliance on both correct and incorrect LLM responses, although sources and visible inconsistencies could reduce reliance on incorrect answers. (Fostering Appropriate Reliance on Large Language Models) Microsoft

This is a useful warning for enterprise AI programs.

Features intended to make AI more understandable can also make it more persuasive. A polished explanation, citation format, confidence indicator or familiar interface may affect human behavior independently of whether the underlying answer deserves reliance.

That makes reliance partly an interface-design issue and partly a workforce capability issue.

Employees need enough understanding to interpret system cues rather than treating them as substitutes for judgment.

Verification cost changes reliance

“Check the AI” is easy advice.

Checking can be expensive.

If verifying an AI-generated answer requires reconstructing the entire analysis independently, the time saved by using AI may disappear. Employees then face a practical trade-off between efficiency and scrutiny, particularly when the system usually performs well.

Microsoft's 2025 research on appropriate reliance identifies the difficulty and cognitive cost of verification as an important part of the problem. Its design recommendations include helping people develop useful mental models of system capability, signaling when verification is particularly important and making verification itself easier. Microsoft

This gives organizations a better way to think about verification requirements.

The appropriate amount of checking should depend on the work. A low-consequence draft, a strategic recommendation and an agent about to modify a production system do not create the same verification burden or consequence if the AI is wrong.

Our Guide to AI Agent Governance and the Missing Human Risk Layer makes this point at the governance level. Human oversight needs to account for reliance and verification as operating behaviors rather than simply requiring a human approval step.

Organizational conditions shape reliance too

Reliance can look like an individual psychological choice while being heavily influenced by the organization around the person.

Suppose a company announces that AI should improve productivity by 30 percent. Managers begin asking why tasks still take as long as they did before Copilot. Employees remain formally responsible for verifying AI-generated work, but careful verification now competes with an explicit expectation that AI should make everything faster.

Over time, local norms can develop around what level of checking feels reasonable.

The opposite can happen as well. In an organization where employees are punished for AI mistakes, people may become reluctant to use even reliable systems in situations where automation would genuinely improve performance.

These conditions matter because appropriate reliance depends on more than system accuracy and user knowledge. Workload, incentives, professional norms, management expectations, accountability and the perceived consequences of challenging AI can all influence whether people accept or reject machine judgment.

This is why AI Workforce Risk Management treats reliance as part of a larger socio-technical system. The same AI tool can produce very different working behaviors across teams because the surrounding environment changes what people feel able, expected or rewarded to do.

Reliance and supervision are inseparable in agentic work

As agents take on more work, reliance begins to affect the quality of human supervision directly.

A supervisor who assumes an agent will usually behave correctly may monitor lightly. A supervisor with low confidence may check every action and create enough friction that the organization gains little benefit from autonomy.

Neither behavior can be judged without considering system performance and task consequence.

This relationship is one reason reliance sits naturally inside the HRS Agentic Supervision Lifecycle. Reliance influences how closely people observe, what they interpret as meaningful deviation and how readily they intervene when something changes.

It also affects the practical value of Intervention Points and Human Override. A perfectly designed intervention mechanism will be used differently by someone who assumes the agent is almost always right than by someone who routinely expects it to fail.

Reliance therefore becomes part of the supervisory state of the human-agent system.

Reliance should move with consequence

One of the most useful enterprise principles is that appropriate reliance should be sensitive to consequence.

People already behave this way in many parts of life. We tolerate imperfect navigation suggestions more readily than imperfect medical instructions because the cost of error differs.

Human-AI work needs the same contextual discipline.

An employee may reasonably accept routine formatting assistance with little verification while scrutinizing an AI-generated financial recommendation closely. The system could be equally accurate in both tasks, yet the appropriate human behavior differs because the consequence of error is different.

That means organizations should avoid trying to define one universal “correct” level of trust or reliance for a particular AI product.

Reliance should be calibrated to the system, task, context and consequence.

This is particularly important for agentic systems because the same agent may move across very different states during one workflow. Research may be low consequence, recommendations more consequential, and external action more consequential still.

The appropriate reliance relationship can therefore change while the agent is working.

Appropriate reliance also depends on human capability

A person can only calibrate reliance well when they possess enough capability to judge the system.

An expert may recognize that an AI answer is superficially convincing but based on a flawed assumption. A novice may see the same answer and have little basis for challenging it.

That creates an important relationship between reliance and expertise.

As AI systems perform more of the underlying work, organizations also need to consider whether the human retains enough skill to continue evaluating it. If the employee becomes increasingly dependent on the system while losing practice in the task, the ability to calibrate reliance can weaken over time.

This links reliance to the emerging problem of Agentic Skill Atrophy, where the system's success can gradually reduce the human capability needed to supervise, verify or recover from failure.

Our work on Human Resilience Engineering for Agentic AI treats retained capability as part of the human-agent control architecture for exactly this reason. Human judgment cannot function as a resilient backstop when the work system has allowed the relevant expertise to disappear.

What should organizations observe about reliance?

Reliance is important enough to measure, but it is easy to measure badly.

A survey question asking whether employees trust AI may contribute useful evidence. It should not be interpreted as a direct measure of appropriate reliance.

Organizations may need to combine several types of evidence:

• reported trust, confidence and perceived system capability;
• the kinds of work people delegate to AI;
• acceptance and rejection behavior;
• verification practices;
• frequency and type of corrections;
• intervention and override behavior;
• task consequence and reversibility;
• changes in human skill or direct task participation;
• workload and time available for verification;
• outcome quality; and
• organizational incentives or norms affecting AI use.

The significance of each measure depends on context.

For example, a declining verification rate could indicate that employees have developed appropriate confidence in a highly reliable workflow. It could also indicate growing complacency, excessive workload or loss of the expertise required to verify effectively.

This is why AI Workforce Risk Intelligence emphasizes interpretation across multiple forms of evidence. Reliance is a condition to understand over time rather than a single score to optimize.

A practical reliance review

Organizations introducing AI into meaningful work can use a small set of questions to examine whether reliance is developing appropriately.

1. What is the person relying on the AI to do?
   Separate generation, analysis, recommendation, process execution and delegated authority.
2. What happens if the AI is wrong?
   Consider the consequence, reversibility and downstream impact of an incorrect result or action.
3. How difficult is the result to verify?
   Identify whether meaningful checking is easy, expensive, ambiguous or dependent on specialist expertise.
4. What does the employee know about system capability?
   Examine whether people understand where the system performs reliably, where uncertainty is higher and what signals deserve attention.
5. How has reliance changed with familiarity?
   Look for changes in verification, delegation, review depth and willingness to allow autonomous action.
6. What organizational pressures affect the behavior?
   Consider workload, productivity expectations, management signals, local norms and the perceived cost of questioning AI.
7. Does the human retain enough capability to challenge the system?
   Pay particular attention where AI is taking over tasks that people still need to evaluate or recover.
8. What evidence would show that reliance has become inappropriate?
   Define the combinations of behavior, outcome and context that should prompt further investigation or intervention.

These questions do not produce a universal reliance score.

They give the organization a way to examine whether human judgment and system capability remain aligned as the work changes.

Reliance belongs inside Agentic Readiness

Organizations preparing employees for AI often concentrate on tool knowledge: how the system works, what data can be entered and which use cases have been approved.

Reliance requires a more behavioral form of readiness.

People need to understand which tasks deserve greater verification, what uncertainty looks like, where agent authority ends, how to challenge plausible outputs and when confidence in a familiar system should be reconsidered.

Managers need similar capability because they shape the environment in which those decisions occur. Productivity expectations, workflow design and local incentives can quietly push teams toward either excessive reliance or excessive caution.

This is one reason Agentic Readiness & Change (ARC) needs to address the relationship between people and agents as well as the mechanics of using the technology. As roles move toward delegation and supervision, reliance becomes part of how well employees perform the supervisory job.

Reliance is dynamic

One of the easiest mistakes in AI governance is to treat trust and reliance as characteristics established during rollout.

They move.

The AI changes. Employees gain experience. Workflows become more automated. Interfaces improve. Verification becomes routine or burdensome. Managers change expectations. People gain skill in some areas and lose practice in others.

The reliance relationship can therefore evolve even when the approved application, written policy and formal workflow remain unchanged.

For Human Risk Management, that makes reliance a particularly important longitudinal signal.

A mature program should be interested in whether people are relying on AI differently than they did three months ago, what changed around that behavior and whether the new pattern remains appropriate for the work being performed.

The objective is not to make people perpetually suspicious of AI.

It is to preserve enough alignment between human judgment, system capability, task consequence and delegated authority that reliance remains useful rather than quietly becoming a new source of risk.

Frequently Asked Questions

What is reliance in human-AI interaction?

Reliance describes the extent to which a person allows an AI system's output, recommendation, process or action to influence or substitute for their own work, judgment or control in a particular context.

What is appropriate reliance on AI?

Appropriate reliance occurs when people use AI when its capability is suitable for the task and withhold, verify or challenge its output when the conditions warrant greater human judgment. Microsoft Research commonly operationalizes appropriate reliance as accepting correct AI outputs and rejecting incorrect ones. (Microsoft Research) Microsoft

What is the difference between trust and reliance in AI?

Trust is an attitude or expectation about the AI system. Reliance is reflected in behavior: whether someone accepts the output, follows the recommendation, delegates additional work or allows the system to act.

Trust can influence reliance, but the two should not be measured or interpreted as if they were the same construct.

What is AI over-reliance?

Over-reliance occurs when a person relies on an AI system in circumstances where greater verification, skepticism or human control would have produced a better outcome. In experimental research this is often measured through acceptance of incorrect AI outputs; enterprise environments may require broader evidence because many tasks do not have simple binary answers.

What is AI under-reliance?

Under-reliance occurs when people reject, avoid or fail to use AI in situations where using the system would improve performance or reduce unnecessary work.

Both over-reliance and under-reliance can reduce the effectiveness of human-AI collaboration.

Is reliance the same as automation bias?

No. Automation bias refers to tendencies in human decision-making that can lead people to favor automated recommendations or fail to notice contradictory evidence. Reliance is broader and describes the extent to which a person uses or depends on the system's outputs or actions.

Automation bias can contribute to inappropriate reliance.

How does agentic AI change reliance?

Agentic AI expands reliance beyond accepting generated output. People may also rely on agents to perform analysis, select actions, execute intermediate processes or operate under delegated authority.

That makes reliance relevant to supervision, delegation, intervention and accountability across the wider human-agent system.

How can organizations measure AI reliance?

Organizations may need to combine evidence about trust, delegated work, acceptance and rejection behavior, verification, corrections, intervention, task consequence, retained expertise and outcomes.

A single usage or survey metric is unlikely to establish whether reliance is appropriate.

How does reliance affect human oversight?

Reliance influences how closely people monitor AI, how readily they question its output and when they choose to intervene. Effective oversight therefore depends partly on whether reliance remains calibrated to the system, task and consequence.

Our Guide to effective human oversight of AI agents examines the wider conditions required for that supervision to work.

How does reliance relate to Human Resilience Engineering?

Human Resilience Engineering treats reliance as part of the relationship between people, AI systems and work. A resilient system should support people in relying on automation where it performs well while retaining enough awareness, capability and control to respond when conditions change.