As the FTC investigates OpenAI, Anthropic and other AI companies over potential consumer risks, a bigger enterprise question is emerging: how do organizations govern human trust in AI, agentic systems and increasingly autonomous technology?
The Federal Trade Commission has opened an investigation into OpenAI, Anthropic and other AI companies over potential safety risks their products may pose to consumers. The probe, reportedly underway for months before becoming public, could examine whether AI companies' safety practices and representations about their systems comply with existing consumer-protection law. The investigation is not a finding of wrongdoing, but it is another sign that AI safety is moving beyond voluntary commitments and model testing into questions of consumer trust, accountability and what people can reasonably expect these systems to do.
That shift matters because the interesting question is no longer simply whether an AI model can be made safe enough. It is whether humans know when to trust it.
The FTC already has a fairly technology-agnostic way of approaching this problem. Section 5 of the FTC Act prohibits unfair or deceptive practices, and the Commission has increasingly applied that framework to AI. Its recent policy work around AI accuracy has focused heavily on the expectations companies create around what their systems do and how accurately they do it. The FTC has also previously asked AI companies how they test for negative impacts, communicate capabilities and limitations, handle user data and monitor what happens after deployment.
There is something important buried in that idea of expectation.
For most of the current AI era, we have talked about safety as though it is predominantly a property of the machine. Does the model hallucinate? Can it be jailbroken? Will an agent follow a malicious instruction? Can it escape a sandbox, misuse tools or reach systems it should not? Those are real engineering problems, and recent incidents involving agentic systems have understandably increased scrutiny of them.
But very few real-world AI outcomes belong solely to the model.
They happen at the junction between what the technology does and what a person believes the technology is capable of doing.
A generated answer arrives in impeccable prose and looks more authoritative than the human expert who would have added three caveats. An employee sees an AI assistant embedded in an approved enterprise platform and assumes that whatever it can access, it is permitted to access. A manager sees an elegantly summarized recommendation and mentally upgrades it from “model output” to “analysis.” Someone asks an agent to “take care of this” without really knowing which credentials, data sources, systems or downstream actions sit behind those four words.
None of those people necessarily made an irrational decision. They responded to cues.
That is where AI safety becomes human risk.
For human risk management, AI changes the problem from simply teaching people how to use technology safely to understanding how technology changes judgment, behavior and decision-making. Organizations need visibility into reliance, verification, intervention, escalation, authority bias, confidence, skill displacement and the cultural pressures that influence all of them.
We are teaching ourselves how much to trust the machine
Human centered trust in technology is learned through experience. If a system gives you a useful answer this morning, another useful answer at lunch, drafts a competent email in the afternoon and correctly analyzes a spreadsheet before you go home, tomorrow you will probably check it a little less carefully. After a month of success, perhaps less again.
This is one of the uncomfortable things about reliable automation: its successes change human behavior too.
A disclaimer saying “AI can make mistakes” does not exist in a psychological vacuum. It competes with hundreds of interactions in which the AI did exactly what the user wanted. Repeated success can create learned reliance, automation bias and cognitive offloading long before anyone consciously decides, I trust this system now.
Which makes disclosure necessary, but nowhere near sufficient for human use.
What is AI trust?
AI trust is the willingness to rely on an AI system under conditions of uncertainty. But trust is not the same thing as trustworthiness. A model may be technically capable without being appropriate for a particular decision, and a person may trust a system far more—or far less—than its actual performance warrants.
The goal for organizations is therefore not maximum trust in AI. It is calibrated trust: people relying on AI when the evidence supports that reliance and knowing when to verify, intervene, override or escalate.
This distinction between trust and trustworthiness is becoming an important part of AI assurance. Trust describes whether people actually rely on a system; trustworthiness asks whether that reliance is deserved. Mature AI governance has to close the gap between the two.
The important behavioral questions become much more specific. When does someone rely on the model? What makes them verify its output? When do they intervene? Override it? Escalate? When does the presence of AI influence a decision even if the final decision is technically still made by a person? At what point do skills begin to decay because the machine usually does the work?
Those are measurable human behaviors, not just abstract principles that matter because the AI itself is changing too.
The familiar chatbot is rapidly becoming agentic AI: systems with tools, memory, permissions and increasing autonomy to perform work across multiple systems.
We think that shift changes human oversight. Reviewing an answer after the fact is very different from supervising a system that can take actions before a human sees them. An incorrect answer is one kind of problem. An incorrect answer attached to an identity, a browser, an API key and permission to execute is.... something else entirely.
Same model. Same guardrails. Completely different risk environment.
Organizational AI risk is not simply model risk. Here's why:
Imagine two organizations deploying exactly the same frontier model.
The vendor's model weights are the same. Its built-in safety controls are the same. The documentation is the same. The organization's license agreement is the same.
Yet the risk for each company could be radically different.
In the first company, executives are pushing hard for AI adoption and celebrating speed. Employees know leadership wants them to use the technology, but nobody has clearly explained where AI authority stops and human accountability starts. People hesitate to challenge outputs because doing so feels like resistance to the transformation agenda. Agents accumulate access as teams experiment. Managers approve AI-supported decisions without being particularly clear about what they are approving.
In the second organization, adoption may be just as ambitious, but the culture behaves differently. People can express uncertainty without looking anti-innovation. High-impact workflows have clear intervention points. Teams understand which decisions require verification, what kinds of anomalies should trigger escalation and who owns the outcome when an AI system participates in the work. Near misses are discussed rather than quietly corrected. Trust grows, but it grows conditionally.
Same model. → Same guardrails. → Completely different risk environment.
This is why AI governance cannot stop at model governance. Technical controls establish what a system is allowed to do. Organizational culture, human behavior, incentives and workflows determine what actually happens when people and machines work together.
The challenge is increasingly one of human-AI collaboration: determining which judgments belong to people, which actions can safely be delegated to machines, and how authority moves between the two as confidence, context and consequences change.
That is the underpinning of human resilience.
Human resilience in an AI-enabled organization is the capacity of people and teams to adapt their judgment and behavior as machine capability, autonomy and authority change.
It means knowing when reliance is appropriate, detecting when something does not fit the expected pattern, intervening when necessary, escalating uncertainty and recovering when either the human or the machine gets something wrong.
International guidance is beginning to use similar language. The OECD AI Principles explicitly call for human agency and oversight, including mechanisms for humans to override AI systems when necessary.
A resilient organization does not require every person to distrust AI. In fact, generalized distrust would make these systems almost useless. The goal is better calibrated trust: enough confidence to gain the value of automation, with enough understanding and behavioral flexibility to recognize the situations in which that confidence should stop.
AI has a shared-responsibility problem
Cybersecurity has seen a version of this movie before.
Cloud providers can secure their infrastructure while customers misconfigure access. Software vendors can ship good security features while organizations grant excessive privileges. Identity platforms can enforce sophisticated controls while humans create bad workflows around them.
AI is developing its own shared-responsibility model, except the boundaries are less mature. Organizations increasingly need an AI management system, not just a model policy.
The frontier lab has responsibilities around model design, testing, transparency and safety controls. The application developer decides how that model is packaged, prompted and connected. The enterprise determines what data, tools and identities it can reach. Governance teams decide what use is permitted. Security teams monitor the technical environment. Leaders create incentives. Employees and increasingly autonomous agents make thousands of decisions inside those constraints.
That is a very long chain in which to place a single word like "safe".
The FTC investigation is therefore interesting even if it ultimately produces little dramatic enforcement. It is one institution asking what responsibility belongs to the companies creating these systems. The harder operational question for everyone consuming the technology is what responsibility remains once the model crosses the enterprise boundary. (Quite a lot, as it turns out.)
Trust itself is becoming part of the attack surface
Cybersecurity has always concerned itself with trust, but usually in fairly technical terms: trusted identities, trusted devices, trusted networks, trusted certificates. Human trust appears mainly when someone abuses it through phishing, impersonation or social engineering.
Generative AI changes the economics of that problem.
Confidence can be generated. Familiarity can be generated. Authority can be simulated. Tone can be adapted. A system can remember previous interactions, respond sympathetically, mirror communication styles and produce explanations perfectly calibrated to the person in front of it. Increasingly, it can then act.
That does not mean AI systems are inherently manipulative. It means the human cues we have historically used as shortcuts for competence, credibility and intention are becoming easier to manufacture.
We therefore have to start treating trust itself as something that can be engineered, measured, misplaced, reinforced and exploited.
Trust is not the same thing as trustworthiness
This distinction matters. Trust describes whether a human is willing to rely on a system. Trustworthiness describes whether the system actually deserves that reliance. AI assurance increasingly exists to bridge that gap by providing evidence—through testing, evaluation, governance and independent assessment—that allows people and organizations to make better-informed judgments about when trust is justified.
That still leaves a human question. Evidence does not automatically determine behavior. Two people can see the same assurance report and respond very differently because of experience, confidence, organizational pressure, perceived authority or their previous interactions with AI.
That takes us well beyond “AI awareness training.”
Organizations need to understand which employees are inclined toward rapid adoption and which are hesitant. That is why human risk assessment increasingly has to include AI workforce behavior, culture and trust—not just knowledge or policy awareness. Where perceived authority changes behavior. Where confidence in a system exceeds demonstrated competence. Whether people verify when verification matters. Whether teams feel comfortable challenging an automated recommendation. Whether organizational pressure for productivity quietly suppresses escalation. Whether repeated successful interactions are increasing reliance faster than controls and skills are adapting.
Technology controls matter enormously. So do models of the workforce, culture, psychology and behavior surrounding them. And one without the other gives us only half the risk picture.
So, who tells us when the AI is safe to trust?
This is where the FTC investigation leaves us with a bigger question than the one it started with.
→ Perhaps frontier AI companies will become extremely good at telling us where their systems are reliable. They certainly have enormous technical capability and more direct access to their models than anyone else. They also have commercial incentives, competitive pressure, enormous capital requirements and considerable power riding on adoption.
→ Perhaps governments will define the boundary. Regulation, consumer protection, standards and enforcement will undoubtedly play a part. But governments are unlikely to move at the speed at which models, agents and enterprise architectures are currently changing. They also cannot see inside every organization's workflows, permissions, incentives and culture.
→ Independent evaluators, auditors and standards bodies will contribute another piece. AI governance programs will improve. Organizations will get better at model selection, testing and assurance. More sophisticated companies will build adaptive and modular harnesses around frontier models, constraining what agents can see, where they can act, which tools they can invoke, and when a human needs to intervene.
All of that is coming, but none of it eliminates the human challenges.
Organizations now need to understand the conditions under which their people trust machines, how that trust changes with experience, and whether their culture makes verification, intervention and escalation normal when the machine appears confident but something feels wrong.
Because there probably will not be one institution, regulator, vendor or safety benchmark capable of telling us when AI is safe to trust. For organizations, this is becoming an AI workforce risk problem as much as a model-risk problem.
The future is likely to be considerably messier than that: different models, different harnesses, changing capabilities, autonomous agents, new regulation, shifting standards and humans continuously renegotiating what work they are prepared to hand over.
So perhaps the question is no longer who told you the AI was safe to trust?
The question we need to get better at answering is:
Who will tell you—and what evidence will you require before you believe them?
What organizations should know about AI trust and human risk
Why is the FTC investigating OpenAI and Anthropic?
The FTC has confirmed an investigation into OpenAI, Anthropic and other AI companies over potential risks their products may pose to consumers. The inquiry may help clarify how existing consumer-protection law applies to AI safety practices, representations and the expectations companies create around their systems. The investigation itself does not establish wrongdoing.
What is trust calibration in AI?
Trust calibration means aligning how much a person relies on an AI system with what that system can actually be trusted to do in a particular context. Both over-trust and under-trust can create risk. Effective AI governance should help people know when to rely, verify, intervene, override or escalate.
Why is AI trust a human risk management issue?
AI changes human judgment and behavior. Repeated successful interactions can increase reliance, while confidence, authority cues, organizational pressure and automation can affect whether employees challenge or verify machine output. Human risk management therefore has to address the relationship between people, technology and organizational culture, not simply AI knowledge.
How does organizational culture affect AI risk?
Two companies can deploy the same AI model with the same technical guardrails and experience very different risk because their incentives, leadership behavior, escalation norms, permissions and decision-making practices differ. AI risk is therefore partly a property of the organizational environment in which the technology operates.
What does human resilience mean in an AI-enabled organization?
Human resilience is the ability of people and teams to adapt as AI capabilities and autonomy change: relying appropriately, recognizing anomalies, verifying important outputs, intervening when necessary, escalating uncertainty and recovering when either humans or machines make mistakes.
How should organizations govern human-AI collaboration?
Organizations need to define which decisions can be delegated to AI, where human judgment remains necessary, how authority is constrained, what evidence requires verification, and what conditions trigger intervention or escalation. Those controls should evolve as systems become more capable and more autonomous.
Who is responsible for deciding whether AI is trustworthy?
No single actor can answer that completely. AI developers can provide evidence about model capabilities and limitations. Regulators can establish legal expectations. Independent evaluators can test systems. Enterprises can govern access, workflows and authority. Humans still have to decide when reliance is warranted in context.
As AI becomes more agentic, trust is likely to become a shared-responsibility problem across model providers, application builders, organizations and the people supervising their use.
