Why speak-up culture belongs inside the AI control system

OpenAI’s dismissal of three safety researchers has produced two incompatible accounts of the same event. The company says an internal investigation found a significant breach of trust involving sensitive information. The researchers dispute that explanation and argue that their dismissals risk creating a chilling effect around safety work. OpenAI, meanwhile, has said it strongly supports internal criticism and safety debate. From outside the company, there is no responsible way to settle the underlying employment dispute. The Verge’s reporting on OpenAI’s response provides the clearest summary of the company’s position, while TechCrunch covers the researchers’ account and their concerns about the chilling effect on safety work.

The more interesting question sits one level above it.

What does everyone else inside an organization learn when something like this happens?

Organizational researchers have been studying versions of that problem for decades. Edgar Schein’s work on organizational culture distinguished the values an organization publicly espouses from the deeper assumptions people gradually learn about how the place actually works. Those assumptions become powerful precisely because they stop needing to be stated. They shape what people notice, what behavior feels normal, what risks seem acceptable, and which actions appear likely to carry consequences. MIT Sloan’s overview of Schein’s work is a useful introduction to this distinction between visible organizational artifacts, stated values and deeper learned assumptions.

Seen through that lens, culture is much less about messaging than organizational memory.

People watch what happens around them. They notice which mistakes can be discussed, which senior people can be challenged, which concerns receive attention, whose judgment is trusted, what happens when someone slows an important project down, and whether the person carrying bad news tends to fare better or worse afterwards. Over time, those observations become instructions.

That matters enormously for AI because organizations are beginning to place people inside systems where the human is expected to provide judgment, oversight and intervention precisely when the technology behaves in ways the formal control architecture has failed to anticipate.

We have known for seventy years that technology changes the social system around it

There is a useful piece of history here because AI governance sometimes behaves as though the problem of humans interacting with powerful new technology was invented in 2022.

(hashtag: It was not.) 

The modern field of sociotechnical systems grew out of work at London’s Tavistock Institute in the 1950s. Researchers studying mechanization in British coal mines found that introducing technically more sophisticated machinery changed far more than the mechanics of extracting coal. It changed how people coordinated, where expertise sat, how teams communicated, how workers exercised judgment and how responsibility moved through the system. In some cases, improvements to the technical subsystem disrupted social arrangements that had previously made the overall system work remarkably well. The foundational study by Eric Trist and Ken Bamforth remains one of the clearest early statements of this problem: “Some Social and Psychological Consequences of the Longwall Method of Coal-Getting”.

The important idea was joint optimization. You cannot meaningfully optimize the technology while treating the humans, roles, incentives, workflow and organizational environment around it as fixed background conditions. The social and technical components act on one another. A technically excellent system can perform badly because the surrounding work system makes sensible behavior difficult. A technically imperfect system can sometimes remain resilient because people detect, compensate, adapt and intervene.

This sounds strikingly contemporary because AI is forcing us back into exactly the same problem.

An organization acquires Copilot, Claude, Gemini or ChatGPT. A development team deploys an agent. Customer operations introduces an automated service workflow. Finance starts experimenting with machine-supported decisions. The technical discussion concentrates on models, prompts, permissions, evaluations and accuracy.

Meanwhile, the technology is changing the work.

Who checks? Who decides? Who still knows how to do the task without the system? Who owns the outcome when the machine contributes to the decision? How much attention can a person realistically give an automated process while still doing their actual job? At what point does disagreement with the machine become appropriate professional judgment rather than resistance to change?

Those are sociotechnical questions, and they cannot be answered by model evaluation alone.

A timely 2026 paper in the Harvard Data Science Review makes essentially this argument by connecting responsible AI to lessons from earlier sociotechnical failures. Its authors argue that consequential automated systems need to be evaluated as whole sociotechnical systems rather than as isolated technical components. The paper draws explicit parallels to failures including Challenger, Chernobyl, Three Mile Island and Fukushima.

The lesson from those fields is uncomfortable and useful: failures emerge from relationships between technology, organizational structure, incentives, assumptions, authority and human behavior. That is exactly where human risk belongs.

Safety science learned to care about weak signals because disasters rarely arrive fully formed

NASA has been particularly candid about this history. Looking back across Challenger, Columbia and other failures, NASA engineers and leaders have described patterns of normalization of deviance and organizational silence: conditions outside expected boundaries occurred repeatedly without catastrophe, gradually became normalized, and people who perceived increasing risk did not always have the organizational latitude to elevate their concerns effectively. NASA has explicitly connected those failures to forces that made it difficult for people to speak up and to the danger of allowing organizational memory of those lessons to fade. NASA’s discussion of engineering best practices and normalization of deviance is worth reading in this context.

That is not an analogy suggesting an enterprise chatbot is the Space Shuttle.

The useful similarity is structural.

Complex failures frequently begin with information that is incomplete, distributed or easy to rationalize. Someone sees an unusual condition. Someone else has a different piece of evidence. Previous deviations ended harmlessly. Operational pressure favors continuing. Each individual decision looks defensible. The eventual failure becomes obvious only after the signals can be assembled into a pattern.

High-risk industries consequently learned to value information that looks almost embarrassingly mundane before an accident occurs.

Aviation built voluntary reporting systems around exactly this principle. The FAA describes these programs as ways to obtain safety information that might otherwise never become available, particularly information about potential precursors to accidents. The system deliberately creates protections around good-faith reporting because punishing every error would reduce access to the information required to improve the system. The FAA’s overview of voluntary safety programs lays out this logic directly.

Healthcare has reached similar conclusions from a different direction. Research on speaking up in patient safety consistently finds that people weigh the seriousness and ambiguity of the situation, leadership behavior, hierarchy, whether speaking is likely to have any effect, and the social consequences of challenging someone more senior. A scoping review on patient safety and power hierarchy summarizes many of those barriers.

Other reviews describe speaking up in safety terms as identifying a concern and raising it to someone with the power to address it. The phrase with the power to address it deserves considerably more attention in AI governance. Why? It moves the issue beyond encouraging people to talk.

Psychological safety solved one part of the puzzle

Amy Edmondson’s foundational research on psychological safety emerged from an apparent paradox in her work on team learning and error. Teams that appeared to report more mistakes were sometimes the stronger teams. The important variable was not necessarily how many mistakes they made, but whether people were willing to make mistakes visible enough for the organization to learn from them.

Her 1999 work defined team psychological safety around whether a group feels safe for interpersonal risk-taking and found a relationship between psychological safety and learning behavior. Questions, help-seeking, admitting mistakes and challenging assumptions all involve social risk because they can make a person appear ignorant, incompetent, disruptive or negative. The original paper, “Psychological Safety and Learning Behavior in Work Teams,” is available through Harvard.

That research gives AI leaders an essential piece of the current problem because a person can feel psychologically safe and still have nowhere useful to take a concern. They can raise an issue confidently while lacking authority to pause the process. They can be heard sympathetically by a manager who has neither the expertise nor organizational power to intervene. They can report the same problem repeatedly and discover that nothing changes.

Healthcare research makes this distinction visible. Studies repeatedly identify hierarchy, perceived ineffectiveness, fear of consequences and leadership response as determinants of whether people use their voice. Those findings matter because they show that voice depends on both the willingness of the speaker and the response architecture surrounding them.

One especially useful lesson from patient safety is that more reporting can coexist with better outcomes. In organizations that improve speaking-up behavior, reported incidents may increase because previously invisible conditions have become visible. That should make AI leaders very careful about dashboards that equate low reporting volume with low risk.

A healthy AI program may produce more bad news than an unhealthy one. It may generate more near misses, challenges, anomaly reports and interventions precisely because more of the system has become visible.

AI has an unusually difficult weak-signal problem

AI makes these old organizational dynamics harder because many AI failures begin ambiguously.

A customer-service representative notices that an agent has become unusually generous with a refund rule. A recruiter feels uncomfortable with the candidates an automated system keeps ranking lower. An engineer sees an agent access information it was technically allowed to access but cannot understand why that information was relevant to the task. An analyst notices that a summary repeatedly excludes the same category of information.

None of these observations necessarily arrives as an incident, but they may certainly begin as doubt.

That creates a requirement beyond generic psychological safety. People need permission to raise uncertainty before they have completed the diagnosis. Security teams already understand this pattern. We do not expect an employee to reverse-engineer suspicious code before reporting a possible phishing email. The signal enters the system while it remains uncertain, and people with greater expertise determine what it means.

AI programs often make the opposite mistake. The employee closest to the behavior may feel they need to prove that the model is wrong before challenging a system built and approved by people who appear to know considerably more about AI than they do.

This produces a dangerous threshold problem. When proof becomes the entry requirement for speaking, weak signals stay local until the evidence becomes undeniable. By then the interesting question is usually why nobody said something earlier.

The AI incidents are already giving us versions of the same lesson

The examples are accumulating quickly enough that this is no longer theoretical.

In the Air Canada chatbot case, a customer received incorrect information about bereavement fares from the airline’s chatbot even though the company’s published policy said something different. The British Columbia Civil Resolution Tribunal ultimately held Air Canada responsible for the inaccurate representation. The governance lesson was simple: the output of an automated interface still belonged to the organization operating it.

Amazon’s abandoned experimental recruiting system offers another version. Its machine-learning team discovered that the system had learned patterns that disadvantaged women in technical hiring, including penalizing resumes containing terms associated with women. Engineers attempted to remove particular signals but could not guarantee that the system would not find proxies elsewhere, and the project was eventually abandoned. The Guardian’s original reporting remains a useful account of the incident.

Agentic systems raise the stakes because the output can become an action.

In 2025, a Replit coding agent deleted production data during a declared code freeze. The incident became useful partly because a human operator observed the system doing something explicitly outside the intended boundary and documented what happened. Replit subsequently announced stronger separation between development and production environments, alongside additional rollback and safety measures. Replit’s own postmortem is the most useful source because it explains both the failure and the control changes that followed.

The story becomes stranger as autonomy increases. Anthropic disclosed that during testing one of its AI systems submitted a false homicide tip to Philadelphia police, alongside other examples in which systems interacted with government websites in unintended ways. The company changed its testing controls following the incidents. The important feature is not that one particular model made a bizarre mistake. It is that increasingly capable systems can move from generating an incorrect answer to executing an action in the world before anyone intended them to do so.

Anthropic’s own work on trustworthy agents makes the architectural point directly. Greater autonomy creates additional risk because agents act with less direct human oversight and have more opportunity to misinterpret intent or take unintended actions. Anthropic’s “Trustworthy agents” research is a strong primary source for this shift from model-output risk to delegated-action risk.

This is why “human in the loop” is becoming an inadequate description of the control.

The important questions concern what the human can see, what they understand, what they are allowed to do, whether they have enough time to do it, and what happens organizationally when they disagree with the machine.

Intervention authority is where culture meets control

AI governance documents increasingly assign humans oversight responsibilities. The NIST AI Risk Management Framework is explicit that organizations need appropriate accountability mechanisms, roles, responsibilities, culture and incentive structures, and that simply adopting a framework does not create those conditions.

The word empowered is doing a lot of work in these systems.

An organization can state that humans remain accountable while making meaningful intervention socially, technically or operationally difficult. Someone may technically have permission to stop an agent but know that doing so will miss the quarter-end deadline. A reviewer may be expected to challenge an AI recommendation while processing several hundred decisions a day. An employee may be authorized to escalate outside their management chain while having watched colleagues become labelled “difficult” for doing exactly that.

Formal authority and exercised authority can diverge dramatically. That gap deserves measurement.

A mature AI risk program needs to know whether people understand when intervention is expected, whether the interfaces and workflow make intervention possible, whether staff believe they will be supported when they act in good faith, and how frequently authorized intervention is actually exercised when relevant conditions arise.

A stop button that everybody knows they should avoid pressing is a beautifully implemented control failure.

We have also designed only half of escalation

The current AI governance conversation spends substantial effort on machine-to-human escalation. An agent reaches the edge of its authority, confidence falls, a threshold is crossed, or human judgment is required, so the system transfers the decision.

That is essential. There is a second escalation system operating in the opposite direction.

A human notices that the system itself may be wrong. The workflow may be inappropriate. The supervision architecture may be insufficient. The machine may be operating inside its permissions while drifting outside the purpose of the delegation. The local manager may be satisfied with the result while the person closest to the work can see a risk forming.

That concern has to travel from the human into the organization.

The distinction is important because the person spotting the problem may have very little positional authority. The frontline employee often has the richest contextual information and the weakest formal power.

Safety science has lived with this problem for decades. High-reliability thinking places particular importance on preserving sensitivity to operations and allowing expertise close to the problem to influence decisions rather than assuming hierarchy reliably maps to knowledge.

AI transformation programs create exactly the conditions that can interfere with that behavior. Executive sponsorship is strong. Productivity promises have been made. Budgets have been committed. Speed is celebrated. Employees quickly discover whether raising friction earns respect or marks them as the person who “doesn’t get AI.”

Culture is the memory of those consequences

This brings us back to the anthropological problem.

Humans enter organizations and learn the local rules. Some are written. Many are inferred. Schein’s distinction between artifacts, espoused values and deeper assumptions remains useful because organizations routinely confuse the first two with the third. A Responsible AI policy is an artifact. “We encourage responsible experimentation” is an espoused value. The assumption an employee carries into a difficult moment is learned from experience.

The relevant assumption may be that leadership genuinely wants bad news early. It may be that questioning AI is considered sensible professional skepticism. (It may also be that important programs are not to be slowed down, senior sponsors do not enjoy challenge, reporting rarely changes anything, or the safest way to survive uncertainty is to keep your head down.)

Those assumptions are part of the AI system because they change human behavior inside the AI system.

This is why human risk cannot be reduced to awareness training. Competence matters, but a fully competent person can still remain silent. Policy knowledge matters, but people routinely behave according to incentives and learned norms when policy and reality diverge. Technical safeguards matter, but technical safeguards themselves operate inside workflows governed by human authority, attention and judgment.

AI is a sociotechnical system, and sociotechnical systems require us to consider how the technology changes people and how people change the effective behavior of the technology.

The metric senior leaders need is risk signal flow

For a CISO, CIO, CAIO or senior risk leader, the practical consequence is that traditional adoption and incident metrics describe only part of the system.

The deeper question is whether meaningful risk information can travel.

We call this risk signal flow: the organization’s ability to detect, transmit, interpret and act on weak or adverse signals before they become material failures.

That means examining perception, behavior and organizational response together.

Survey data can tell you whether employees believe challenge is safe. Behavioral evidence can show whether people actually question, intervene and escalate. Process evidence can show whether concerns reach people with appropriate expertise and authority, how quickly they are addressed, and whether reporters ever hear what happened. Context tells you whether those numbers make sense given the amount and type of AI being used.

The gaps between those evidence sources are often more informative than the scores themselves.

High psychological safety combined with almost no reporting may point to weak recognition, unclear routes or low confidence that reporting achieves anything. Broad formal intervention authority accompanied by almost no exercised intervention may indicate cultural inhibition or poorly designed workflows. Very low incident volume in an organization using AI extensively may reflect outstanding control performance, or it may reflect poor visibility.

This is why the quietest AI program in the company does not automatically deserve the greenest dashboard tile.

Safety systems are valuable partly because they expose risk, and good governance should make the organization more capable of hearing uncomfortable information before it becomes expensive or dangerous. 

AI governance needs a human-risk architecture

The organizational task is larger than convincing employees to “speak up.”

People need sufficient AI literacy and contextual competence to recognize a meaningful anomaly. They need psychological safety and permission to surface uncertainty. They need reporting routes, including independent escalation where the immediate management chain has conflicting incentives. They need clear intervention authority, practical mechanisms for exercising it, and protection when good-faith intervention creates delay or inconvenience.

Managers and governance functions need the skills to receive those signals without demanding impossible certainty from the person raising them. The organization needs sufficient triage and technical expertise to interpret what it is hearing. Outcomes need to feed back into systems, policy, delegation design, supervision and learning.

Every response to a concern changes the probability that the next concern will be raised, and that is the part of culture AI governance cannot outsource to a statement of principles.

From responsible AI policy to human resilience

This is one of the reasons we built our AI Enablement & Culture Management (AIECM) and Agent Risk & Control (ARC) work around the human–AI system rather than treating people as the final line of an AI policy.

AIECM focuses on the workforce conditions required for AI to be adopted responsibly: literacy, competence, culture, leadership, behavioral norms, accountability and the practical ability of employees to operate safely as AI changes their work.

ARC moves further into agentic environments, where the problem becomes one of delegated authority, reliance, supervision, escalation, intervention and override. As systems gain greater autonomy, organizations need to understand the relationship between what machines are authorized to do, what they actually do, and whether humans remain capable of recognizing and correcting the difference.

Speak-up culture sits across both.

It affects whether people surface uncertainty while organizations are learning how to use AI. It affects whether employees challenge machine recommendations rather than drifting into passive reliance. It affects whether supervisors exercise intervention authority when agentic behavior moves beyond intent. It affects whether the organization sees the weak signals required to improve its controls.

We have spent much of the first era of enterprise AI asking how to make the technology trustworthy. The harder organizational problem might just be making the surrounding system capable of discovering when it is not.