Artificial intelligence changes impersonation and social engineering risk by making convincing deception easier to produce, personalize, scale, and adapt. Generative AI can help attackers create credible messages, synthetic voices, manipulated images, and fabricated information that exploit how people establish identity and trust. At the same time, organizations are introducing AI into legitimate business workflows, creating new questions about what employees should verify, which decisions they can delegate, and how they maintain control over consequential actions.
For cybersecurity leaders, the challenge extends beyond teaching employees to recognize AI-generated content. Organizations need to understand where their business processes depend on human judgments about identity, authority, credibility, and authorization. They then need to establish verification controls that remain effective even when a message sounds familiar, a video looks authentic, or a request contains convincing details about the business.
This is becoming an important intersection of cybersecurity, fraud prevention, Human Risk Management, and AI governance. The technologies involved may be changing rapidly, but the underlying question is familiar: when someone or something asks us to act, how do we establish whether that action is legitimate?
AI is changing social engineering by reducing the effort required to create persuasive communications, imitate trusted individuals, and tailor attacks to specific targets. Attackers can use generative tools to draft messages, translate content, develop plausible pretexts, and produce synthetic media. Some AI-enabled tools can also support reconnaissance, interaction, and adaptation during an attack.
Historically, many phishing and impersonation attempts contained clues that made them easier to recognize. Poor grammar, awkward phrasing, generic greetings, and inconsistent branding were common warning signs. Those clues were never reliable indicators of safety, but they became familiar features of cybersecurity awareness training.
Generative AI makes that style of detection increasingly inadequate.
A fraudulent supplier email may now use polished business language and accurately reference a procurement process. A voice message may resemble a known executive. A fabricated video call may create the impression that the person requesting an action is physically present and speaking directly to the employee.
The danger is that people naturally use familiarity, coherence, and apparent credibility when deciding whether to trust information. AI can make those signals easier to imitate.
Organizations therefore need to distinguish between evidence that something appears authentic and evidence that a request has been independently verified.
AI can increase the potential effectiveness of phishing and impersonation attacks by improving the quality, relevance, and scalability of fraudulent communications. However, the effectiveness of a particular attack still depends on the target, the circumstances, the controls in place, and whether the attacker can persuade someone to take a consequential action.
It is important to avoid assuming that every sophisticated phishing campaign uses AI or that AI automatically makes an attack successful.
Cybercriminals have used convincing impersonation, compromised accounts, supplier relationships, and organizational knowledge for years. AI adds capabilities that can make some of those activities faster or more accessible, but the fundamental mechanisms of deception remain recognizable.
Microsoft's October 2026 action against industrialized impersonation services illustrates the scale at which cybercriminal infrastructure can support fraudulent communications. The operation targeted services associated with more than a billion malicious messages directed at tens of thousands of organizations. That scale is significant, although it should not be interpreted as evidence that every message involved generative AI.
We examine the broader implications in What Microsoft's Billion-Message Cybercrime Takedown Reveals About Human Risk.
The lesson for defenders is that attackers can industrialize the delivery of deception while increasingly improving its presentation. Organizations cannot build their defenses around the expectation that fraudulent communications will always look suspicious.
Organizations should prepare for AI-enabled attacks that impersonate trusted people, organizations, systems, and business relationships.
Several scenarios are particularly relevant to enterprise risk.
An attacker may use AI-generated messages or synthetic voice recordings to imitate a CEO, CFO, or other senior executive. The request may involve an urgent payment, disclosure of sensitive information, or an exception to an established procedure.
The attack exploits both apparent identity and organizational authority. Employees may feel pressure to respond quickly because the request appears to come from someone senior.
AI-generated communications can make fraudulent supplier requests more polished and contextually convincing. Attackers may impersonate established contacts, fabricate explanations for banking changes, or imitate the language and format of normal procurement correspondence.
These scenarios are particularly dangerous when the organization relies on familiar email threads or recognizable writing styles as evidence of legitimacy.
An attacker may impersonate an employee seeking urgent assistance with account access, authentication, or device recovery. Synthetic voice, personal information, and convincing explanations can increase the apparent credibility of the request.
The important control question is whether identity recovery procedures establish identity independently of the caller's presentation.
Organizations may encounter fabricated candidate identities, manipulated interview materials, synthetic media, or fraudulent employment-related requests. These attacks can affect recruitment, onboarding, access provisioning, and the handling of personal information.
The risk extends into the procedures used to establish identity and grant access to company systems.
Attackers may impersonate customers or authorized representatives to request changes to account details, contact information, access permissions, or payment arrangements.
Employees need reliable ways to distinguish a convincing explanation from a properly authorized request.
These scenarios share a common weakness: a person may accept the appearance of legitimacy as sufficient evidence to act.
Deepfakes are a cybersecurity concern because synthetic audio, video, and images can undermine the visual and auditory cues people traditionally use to recognize trusted individuals.
A familiar voice can create a powerful impression of authenticity. A video call can appear to provide direct confirmation that a request comes from the expected person. A realistic image may seem to support a story or establish an identity.
Yet none of those signals necessarily proves that the communication is genuine or that the requested action is authorized.
Deepfake detection technologies can help identify some manipulated media, but detection alone is not a sufficient business control. The reliability of detection depends on the content, techniques involved, operating conditions, and capabilities of the tools being used.
For high-risk activities, organizations need verification methods that do not depend entirely on deciding whether a voice or video looks real.
A finance employee receiving a voice request from someone who sounds like the CFO should still follow the required payment authorization procedure. A help-desk analyst interacting with someone who appears to be an executive should still apply established identity verification controls.
The practical defense is to make the legitimacy of the transaction independently demonstrable.
This is also why deepfake preparedness should include realistic employee scenarios, clear escalation paths, and communications that explain how existing business controls apply to new forms of impersonation.
Detecting AI-generated content is difficult to treat as a definitive security decision because authentic communications may contain AI-generated material, while fraudulent communications may be entirely human-written.
An employee may receive a legitimate email drafted with an AI assistant. A criminal may send a carefully written message without using AI at all. The presence or absence of AI-generated language does not establish whether the sender has authority to request an action.
The same problem applies to synthetic media.
A suspicious voice recording may be generated by an attacker, but a genuine executive may also use an AI voice tool for legitimate communications. The relevant business question is whether the requested transaction, access change, or disclosure is authorized.
This distinction has important implications for training.
Employees should understand that unusual language, manipulated media, and unexpected requests can be warning signs. However, they should also know that a polished, familiar, or apparently authentic communication may still require independent verification.
Organizations should teach employees to evaluate the request and its consequences rather than relying on their ability to identify the technology used to create it.
Authentication, verification, and authorization are related concepts, but they answer different questions.
Authentication establishes that a person or system has satisfied an identity-checking mechanism, such as a password, passkey, or multifactor authentication process.
Verification establishes whether a particular claim, request, or transaction is legitimate using appropriate evidence. In an impersonation scenario, this may involve confirming a request through a trusted, independently established channel.
Authorization establishes whether the identified person or system has permission to perform the requested action.
These distinctions become important when a fraudulent request appears to come from a legitimate account.
A compromised supplier mailbox may pass ordinary email authentication checks. A message may originate from an account belonging to a real employee. An AI agent may have legitimate credentials to access a system.
None of those facts automatically establishes that a particular payment, disclosure, account change, or delegated action is appropriate.
Organizations need controls that establish legitimacy and authority at the point where a consequential decision is made.
Our guide to designing verification processes to prevent impersonation fraud examines how independent verification, approval requirements, and escalation procedures can be incorporated into business workflows.
Organizations should update verification procedures by identifying where high-risk decisions rely on communications or evidence that AI can convincingly imitate. They should then establish independent confirmation methods, clear authorization requirements, and practical escalation procedures.
The objective is to make a successful fraudulent transaction depend on more than the attacker producing a convincing message or impersonating a familiar person.
For example, a supplier banking change should require verification through contact information already established as trustworthy. An urgent executive payment request should remain subject to appropriate approval and financial controls. A help-desk account recovery request should follow identity-proofing procedures that cannot be bypassed because the caller sounds familiar.
Verification also needs to be usable.
A procedure that requires employees to obtain confirmation from an unavailable contact may encourage informal workarounds. A policy that provides no escalation path for urgent exceptions may leave employees choosing between business pressure and security requirements.
Effective verification processes anticipate those conditions.
Organizations should also evaluate whether employees know how to perform the required checks and feel comfortable applying them when a requester appears authoritative or impatient.
That connects verification design to competency, behavior, and culture rather than treating it as a purely technical requirement.
AI can affect business email compromise by making fraudulent requests more credible, improving the personalization of messages, and supporting impersonation across multiple communication channels.
A criminal may combine an email request with a synthetic voice message, fabricated supporting documents, or convincing background information. The communication may appear consistent with the organization's normal business activities.
The financial exposure, however, still depends on the business process.
A fraudulent message cannot redirect a payment unless the attacker can persuade someone to change banking information, authorize a transaction, or bypass a control. That makes supplier master-data management, independent verification, segregation of duties, payment authorization, and escalation essential parts of BEC prevention.
Our guide to reducing business email compromise risk in finance and supplier workflows examines those controls in detail.
AI increases the importance of understanding where people make consequential decisions and what evidence they use to justify those decisions.
Organizations that already have strong financial controls should test whether those controls remain effective when employees encounter more convincing impersonation attempts. Organizations with inconsistent procedures may find that AI-enabled deception exposes weaknesses that have existed for years.
Employees should verify high-risk requests through trusted procedures and independent sources rather than relying solely on the apparent authenticity of a message, voice, or video.
The appropriate method depends on the action being requested.
For a supplier payment change, the employee may need to contact the supplier using previously validated contact information. For an account recovery request, the help desk should use the organization's approved identity-proofing procedure. For a sensitive disclosure, the employee may need to confirm the requester's authority and the approved destination for the information.
The employee should also understand what to do when verification cannot be completed.
An attacker may create urgency or claim that normal procedures are temporarily unavailable. Those circumstances should increase the need for appropriate escalation rather than justify abandoning the control.
This can be challenging in practice.
People generally want to be helpful, efficient, and responsive. They may also feel uncomfortable questioning someone who appears to be a senior executive, an important customer, or a familiar colleague.
That is why verification training should include realistic interpersonal pressure and the language employees can use to explain a required check.
Our guide to verification behavior in cybersecurity explores how organizations can evaluate whether employees recognize verification requirements, choose appropriate methods, execute the required actions, and escalate uncertainty.
AI can change the human risk of trusting information by making content appear coherent, authoritative, and personalized even when its underlying claims are unreliable or deliberately deceptive.
People use numerous signals to judge credibility. Familiar language, professional presentation, confidence, consistency, and apparent expertise can all influence whether information is accepted.
Generative AI can reproduce many of those signals.
This creates risks beyond traditional phishing.
An employee may trust an AI-generated explanation because it is fluent and detailed. A manager may accept an automated recommendation because the system has performed well in previous situations. A team may gradually stop independently checking outputs because doing so seems inefficient.
The organization therefore needs to consider how employees develop trust in AI systems and how that trust affects their decisions.
Appropriate reliance requires understanding what the system can do, where its limitations matter, and which claims or actions require independent evidence.
Cybermaniacs examines these questions through its work on human competency, psychology, behavior, and AI-enabled decision-making. The challenge is to understand when trust supports effective work and when it becomes a source of exposure.
AI agents introduce human risk questions because they may perform tasks, access information, interact with systems, and initiate actions on behalf of employees or organizations.
The employee's responsibility can shift from performing each step directly to delegating work, establishing constraints, supervising execution, reviewing results, and intervening when necessary.
That changes the control environment.
An employee may authorize an agent to collect supplier information, prepare a transaction, modify records, or communicate with third parties. The risk depends on what the agent can access, which actions it can perform, what evidence the employee receives, and where human approval is required.
An agent may also encounter malicious instructions embedded in external content, including emails, documents, webpages, or retrieved information. These prompt-injection scenarios can attempt to redirect the agent away from its authorized task.
Organizations need to examine both the technical controls surrounding agents and the human responsibilities associated with their operation.
For example, a manager may need to understand which actions an agent can perform autonomously, which require approval, and how to recognize when the agent's behavior falls outside expected boundaries.
An employee reviewing an AI-generated supplier recommendation may need to verify important claims before approving a change.
These are human risk questions because the effectiveness of the control depends partly on what people understand, how they behave, and whether the organization has established meaningful oversight.
AI agents can introduce additional impersonation and authorization risks when systems act using delegated credentials, communicate on behalf of people, or interact with other automated systems.
A message generated by an authorized agent may appear to represent an employee or business function. A malicious actor may attempt to imitate that agent, manipulate the information it consumes, or exploit weaknesses in how its actions are authorized.
The organization needs to establish how automated identities are authenticated, how delegated authority is constrained, and how consequential actions are approved and recorded.
Employees also need to understand the difference between a system being technically permitted to perform an action and the action being appropriate in a particular business context.
An agent with access to supplier records may be able to propose or prepare a banking change. That does not mean every requested change should be accepted without independent validation.
The principle is familiar from conventional impersonation prevention: the apparent source of an instruction and the legitimacy of the requested action are separate questions.
Agentic systems make that distinction more important because a single delegated workflow may involve multiple systems, sources of information, and automated decisions.
Human oversight should be designed around the decisions and actions that people are expected to control, rather than assuming that placing a human somewhere in a workflow automatically makes the system safe.
The organization should establish what the human needs to know, what evidence they must review, which decisions they retain, when intervention is required, and what authority they have to stop or correct an action.
Oversight should also be proportionate to the consequences of the workflow.
An employee reviewing a low-impact draft may need relatively little intervention. A person approving an AI-assisted payment, access change, or critical operational decision may require more rigorous verification, authorization, and review.
The quality of oversight depends on whether the human can meaningfully perform the assigned responsibility.
If the system produces too many decisions for effective review, provides inadequate evidence, or makes intervention impractical, the presence of an approval button may offer little assurance.
Organizations should assess the competencies, behaviors, workload, authority, and organizational conditions associated with oversight.
This is an emerging area of Human Risk Management because AI changes both the work people perform and the evidence needed to understand whether human controls remain effective.
Organizations should assess AI-enabled social engineering risk through the business processes where deception could influence consequential human decisions.
The assessment should identify the relevant threat scenarios, the employees and roles exposed, the actions attackers want to induce, and the controls intended to prevent those actions.
It should then evaluate whether employees have the necessary competencies, perform the expected behaviors, and operate within conditions that support effective control execution.
A finance assessment might examine whether employees independently verify unusual supplier changes even when requests include convincing documents or synthetic voice confirmation.
A help-desk assessment might evaluate whether analysts follow identity-proofing requirements when callers provide credible personal details and appear to be senior employees.
An AI-enabled procurement workflow might require assessment of how employees verify agent-generated recommendations, review delegated actions, and respond to unexpected results.
These scenarios should be connected to actual business responsibilities rather than treated as generic AI awareness exercises.
Our guide to assessing human risk in high-risk business processes explains how competency, behavior, psychology, culture, role exposure, and process conditions can be evaluated together.
The objective is to determine where the organization is vulnerable, why the vulnerability exists, and what intervention would reduce the risk.
Organizations can evaluate preparedness through competency assessments, realistic scenarios, observed verification behavior, escalation performance, and evidence from relevant business processes.
Knowledge assessments can establish whether employees understand synthetic media, impersonation techniques, and the limitations of familiar authenticity cues.
Scenario-based exercises can evaluate whether they recognize when verification is required and select the correct response. Operational evidence may help establish whether verification and approval controls are performed during actual transactions.
Cultural assessment can reveal whether employees feel comfortable challenging apparent authority or delaying suspicious requests.
These forms of evidence should not be collapsed into a single measure without a defensible interpretation model.
An employee may understand deepfake technology yet fail to challenge a convincing executive request. Another may have limited technical knowledge of synthetic media but consistently follow strong verification procedures.
The second employee may be better protected in the particular business scenario being assessed.
That distinction matters because the purpose of Human Risk Management is to understand and improve the conditions that influence security outcomes.
Cybersecurity awareness programs should evolve to address the decisions employees make in an environment where communications, identities, and information can be convincingly manipulated.
Employees still need to understand phishing, impersonation, and common warning signs. However, training should increasingly emphasize independent verification, appropriate reliance, authorization, escalation, and the consequences of actions within specific business workflows.
The content should also reflect the organization's AI adoption.
Employees using generative AI need to understand information handling, reliability, and verification responsibilities. Employees supervising AI agents may require additional competencies related to delegation, oversight, intervention, and accountability.
A single annual AI awareness course is unlikely to address the full range of these requirements.
Organizations should consider role-based learning, realistic exercises, targeted communications, and ongoing measurement that reflects how employees encounter AI-related risks in their work.
Cybermaniacs supports these needs through its Cyber Learning Experience, Human Risk Assessments, and targeted engagement and culture programs.
Our work in AIECM (AI Enablement & Culture Management) and ARC (Agent Risk & Control) extends this approach into AI-enabled and agentic environments, connecting workforce preparedness to the responsibilities people hold within evolving business processes.
Cybermaniacs helps organizations assess and improve the human competencies, behaviors, cultural conditions, and operational practices that influence cybersecurity and AI-related risk.
Our Human Risk Management approach begins with understanding the relevant workforce populations, business activities, risk scenarios, and controls. From there, organizations can identify where people need additional competency, where behavior is inconsistent, and where processes or organizational conditions undermine the intended safeguards.
Our Human Risk Assessments and Baselines provide a foundation for identifying and interpreting those conditions. Our learning, communications, culture, and advisory capabilities help organizations translate assessment findings into practical interventions.
For AI-enabled work, AIECM addresses the human dimensions of AI adoption, enablement, and organizational culture. ARC extends the focus to the responsibilities and controls associated with human-agent interactions, delegated work, and agentic operations.
The common objective is to help organizations understand how human decisions affect risk and establish the competencies, behaviors, and controls needed for resilient operations.
Social engineering has always exploited the gap between what appears legitimate and what has actually been established as legitimate.
AI makes that gap harder to recognize.
A familiar voice may be synthetic. A professional email may be fraudulent. A convincing document may contain fabricated information. An AI agent may perform an action using legitimate access while relying on manipulated instructions or incomplete evidence.
The response cannot depend entirely on employees becoming expert detectors of every new deception technique.
Organizations need reliable verification and authorization procedures, clear escalation paths, role-specific competencies, and cultures that support people when they question apparently legitimate requests.
They also need to understand how AI changes the decisions people make within ordinary business processes and the responsibilities they retain when work is delegated to automated systems.
The most useful question for security leaders is therefore not simply whether employees can recognize AI-generated deception. It is whether the organization's people and controls can establish legitimacy, exercise appropriate authority, and prevent consequential actions when the evidence presented to them cannot be taken at face value.
AI can increase social engineering risk by making fraudulent communications easier to produce, personalize, scale, and adapt. Generative AI can support convincing impersonation through polished messages, synthetic voices, manipulated images, and fabricated information, making it more difficult for employees to rely on familiar warning signs.
Employees may recognize some suspicious characteristics, but identifying AI-generated content is not a reliable standalone defense. Legitimate communications may use AI, and fraudulent communications may be entirely human-created. Organizations should emphasize independent verification and authorization controls for consequential requests.
Effective defense combines technical security controls with independent verification, appropriate authorization, role-specific training, realistic testing, and clear escalation procedures. High-risk actions such as payment changes, account recovery, and sensitive information disclosure should not depend solely on the apparent authenticity of a message, voice, or video.
AI can make business email compromise attempts more credible through personalized messages, convincing explanations, synthetic media, and coordinated communications. Organizations should protect finance and supplier workflows with independent verification, strong payment authorization, secure identity practices, and employee training based on realistic business scenarios.
AI agents introduce risks associated with delegated authority, reliance on automated outputs, inadequate oversight, prompt injection, and uncertainty about when people should intervene. Organizations need to define human responsibilities, establish authorization boundaries, provide appropriate evidence for oversight, and assess whether employees can perform their assigned control activities.
Organizations should evaluate employee competency, performance in realistic scenarios, verification and escalation behavior, and the organizational conditions that influence security decisions. Findings should be interpreted in the context of roles, business processes, controls, and the quality of the available evidence rather than relying solely on training completion or phishing click rates.