An AI hallucination is a model output presented as factual but not grounded in the model's training data, the retrieved context, or reality. Hallucinations range from minor factual drift through to fabricated citations, APIs, and identities. They are an architectural property of probabilistic generation rather than a bug awaiting a patch.
- Output presented as factual but grounded in neither context nor reality
- An architectural property of probabilistic generation, not a fixable defect
- Failure shapes run from small factual drift up to wholly invented sources
- In agent settings a hallucinated tool call has real consequences
Why is AI hallucination important?
Hallucination matters because it is the AI failure mode most visible to non-technical stakeholders, and the one that most often kills enterprise AI deployments before security concerns even surface. A single fabricated citation in front of an executive does more to stall a program than a posture finding ever will.
In agent settings the character of the problem changes. A hallucinated answer is a credibility problem a reader can catch. A hallucinated tool call or a hallucinated parameter produces real-world consequences in production systems – an agent invoking a function that does not exist, or invoking a real one against a fabricated record identifier. That moves hallucination out of user experience and into security and reliability.
The framing that matters for a security team is that hallucination cannot be eliminated at the model layer. It is a property of how generation works. What can be done is bounding the consequences, which means controls at the point where output becomes action.
What is an AI hallucination?
An AI hallucination is an output a model presents with the same confidence as a correct one, which is not supported by anything it was given or trained on. The model is not lying, since it has no representation of truth to depart from. It is producing the most probable continuation, and probability and accuracy diverge.
Several shapes recur, and they escalate. Factual drift produces answers that are broadly right and wrong in specifics – a date, a figure, an attribution. Fabricated citations invent sources with plausible titles and authors. Fabricated APIs invent functions or endpoints that do not exist, which is common in code generation. Fabricated identities invent people, companies, or records that were never there at all.
The distinguishing feature is confidence. A model rarely signals uncertainty in proportion to its actual reliability, which is why hallucinations pass review – nothing in the output marks it as different from the correct material around it.
Retrieval reduces hallucination by grounding answers in real sources. It does not remove it, since a model can still misread retrieved content or blend it with training knowledge.
Types of AI hallucination
Hallucinations divide usefully by what is fabricated, because each type is caught differently.
Factual hallucinations misstate verifiable information. They are detectable by checking against a source, which requires knowing which claim to check.
Citation hallucinations invent or misattribute sources. They are detectable because a citation either resolves or does not, which makes them the most mechanically checkable type.
Structural hallucinations invent APIs, functions, parameters, or schema fields. In code and agent contexts these surface as errors at execution, which is late but at least visible.
Contextual hallucinations contradict material the model was actually given. These are the most concerning in RAG deployments, because grounding was supposed to prevent exactly this.
Action hallucinations are the agent-specific class: a tool call to a function that does not exist, or a real function called with fabricated arguments. The consequence depends entirely on whether anything evaluates the call before it executes.
AI hallucination & Onyx
Onyx treats hallucination resistance as a runtime control surface rather than a model-selection problem. Action-boundary inspection applies to tool calls and outputs, which is where a hallucinated call or a fabricated argument can be caught before it reaches a production system.
That placement is the substantive point. Improving a model's accuracy reduces how often hallucinations occur and never reaches zero, so the durable control is at the boundary between output and effect. Where a call is destructive or irreversible, escalation to a person is available as a policy outcome rather than a separate process. Hallucination is also listed among the attack-goal categories covered by Onyx Agentic Red Teaming, which is read-only, so agents can be probed for the conditions that produce it before deployment widens through AI Security.
