Consider a customer service assistant deployed by a retail bank. A user asks: "What is the maximum loan tenure for home mortgages?"
If the AI confidently responds: "35 years with zero down payment," when bank policy strictly caps tenures at 20 years with 30% down, the application has suffered a catastrophic failure. No database crashed, no API threw a 500 error code, yet the business has incurred severe operational liability.
This failure mode is hallucination: the tendency of probabilistic neural networks to generate syntactically convincing but factually unsupported statements.
Temperature = 0 does not solve truth
A common developer misconception is that setting an LLM's sampling temperature to zero eliminates hallucination. Lowering temperature makes generation deterministic (selecting the highest-probability next token), but if the underlying model lacks access to verified external facts, it will simply generate the exact same plausible lie deterministically every time.
The three layers of RAG failure modes
Grounded AI architectures prevent hallucination by enforcing a strict constraint: the model is instructed to answer only using the retrieved passages provided in the prompt context. However, hallucinations can still manifest across three structural layers:
- Source Layer Failure: The ingested internal knowledge base contains outdated, conflicting, or unverified documents.
- Retrieval Layer Failure: The vector search algorithm fails to fetch the correct document passage, forcing the model to operate on incomplete context.
- Generation Layer Failure: The LLM ignores negative constraints and extrapolates beyond the retrieved text.
Production enterprise AI systems must implement a explicit fallback state: when retrieved evidence confidence falls below threshold, the assistant must return a structured response: "I do not have sufficient internal documentation to answer this query reliably."