When AI doesn’t know: the importance of recognizing limits and avoiding hallucinations

The problem of hallucinations

Ask an artificial intelligence (AI) assistant a question, and it will almost always provide an answer, even if it doesn’t know the correct response. This phenomenon is known as hallucination and represents one of the most persistent challenges in AI technology: models that fabricate data, citations, and sources with the same confidence as they present factual information. A recent study identified cases where models generated entire court decisions, attributing them to non-existent proceedings.

Medical experience and the culture of doubt

For Peter Ruppersberg, the explanation lies in how these systems are built. Large language models are trained to predict the next word and continue a conversation, rather than to assess their own knowledge. “A fluent answer is not necessarily a good decision,” he said in an interview. Ruppersberg developed AI technology to guide doctors during cardiac procedures, which was sold to Boston Scientific in 2025. He now leads the Swiss startup Ammonix.

His experience in the hospital changed his perspective on the problem. “In medicine, you repeatedly learn that being right most of the time is not enough,” he said. “In an operating room, you can’t simply tell a doctor that the model thinks this is the answer. The important questions arise immediately: Why? What evidence supports this? And perhaps most importantly, does the system know when it doesn’t know?”

The most important lesson he took from this experience is that doubt is also a valid outcome. “In conventional software, people often think in terms of input and response. In medicine, a responsible system sometimes needs to produce: ‘I don’t have enough evidence to make this decision,’” he explained.

For this to happen, the system must understand the limits of its own knowledge. In Ammonix’s approach, AI operates within a defined set of decisions and compares each new situation to the cases it was trained on.

When a situation doesn’t resemble anything known or when confidence is low, the system abstains. Depending on the application, it may request more information, refer the case for further analysis, or consult a human expert. “Sometimes the safest and smartest response is simply: I don’t know this case well enough,” he said.

Industry responses and best practices

The industry has begun to address this problem. A study published by OpenAI in 2025 already indicated that traditional evaluation methods can encourage risky responses. If a model is primarily rewarded for being correct, it has an incentive to attempt an answer even without certainty, rather than admit uncertainty.

When launching Claude Opus 5.5, Anthropic conducted a test in which models were asked to write financial reports with an automatic checker verifying every number and citation. Any fabricated data would result in a failed report. According to the company, the new model passed 16 out of 18 attempts, while previous models failed all of them. However, this advancement does not yet eliminate the problem.

In the meantime, a practical tip for daily AI users is to treat very confident answers with the same caution as hesitant ones. Requesting sources, verifying numbers in original documents, and asking the assistant about its level of certainty can help reduce risk. Simply include a command like “If you don’t know, say you don’t know, don’t make it up” to change the model’s behavior. For decisions involving health, money, or legal issues, this rule applies to any tool: the greater the consequences of an error, the greater the need for human verification.