Why AI Gets Things Wrong With Such Confidence

You ask AI for the legal capacity of a venue, a figure from your own company, or a supplier’s price, and it replies with a specific number, written with total confidence. It sounds perfect. And it is false. Generative AI gets things wrong with such confidence because it is not looking for the truth: it predicts the most plausible text that follows your question. The confident tone is always the same, whether it is right or wrong.

This is not a rare glitch that gets fixed by updating to the latest version. It is how the tool works under the hood. And the good news for anyone running a business is that the error does not fall at random: it shows up in places you can anticipate. If you know where to look, you can keep using AI for what it does well without blindly signing off on what it makes up.

The short answer: it predicts, it does not consult

A large language model (an LLM: the engine behind tools like ChatGPT) is, in essence, a very advanced autocomplete system. It has read an enormous amount of text and learned which words tend to follow others. When you write it a question, it does not open a file with the correct answer. It calculates, word by word, the most probable continuation given everything it has read.

The unit it works with is the “token”: a piece of a word, a short word, or a symbol. The model keeps choosing the most probable next token, and then the next, until it forms a complete answer. Almost all the time, the most probable continuation matches the truth, because in real text true things are repeated a lot. That is why AI gets so much right and turns out to be so useful.

The problem comes when the most plausible continuation is not the true one. There the model fills the gap with something that sounds right: a believable figure, a source that looks real, a legal article with a number that fits. This has a technical name: hallucination. The system does not “lie” in the human sense of knowing the truth and hiding it: it generates plausible text without telling whether it corresponds to a fact.

Why it sounds so confident even when it is wrong

The confident tone is not tied to the truth. A language model writes with the same firmness whether it is right or making the answer up, because its goal is to produce fluent, convincing text, not to signal its own level of doubt. It has no internal meter that lights up with an “I really do not know this one”.

Compare it to a calculator. You type 2 times 2 and it always gives 4. It is a deterministic machine: the same inputs always produce the same output, and that output is correct by design. An LLM does not work like that. Faced with the same question it can give you different answers at different moments, and none of them comes with the calculator’s guarantee. Mistaking the confidence of the tone for the reliability of a calculator is the error that costs the most.

For a decision-maker, the practical lesson is direct: the confidence with which the AI speaks to you tells you nothing about whether it is right. A nervous intern who hesitates out loud may be giving you a correct figure, and a flawless, confident answer may be pure invention. Judge the content, never the tone. If you want the detail on why it makes up specific data, I explain it in what an AI hallucination is.

The three places where the error is predictable

The failure is not random. It concentrates in three situations you can recognize before trusting an answer.

One: you ask it for private data it has never seen. The model learned from public text. It does not know your company’s internal figures, your contracts, your inventory, or the notes from your last meeting, unless you give them to it in the conversation. If you ask it “how much did my Seville branch bill last quarter?”, it has no way of knowing, but it will give you a number anyway, because its job is to complete the sentence.

Two: you demand an exact figure. Specific dates, prices, capacities, legal deadlines, article numbers in a regulation. These are precise, verifiable data points where there is no room for “close enough”. The model produces something with the right format (a date that looks like a date, an article with a believable number) that may be wrong. The format is right, the data is not.

Three: the question falls outside its domain. Very niche, very local, or very recent topics, ones it has barely read about. The less reliable text exists about something, the more the model leans on filling the gap with what sounds reasonable. A regulatory change from last week or a particularity of your sector can land right in that zone.

Situation	What the user asks	Why it fails	What to do
Private data	Internal figures or facts about your company	The model has never seen them	Give them to it yourself in the conversation
Exact figure	Dates, prices, capacities, legal articles	It fills in with a plausible value	Check the original source
Out of domain	Niche, local, or very recent topics	It has read little reliable material on it	Cross-check with a human expert

What this means for your company

The conclusion is not to stop using AI. It is worth using where the cost of an error is low and there is human review, and being wary where a false data point does damage. The same tool is excellent for some things and unsafe for others, and you set the difference by deciding where you put it.

Where it performs well: first drafts of an email, summaries of a long document, brainstorming, rewording a text, translating the tone of a message. In all these cases a human reads the result before it goes out, and a slip is corrected without consequences.

Where it is worth slowing down: figures that go to a client or an offer, legal compliance text, decisions that get executed without anyone reviewing them, any number that someone will later take as certain. Here the AI’s confident tone is exactly what is dangerous, because it invites you to hit “send” without checking.

Deciding that boundary with judgment, in your context and without depending on someone on the team who “knows about this stuff”, is what we work through step by step in the IA sin hype course. The goal is for you to direct the tool with judgment, knowing when to trust it and when to stop, without needing to become technical.

How to reduce the risk without being technical

You do not need to understand the engine to drive with care. With a few simple habits you cut out most of the costly errors:

• Ask for sources and links, and open them. If the model cannot cite where a data point comes from, treat it as a draft, not a fact.
• Provide the context yourself. Paste the document, the data, or the real figures into the conversation instead of expecting it to guess them. Giving it your own data is the most reliable way to reduce the first type of error, something I go into in how to give AI your company’s data.
• Leave the door open to “I do not know”. Explicitly ask it to answer “I do not have that data” when it does not have it, instead of forcing it to invent.
• Check every figure before using it. Any number that goes to a client, an invoice, or a legal document gets verified against the original source.
• Do not automate sensitive decisions without a human reviewing. The greater the damage of an error, the more justified the review step.

If you want to better understand the engine underneath all of this, the starting point is how generative AI works, the base guide in this series.

One new concept every week

Subscribe

Frequently Asked Questions

Does AI lie on purpose when it gets things wrong?

Not in the human sense. Lying means knowing the truth and hiding it, and a language model has no record of facts to consult. It generates the most plausible answer word by word, and when that answer does not match reality, the result looks like a lie even though there is no intent behind it. That is why it is more useful to talk about error or hallucination than about lying.

Why does AI make up sources, citations, or links?

Because a citation has a very recognizable shape: an author, a title, a year, a web address. The model has seen thousands of citations and knows how to imitate that pattern perfectly, even when the specific combination does not exist. It produces something with the flawless format of a real source. That is why any reference it gives you has to be opened and checked to exist before you rely on it.

Will this be fixed by more advanced models?

Newer models get more right and make up fewer things, but the underlying mechanism is still predicting plausible text, so the risk does not fully disappear. It improves the average, not the guarantee. It is worth treating human review as a fixed part of the process, not a temporary patch until the next version.

Can I trust the numbers AI gives me?

As a starting point yes, as a final figure not without checking it. A number can be correct or it can be an invention with the right format, and at a glance you cannot tell them apart. For any figure that goes to a client, an offer, or a legal document, verify it against the original source before using it.

Is it the same as an internet search engine?

No. A search engine returns pages that exist and that you can open to check the source. A language model generates a new answer from what it learned, without guaranteeing that it corresponds to any real page. Some tools combine the two things and search the web before answering, which helps, but it is still worth reviewing the links they cite.