If You Cannot Ask Where a Number Came From, It Does Not Exist

There is an old rule in finance: a number whose source you cannot show is not a number.

An auditor sees a figure on a balance sheet, and their first instinct is:

Where does this come from?

Which document supports it? How was it calculated? Who produced it? Who performed the final check?

Because the number itself says little on its own. What makes it reliable is the trail behind it. Without that trail, even a correct number is not yet trustworthy.

Now put that same auditor in front of an AI.

The AI gives an answer. The figure is precise. The sentence is polished. The tone is confident.

The auditor asks the same question:

‘Where does this come from?’

This is where the real issue begins.

One of the least discussed issues in enterprise AI today is traceability, even before the accuracy of the answer.

A number may be correct. But if the system cannot show which document, dataset, or calculation supports it, then in financial terms, that number has not yet been audited.

The dangerous thing about AI is not that it makes mistakes. People make mistakes too.

The problem is that it often states what is wrong and what is right with the same confidence.

Whether it finds a number or guesses one, you see the same polished sentence on the screen.

So why does it make things up?

We often use the word ‘hallucination.’ But the term can make the problem sound stranger than it is, as though the system occasionally malfunctions and talks nonsense.

The issue is more fundamental.

The basic learning mechanism of large language models involves predicting the next piece of text. Later training stages also develop behaviors such as accuracy, instruction following, and safety.

But when a model does not have enough information, a fundamental problem remains:

What will it do with the gap?

The 2025 paper by OpenAI researchers Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang, Why Language Models Hallucinate, highlights an important issue: many evaluation systems reward correct answers more than expressions of uncertainty.

If ‘I do not know’ and a wrong answer lead to the same outcome, guessing becomes more attractive to the system.

Like a student who does not know the answer choosing an option on a test instead of leaving it blank.

The issue is not the model’s morality. It is the incentive structure.

That is why the central question in enterprise use should not be how intelligent the model is.

The more important question is:

What does it do when it does not know?

A better model is not always the solution

When companies encounter hallucinations, their first instinct is usually to move to a more powerful model.

We improve prompts. Change settings. Try a newer model.

These steps help.

But sometimes we are repairing the wrong layer.

Because the issue lies not only in the model’s capabilities, but also in the quality of the information it receives.

If a company’s price list contains three different figures in three spreadsheets, customer definitions vary by department, contracts are outdated, and nobody owns the question of which data is correct, expecting flawless AI answers is hardly realistic.

This is not an unfamiliar problem for a finance professional.

Bad records do not produce good financial statements.

However talented an analyst may be, disorganized and unreliable data limits the quality of their report.

The basic principle does not change with AI.

A system that can show, not merely guess

The goal of enterprise AI should not simply be to build a model that gives correct answers.

A more valuable goal is:

Build a system that can show the basis for its answers.

Grounding a model in verified documents and current corporate data significantly reduces the scope for fabrication.

But it is not enough on its own.

A model can find the right document and misinterpret it. Add a detail the document does not contain. Combine information from two sources incorrectly.

That is why I believe the critical rule for enterprise use is simple:

Every important claim must have support. If it does not, the system must say so clearly.

‘I cannot verify this information.’

Sometimes that is the best answer an AI can give.

It is not very different from telling a finance employee, ‘If you are not sure, do not sign.’

Building an internal control system for AI

The issue is actually more familiar than technical.

If you use AI inside an organization, you need to establish a form of internal control for it too.

The first step is to identify where the organization’s authoritative information resides.

What does the company sell? At what price? To which customer segment? Which contract is in force? Which figure is current?

Rather than being scattered across folders, personal computers, or old presentations, this information needs to be kept in an up-to-date source structure with clear ownership.

In a sense, the company must create its own master ledger for AI.

The second step is to make sources visible.

The model should not only provide an answer; where possible, it should also show the document or data on which that answer is based.

The third step is even more important:

Give the system permission to say, ‘I do not know.’

If the information is absent from the source, it should not guess. It should distinguish approximate information from confirmed information.

Finally, the answer must be checked again.

In smaller organizations, a person does this. In larger systems, automated validation layers can help.

It is the digital equivalent of one of accounting’s oldest control principles:

The person making the entry should not be the person checking it.

If AI produces the output, another mechanism should verify it.

Why is this not just an IT department issue?

Because ultimately, the issue is about management, not technology.

According to a Gartner forecast from 2025, a significant share of AI projects not supported by AI-ready data will be abandoned during 2026.

That is not surprising.

A company can buy the most advanced model. Commission the best integration. Build an impressive interface.

But if the company itself cannot agree on what its own data means, you cannot expect AI to resolve that.

Responsibility extends beyond the technology team.

Finance, operations, legal, data owners, and management must answer certain questions together:

Which information is authoritative?

Who approves it?

When is it updated?

When does old information cease to be valid?

Who owns a number?

AI cannot determine these things for you.

It can only work within the structure you define.

One question is enough

Sometimes you do not need dozens of technical questions to assess the reliability of an organization’s AI.

One question is enough:

‘Can you show me where this answer came from?’

If it cannot, the answer may be correct.

But it may also be a guess.

The real risk is that the two look very similar from the outside.

What we have known for years in finance applies here too:

A fast answer is valuable.

But an answer you can stand behind is far more valuable.

Just as a number without a source is merely a number, an AI answer whose source cannot be shown is merely an answer.

It is not yet knowledge.

Sources

• Kalai, A. T., Nachum, O., Vempala, S. & Zhang, E. — Why Language Models Hallucinate, OpenAI, 2025. arXiv:2509.04664.

• OpenAI — Why Language Models Hallucinate, 2025.

• Atlan — Work on enterprise data governance, context management, and AI reliability.

• Gartner — Lack of AI-Ready Data Puts AI Projects at Risk, February 26, 2025.