The Compliance Risk AI Cannot Talk Its Way Out Of
Why large language models are not safe as direct decision engines for tax and compliance outputs
The industry has already admitted the problem
The most important warning about using large language models in tax and compliance does not come from critics of artificial intelligence. It comes from the companies and institutions closest to the technology itself. OpenAI states that hallucinations remain a stubborn challenge for language models and describes them as plausible but false statements generated with confidence. Google Cloud explains that grounding can reduce the chances of inventing content, which is another way of admitting that the underlying risk remains part of the system rather than a cosmetic defect. Microsoft’s Azure OpenAI documentation says that even reproducible output with a seed is only a best effort and that determinism is not guaranteed. NIST, in its Generative AI Profile, uses the term confabulation for confidently presented erroneous or false content and describes it as a natural result of the way generative models are designed. The OECD, looking specifically at AI in tax administration, warns that transparency, explainability and interpretability problems can create risks for rule of law and taxpayers’ legal recourse. These are not marginal comments. They are the operating instructions for understanding the boundary of the technology.
That boundary matters because the current AI debate often confuses intelligence with reliability. A model may be impressive, fast, fluent and useful in many situations while still being unsuitable for an environment in which the correct result is not a preference but a legal obligation. Tax and compliance are exactly such environments. They do not tolerate an answer merely because it sounds reasonable. They require a result that can be repeated, audited, defended and reconciled with the applicable rule.
Fluency is not the same as legal reliability
Large language models are trained to produce language that is statistically plausible in context. This is the source of much of their power, because they can summarize, classify, explain and draft at a speed that changes how professionals work. It is also the source of their central weakness. The model does not know the law in the way a legal system knows the law. It generates a continuation based on patterns learned from data, guided by prompts, system instructions and model behavior. In many cases, this produces a correct or useful response. In other cases, it produces a wrong response that looks just as polished as a correct one.
This distinction is not academic. In tax, the form of the answer is often less important than its legal effect. A VAT rate applied to the wrong product, a fiscal receipt created with an incorrect sequence, a reporting obligation misunderstood by one field, an audit export produced with missing evidence or a local exception overlooked in a cross-border process can all create consequences that are disproportionate to the size of the error. Compliance systems are judged by what they do, not by how convincing their explanation sounds.
Hallucination is a structural risk, not a normal software bug
The common temptation is to treat hallucination like a defect that will disappear as models become larger, smarter and better trained. Better models will reduce the frequency of errors, and better product design can make many consumer and productivity use cases safer. That improvement should not be confused with elimination. OpenAI’s own description is precise: hallucinations still occur even as models become more capable. NIST’s explanation is equally important because it links confabulation to the way generative systems produce outputs that approximate statistical distributions. The issue is therefore not only whether a model has enough data or whether the prompt is well written. The issue is that the underlying mechanism remains probabilistic and generative.
In a normal business document, a non-zero risk of a wrong sentence may be acceptable because the cost of correction is limited. In a marketing draft, an internal memo or a first version of a policy summary, the model’s uncertainty can be managed by the user’s tolerance for imperfection. Tax compliance is different. The relevant standard is not whether the model is usually right. The relevant standard is whether the system can be trusted when the cost of one wrong output is regulatory exposure, penalties, rejected reporting, failed certification, customer disputes or damaged audit evidence.
Repeatability is a compliance requirement
Tax systems depend on repeatability. The same facts, the same transaction and the same rule must lead to the same result. This is not only a technical preference but a compliance expectation. A business must be able to explain why a tax amount was calculated, why a receipt was created in a certain form, why a transaction was reported in a specific field and why an audit trail contains the evidence it contains. If the same input can produce a slightly different answer in another run, the system may be useful as a drafting assistant, but it is not suitable as the source of a legally relevant compliance output.
Microsoft’s own documentation on reproducible output is useful here because it shows the problem in plain operational terms. Even when a seed parameter is used and other parameters are kept the same, determinism is described as a best effort rather than a guarantee. That may be acceptable for generating a story, a support draft or an analytical summary. It is not acceptable for tax logic. Compliance cannot be based on a system that is only mostly reproducible, especially where the output may trigger reporting, evidence creation, financial calculation or a legally relevant action.
Grounding reduces risk without removing the boundary
The industry often presents grounding as a way to make generative AI safer, and it can certainly improve the quality of responses. Google Cloud describes grounding as connecting model output to verifiable sources of information and says it reduces hallucinations by tethering output to those sources. That is valuable in many knowledge workflows, but the word reduce is doing important work. A reduced risk is not the same as a removed risk. In compliance, the difference between those two words is the difference between convenience and legal certainty.
Grounding also does not change the basic nature of the model’s output. The model may retrieve a source, quote a source, summarize a source or connect several sources in a useful way, but the final language remains generated. It can still select the wrong passage, miss an exception, combine rules incorrectly, misunderstand the transaction context, produce a misleading conclusion or present an answer with more confidence than the source supports. For tax and compliance, the danger is not only a hallucinated fact. It is also a plausible interpretation that crosses the line between explanation and obligation without the precision that the domain requires.
The risk is highest where the output becomes action
The safest way to understand LLM risk in tax and compliance is to focus on the distance between language and action. The risk is lower when the model produces a general explanation that a professional reads as background. The risk rises sharply when the model output becomes a calculation, a classification, a report, a filing instruction, a fiscal receipt decision, a refund treatment, an audit position or a compliance conclusion. At that point, the model is no longer helping someone think about a problem. It is shaping an outcome that may be examined by a tax authority, a customer, an auditor or a court.
This is where the hype around AI becomes dangerous. A model that performs well in a demonstration can still be unsuitable for business-critical operation. Demonstrations show that a model can produce a persuasive answer under selected conditions. Compliance requires confidence under all relevant conditions, including edge cases, outdated rules, conflicting sources, local exceptions, incomplete transaction data and changes in regulation. The harder the domain becomes, the more costly it is to mistake fluency for control.
Tax and compliance need certainty, not probability
The fundamental problem is therefore simple. Large language models can be useful, but they cannot be trusted as direct engines for tax and compliance outputs because hallucination is not fully removable inside the language model itself. The probability may become smaller. The model may become more cautious. The response may be grounded in a source. The prompt may be improved. The system may sound more professional. None of that changes the core fact that a non-zero risk remains, and in compliance a non-zero risk of an uncontrolled false result is not a detail.
This does not make AI irrelevant to tax technology. It makes the boundary of AI more important. The most responsible position is not to reject the technology, but to refuse the illusion that a fluent model can replace the certainty required by compliance. Tax is not a field where a plausible answer is enough. Compliance is not a field where probability can quietly replace determinism. The burden of proof is on the system that produces the result, and a language model output by itself cannot carry that burden.
The conclusion for compliance leaders
For executives, tax teams and software vendors, the message should be clear. LLMs should not be treated as reliable authorities for tax calculations, compliance decisions or legally relevant reporting outputs. They are not unsafe because they are weak. They are unsafe in this role because they are strong enough to sound convincing even when they are wrong. That is precisely what makes the risk difficult to detect.
The next phase of AI adoption will separate companies that understand this boundary from those that confuse experimentation with operational trust. In tax and compliance, the central challenge is not to make the AI sound smarter. The central challenge is to know where the AI must stop.
Sources and further reading
The following sources were used to validate and strengthen the article. They are included for editorial review and can be kept or removed before publication.
1. OpenAI, “Why language models hallucinate”. OpenAI states that hallucinations remain a hard problem, defines them as plausible but false statements generated by language models, and explains that training and evaluation often reward guessing rather than acknowledging uncertainty. Link
2. NIST, Artificial Intelligence Risk Management Framework: Generative AI Profile, NIST AI 600-1. NIST defines confabulation as confidently presented erroneous or false content and connects it to how generative models approximate statistical distributions of training data. Link
3. Google Cloud, “What are AI hallucinations?”. Google Cloud describes hallucinations as incorrect or misleading AI results and notes that they can be problematic when AI systems are used to make important decisions. Link
4. Google Cloud, “Grounding overview”. Google Cloud explains that grounding connects model output to verifiable sources and reduces the chance of invented content, while the formulation itself shows that the risk is reduced rather than eliminated. Link
5. Microsoft Learn, “How to generate reproducible output with Azure OpenAI”. Microsoft states that determinism is not guaranteed even when reproducibility parameters are used, which is directly relevant to repeatability requirements in compliance systems. Link
6. OECD, “AI in tax administration: Governing with Artificial Intelligence”. The OECD explains that transparency, explainability and interpretability challenges can create risks for rule of law and taxpayer legal recourse in tax administration. Link
