There is an obvious irony in using AI to help draft an article about validating AI-generated analysis, but it also makes the point fairly well.
I use AI regularly in analytical work. It helps me write SQL and Python, work through unfamiliar libraries, generate hypotheses, check formulas, explain code and get through repetitive work faster. I am very happy for it to do those things, but I do not assume an answer is correct because the code ran, the chart looks sensible or the explanation sounds confident.
AI can make analytical work much faster, including producing the wrong answer much faster.
Before I trust AI-assisted analysis, I work through the same basic checks I would apply to another analyst's work, a spreadsheet model or code I wrote myself. The main difference is that AI makes it unusually easy to jump from a question to a plausible-looking answer without seeing much of the reasoning in between.
1. Check what the data actually represents
Before checking a calculation, I want to know whether the dataset has been interpreted correctly.
What does one row represent? What period does the data cover? What is being counted? Are the measures additive? What has been excluded, and do similarly named fields actually mean the same thing?
A model can write perfectly valid code against the wrong interpretation of a field. If a column called sales contains gross sales in one source and net sales in another, the code can still execute. If customer_id represents an account rather than an individual customer, a customer count can still look reasonable. If one table is at transaction level and another is already aggregated, a join can quietly multiply the result.
So the first question is not whether the AI calculated something correctly, but whether we gave it the right analytical problem in the first place.
This is also why I tend to start data-quality work with definitions rather than null checks. A technically clean dataset can still be conceptually wrong.
2. Reconcile the result before explaining it
One of my favourite checks is also one of the least sophisticated: add everything back up.
If I have analysed individual products, do they reconcile to the category? If I have projected thousands of local areas, do they produce a believable national population? If I have calculated customer segments, do the segments reconcile to the total customer base?
If they do not, I am not particularly interested in explaining the result yet.
I ran into this while working with Australian Census data. I had built population projections at a very granular geographic level and many of the local results looked plausible on their own, but when I rolled everything back up Australia suddenly had roughly 52 million people instead of around 26 million.
This was not an AI error but a modelling error, although the validation principle is exactly the same: a result can look reasonable row by row and still be completely wrong as a system.
I changed the projection approach and kept reconciling the granular output against broader totals. I use the same test with AI-generated analysis because before asking what a result means, I want to know whether the parts still equal the whole.
3. Read the generated SQL or Python
Successful execution is not a validation test.
If AI generates SQL or Python for an analysis, I read it and pay particular attention to joins, filters, grouping logic, date conditions, denominators and the treatment of missing values.
I want to know whether a join can create duplicates, whether the denominator is correct, whether nulls are being excluded or treated as zero, whether the comparison period is genuinely like-for-like, and whether something has been hardcoded that should have been calculated.
I also look for assumptions the model has introduced without being asked. AI will sometimes fill in an ambiguity because producing something feels more useful than stopping to ask a question, and those assumptions can disappear very easily inside otherwise sensible-looking code.
I do not need to rewrite every line myself because that would defeat much of the point of using the tool, but I do need to understand the important logic well enough to explain why the output should be trusted.
4. Check sample records and edge cases
Aggregate numbers can hide some very strange behaviour, so I usually trace a handful of records manually through the logic.
I deliberately choose awkward cases: missing values, very small groups, unusually large values, products that changed hierarchy, dates around the edge of a reporting period or customers that appear in more than one place.
AI-generated code often handles the normal case perfectly well, while the problems sit around the edges. A calculation might work for almost every record but fail when a denominator is zero, a category mapping might break for products introduced halfway through the year, or a time-series comparison might accidentally include a partial week.
Those exceptions may barely move the total, or they may change the entire conclusion. Looking at individual records is usually a quick way to find out which.
5. Separate the calculation from the interpretation
An AI model can move very quickly from "sales declined 12%" to "this suggests weakening customer demand", even though the second statement does not follow automatically from the first.
There could be fewer stores ranging the product, an out-of-stock issue, a price change, less promotional support, a competitor launch or customers moving to another product in the same range.
I therefore validate what happened first and investigate why separately.
This mattered in a BIC product range that declined 48% within one month. Sales data established the scale of the problem, but it did not explain the cause, and the obvious explanations such as distribution, price and product quality did not fit what was happening.
I combined scan data, consumer panel data and custom shopper research. The panel showed previous buyers switching away, while the research explained why: the packaging had changed from transparent to opaque and shoppers could no longer clearly understand what they were buying.
An AI model could have generated a long list of plausible explanations from the initial 48% decline, but the useful part of the analysis was eliminating those explanations against the evidence until there was a defensible cause.
6. Compare the answer with what is happening outside the dataset
Sometimes an output is statistically correct and commercially ridiculous.
That is where domain knowledge and organisational context matter. If an AI-generated result says a product is declining because of weaker demand, does that fit what happened with distribution? If a customer metric suddenly changes, did the business alter the process used to capture it? If a category appears to accelerate, did its definition change?
A result that conflicts with everything the business knows is not automatically wrong. Unexpected findings can be useful, but they should trigger investigation rather than immediate confidence.
AI only sees the context it has been given. It does not automatically know what happened in last week's sales meeting, which retailer changed its range, which internal definition was quietly updated or which operational process broke.
Those details often determine whether a technically correct piece of analysis is actually useful.
7. Ask how the conclusion could be wrong
One of the more useful ways I use AI during validation is to ask it to attack its own analysis rather than keep strengthening the same conclusion.
What assumptions does this depend on? What alternative explanations fit the evidence? What additional data would separate those explanations? What result would contradict the current interpretation? Where is the analysis most sensitive?
I find those questions more useful than simply asking the model to "double-check" its answer, because a model that has already produced a conclusion can quite easily produce another confident explanation of why that conclusion is correct.
Asking for failure modes gives me a better list of things to test, although it still does not replace independent validation.
8. Keep the model's role visible
The more important the analysis becomes, the more useful it is to know which parts came from deterministic logic and which parts involved model judgement.
Revenue divided by units does not need an LLM. A reconciliation check should also be deterministic. Classifying hundreds of open-text responses or generating possible explanations for an anomaly may be reasonable uses of a model.
Keeping those jobs separate makes the workflow easier to inspect and reduces the risk of an LLM becoming one large black box between the source data and the recommendation.
I used a similar principle when building Revisi, an LLM-powered website auditor. Instead of simply asking a model whether a website's writing is "good", the workflow breaks the assessment into defined brand-voice measures, evaluates pages individually and turns gaps into specific edits.
There is still judgement in the process, but the structure around it makes that judgement easier to review.
9. Make sure I can defend the final recommendation
Eventually the validation has to end and someone has to make a decision.
At that point, I want to be able to explain what the data says, how the result was calculated, which assumptions matter, what I checked, what remains uncertain and why I think the recommendation is justified.
If the only explanation is that the AI found it, the work is not ready.
That does not mean recreating every AI-assisted task manually, because if that were necessary there would be little value in using the tool. It means having enough independent evidence to trust the result because the analysis survived scrutiny rather than because the model delivered it confidently.
AI makes validation more important, not less
The useful thing about AI is that it lowers the cost of analytical work. I can explore more hypotheses, write technical work faster and test approaches that previously would have taken much longer.
When producing an answer becomes cheaper, however, checking whether that answer deserves to be believed becomes more important.
My validation process is not particularly exotic: understand the data, reconcile the output, inspect the logic, test difficult cases, separate fact from interpretation, compare the result with the real world and actively look for reasons the conclusion could be wrong.
AI helps me do more analysis, while those checks determine which parts of it I am actually willing to trust.
This article is part of my broader practical AI for data analysts guide.