There is an obvious irony in using AI to help draft an article about validating AI-generated analysis, but it also makes the point fairly well.
I use AI to write SQL and Python, work through unfamiliar libraries, generate hypotheses, check formulas and get through repetitive tasks faster. I am happy for it to do those things, but I do not assume an answer is correct because the code ran, the chart looks sensible or the explanation sounds confident.
Before I trust AI-assisted analysis, I apply the same checks I would use when reviewing another analyst’s work, a spreadsheet model or code I wrote myself. The difference is that AI makes it unusually easy to move from a question to a plausible-looking answer without seeing much of the reasoning in between.
1. Check what the data represents
Before reviewing a calculation, I want to know whether the model has interpreted the dataset correctly.
What does one row represent? What period does the data cover? What is being counted? Are the measures additive? What has been excluded, and do similarly named fields mean the same thing across different sources?
A model can write valid code against the wrong interpretation of a field. If a column called sales contains gross sales in one source and net sales in another, the code can still execute. If customer_id represents an account rather than an individual customer, the resulting customer count can still look reasonable. If one table is at transaction level and another is already aggregated, a join can quietly multiply the result.
The first question is therefore not whether the model calculated something correctly, but whether it was given the right analytical problem. This is also why I begin data-quality work with definitions rather than null checks, because a technically clean dataset can still be conceptually wrong.
2. Reconcile the result before explaining it
One of my most useful checks is also one of the least sophisticated: add everything back up.
If I have analysed individual products, they should reconcile to the category. Customer segments should reconcile to the customer base, while projections for thousands of local areas should produce a believable national population.
I ran into this while building Australian population projections at a very granular geographic level. Many of the local results appeared plausible on their own, but when I rolled them back up, Australia suddenly had approximately 52 million people instead of 26 million.
That was a modelling error rather than an AI error, but the validation principle is the same. A result can look reasonable row by row while being completely wrong as a system, so I reconcile the parts to a known whole before spending time interpreting them.
3. Read the generated SQL or Python
Successful execution is not a validation test.
When AI generates SQL or Python, I review the joins, filters, grouping logic, date conditions, denominators and treatment of missing values. I want to know whether a join can create duplicates, whether the denominator matches the question, whether nulls are being excluded or treated as zero, and whether the comparison period is genuinely like-for-like.
I also look for assumptions the model introduced without being asked. When a request is ambiguous, AI will often fill the gap because producing an answer appears more useful than stopping to ask a question. Those assumptions can disappear inside otherwise sensible-looking code.
I do not need to rewrite every line myself, because that would remove much of the benefit of using the tool. I do need to understand the important logic well enough to explain what it does and why the output should be trusted.
4. Trace sample records and difficult cases
Aggregate results can hide strange behaviour, so I trace a selection of records through the logic manually.
I deliberately choose awkward cases, including missing values, small groups, unusually large values, products that changed hierarchy, dates around the edge of a reporting period and customers that appear in more than one place.
AI-generated code often handles the normal case correctly while failing at the edges. A calculation may break when a denominator is zero, a category mapping may omit products introduced halfway through the year, or a time-series comparison may include a partial week on only one side.
Those exceptions may have almost no effect on the total, or they may change the conclusion completely. Reviewing individual records is usually a quick way to find out which.
5. Separate calculation from interpretation
AI can move quickly from “sales declined 12%” to “customer demand is weakening”, even though the second statement does not follow automatically from the first.
The decline could reflect fewer stores ranging the product, poor availability, a price change, reduced promotional support, a competitor launch or customers moving elsewhere in the range. The calculation establishes what happened, while explaining why requires additional evidence.
I therefore validate the measure first, confirm that the comparison is sound and label any explanation as a hypothesis until it has been tested. This prevents a plausible narrative from becoming attached to a number before the underlying number has earned that interpretation.
6. Ask how the analysis could be wrong
One useful role for AI during validation is to attack its own analysis rather than continue strengthening the same conclusion.
I ask which assumptions the result depends on, what alternative explanations fit the evidence, which additional data would separate those explanations and what result would contradict the current interpretation. These questions usually produce a better test plan than simply asking the model to “double-check” its answer.
However, the model’s critique is not independent validation. A model can identify a possible failure mode, but I still test that possibility against the source data, known totals or separately written logic.
7. Keep the model’s role visible
The more important the analysis becomes, the more useful it is to separate deterministic calculations from model judgement.
Revenue divided by units does not require an LLM, and neither does a reconciliation check. Classifying hundreds of open-text responses, proposing explanations for an anomaly or identifying possible weaknesses in an argument may be reasonable uses of a model.
I used this principle when building Revisi, an LLM-powered website auditor. Rather than asking a model whether a website’s writing is simply “good”, the workflow breaks the assessment into defined measures, evaluates pages separately and converts identified gaps into specific edits. The model still exercises judgement, but the structure makes that judgement easier to inspect.
Before I rely on AI-assisted analysis, I want to be able to explain what the data represents, how the result was calculated, which assumptions matter, what I tested and where uncertainty remains. If the only explanation is that the AI found it, the work is not ready.
AI lowers the cost of producing analysis, which makes validation more important rather than less. It helps me examine more possibilities and build technical work faster, while these checks determine which results I am willing to trust.