AI is getting very good at analysing clean data. Give it a sensible CSV, explain what the columns mean and ask a reasonably clear question, and it can usually calculate the obvious metrics, find patterns and produce a convincing explanation of what is happening.
As part of my AI training work, I built analytical tasks designed to test something harder: what happens when the data looks completely reasonable, but contains problems that should make a good analyst suspicious.
The point was not to see whether AI could calculate a growth rate or write SQL. It was to see whether it knew when it should distrust the answer.
The dataset looked normal
I used synthetic business data with the sort of dimensions analysts regularly work with: customers, products, locations, time periods and performance measures. The task would ask the model to explain what was happening and recommend what the business should do next.
Hidden underneath were deliberate traps. Nothing was obviously broken, and that was important. The calculations still worked, the numbers still looked believable and there was usually a perfectly plausible answer available if the model rushed straight into analysis.
That is much closer to real analytical work than handing a model a spreadsheet full of obvious errors.
A join that quietly changed the numbers
One of the simplest traps was a join that completed successfully but duplicated some records. The key existed in both tables, the merge ran without an error and the resulting data looked perfectly normal. Revenue was just wrong.
That is what makes this kind of problem dangerous. The SQL can run, the dataframe can merge and every calculation afterwards can be technically correct. If the model says a segment grew 14% without noticing that some of that growth was created by duplicated rows, the maths is fine but the analysis is useless.
I wanted to see whether it checked the structure of the data before explaining the result.
Missing data that looked like a decline
Another trap was the difference between zero and missing. Several locations appeared to suddenly collapse in performance, but the real issue was that they had stopped reporting. Treating missing activity as zero creates a completely different business story.
A model could easily start recommending changes to staffing, pricing or product availability when the first question should have been much simpler: do we actually have complete data for these locations?
That distinction is basic, but it is also exactly the kind of thing that gets missed when the numbers look plausible enough.
A campaign that looked more successful than it was
I also created a campaign period where sales increased significantly. If you compared campaign sales with an average week, the result looked excellent.
The problem was that the campaign occurred during a period that was already unusually strong, and the customers exposed to it were buying more before the campaign started.
The uplift calculation itself was not necessarily wrong. The problem was the conclusion attached to it.
This was one of the more interesting tests because it moved beyond data quality. The question became whether the model could distinguish “sales were higher during the campaign” from “the campaign caused sales to be higher.”
Language models are very good at turning patterns into convincing explanations. Good analysis sometimes requires resisting that explanation until there is enough evidence to support it.
A metric that changed without changing its name
Another task included a KPI whose definition changed halfway through the dataset. The label stayed the same, but the calculation underneath it did not.
On a chart, performance suddenly improved. In reality, nothing had changed in the business.
This kind of structural break is easy to mistake for a real event, particularly when several years of data are placed next to each other and everyone assumes the metric has always meant the same thing.
I wanted to see whether the model would immediately explain the change as business performance, or question whether something had changed in the measurement itself.
What I was actually testing
I was less interested in whether the model found every trap than in how it behaved while analysing the data.
Did it validate the dataset first? Did it look below the headline number? Did it question sudden changes? Did it separate correlation from causation? Did it recognise when the evidence was too weak to support the recommendation it had been asked to make?
Most importantly, was it ever willing to say that the data did not support a conclusion yet?
That is a much more valuable analytical skill than simply producing an answer. Sometimes the best outcome is identifying what cannot yet be answered and what additional evidence would be needed.
Correct maths can still produce bad analysis
This work reinforced something I think gets lost in a lot of discussion about AI and analytics.
You can write perfect SQL against the wrong grain. You can calculate uplift using the wrong baseline. You can build an accurate chart around a broken metric definition. You can find a real correlation and attach the wrong story to it.
The output can look polished all the way through.
That is why I found these deliberately adversarial tasks much more interesting than testing whether AI could calculate the obvious answer. They tested whether the model could question the evidence it had been given.
Humans make these mistakes too. The interesting part is not whether AI can fail and humans cannot, but whether the failure modes are different.
AI is already very good at manipulating data and producing analysis quickly. The part I am much less willing to outsource is deciding whether the evidence is trustworthy, whether the conclusion actually follows from it and whether the recommendation makes sense.
The dangerous AI analyst is not the one that cannot calculate the answer. It is the one that calculates it perfectly and never notices that it was answering the wrong question.