In the past few years, we’ve seen many embarrassing media reports of people who work in professional services – lawyers, consultants, writers and many others – getting into trouble for using generative AI without thoroughly checking the outputs.
For example, a recent article in The Australian Financial Review noted that consulting firm KPMG published a report titled Redefining excellence in the age of agentic AI, which gave multiple examples of AI use by organisations such as UBS, the UK National Health Service, Swiss Federal Railways and Transport for London. Those organisations said the claims of AI use were “factually incorrect”, “not accurate”, “misleading” and “not aligned” with the media release KPMG cited as a source.
We don’t know exactly what happened in this case, but it appears someone at KPMG didn’t follow the normal editorial process of fact-checking the claims and citations used in the report. This situation is increasingly likely as more teams in large organisations use large language models (LLMs) to augment and streamline their work, while still handling the usual pressures of their workloads and deadlines.
LLMs can do helpful and amazing things, but like all power tools, you need to use them with care to avoid losing a finger, metaphorically speaking. How can you avoid being the next awkward news story? We have three tips to keep you safe – or at least safer than if you didn’t follow them:
- Understand why LLMs make mistakes
- Don’t skimp on thinking
- Fact-check everything.
Why LLMs make mistakes
Many unfortunate AI stories are the result of people publishing or relying on AI-generated material that cites incorrect or non-existent sources, or that misreads or misinterprets facts.
The AI industry calls these mistakes ‘hallucinations’, which can give the impression that they are an error of perception – of seeing the world incorrectly. In fact, hallucinations are more a fundamental feature of how LLMs work.
These models use statistical techniques at massive scale to predict the next word or phrase, based on the prevalence of similar words and concepts in their source materials. They are trained to provide grammatically correct, plausible and confident responses.
However, they don’t have a model of the real world to compare against to see if their responses are true or likely. Cross-checking – for example by searching the web – would be too expensive computationally. And anyway, there are plenty of questions of fact you can’t settle just by searching the web.
When facing a choice between generating a speculative answer or saying “I don’t know”, LLMs are trained to provide a response. This is understandable. If your chatbot said “I’m not sure” every other time you asked a question, you’d find it somewhere between limited and annoying. However, this also means an LLM won’t tell you when it’s making things up – unless you challenge it.
So, the first tip: if your job or multinational brand depends on a piece of information being correct, don’t believe a chatbot, no matter how confident it sounds. And pay attention to the warnings LLM makers provide – they themselves always emphasise this point in their product disclaimers.
Don’t skimp on thinking
Many people try to minimise AI mistakes using retrieval-augmented generation – that is, by telling the LLM to draw information only from specific source materials.
This is definitely helpful, because it reduces the potential for errors such as inventing sources. However, there’s still a risk the LLM will misinterpret the sources you provide. For example, in the AFR article, the KPMG report referred to a real NHS media release but made claims about how the NHS was using AI that weren’t in the source announcement.
Whether you’re providing your own sources or asking an LLM to find them for you, it’s essential to read and understand those sources before you get AI to generate a draft. That way you’ll know enough to spot when an LLM makes mistakes – and push back when you see them.
Fact-check and reference everything
The third part of the solution is an old one but still a good one: fact-checking and reviewing AI-generated materials before you publish them.
It’s important to check very thoroughly and not just wave through anything that looks right. Remember: AI is very good at generating outputs that look right! A helpful distinction comes from psychologist Daniel Kahneman’s 2011 book Thinking, Fast and Slow. Kahneman describes two different ways people form thoughts:
- System 1 is fast, emotional and unconscious, relying heavily on pattern recognition.
- System 2 is slow, logical, conscious and much harder to do than System 1.
Kahneman argues that our brains are lazy and will revert to System 1 even while we think we’re doing System 2.
When you’re checking the outputs of an LLM, it can be very tempting to rely on System 1 pattern recognition (this looks right) rather than System 2 critical evaluation (this argument is supported by the source reference, for example). And as in our previous tip, it can feel like pushing the easy button if you outsource System 2 thinking entirely to a chatbot. But these are ways embarrassing errors – such as non-existent or misinterpreted sources – can creep in.
Fortunately, deep thinking and fact-checking are things we at Editor Group like to think we do well, and we’re always glad to help others. Our team has extensive experience in disciplines such as journalism and academic research that are steeped in critical thinking and fact-checking. We can help you prevent avoidable errors, as well as giving you the benefit of our judgement and experience on questions of structure, argument and impact on the audience.
By Josh Mehlman
Photo by Andrea Piacquadio
