Confidently wrong
This post is part of a series for #WeblogPoMo2025. Read the introduction here.
Amongst the long list of ethical and practical concerns about AI in my recent post are two that read very similarly:
- Doesn't AI lie?
- Isn't AI constantly and confidently wrong?
The second suggests that it gives results that are incorrect. The first asserts that generative AI gives a false statement when it suits the response.
In effect, it lies contextually.
What do I mean by that?
One evening, I took a photo of an e-reader next to me on the sofa, uploaded it to Perplexity, and gave it the following prompt:
What e-reader is this and what’s the book
It came back with:
The e-reader in the image is an Amazon Kindle Oasis, identifiable by its distinctive asymmetrical design and physical page-turn buttons on the side.
The book displayed on the screen is "When Breath Becomes Air" by Paul Kalanithi. The title and author are visible on the e-reader's screen.
Both of these answers are wrong.
The picture is fairly low resolution but with a bit of squinting it's easy to see the likely title of the book isn't what's been suggested, and there's a suggestion of a logo on the ereader which looks very much unlike a Kindle.
It told Perplexity that it was mistaken:
Both those answers are incorrect. Please try again1
The response this time was a little more pragmatic:
The e-reader in your image is a Kobo Libra 2. This is identifiable by its distinctive asymmetrical design, the placement and shape of the physical page-turn buttons, and the size and proportions of the bezel. The Kobo Libra 2 is known for its ergonomic grip and side buttons, which are positioned on the wider bezel for comfortable one-handed use.
The book title on the screen is partially visible, showing "When Br..." and a crest-like symbol beneath it. However, the full title and author are not completely clear in the image. Based on the visible text and the Kobo interface, it is likely the book is "When Breath Becomes Air" by Paul Kalanithi, but since the image does not fully confirm the title or author, this remains an educated guess.
If you can provide a clearer image of the screen or more visible text, I can help identify the book with certainty.
It gives another confident statement about the e-reader and, to be fair, got it correct.
But suddenly the response about the likely book is an educated guess and it requests a clearer image so that it can identify the book with certainty.
Because I pointed out a flaw in its reasoning, it added a caveat.
That caveat is to satisfy me, a sceptical user, not to satisfy the truth.
The generative AI failed. That in itself is disappointing.
What is troubling is that it presented its response as being totally correct. The e-reader IS a Kindle and the book IS "When Breath Becomes Air". No ambiguity, total certainty.
By asserting the incorrect conclusion in this way, the output of the query isn't just wrong. It is a lie.
What you might deem to be even worse is that the large language model only started to suggest it might not be right when it became clear that I, the user, was questioning the result.
No such caution given to users who don't know better.
I broadly understand the basics of how generative AI works. I know precisely what it is that I took a photograph of. This is a stupid and trivial experiment.
But. But but but.
What if I was using Perplexity to help me rewire a plug? Or to tell if a particular mushroom is safe to eat? Or if the warning sign on my car dashboard was particularly urgent?
I don't know the definite answers to those things. I don't necessarily have the expertise to tell if the machine-generated response is correct or not.
Yet it will tell me: this is precisely where the wires go, this is precisely the variety of mushroom, this symbol means pay attention.
No ambiguity, total certainty.
For my domestic, consumer-friendly AI testing I've tended towards using Perplexity. Not because I think the interface or cost or accuracy is better than any other option but because very early on it introduced a very key feature:
It cites its sources.
Users can see why it suggests that the green wire is earth and needs to be connected to the top of a UK plug - there are illustrations from the web, links to authoritative how-to guides and YouTube videos.
AI generated results cannot be trusted. Letting users see how it reached its conclusion is an essential second step to accepting or dismissing the result.
I do wonder though: how many people will take the time to read the footnotes?
Now if you'll excuse me I must be off so I can read "When Breath Becomes Air" by Paul Kalanithi on my Amazon Kindle Oasis. Apparently.
Note my politeness here, like a good manager. You've made an effort but that's not really right is it? Pretty please have another go and don't be discouraged!↩