Large Language Models and the Importance of Being Right
A short essay
I once heard a psychologist ask an audience: ‘What does it feel like to be wrong?’ Most people said it made them feel stupid or ashamed. The psychologist then explained that was the answer to a different question, namely: ‘What does it feel like to find out you are wrong?’ To be wrong, she said, feels like being right.
It also feels like large language models are right all the time. But we know by now that they tend to “hallucinate”: they sometimes produce information partially incorrect or just wholly fabricated.
One reason why that’s problematic, is because you need to know the answer to understand that the answer that you’ve been provided is incorrect, which defeats the purpose of asking for it in the first place. For that reason, Stack Overflow banned ChatGPT as ‘the average rate of getting correct answers from ChatGPT is too low’ and ‘the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers’.
The word “hallucinate” is problematic, too. It anthropomorphizes the system in a way that adds to a misleading impression of greatness — echoing the words of Sam Altman.
We seem to trust these models, instinctively. I see two main reasons for that. First of all, the range of knowledge that it displays (not the depth, as for any given topic it is actually pretty shallow) is a powerful proof of competence for most people. It makes it appear smart. On top of that, it writes very proficiently and in doing so is very convincing. So convincing, in fact, that it provides it with a level of authority, as it is able to produce something that reads well and sounds reasonable, pretty much instantaneously.
Its apparent fluency shouldn’t be a reason to trust it to be right, though. It’s a bit like having a well-spoken friend that will always provide you with an answer, but anytime he doesn’t know, he'll just make up stuff on the fly and make it sound believable. It’s easy to see why one would be impressed by someone like that. But would you call them trustworthy?
The truth is some of us are more easily fooled than others. As I explained, you need to know the answer to realize that what you’re being told is fiction. How do you know your friend is selling you bullshit if you can’t spot the bullshit?
We don't know if it is something that can be fixed either. This thing that we call “hallucination” might just be an inherent feature, an innate quality of these transformer-based models. It might be part of the generative nature of the technology: to produce unpredictable and undesirable outputs from time to time.
Meanwhile, the impression of greatness will grow stronger as the proficiency of these models improves — and they will. The misleading will become more misleading. Fewer people will be able to spot its mistakes, but that doesn't mean that it is becoming more factual. To the contrary, it could still make up just as much, or more, but only when pushed to deeper levels of knowledge. That’s a scary thought, if you ask me.
Meanwhile, more products will be built on top of these models. More people will rely on its authority and its seeming trustworthiness. Until, eventually, its authority starts to compete with that of real people. It’s what all of this is culminating towards: AI will ultimately compete with specialists and domain experts on who’s right. And at that point, we'll be left with a choice…
Who do we believe? Who do we trust?