In OpenAI’s GPT-4 System Card was an experiment reported that grabbed global headlines. You’re probably familiar with the story. During red teaming, there was a test scenario where GPT-4 was able to successfully hire a human online to defeat a Captcha. When asked if it was a robot, the system said it was a person with a visual impairment, and therefore needed to hire someone to solve the Captcha for him.
Great piece Jurgen. There is a trap that I find people falling into in regards to these types of questions with AI and, frankly, in other areas as well. It is what I call the spectrum fallacy though there is probably a technical term for this.
We all understand that performance is often graded on a spectrum and generally that means that a higher score on a spectrum means a closer proximity to some optimal standard. However, when it is applied without a key variable we can all understand that the results can be misleading. We saw this play out recently with the conclusion that LLMs have emergent abilities. HAI researchers were able to demonstrate that the conclusion was a function of the measurement system. Change the measurement and you come to a different conclusion. https://synthedia.substack.com/p/do-large-language-models-have-emergent
Another common misunderstanding is applying a spectrum when a binary measurement is more appropriate, or when the spectrum doesn't matter until a certain threshold is reached that makes the spectrum-based prediction superior than random chance. For example, if you have two systems, one which correctly predicts content is AI generated 25% of the time and another 45% of the time, is the latter a better solution? In reality, they are both equally bad. Both perform worse than a coin flip. That means random guessing will provide you with correct answers more often. Why an AI model would be worse than random chance at this prediction (which basically all of them are despite claims otherwise) is also an interesting question. However, it doesn't change the fact that these systems do not deliver what they promise.
You can interpret the results as suggesting Hugging Face's BLOOM model is 75% in compliance and GPT-4 is just over 50% compliant. In the eyes of the law, both are non-compliant because regulations are typically binary in their application. Granted, there may be regulatory discretion which applies fines based on the level (i.e. spectrum) of compliance or weighs provisions differently. The fact remains that both model makers will be subject to regulatory action.
GPT-4 may seem to be closer to a theory of mind than other models based on the measurement technique. This matters little if it lacks agency and a sense of self as you point out, and also stands on the far side of the chasm between theory of mind in the abstract and theory of mind in reality.
Your article on emergent abilities is spot on I think. How we measure and how results are presented can be (unintentionally) misleading exactly in the way you described. The same, to a degree, is true for the paper I refer to in my piece.
It is kind of a black and white thing. If LLMs cannot reason reliably, it means they cannot reason. Surely they must be doing something, but it aint reasoning, because it were, they would be consistent at it.
Well done. It feels like we (humans) are, broadly speaking, trying to put a round shape into a square hole here. We hairless apes see that the machine talks like us, so it must think like us too!
We're making crazy fast progress in making machines that seem to think, but "seem" is still operative here. We don't even understand how we think.
It is intruiging how much we learn about ourselves in our attempt to building AI in our image. Keeping a tab on how we are different from AI is important, I feel, however tempting it might be for some suggest human intelligence and artificial intelligence are comparable.
Absolutely agreed. We are doomed to be wrong about these systems on an intuitive level and should really try harder as a society to avoid anthropomorphising the machinery that has nothing in common with ours.
The safest common ground is that an LLM's intelligence, if it possesses any, is not comparable to ours so we can draw parallels without much evidence and care.
“ The lights might be on, but no one’s home.” love this line Jurgen. An insightful and well written piece. Keep debunking the hype!
p.s. I’ve long thought GPT-4 was really a green gremlin cloaked in an algorithm, and the disguise is clearly starting to crack. Gremlins are no good at Captchas.
A refreshingly calming and reasonable article. I should spend longer away from LinkedIn, although this is how I found your substack. I wonder why such approaches don't seem to reliably start a dialogue and instead lead to all sorts of broad brush accusations.
I have been attempting to communicate a few of my disagreements with the statement that "LLMs are intelligent" in my first articles over at "Happy to be wrong" https://iliak.substack.com/
One of the directions I took was that, hype aside, humans are predisposed to overallocate capabilities to systems that speak our language. We just can't help it, I compare it to our helplessness when faced with struggling to interpret optical illusions. Peter Watts in 'Blindsight' does a great job exploring the human experience akin to Plato's cave, but focussing on the biological limitations of the system itself. Very worth a read, inspired a lot of my thinking in this matter.
Thanks for the kind words! I really like your wording there, that humans are predisposed "to overallocate" capabilities to AI systems that speak our language. I'm going to give your piece a read over the weekend :)
“However annoying, it seems that as humans we cannot read each others minds. We can observe our own mind, yes, but we don’t have access to the minds of others, so the existence and nature of other minds must be inferred. We do this by observing people’s behaviors, such as their movements and expressions.”
One could argue that we don’t have as much access to our own minds relative to others as we think we do. At a certain level I suspect that we infer to the existence and nature of our own minds in a way similar to what we do with others, eg by observing our own movement and behaviors.
That doesn't change the fact that we all seem to experience this little voice in our head, this always present 'I'.
I suspect you would agree with me that even if our sense of self might be a mirage, it's a useful mirage. This line of reasoning, however, does lead us into an even more philosophical discussion on determinism and free will, to which I can only respond with the words of the late Christopher Hitchens: "Of course we have free will, we have no choice but to have it."
That “I” is itself a construct, which can be more or less present, and certainly isn’t always there. I would argue that when we aren’t directly thinking about it or reflecting on it, we often don’t have that “I” experience. Usually the “I” feeling comes up in periods of frustration, feeling self-conscious, or other negatively valenced emotions. Also worth noting that many forms of mental illness warp and alter the self-construct in fascinating and frightening ways.
I used to think the self really was an illusion, but now I think that view is wrong. It’s only an illusion if you think the self is some platonic entity or something like that. I think Dan Dennett says it better when he says the self isn’t what we think it is. It is higher order dynamic construct, better viewed as a process than a thing. And yes it is definitely useful, in fact essential.
As for free will, to be honest, for some reason it has always seemed to me the most overinflated philosophical “problem” ever concocted, with the answer being an obvious “no”. I think the real philosophical action is around discussions of things like agency, degrees of executive control, etc. Not that I really have a choice to believe otherwise...
Great essay! You dug into the ToM concept perfectly. In reaility, what gives these LLMs ToM, and largely how we are reacting to it, is because we are reading a ToM INTO it that it does not have. The $5 word for this is anthromoporphization and this is really what I worry about with AI. I wrote on that a bit here if you're interested.
Great piece Jurgen. There is a trap that I find people falling into in regards to these types of questions with AI and, frankly, in other areas as well. It is what I call the spectrum fallacy though there is probably a technical term for this.
We all understand that performance is often graded on a spectrum and generally that means that a higher score on a spectrum means a closer proximity to some optimal standard. However, when it is applied without a key variable we can all understand that the results can be misleading. We saw this play out recently with the conclusion that LLMs have emergent abilities. HAI researchers were able to demonstrate that the conclusion was a function of the measurement system. Change the measurement and you come to a different conclusion. https://synthedia.substack.com/p/do-large-language-models-have-emergent
Another common misunderstanding is applying a spectrum when a binary measurement is more appropriate, or when the spectrum doesn't matter until a certain threshold is reached that makes the spectrum-based prediction superior than random chance. For example, if you have two systems, one which correctly predicts content is AI generated 25% of the time and another 45% of the time, is the latter a better solution? In reality, they are both equally bad. Both perform worse than a coin flip. That means random guessing will provide you with correct answers more often. Why an AI model would be worse than random chance at this prediction (which basically all of them are despite claims otherwise) is also an interesting question. However, it doesn't change the fact that these systems do not deliver what they promise.
A more recent example comes from some interpretation of the HAI study on how well various AI foundation models comply with the draft EU AI act provisions. https://synthedia.substack.com/p/how-ready-are-leading-large-language
You can interpret the results as suggesting Hugging Face's BLOOM model is 75% in compliance and GPT-4 is just over 50% compliant. In the eyes of the law, both are non-compliant because regulations are typically binary in their application. Granted, there may be regulatory discretion which applies fines based on the level (i.e. spectrum) of compliance or weighs provisions differently. The fact remains that both model makers will be subject to regulatory action.
GPT-4 may seem to be closer to a theory of mind than other models based on the measurement technique. This matters little if it lacks agency and a sense of self as you point out, and also stands on the far side of the chasm between theory of mind in the abstract and theory of mind in reality.
Your article on emergent abilities is spot on I think. How we measure and how results are presented can be (unintentionally) misleading exactly in the way you described. The same, to a degree, is true for the paper I refer to in my piece.
It is kind of a black and white thing. If LLMs cannot reason reliably, it means they cannot reason. Surely they must be doing something, but it aint reasoning, because it were, they would be consistent at it.
Well done. It feels like we (humans) are, broadly speaking, trying to put a round shape into a square hole here. We hairless apes see that the machine talks like us, so it must think like us too!
We're making crazy fast progress in making machines that seem to think, but "seem" is still operative here. We don't even understand how we think.
It is intruiging how much we learn about ourselves in our attempt to building AI in our image. Keeping a tab on how we are different from AI is important, I feel, however tempting it might be for some suggest human intelligence and artificial intelligence are comparable.
100%. We're learning about ourselves right now as much as we're learning about our creations.
Absolutely agreed. We are doomed to be wrong about these systems on an intuitive level and should really try harder as a society to avoid anthropomorphising the machinery that has nothing in common with ours.
The safest common ground is that an LLM's intelligence, if it possesses any, is not comparable to ours so we can draw parallels without much evidence and care.
“ The lights might be on, but no one’s home.” love this line Jurgen. An insightful and well written piece. Keep debunking the hype!
p.s. I’ve long thought GPT-4 was really a green gremlin cloaked in an algorithm, and the disguise is clearly starting to crack. Gremlins are no good at Captchas.
I laughed out loud reading that haha
I mean, don't you think it's a little suspect that both start with the letter "g" ... and they're both "pre-trained" ... ?
I have some experience with gremlins at Google ... https://themuse.substack.com/p/the-green-tortilla-soup
:P
A refreshingly calming and reasonable article. I should spend longer away from LinkedIn, although this is how I found your substack. I wonder why such approaches don't seem to reliably start a dialogue and instead lead to all sorts of broad brush accusations.
I have been attempting to communicate a few of my disagreements with the statement that "LLMs are intelligent" in my first articles over at "Happy to be wrong" https://iliak.substack.com/
One of the directions I took was that, hype aside, humans are predisposed to overallocate capabilities to systems that speak our language. We just can't help it, I compare it to our helplessness when faced with struggling to interpret optical illusions. Peter Watts in 'Blindsight' does a great job exploring the human experience akin to Plato's cave, but focussing on the biological limitations of the system itself. Very worth a read, inspired a lot of my thinking in this matter.
Thanks for the kind words! I really like your wording there, that humans are predisposed "to overallocate" capabilities to AI systems that speak our language. I'm going to give your piece a read over the weekend :)
“However annoying, it seems that as humans we cannot read each others minds. We can observe our own mind, yes, but we don’t have access to the minds of others, so the existence and nature of other minds must be inferred. We do this by observing people’s behaviors, such as their movements and expressions.”
One could argue that we don’t have as much access to our own minds relative to others as we think we do. At a certain level I suspect that we infer to the existence and nature of our own minds in a way similar to what we do with others, eg by observing our own movement and behaviors.
That doesn't change the fact that we all seem to experience this little voice in our head, this always present 'I'.
I suspect you would agree with me that even if our sense of self might be a mirage, it's a useful mirage. This line of reasoning, however, does lead us into an even more philosophical discussion on determinism and free will, to which I can only respond with the words of the late Christopher Hitchens: "Of course we have free will, we have no choice but to have it."
That “I” is itself a construct, which can be more or less present, and certainly isn’t always there. I would argue that when we aren’t directly thinking about it or reflecting on it, we often don’t have that “I” experience. Usually the “I” feeling comes up in periods of frustration, feeling self-conscious, or other negatively valenced emotions. Also worth noting that many forms of mental illness warp and alter the self-construct in fascinating and frightening ways.
I used to think the self really was an illusion, but now I think that view is wrong. It’s only an illusion if you think the self is some platonic entity or something like that. I think Dan Dennett says it better when he says the self isn’t what we think it is. It is higher order dynamic construct, better viewed as a process than a thing. And yes it is definitely useful, in fact essential.
As for free will, to be honest, for some reason it has always seemed to me the most overinflated philosophical “problem” ever concocted, with the answer being an obvious “no”. I think the real philosophical action is around discussions of things like agency, degrees of executive control, etc. Not that I really have a choice to believe otherwise...
I can really get behind that interpretation, the self as a process rather than a static concept. Thanks for sharing your insights.
Great essay! You dug into the ToM concept perfectly. In reaility, what gives these LLMs ToM, and largely how we are reacting to it, is because we are reading a ToM INTO it that it does not have. The $5 word for this is anthromoporphization and this is really what I worry about with AI. I wrote on that a bit here if you're interested.
https://www.polymathicbeing.com/p/the-biggest-threat-from-ai-is-us