How the novel nature of LLMs confounds our ability to evaluate them

Lack of linguistic context

One of several significant problems in discussing LLMs lies in the language available to do so. LLMs are conceptually unlike anything we’re used to dealing with in either a technical or social context; they are neither machines nor minds, but some third thing that is neither logical nor capable of conceptualizing.

And we’ve never really had to talk about something that, so, lacking a vocabulary to do, we tend to resort to a language of minds and of thought. And those words shape our own thought, unintentionally manipulating our own minds to perceive minds where there are none.

We speak of these artifacts “thinking” while they process. We blame “hallucination” when errors occur - though even the concept of “error” is suspect here. We even believe in “being nice” to them to get better results, or coercing them to “try harder”.

They’re not thinking. They’re not hallucinating - or at least, no more when they’re producing inaccurate than accurate information. They don’t “feel better” when we’re nice, and they don’t “try”. Nor, really, do they make errors, as that implies distinct states of correctness and deviation from correctness. These things have no concept of correctness or error. They have no concept of true or false. Indeed, they have no concept of concepts, or indeed of anything else.

At this point, the language we’re using starts fraying at the edges.

So why is it so hard to describe what they’re doing?

Can we analyze the logic they’re applying? No - there’s no logic, nothing to be applied. If there were, they’d probably be capable of math. Instead, we’ve created machine-based entities that can’t count.

And you’ll have noticed I’m avoiding calling them “machines”. Machines follow visible, predictable processes that can be analyzed. Nor are they “programs”, following defined rules in a predictable fashion.

They’re not machines, they’re not minds, they’re not programs. They’re a trillion numbers in a trenchcoat; not logical, in either a machine or a mental sense, but stochastic. Which is a word few of us have met before, and one that’s hard to handle. Essentially, it means “there’s some math in there, but there’s no way of predicting what it’ll do”.

We have to change our approach somewhat to be able to discuss these things. My personal feeling is that we’re better served by the language of fiction here than that of computation or cognition. And that language serves us in two useful ways.

Firstly, we can look into fiction for analogues of these entities. And we can use words like “entities” and “artifacts”, far more graceful than “things”, and more accurate, with less semantic baggage, than “machines”.

We use the term “artifact” in both science fiction and fantasy to mean “something, alien to our conception, that we don’t fully understand”. “Entity”, more used in science fiction means “we’re still not sure what it is, but we get the impression it might be thinking”.

When we meet either of those in novels, we tend to proceed with caution. If the protagonists don’t, the reader knows what to expect. And if they do proceed appropriately, they tend to do so with few assumptions.

Most critically, they don’t tend to assume that the artifact is benign, friendly, and honest. Which are all, of course, attributes of mind - our language invariably bends back this way. So let’s push it back the other way. Next time you hear “LLM”, think “talking rock”.

Although, of course, you don’t tend to hear the term “LLM”. You tend to hear “AI”. Which is a whole other world of problems, but without opening that box up, let’s say: AI, in computer science, is an attempt to create reasoning intelligence, and so AI researchers tend not to consider LLMs as AI / part of that process. AI in sci-fi, is a reasoning intelligence, generally on a non-biological base. LLMs, on the other hand, are an attempt to simulate a reasoning mind.

And I’m mostly joking when I say that, if “real” AIs ever turn up, they’re going to be really pissed that we called these simulacrums by their name.

Pareidolia of mind

Pareidolia is the strong, innate human tendency to see patterns, particularly faces, where none really exist. In the case of LLMs, it predisposes us to assume the presence of something familiar - a mind - in something superficially resembling it, rather than inclining us towards a purely objective analysis. Indeed, a sufficient resemblance to a mind is often seen as proof of intelligence. The Turing Test, in particular, is sometimes formulated as assuming that a machine that cannot be distinguished from a person, when interrogated through a mechanism that prevents direct observation of the respondent, is inherently intelligent.

In fact, that’s not quite what Turing was saying with his proposal. He sidestepped the term “thinking” itself as too nebulous. He phrased this test as his “Imitation Game”, suggesting that an entity that can sufficiently mimic another to fool an observer might be “thought-equivalent”. In many ways, Turing’s original paper, “Computing Machinery And Intelligence” is better seen as a discussion, albeit a fascinating one, than a hard answer. It’s also worthy of note that it both inherently assumes a “learning machine”, rather than a statistical one, and worries that telepathy, for which it claims “the evidence… is overwhelming” would ruin the test.

So, in the spirit of that discussion, if an LLM can appear human, is it “thought-equivalent”? Well, we know what LLMs are doing, and it’s not doing anything intelligently; these are purely statistical devices executing pure mimicry. In as much appear to “pass” the “Turing Test”, it’s by copying the answers.

It might look like a duck, but it won’t quack.

The Chat Factor

The “chat” interface generally used by LLMs also has subconscious effects; when we engage with something this way we both tend towards considering it somehow equivalent to us, and - because the chat seems to make us an equal partner - invest ourselves in its success. Indeed, precisely because we expect to have to iterate repeatedly on a response to achieve an acceptable outcome, that we feel we’ve “put in the work” - and once we reach “acceptable”, we’re not inclined to iterate further.

The “chatty” (or often obsequious) language used also inclines us to believe in the presence of a mind, regardless of the quality of the responses themselves. Producers of such models increasingly play into this tendency by moving from “machine” names (ChatGPT-*) to “human” ones (Claude et al).

Talking Dog Syndrome

We tend to be so astounded that these artifacts are appearing to think, that we fail to adequately analyze how well they’re doing so. This always reminds me of a passage from Pratchett’s “Moving Pictures”, where the protagonist suddenly realizes that the unseen musician he’s been criticizing is a dog:

“You’re not supposed to recognize the bloody tune,” said Gaspode, sitting down heavily and industriously scratching one ear with his hind leg. “I’m a dog. You’re supposed to be bloody amazed I can bloody well get a squeak out of the bloody thing.”

When we have our “Gaspode moment” with LLMs, we do see the dog. And we are indeed so bloody amazed by a computer seeming to think, that we fail to analyze the quality of those “thoughts”, instead extending a “willing suspension of critique” that we wouldn’t accord to any other agent, human or software. We are often sufficiently amazed that we accept a level of performance from LLMs that we’d accept from no other source.