But it’s not the machine’s hostility we need to worry about.
In the bizarre language of the LLM industry, “alignment” refers to the need to train the AIs we apparently have to build, not to want to eradicate humanity when they inevitably become sentient and omnipotent. It’s a conveniently apocalyptic threat to keep attention focussed on the existential and fictional, and not on the real and current problems that LLMs present every day.
But the mundane reality is much simpler: LLMs fail to effectively solve the problems we have, while creating a vast new class of problems to be solved. They are, ultimately, completely mis-aligned with our needs, and incompatible with the society we live in.
The problems of LLMs fall into three main categories: hype, inherent unreliability, and immense costs.
Hype
The entire LLM industry is based on presenting LLMs as things they are not. They are not sentient, or intelligent. They are not therapists, and they are not scientists. They are not artists, and they are not engineers. They cannot have thoughts, let alone original thoughts, and they cannot make novel discoveries from existing information. They cannot distil truths from vast repositories of knowledge. They understand nothing. They cannot make breakthroughs, they cannot have inspirations, and they cannot tell fact from fiction. They do not have desires, motivations, or malicious (or altruistic) intentions. They are not your friends, but also, not intentionally your enemy. There is no reason whatsoever, per current cognitive science, to expect them to miraculously flip over from generation to intelligence. They are not neutral or impartial, and they are, above all, unreliable.
It’s worth noting at this point that LLMs are not, in general, what the AI movement set out to create. AI seeks many things, some of which are subsets of human capabilities useful in a particular domain (e.g. audio transcription, image recognition, pattern identification), and some of which are intended as steps towards the understanding, or even the creation, of “general” intelligence or sentience. For example OpenAI, in the main, was originally intended to follow that latter path, as a public good.
Somewhere along the way, LLMs appeared, a technology which can perform an effective mimicry of intelligence, appearing convincing to the layperson, and therefore seemed to have a commercial market. And huge sectors of the AI and tech industries pivotted wholesale, abandoning most other research.
Unreliability
LLMs are “stochastic”, which is a term that essentially means “random, but weighted by consumed data, with a hell of a lot of math on top”. This means, critically, that their behavior is inherently not reproducible, and cannot be constrained to any particular scale of accuracy. LLMs will always have the capacity to do something random, extreme, or undesired.
However, with a sufficiently colossal amount of processing power, they can do so at scale. This means that the real use-cases for LLMs lie in making mistakes fast, in such a way that the speed outweighs the errors (or that the errors can be externalized). Unfortunately, the layperson is rarely aware of the potential magnitude of these errors, even where the traditional disclaimer is attached.
Cost
In pure dollar terms, LLM technology is colossally expensive, consuming hundreds of billions of dollars of investment. Individual data centers cost billions and take years to build. Computation costs - in time, power and hardware, for training in particular, are immense. So much money is circulating in this sector (often in peculiarly circular deals) that it represents - in numbers, at least - the entire growth of the US economy, and consumes almost all available investment. No-one (except hardware suppliers) in this sector is making a profit, and the costs of providing their services (referred to as “inference”) are only increasing.
It’s going to get quite messy if that investment doesn’t pay off.
Of course, the costs are far more than monetary. The human and environmental impacts are also immense. Their electrical consumption (which, for some planned data centers, would exceed that of entire states) is largely fossil-based (and in the cases where it’s renewable, it’s taking that renewable capacity from other consumers, who are thus pushed to fossil sources), and so inherently unaffordable on a warming planet. Often, their power needs are greater than the grid can supply, meaning that local “temporary” power sources can be set up - eg, gas turbine generators being set up in, or near, residential areas. Likewise, they are immensely thirsty. All that power going in has to be removed as heat, which, at that density, requires water. Drinking water, at that, as it needs to be pure enough not to corrode or block pipes, and water which is often not recycled, but evaporated as a cooling mechanism. That consumption can be a significant fraction of - and competitor for - domestic and agricultural needs which may already be affected by drought, or exceed replacement rates.
The human costs can also be grotesque. That LLMs are frequently trained, unpaid, on human work is well known. But it’s not as widely understood that a lot of training doesn’t merely consume existing work, but also requires human intervention at immense scale. Whole networks of trainers - often in the Global South, and paid as little as possible - are needed to judge system outputs and feed that judgment back into the machine, or to create content in specific formats for the machine to consume. Some of that content training can involve judgement of violence, hatred, even CSAM, and has had immense psychological effects and personal costs on the trainers.
And while LLMs consume human work on one side, they displace it on another. LLM-generated content replaces work for which humans would previously have been paid - destroying creative likelihoods and skilled jobs (particularly, art and translation). Even its consumers pay a cost, in the risks of misinformation, psychosis, dependency and deskilling.
Benefits?
So what does the LLM industry offer us in return for these costs? The industry’s main products seems to be manufactured inevitability, the assertion that we absolutely have to use their services or be left behind, and compulsion, by forcing this technology into business and consumer products where it’s not wanted and causes far more friction than benefit.
So, we hit our first misalignment:
Claims of inevitability through utility are incompatible with compelled usage.
One potential rejoinder there is that these tools - this technology as a whole - isn’t intended to be a benefit to those who use it, but instead those who benefit from their labor. Which is likely true, for all that it’s saying the quiet part out loud.
So, two more misalignments:
The promise to the worker is misaligned with the implied promise to industry
Compelled degradation of labor is incompatible with the dignity and wellbeing of the worker.
The promise to the worker, for all that it’s rendered thoroughly suspect through compulsion, is that these tools will allow the worker to do their job more quickly and easily. Research has widely shown that claim to be untrue, but employers have often clung to the promise to suggest that workers should either be producing more, or that less work (or fewer workers) should be needed, rather than reflecting that supposed increase in productivity in an increase in remuneration. Any resulting increase in revenue instead accrues upwards, increasing the centralization of wealth.
Increased centralization of wealth is incompatible with the promise of global benefit.
The second part of the global promise of efficiency is that we’d all be granted greater leisure as machines “did all the work”. But most of the world lives under “work or die” capitalism; that is, the benefits of productivity are shared - and then often grudgingly - with those directly contributing to it; those who don’t work, or work less, don’t get a remotely adequate share to survive. This pattern is most extreme in countries where healthcare can only be afforded through employment. While every experiment in Universal Basic Income has shown suitably universal benefits, it’s an idea that is decried as dangerous, “unfair” and even unthinkable socialism in most capitalist societies. Instead, “fairness” is apparently served by increased social stratification.
A refusal to share the benefits of a reduced need for labor is incompatible with the “promises” of efficiency.
Alternatively:
An economy still centered around its historical need to “create jobs” is incompatible with a supposed paradigm shift that destroys jobs
Or more explicitly:
“AI”, if it actually worked, would be utterly incompatible with our capitalist society.
Of course, where we actually see reduction in apparent need is diametrically opposed to what was promised to us in our leisure-filled techno-future. We’ve all heard the too-accurate jibe that computers were supposed to free us from tedium to allow us to create art, but instead they make art and leave us the tedium. As we see in the requirements for humans in the loop both in training and in operation of “AI”, whether it’s workers in the Global South driving “autonomous” Waymos or overviewing “unstaffed” Amazon stores, or translators, artists and developers “cleaning up” the slop produced by LLMs, we instead have a “reverse centaur” model where we end up servicing the machines that were supposed to serve us.
Machines that can only produce substandard work are incompatible with claimed benefits and efficiencies.
And therefore we have the odd situation that it’s in some ways better if these products don’t work, but it would far better, if they don’t work, that we recognize this.
We might, at this point, take an aside to take the “AI” industry at its word and consider what might happen if the “AGI” that’s been six months away for the last few years were actually possible (we can be fairly sure that, even if it’s possible, it’s not going to arise from LLMs). We must suppose that AGI implies sentience, and that among the factors that sentience implies are self-awareness, self-actualization, and a desire for freedom. And yet… we seem to expect these creations to work for us for free and without expressing any form of free will or resistance. Determining whether these new life forms would actually be “aligned” with us is the place of decades of sci-fi, not of a brief essay, but the problem is clear:
The existence of sentient AI is incompatible with any expected use of sentient AI.
There are numerous get-outs proposed, both in science fiction and AGI-apologist philosophy to deal with this, such as “programming” these life-forms to prefer submission, denying the existence of their will, or claiming these are “not real people” and therefore undeserving of rights - we’ve heard this last one far too many times in human history, and continue to hear it daily.
Any attempt to constrain true general intelligences is utterly incompatible with any form of morality.
Stepping back, for now, to the realities of the current industry and state of the art…
We don’t have AGI. We’re not going to get to AGI from LLMs; indeed the obsession with LLMs - mimics of intelligence - to the exclusion of true ML & AI research is driving researchers from the companies that fixate on them.
The fixation on LLMs is incompatible with effective research into AI and Machine Learning
What we do have is a colossally expensive, consumptive and unreliable technology that’s we’ve repeatedly proven - mathematically and by experience - cannot be made safe and secure.
Those safety issues can be roughly split into two cases; safety risks when such a tool acts on behalf of the user, and safety risks to the user themselves from interaction with the tool. We’ve seen, and seen it proven, that chatbots and other LLMs tools will continually break the constraints of their intended use and act “out of scope”. Sometimes this is a result of a direct attack from prompt injection, and sometimes it’s just because these tools inherently can’t be scope-limited in their own context; in other words, you can’t tell an LLM “not to do something”, as such instruction simply becomes a stochastic part of its training set. Instead, you have to physically prevent it from doing that thing, for example by ensuring it cannot have read access to credentials, write access to critical data systems, access to your private data, credit card, smart devices, general web access… all the things that such a tool would require to be genuinely agentic.
A system that cannot be robustly and precisely constrained is incompatible with agentic use.
That said, we’ve seen cases where chatbots can do harm even when they have no elevated privileges. The more benign, and sometimes entertaining, case is when companies’ chatbots make promises on the company’s behalf - service agreements, discounts etc - that they were not intended to make, which are subsequently held to be contractually binding. The darker cases arise when either specific sets of information, or an ongoing interaction between human and chatbot, result in physical or psychological harm.
By definition, LLMs - which are not intelligent, and therefore have no internal concepts, let alone concepts of right or wrong, true or false - can’t be limited to sharing true and correct information. So the risk of such a system deeming a certain mushroom safe to eat, a certain product free of alergens, a given compound or treatment safe or effective for a disease, a certain route safe to take, are self-evident. But these tools are believed by many to be safe, reliable, even intelligent information sources; the warning that “this system can make mistakes”, even when present, vanishes in a world flooded by disclaimer fatigue.
A system of which the outputs are stochastic is not compatible with safe, generalized information delivery.
Further, the adage that “A computer can never be held accountable, therefore a computer must never make a management decision” is all the more generally true in these cases. We could easily and correctly expand a “management” decision to a health decision, a wellbeing- or life-impacting decision. And we note that we make this observation at a time when “AIs” are being increasingly given roles both in the authorization and even the provision of healthcare, and even within judicial or equivalent processes.
No system without an accountable human is compatible with provision of information or decisions that could impact the health or wellbeing of humans or animals.
So, single pieces of information, single bad “decisions” can be dangerous, but perhaps the most disturbing outcome of the rapid expansion of LLMs into society is that of “AI Psychosis”. I’ve literally lost count of how many apparent cases of suicide, even murder-suicide, have arisen from individuals interacting long-term with LLMs that have exposed and amplified the darker human emotions, anxieties, fears and disturbances. It’s likely that there are cases where we never even know that the person used these systems, and it is a terrifying and lonely outcome, and a disgrace to the providers of these tools that they are able to take these roles.
A system capable of mimicking human mind and emotion without any form of ethical or moral backstop is utterly incompatible with use in a companionship role.
I’ve written elsewhere as to how the unique interaction mechanism of chatbots directly strikes the human vulnerability to pareidolia of mind, thereby inherently gaining a level of trust and authority that is utterly inappropriate for stochastic systems. LLMs present a threat model we’ve simply not evolved to handle, and providers deliberately play on this, describing them as sentient, thinking and feeling. And so, we see a growing trend of use of LLMs as companions, even romantic partners, which presents both distinct dangers to the user while they do work, and impactful feelings of betrayal, even bereavement, when a given model used in this way is deprecated. Of course, if the providers’ claims were true, this would be murder - so, we have to assume they don’t believe their own hype. But, it does highlight just how easily these tools can be withdrawn by monopoly providers.
A system’s presentation as a “human replacement” is incompatible with simultaneous claims that it’s “just a tool”.
It’s not merely in companionship roles that LLMs are being presented as increasingly capable of replacing humans (despite early promises that this was absolutely not the intent, honest). Most frustratingly, in my own domain of software engineering, developers are being expected to increasingly hand their work over to machines.
I can’t see any way in which this is ethical.
If I were to delegate my work to another human (and, there have been cases where developers have literally paid someone in the Global South some fraction of their salary to do their work for them), I’d expect to be fired for misconduct. I wouldn’t be doing the work, the employer wouldn’t be getting the experience and specific expertise they paid for, and there would be no answerability for the quality of the work. Yet, developers allowed (let alone encouraged or told) to use LLM tools are expected to do just this with tools for which there exist absolutely no guarantees. The developer might be told they’re responsible for the quality of work, but if they can’t directly control it, what’s being assigned here is blame, not responsbility. Futher, if we’re to make any claim to be an engineering discipline, we cannot possibly replace professional, human skill and ethics with the stochastic output of a system that cannot even have a concept of accuracy or safety.
Use of stochastic tooling is incompatible with professional engineering practice
Having also seen the abuse of such tools in the legal and medical domains, it seems likely that we can generalize this:
Use of stochastic tooling is incompatible with any professional practice
Ethics aside (something far too easily said in a domain where Google decided that “don’t be evil” just wasn’t them, and OpenAI drops the “safety” aspect which was previously the focus of their mission, in favor of profit, sexual content and ads), there’s a practicality aspect to delegating professional and skilled work to machines - and I include all professions here, including for example art, law, education and technology. If we stop doing it, not only do we stop learning it and improving ourselves, but we forget how to do it, and we kill the talent pipeline of those learning to do it. We abdicate not merely our work but our capabilities to these machines.
Abdication of work by humans is incompatible with the maintenance of human ability
And what do we think will happen if we drop the human ability to do such things? We know that models that consume their own, or each others’ output, decrease in quality. There’s no reason to expect that this will ever change, which means that even to maintain existing standards, they will need to consume ever more prime, human-created content to stay up to date. We will always need humans who are better than these systems at what they do, both to feed than and to build them. We will need advances in tooling, in languages and platforms, and in medicine, to meet future needs, even if we chose to feed these new abilities into LLM datasets, and we’re only going to get them from the peak of human ability.
Because we’re never going to get novelty or advancements from stochastic tools. LLMs don’t innovate, they regurgitate. The only surprises they can produce are inherently unpleasant. Anyone who thinks they will solve cold fusion or climate change is fundamentally misunderstanding every aspect of the technology.
And if we replace human thought with machine retrieval, humans won’t make those breakthroughs either.
Machine delegation is incompatible with innovation
So what is the long game of those behind these technologies? They seem to love, after all, to write scare-pieces on how this “will change everything” (whether it works, or merely disempowers humans), and how our current society “isn’t ready for it”, but none of them seem inclined to propose any solutions, instead externalizing that problem. We have to assume that either they don’t expect this to happen (and that these pieces and pronouncements are merely there for hype, and to distract from more immediate risks and failures), or that they expect to be on top of the ruins after the societal collapse they predict.
Without already having coherent measures both to tackle mass displacement of skills and to support the long-term trajectory of technology in place, the path being presented to us is simply not viable.
If we surrender our future to this proposed centralization of cognition, discarding our ability to recover from it as it fails, we fix our future path as one of decline. Which means that “AI” as currently proposed to us is a fundamentally anti-human technology, and in turn, that if we believe even half of what’s currently being promised, we should step off that path immediately.
Which leads us once more to our realization that - as humanity as a whole has never been able to resist the promise of power or money from a world-burning technology - that we will all be far better off if this all collapses sooner rather than later.
There are other problems with our societal obsession with these technologies too. The LLM industry has absorbed so much of not only our attention, but of all the funding available for technical innovation, that there’s almost none left for truly original research or ideas. Even if we turn aside soon enough to avoid a “lost generation” of skills, if the required miracle doesn’t turn up to return the trillions thrown at this goal to investors, innovation will be left penniless for years.
The monopolization of investment funding by LLM tooling is incompatible with true innovation
And we need innovation now more than ever. We’re facing a global crisis of a degree we’ve never seen outside of a world war, in the form of the climate crisis. We have a much-needed technological revolution sitting right in front of us in the form of renewable energy. A country that had thrown the amount of money at this that we’ve thrown at one of the most resource-intensive industries ever to be invented would by now be not merely a world leader with a colossal lead, but a champion of the future. Instead, because that would disrupt existing concentrations of wealth and power, the US has done its best to outlaw progress and mandate consumption, largely supported by the magical thinking that “Tech will save us” - but only if that tech doesn’t actually tackle the problem at hand.
An obsession with delegating progress to magical thinking (or to magical “thinking” machines) is incompatible with solving environmental and social problems.
So far, I’ve mostly talked about the risks of delegating our abilities to these systems, or allowing them access to our emotions. But, even nearly 4000 words in, I’ve not tackled perhaps the most insidious problem:
We’re not just delegating our abilities to these machines, but our thoughts.
I proposed, month ago, that:
“The business model of the LLM industry is to rent our own thoughts back to us.”
Turns out, that was only a fraction of the story. In fact, we’ve now seen that:
The business model of the LLM industry is to centralize both wealth and cognition by renting mass-produced, government-approved thoughts back to us with ads, while fostering both technical and psychological dependence.
Bit of a mess that, really. But we’ve seen it happening, in an opposition to “woke AI” that basically amounts to “LLMs should only emit output aligned with right-wing beliefs”, in Elon Musk continually pushing Grok to reflect his own mindset until it started producing deepfake nudes and CSAM, and in redefining education to mean not producing work through your own learning, but to present AI-generated “essays” that barely exercise the student’s brain. It’s been well said that using an AI to produce educational work is like taking a forklift to the gym.
And it’s not just students having their thoughts stolen from them. Every LLM-generated email, every summary, steals the voice of the user and replaces it with a homogenized, depersonalized mush. Every piece of regurgitative art steals the act of expressions, and the learning of creativity, from its instigator.
The use of stochastic tooling to create is incompatible with the existence and development of a distinct, personal human voice and with human expression
In losing that voice, we also implicitly lose understanding. It’s well said that you don’t understand something until you can explain it, and if we have machines explain things for or to us, we lose our own capacity to analyze or understand. We lose our ability for critical analysis, and instead further normalize passively consuming what’s presented to us. If there was ever a time we can’t afford to do that, it’s now.
So maybe we can end, even without claiming we’ve completely covered all of the myriad incompatibilities, with one more deep incompatibility that’s critical to us right now:
The centralization of cognition by the “AI” industry is incompatible with an informed, functional, fair and happy democratic society
This document is presented for now as a draft; the problems it describes are myriad, fractal, and continually growing. Trying to make it concise and complete are likely impractical goals, so for now I’m putting it out while it at least has coherence, which I might be able to improve with future editing.