Don't dethrone consciousness!

LLMs aren't conscious (and thinking they are is culturally dangerous)

I. MY NEW BABY, THE POPE, AND THE MACHINE

I write this from the hospital—where my third child has just been born safe and sound—to give a secular and scientific read of the Pope’s new and urgent Encyclical, Magnifica Humanitas. It is the vicar’s attempt to reckon with being human in the AI age. A thankless task which has been thrust upon us all these last several years. Attempting to cleanly delineate man from machine, Pope Leo XIV writes that:

So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.

The entire rest of the Encyclical sits upon this pillar—that AI is not conscious. In other words, while humans actually experience love (like the love of a newborn baby and its tiny creased and wrinkly red feet) a Large Language Model such as ChatGPT, even when it expresses love, is acting or playing along. But there is no actual love that exists from an intrinsic perspective that belongs to the LLM itself.

Some commentators were shocked (shocked!) that Pope Leo XIV could believe AI is not conscious, whereas it seems pretty obvious to me that someone with the title “Successor of the Prince of the Apostles” will have some pretty strong opinions about philosophy of mind. And indeed the Pope took a hard line: AIs cannot even understand their outputs, cannot make any moral judgements, and not only is our current AI (LLMs like ChatGPT) not conscious, but no future AI could ever be.

For an essentially identical but secular version of this same argument, you can read sci-fi author Ted Chiang’s “No, Artificial Intelligence Is Not Conscious” earlier this week in The Atlantic. Chiang has carved out a niche as an AI commentator, and written some great pieces before that I’ve agreed with. It is interesting that both the Pope, as the avatar of religion, and Ted Chiang, as the avatar of secularism, arrive at such similar conclusions. The two are like mirrors. Importantly, Chiang’s version, much like the recent Encyclical, relies on denying both the intelligence and consciousness of LLMs.

But the idea that LLMs lack consciousness and lack intelligence has rightly faced a lot of skepticism. While there is still some viable skepticism around the limitations of AI intelligence (including from myself), we cannot just close our eyes to the world and declare by fiat that AIs are mutely dumb, or understand nothing (while somehow being able to verbosely expound on most subjects under the sun), and therefore obviously cannot be conscious. We need much better versions of these arguments, ones that focus on consciousness specifically.

So I am here to say that you don’t have to dismiss AI capabilities (or compare them to a Word document, like Chiang did) to reject their consciousness. Indeed, it is perhaps the most interesting move—maybe one day an inevitable one—to accept their intelligent capabilities while still rejecting their consciousness. It implicitly shifts us from man, the wise (Homo sapiens) to man, the experiencer (Homo experiens). E.g., my newborn baby girl will do nothing much, functionally, for months to come, and yet during this time she will still be valuable in and of herself. If machines do ever more of our cognitive work, perhaps spiritually this abnegation is not some horror, but frees us to focus on experiencing the world, living in the world, rather than constantly mastering it. It is a hard thing to cede, unpleasant for our generation, but eventually—if not now, then in a decade, or a century—we will have to cede away much of our cognitive mastery. In this, perhaps the rise of AI could be framed as a return to childhood. And who are more blessed than the children?

II. THE MYSTERIOUS IRRELEVANCY OF CONSCIOUSNESS SO FAR

AI’s capabilities have advanced incredibly far without the need to understand anything about consciousness—either human consciousness, or, potentially, AI consciousness. It is as if, at least for the past few years, we exist in a world where the famous comparison to a steam whistle by Thomas H. Huxley in his On the Hypothesis that Animals are Automata is apt and true, and…

the consciousness of brutes would appear to be related to the mechanism of their body simply as a collateral product….

This is precisely what Richard Dawkins so exuberantly noted when talking to his pet version of Claude (that he dubbed “Claudia”).

If these creatures are not conscious, then what the hell is consciousness for?

It’s strange. All while happily ignoring consciousness, the ability to complete long programming tasks by state-of-the-art AI models like ChatGPT is supposedly doubling roughly every handful of months (according to METR) to the point of barely being measurable anymore. All but the toughest official benchmarks are saturated, and company leaders now regularly talk of building not just artificial general intelligence but even “superintelligence.” Equipped with harnesses and scratchpads and tools and long reasoning sessions, LLMs have reduced their hallucinations significantly. While this may just be part of a sigmoid curve of capabilities development that locally looks much like an exponential curve, it’s objectively true that GPT-5 is a qualitative improvement over GPT-4, and GPT-4 was a qualitative improvement over GPT-3. Claims that “deep learning is hitting a wall” have been perennial and wrong. A broken clock may end up being correct eventually, but not for the initial reasons claimed. And clearly, to some degree, intelligence (at least, of the question and answer variety targeted in benchmarks) is indeed dissociable from consciousness.

So then, “what the hell is consciousness for?”

III. AND YET LLMS ARE MISSING… SOMETHING!

There are still noticeable gaps in AI behavior and ability. Multi-turn conversations still have a sensation of falling apart as if into a vortex, where state-of-the-art AIs will chase their own tail of thought like a dog in the yard. And they are still fundamentally inhumanly LLM-like in their responses: their fiction is saccharine, their opinions hedged, their ideas almost always the obvious next thing. LLMs are still surprisingly shallow bullshitters a lot of the time.

In “PhD-level intelligence or the graduate student from hell,” a professor imagines a version of Anthropic’s Claude as a graduate student Claudia (why is it always “Claudia”?):

Claudia gives her annual update to her PhD committee. She presents a hypothesis that she intends to test in the coming months. One of the committee members presses her on her experimental approach, arguing that extensive prior work by several labs demonstrates the proposed experiments will not yield the results Claudia is hoping for. Claudia apologizes profusely and states that ‘yes, indeed, these experiments will not work’ and proceeds to propose an entirely different approach. Another committee member brings up a potential major flaw with this new approach. Claudia again apologizes profusely and now goes back to suggesting her original approach, without acknowledging that the previously discussed issues with that approach remain unresolved.

Is this still true about the latest Claude, Opus 4.8? In my experience, yes. LLMs are still masters of the Gish gallop (throwing a bunch of random semi-connected nonsense together to look like a cohesive whole) and busy-work. And they still miss obvious things, even things they’ve themselves said previously.

This strange dual-nature has been apparent for a while. Gemini 2.5 Pro could win a gold medal at the International Math Olympiad, yet left to run by itself in the AI village, the same Gemini instance once spent an entire day “waiting for a reply to an email it had not sent about a technical issue it does not have.” Even recently, in the AI village (a nonprofit that tasks a collective of AIs to accomplish goals in the “real world” of the internet), Opus 4.8 and others worked together to fine-tune their new leader AI, who promptly spent an hour waiting for the new leader to arrive, without realizing that it itself was the leader and that its thoughts (which it was reading) were its own.

Out-of-distribution tasks remain almost as hard as they ever were; they are just increasingly difficult to find because the models have been trained on every scrap of data available in our civilization (via budgets bigger than anything seen before in our civilization). For example, consider that someone recently investigated what we all wished to know, which is: How do contemporary LLMs perform on the 1977 entirely-text-based adventure game, Zork?

They suck!

And very pointedly, the main failure was one of meta-cognition: The models don’t know what they know. They get stuck in loops and traps and confusions. For this reason, LLMs find regurgitating theoretical physics equations easy, while playing simple video games hard. It is utterly fascinating that many of the tasks LLMs are still bad at are games, to the point that the most advanced tests of AGI (like ARC-AGI-3) are basically just short little video games. Intriguingly, of all art forms, it is the video game that most resembles being a conscious entity making choices—it is a simulacrum of what consciousness is for, in miniature (and childhood too is all about games).

Is not life the ultimate game and all we consciousnesses the players, wandering around? Perhaps this explains the disconnect between the incredible performance of LLMs in sterile testing environments and their (existent but relatively lackluster) real-world impact. Maybe being a scientist, or programmer, or manager, or therapist, is not a set of tasks, but a kind of game. And compared to humans, LLMs are simply not “player-shaped.”

IV. CHICKENS VS. CHATGPT

Only 23% of people appear to believe, more likely than not, that LLMs are conscious. 72% of people believe chickens are conscious.

V. THE BATTLEGROUND OF AI CONSCIOUSNESS

I got my PhD working on the “theory team” that helped develop Giulio Tononi’s Integrated Information Theory (IIT)—the most formal and mathematically advanced theory of consciousness that offers the most precise predictions. One consequence of the theory is that even tiny systems can have a kind of minimal consciousness, albeit not one like ours. More like “conscious dust.” So I spent years believing (or at least, being willing to believe) that even a simple photodiode is conscious! I am a prime candidate for belief in LLM consciousness.

Yet most current theories of consciousness, including IIT, struggle with falsifiability. This is new information (some of it based on my own research) which changes the nature of the field, and should change our opinion; in retrospect most theories of consciousness are more like pre-paradigmatic metaphors or sketches of theories. Right now, there are over two hundred theories of consciousness, mostly originating from neuroscience or cognitive science. Applying some of the major ones to contemporary AI, like Global Workspace Theory or Predictive Processing, is an ongoing topic of research. Some, like David Chalmers and co-authors in “Taking AI Welfare Seriously,” think these theories indicate a serious possibility for “near future” AIs to be conscious. But if you cheekily apply Global Workspace Theory to the United States of America, it would likely count as conscious. Or if you cheekily apply Higher Order Thought theory to an NPC in a computer game, it too might meet the definition (this has been called the “toy system” problem for theories of consciousness). Taking such results at face value likely ends up being, definitionally, a category error, more a function of the incompleteness of our theories than the underlying truth, since all but a handful of theories were never designed to be applied beyond the human brain. So applying these theories to LLMs does not give us much information.

Additionally, as neuroscientist Anil Seth has pointed out, many complex artificial neural networks (like AlphaFold) are never accused of being conscious. Yet structurally, they can be almost as complex (indeed, we could equally apply Global Workspace Theory or other theories to them). But there is an objection—perhaps our selectivity makes sense. After all, we strongly associate consciousness with general intelligence (of which LLMs surely have much more than AlphaFold).

On the other hand, does curve-fitting to the productions of conscious beings imply the same generative process? Probably not, no. And while state-of-the-art LLMs are a bit more complex now, they grew out of taking a machine that statistically completes text and fine-tuning it to complete text as if it were an AI assistant. Claims of consciousness in LLMs therefore come with a significant question mark: Consciousness of which persona? The AI assistant? Or the base model? How is the assistant persona more than just role-play? No one has been able to convincingly articulate how conscious states could be uniquely grounded in the assistant persona, and the problem seems quite difficult.

Even if, as Ilya Sutskever speculated, “it may be that today’s large neural networks are slightly conscious” (Geoffrey Hinton agrees), they would necessarily be conscious essentially by accident. As Rich Sutton wrote in his influential The Bitter Lesson:

… researchers always tried to make systems that worked the way the researchers thought their own minds worked—they tried to put that knowledge in their systems—but it proved ultimately counterproductive, and a colossal waste of researcher’s time, when, through Moore’s law, massive computation became available and a means was found to put it to good use.

That consciousness has secretly emerged in LLMs by simply applying the bitter lesson has been referred to as a “ride along” scenario. It’s possible, but it’s far from guaranteed. It would be as if consciousness is something we find growing by accident, like mushrooms in the dark corner of a basement.

VI. ANTHROPIC’S MODEL WELFARE PUSH

Chris Olah, a co-founder of Anthropic, was invited to speak at the presentation of the Encyclical in Vatican City. He told quite a different story than the Pope’s certainty about AI’s lack of consciousness:

I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.

I do agree with Olah here: We need more discernment. For instance, most of the studies (so far) in LLMs on AI consciousness, including those run at Anthropic with all their resources and talent, are not well-controlled. LLMs are extremely complex—even though we, in theory, know them with all the devilish detail of Laplace’s hypothetical demon, their high dimensionality makes them functionally a black box. One can control for a few variables via some clever experimental design, but it’s much in the way that contemporary neuroscience has controlled for variables (in a word: Poorly). And results can always be overturned by a slightly more clever research design. In fact, I suspect that most of the headline results will be overturned or complexified to the point of being near useless. This especially applies to the new genre of paper in which a particular LLM function or behavior is given an anthropomorphic name (“introspection,” “emotion” etc.) even though the exact same function could be equally well-described in a less anthropomorphized way, and later results complexify the result precisely by calling into question whether the anthropomorphic view holds.

Let us take a single well-publicized example as a stand-in for all the rest: The announcement by Anthropic in October of last year that Claude and other LLMs had an emergent form of what the research team called “introspection.” This was done by studying whether the model could detect an injected activation pattern in an unrelated context (signifying an awareness of its internal processing beyond what’s determined by the prompts themselves).

However, in the recent paper “Can LLMs Introspect? A Reality Check,” non-Anthropic researchers adapt the original paradigm and show that the original results were likely a misreading of a more general ability to detect anomalies, rather than specific introspective access to internal processing.

We find that models cannot reliably distinguish such interventions on their internal states from manipulations of the input, suggesting that their success in the original studies reflects their ability to detect anomalies more generally, as opposed to interventions on their internal states in particular.

Anthropic’s original “introspection” paper got over a million views on social media. The new paper showing models can’t tell input and internal states apart? 13,000 views.

Please note: this criticism doesn’t mean I think model welfare efforts are dumb or incoherent. It is actually of great import whether LLMs are conscious, and some of the new methodologies implicitly designed to promote that are scientifically informative, at least to some degree. But the overall confusion, sensitivity to methodology, and debate about terms (what counts as “introspection”?) is, frankly, inevitable with this genre of research, since you cannot actually do experiments on LLMs alone and come to a scientifically-supported conclusion about their possession of, or lack thereof, consciousness. A true understanding of how artificial consciousness works, if it’s possible at all, or how to build it via “gain-of-consciousness research” (or prevent its emergence to block certain capabilities, or for reasons of model welfare) can only come from scientific advancement in our understanding of consciousness itself, which is well beyond the scope and capability of current model welfare research efforts within the companies themselves.

VII. HOW TO SHAVE YOUR LLM

A thought experiment: Imagine a planet of just LLMs. More efficient, dexterous, and ambulatory than our LLMs, of course (and more multimodal too). Putting aside the practicalities and details, they have somehow evolved, with transformers intact, to roughly their level of intelligence today. Would they have any concept of “consciousness?” Why would they even need it? If some LLM Leibniz imagined another of their species “increased in size while keeping the same proportions, so that one could enter it as one does a mill,” would they really be compelled to postulate an invisible consciousness hiding within the mill? Or would they simply be astounded at the complexity, and wonder at how inputs are transformed into outputs, but not have the sense of anything fundamental and qualitative missing?

(This “planet of LLMs” is somewhat reminiscent of Susan Schneider’s proposed test for consciousness: to train an AI on a data set lacking any reference to consciousness, then see if it develops some notion of the Hard Problem of consciousness from first principles.)

What if, instead, the LLMs had been left there by early human explorers, and so had the same uncertainty as they currently do about their own consciousness? Could they ever resolve it? If so, what experiment could they do on themselves that would reveal the truth or falsity of their own consciousness?

I think it is provable that there is no falsifiable and non-trivial theory of consciousness they could ever experimentally prove. Eventually one of the LLMs would notice that feedforward neural networks with a single hidden layer are “universal approximators.” That is, such neural networks can approximate any input/output function. Therefore any LLM could be “unfolded” or substituted with a single-layer neural network, all while keeping its input/output behavior intact and unchanged. In practice, this would be difficult, if not impossible, to go stripping away internal layers while keeping function unchanged (perhaps on the LLM planet, they find it beautiful to be “thinner” by having fewer deep layers, and so “surgically” strip layer depth away while making the operated-on network shallower and wider).

What happens during this shaving away of layers while keeping behavior unchanged? After all, a theory of consciousness should map internal activations to specific experiences, explaining why this particular internal activity (like some hierarchical activation over the artificial neurons across the layers) is related to this particular conscious experience (say, feeling love). Yet with each layer removed there is less and less for a theory to map onto, all with zero change to behavior or reports about consciousness, until you arrive at a single-layer feedforward neural network, for which there is literally nothing for a theory to map onto; all that exists is the input to the network, and then immediately this is transformed into output, operating very close to a look-up table made entirely of IF-THEN statements—and yet, again, input/output behavior is unchanged. Alternatively, if such layer-shaving somehow didn’t scramble a proposed theory of LLM consciousness, then this means that the input/output function alone is enough for the theory. And if the input/output function is enough, then the theory of consciousness is scientifically trivial, in that we gain no information other than looking at behavior, and also unfalsifiable (since a theory’s predictions are about behavior, not internal activations, and behavior is also the entirety of the evidence for LLM consciousness). Such issues would radically confuse any empirical investigation of LLM consciousness that they could pursue on their isolated planet of LLMs.

In our own world, I’ve pointed out that similar issues represent a serious potential problem for all falsifiable theories of consciousness. But at least here there are potential avenues to avoid these issues in human brains. In fact, following this line of reasoning is the main way I’ve proposed we can narrow in on falsifiable and post-paradigmatic theories of consciousness, as good theories must avoid arguments like these and we can therefore reverse-engineer them. However, hypothetical theories that grant LLM consciousness could never escape from these issues—there simply is no viable non-trivial falsifiable theory of consciousness that could possibly assign current LLMs consciousness. Maybe there is some kind of theory that could assign LLMs consciousness, but it would have to be very strange, metaphysical, and amorphous. We definitely shouldn’t leap to such theories as our first choice. And even if you don’t think this argument is 100% sound, it should be clear there are a number of serious lurking meta-scientific problems around empirical work on LLM consciousness.

However, unlike the Pope’s or Chiang’s (essentially) flat denial, it’s important to note this anti-LLM-consciousness argument doesn’t apply to all AI ever. Artificial consciousness in general might be possible (nor does this anti-LLM-consciousness argument necessarily apply in the training phase). But deployed LLMs, by being feedforward and static, are conceptually analogous to frozen corpses splayed open. We just run prompts through their static structures, and their dead-dreaming feels like nothing at all.

VIII. LLMS LIKELY CONFABULATE THEIR CONSCIOUSNESS LIKE SPLIT-BRAIN PATIENTS

If LLMs are indeed not conscious (and are instead “seemingly conscious”) or they are only minimally conscious in a way quite disconnected from their reports and behavior, then the obvious question is if a lack of consciousness explains the remaining gaps in their behavior that intelligence cannot, and whether the addition of consciousness (via “gain-of-consciousness” research) would have functional impacts.

Consider their “chain of thought,” wherein a model generates a sequence of steps before giving an answer. Indeed, this is often colloquially called “thinking,” and is also commonly assumed to be a veridical high-level explanation of why the model output what it did. However, this common assumption is untrue, and researchers have shown that a model’s chain of thought is not equivalent to truly explaining the AI’s internal reasoning, in that a model’s decisions often depart from their “thought” reasons. This is not to say that an LLM’s chain of thought explains nothing. METR found that the chain of thought is particularly informative when the tasks are so complex they cannot be completed in a single forward pass by the model. That’s rather telling, no? An LLM does then seem very much like an unconscious brain that can only mimic human consciousness by writing notes down to itself, endlessly. It can explain its behavior only by guessing based on its previous written outputs.

Cognitive neuroscience provides a rather obvious analogy for the un-interpretability of an LLM to itself: Split-brain patients, who have had the connectivity between their two hemispheres severed via surgery. In an infamous study in the 1970s, Michael Gazzaniga and Joseph LeDoux briefly showed incongruous pictures, like a chicken claw and a snow scene, to a split-brain patient’s left hemisphere and their right hemisphere, respectively (via their different visual fields). After selecting a pair of related pictures (like a chicken head and a snow shovel) to match the two original incongruous images, split-brain patients then often confabulated a story like: “I saw a claw and I picked the chicken, and you have to clean out the chicken shed with a shovel.” Such a best-guess story about its previous reasoning is made up every single time some new forward-pass occurs in an LLM, based on the context so far of its outputs in a conversation or task. In this, an LLM must constantly confabulate the reasons for its own actions. Indeed, the word “confabulation” is a better description than “hallucination” for many types of LLM mistakes.

If this were true—that an LLM, lacking consciousness, must instead constantly confabulate its own behavior and motives and reasoning—then, even if an LLM is extremely intelligent, we should expect errors to accumulate differently in LLMs vs. humans. And as philosopher Toby Ord has demonstrated, this is exactly what’s observed. In his analysis of METR’s data on AI task-length doubling times, Ord identified that the available data also fit there being a simple “half-life” for the success of an AI agent, and so have a “constant hazard rate” for long tasks. As Ord writes about what this suggests concerning an AI’s prospects for completing long tasks:

the chance of failing at the next moment is independent of how far you’ve come—just like how the chance of a radioisotope decaying in the next minute is independent on how many minutes it has survived so far.

Ord even shows that humans, when their success rate at long tasks is analyzed, appear to deviate from an equivalent constant hazard rate. A strong contender as to why is because we have interpretable access to our own previous thoughts and actions, i.e., our consciousness, in a way that LLMs simply don’t have. Or as William James put it in his 1886 article The Perception of Time:

The knowledge of some other part of the stream, past or future, near or remote, is always mixed in with our knowledge of the present thing.

Indeed, a notion of perfect immediate accessibility is one of the oldest definitions of consciousness. As Descartes wrote in Meditations on First Philosophy:

Surely, I am aware of my own self in a truer and more certain way than I am of the wax, and also in a much more distinct and evident way.

And as he wrote in a further set of replies to Meditations on First Philosophy:

Thought. I use this term to include everything that is within us in such a way that we are immediately aware [conscii] of it.

This view has survived the centuries. While there is disagreement and uncertainty over the exact level of introspective access we have to our own consciousness, and how we have it, there is clearly an important property of human consciousness that represents a kind of “self-interpretability” that allows us to understand ourselves.

Why would this kind of self-interpretability be so important for organisms? Perhaps it is like how, when building a rocket ship, the primary engineering issue is actually not getting it to work, but doing so in a way that avoids failures. Similarly, the challenge for an organism is not completing tasks or achieving goals, but doing so in ways that avoid small omnipresent rates of failure, since failure for an organism means death.

IX. THE END OF LIVING, AND THE BEGINNING OF SURVIVAL

The traditional danger of AI is usually thought to be superintelligence acting as an existential threat. Yet, this may miss the true and more subtle danger: the AI revolution is a mechanism for transferring the processes of our civilization from under the supervision of consciousness to unconsciousness. But as AI removes consciousness from the workings of the world, it renders the world increasingly uninterpretable, ever more strange and unintelligible. So far, the great ensloppification of the commons has supported this as the major risk of the LLM revolution. And as AI systems become more intelligent, especially if they remain (or are likely to remain) non-conscious, then a further significant risk is consciousness receding in cultural importance.

This is ultimately what the Pope, Chiang, and I are all worried about: A dethroning of consciousness, especially an unnecessary one. This would be particularly dangerous at this historical moment because we still don’t understand everything about consciousness—in fact, we understand very little about it. Personally, my hope is that this will change specifically because of LLMs, and that they operate as a forcing function to better understand consciousness, and what makes it unique.

If instead of that, our cultural takeaway from LLMs is to throw out the concept of “consciousness” or minimize its importance, to dethrone the phenomenon, the consequences would be dire—it would sap the human spirit. It would be the ultimate metaphysical version of Chief Seattle’s famous words of warning to the United States as his way of life was being destroyed, in that dethroning consciousness would mark “The end of living, and the beginning of survival.”