With much of the company literature (not merely Anthropic's), we are getting "If it looks like a duck and quacks like a duck, then (maybe) it's a duck." This avoids the problem that we don't understand what a duck is in the first place.
One more observation, as an AI researcher. Even beyond the protein issue I identified earlier, the paper is flourish. It is just a standard interpretability result. The claim reduces to: a transformer keeps a small set of readable directions in its residual stream that hold intermediate values of multi-step computation and get reused by many downstream components. This statement is about representational format and reuse, which you should expect from any neural network that has to chain steps efficiently. Even the author acknowledge "computationally efficient." The global workspace framing adds no predictive content. Every result the paper reports follows directly from representational efficiency. The one property that I would want to see (which would distinguish the workspace), is sharp nonlinear ignition. This is the one they cannot show. The paper's real contributions are narrower. (1) The Jacobian lens recovers intermediate representations that the logit lens misses. (2) The directions it finds are the same ones that causally drive behavior.
What I'm getting from this is that Anthropic has published not so much a theory as an in-depth analogy to a theory, sort of like saying a toaster is like a wildfire because they both produce heat at first gradually and then intensely, and therefore toasters in the right circumstances could have the ability to restore health to forest ecosystems. Is that about right? I don't have the knowledge to fully follow what is being said. But I'm having trouble understanding why Anthropic's "research" would be taken seriously, and why we would have an expectation at all that a mechanical computation system, no matter how sophisticated/powerful it is at analyzing massive inputs consisting of the output of biological systems (ie human brains), would be able to replicate a biological system? (I do sort of get what you are saying about the difficulty of measuring consciousness, which is interesting.)
The burden of proof is on people who say that carbon-based systems would for some reason be _intrinsically_ different from silicon-based systems. Brains simply are (crappy and overloaded on heuristics and parallel processes) mechanical computation systems.
I still maintain that theories of consciousness keep ending up trivial because consciousness just _is_ trivial and non-interesting and you are fighting over definitions with no useful information provided by "is this system conscious". Like, if you keep running into a wall, that is usually a sign of you using a wrong road entirely.
I think it’s a fine reinterpretation. But I don’t know if I’d go that far - the thing is, working memory is supposed to be something more the smeared reportability too!
I don't understand this reverse-engineering approach. It appears unscientific to assume a scientific theory must have a certain shape. Science works by:
1. A new phenomenon is observed.
2. There is no model for how this phenomenon works.
3. Through experiments, a model that explains how the phenomenon works is developed.
But with consciousness, we have not observed it as a phenomenon. We have access to our own consciousness, through common sense we assume other humans are conscious, but being scientifically skeptic about it leads to solipsism.
What we really need is a consciousness detecting instrument that could be applied to anything, if we have that, then observations could be gathered to develop a theory of consciousness.
First, that's really not how science works. Science is not just some downstream explanation of measurement devices that come out of thin air. You often have to build a specific measurement device first, based on what you hope to see, in the hopes of making your theories better. People do this all the time, but an obvious example in physics.
Second, we do have a primitive consciousness-detecting instrument! The human brain. You yourself said that "we have access to our own consciousness." But if your point is that you need some sort of objective 3rd-party consciousness detector beyond that, which you can rely on 100%, that's something that gets built after, or in tandem with, a theory of consciousness.
It just all has me wondering - is this actually something we can even approach a theory for? Like, I see your argument - that these reportability-based theories we're going with aren't good ones and we need something better. But how much can measurement actually cross over from observable into the experienceable? Granted, I write from a phenomenological point of view here - I don't think we can reduce something like consciousness into something so easily measured with any kind of theory or experimentation in humans, birds, trees, or LLMs. We can map behavior and physiology with plenty of precision and STILL have no idea what it's actually like to BE that thing. It's not that we haven't closed that gap yet, it's that that gap simply isn't closable.
So I guess my big question after reading this (very detailed well articulated) piece is what if the problem isn't that consciousness science is pre-paradigmatic but that measurement was never going to be the way in in the first place?
(And maybe this is something you've talked about before - this is my first time coming across your substack and I'm looking forward to taking a look through your archive!)
There is a fair bit of gloss in this paper. The one that I can directly call out is the party trick of showing something on proteins (something other than language), where most people will find something magical happening. This is a general chat model (Sonnet 4.5, Haiku 4.5 for the oracle lens), not a protein language model. The recognition they claim fires roughly five residues into the sequence. MSKGEELFT is the canonical avGFP N-terminus, one of the most-memorized sequences in all of biology! What the experiment demonstrates is that a general LLM has stored the canonical GFP prefix well enough to retrieve its identity, and that this retrieval is an unspoken intermediate in the workspace rather than in the output. This is not evidence of sequence-to-function inference. There is no held-out or novel-sequence control, no test on an obscure or mutated protein, no function prediction where memorization is unavailable, and no generalization claim. The relevant test would be the one they do not do in this paper: does the workspace encode anything predictive for a protein sequence the model has actually never seen, or for a point mutation that changes function. As stated, "identifying biological function from raw sequence" reduces to "recognizing a famous sequence from its memorized head."
At this point I’m convinced that the neuroscientists, neuro”scientists” (pejoratively), and especially the financially-incentivized software technicians that contribute work to AI research (propaganda?) are terrified of philosophy. Not just because they seemingly have no honest skillset of the sort if they’re willing to prosthelytize for kickbacks from AI advancement, but because they must have an incapability to address the philosophical demands that threaten their legitimacy and value at all.
Philosophy isn’t a threat to science; they are mutually beneficial to each other! Metaphysics isn’t some boogyman unrelated to scientific progress. Philosophy of Mind and consciousness research demands rigorous (and scientific) philosophical work to tackle!
As just a dumb engineer, I recognize a Jacobean as being a matrix of first derivatives with respect to the variables such as position xyz in structural analysis, ie stiffnesses. Sooo what the hell is being referred to in a J lens? Derivatives of what?
I'm not up on transformers, but I think it's the same thing as what you are used to. There is a vector of functions (the final layer at different "times", or different places in a prompt sentence, all real numbers, maybe probabilities of different words, so it's actually a 2-tensor), and the derivatives are of this vector with respect to whatever parameters are being learned in some other layer, which are also a vector indexed by "time". It's a bit more complicated because they are looking at differences in parameter values between iterations (residuals) instead of just the parameter being learned. Someone correct me if I'm wrong.
With much of the company literature (not merely Anthropic's), we are getting "If it looks like a duck and quacks like a duck, then (maybe) it's a duck." This avoids the problem that we don't understand what a duck is in the first place.
Especially because without a knowledge of ducks this rapidly breaks down into "Look, it quacks!"
It's akin to Wittgenstein's "beetle in the box" observations.
One more observation, as an AI researcher. Even beyond the protein issue I identified earlier, the paper is flourish. It is just a standard interpretability result. The claim reduces to: a transformer keeps a small set of readable directions in its residual stream that hold intermediate values of multi-step computation and get reused by many downstream components. This statement is about representational format and reuse, which you should expect from any neural network that has to chain steps efficiently. Even the author acknowledge "computationally efficient." The global workspace framing adds no predictive content. Every result the paper reports follows directly from representational efficiency. The one property that I would want to see (which would distinguish the workspace), is sharp nonlinear ignition. This is the one they cannot show. The paper's real contributions are narrower. (1) The Jacobian lens recovers intermediate representations that the logit lens misses. (2) The directions it finds are the same ones that causally drive behavior.
What I'm getting from this is that Anthropic has published not so much a theory as an in-depth analogy to a theory, sort of like saying a toaster is like a wildfire because they both produce heat at first gradually and then intensely, and therefore toasters in the right circumstances could have the ability to restore health to forest ecosystems. Is that about right? I don't have the knowledge to fully follow what is being said. But I'm having trouble understanding why Anthropic's "research" would be taken seriously, and why we would have an expectation at all that a mechanical computation system, no matter how sophisticated/powerful it is at analyzing massive inputs consisting of the output of biological systems (ie human brains), would be able to replicate a biological system? (I do sort of get what you are saying about the difficulty of measuring consciousness, which is interesting.)
The burden of proof is on people who say that carbon-based systems would for some reason be _intrinsically_ different from silicon-based systems. Brains simply are (crappy and overloaded on heuristics and parallel processes) mechanical computation systems.
I refute it thus (*kicks stone*)
I still maintain that theories of consciousness keep ending up trivial because consciousness just _is_ trivial and non-interesting and you are fighting over definitions with no useful information provided by "is this system conscious". Like, if you keep running into a wall, that is usually a sign of you using a wrong road entirely.
What do you think of reclassifying the J-space as closer to being considered a working memory than any simulation of consciousness? https://izaktait.substack.com/p/working-memory-or-global-workspace
I think it’s a fine reinterpretation. But I don’t know if I’d go that far - the thing is, working memory is supposed to be something more the smeared reportability too!
I don't understand this reverse-engineering approach. It appears unscientific to assume a scientific theory must have a certain shape. Science works by:
1. A new phenomenon is observed.
2. There is no model for how this phenomenon works.
3. Through experiments, a model that explains how the phenomenon works is developed.
But with consciousness, we have not observed it as a phenomenon. We have access to our own consciousness, through common sense we assume other humans are conscious, but being scientifically skeptic about it leads to solipsism.
What we really need is a consciousness detecting instrument that could be applied to anything, if we have that, then observations could be gathered to develop a theory of consciousness.
First, that's really not how science works. Science is not just some downstream explanation of measurement devices that come out of thin air. You often have to build a specific measurement device first, based on what you hope to see, in the hopes of making your theories better. People do this all the time, but an obvious example in physics.
Second, we do have a primitive consciousness-detecting instrument! The human brain. You yourself said that "we have access to our own consciousness." But if your point is that you need some sort of objective 3rd-party consciousness detector beyond that, which you can rely on 100%, that's something that gets built after, or in tandem with, a theory of consciousness.
It just all has me wondering - is this actually something we can even approach a theory for? Like, I see your argument - that these reportability-based theories we're going with aren't good ones and we need something better. But how much can measurement actually cross over from observable into the experienceable? Granted, I write from a phenomenological point of view here - I don't think we can reduce something like consciousness into something so easily measured with any kind of theory or experimentation in humans, birds, trees, or LLMs. We can map behavior and physiology with plenty of precision and STILL have no idea what it's actually like to BE that thing. It's not that we haven't closed that gap yet, it's that that gap simply isn't closable.
So I guess my big question after reading this (very detailed well articulated) piece is what if the problem isn't that consciousness science is pre-paradigmatic but that measurement was never going to be the way in in the first place?
(And maybe this is something you've talked about before - this is my first time coming across your substack and I'm looking forward to taking a look through your archive!)
There is a fair bit of gloss in this paper. The one that I can directly call out is the party trick of showing something on proteins (something other than language), where most people will find something magical happening. This is a general chat model (Sonnet 4.5, Haiku 4.5 for the oracle lens), not a protein language model. The recognition they claim fires roughly five residues into the sequence. MSKGEELFT is the canonical avGFP N-terminus, one of the most-memorized sequences in all of biology! What the experiment demonstrates is that a general LLM has stored the canonical GFP prefix well enough to retrieve its identity, and that this retrieval is an unspoken intermediate in the workspace rather than in the output. This is not evidence of sequence-to-function inference. There is no held-out or novel-sequence control, no test on an obscure or mutated protein, no function prediction where memorization is unavailable, and no generalization claim. The relevant test would be the one they do not do in this paper: does the workspace encode anything predictive for a protein sequence the model has actually never seen, or for a point mutation that changes function. As stated, "identifying biological function from raw sequence" reduces to "recognizing a famous sequence from its memorized head."
At this point I’m convinced that the neuroscientists, neuro”scientists” (pejoratively), and especially the financially-incentivized software technicians that contribute work to AI research (propaganda?) are terrified of philosophy. Not just because they seemingly have no honest skillset of the sort if they’re willing to prosthelytize for kickbacks from AI advancement, but because they must have an incapability to address the philosophical demands that threaten their legitimacy and value at all.
Philosophy isn’t a threat to science; they are mutually beneficial to each other! Metaphysics isn’t some boogyman unrelated to scientific progress. Philosophy of Mind and consciousness research demands rigorous (and scientific) philosophical work to tackle!
People are terrified of something that has power. Philosophy is instead, correctly, dismissed as empty and powerless.
As just a dumb engineer, I recognize a Jacobean as being a matrix of first derivatives with respect to the variables such as position xyz in structural analysis, ie stiffnesses. Sooo what the hell is being referred to in a J lens? Derivatives of what?
I'm not up on transformers, but I think it's the same thing as what you are used to. There is a vector of functions (the final layer at different "times", or different places in a prompt sentence, all real numbers, maybe probabilities of different words, so it's actually a 2-tensor), and the derivatives are of this vector with respect to whatever parameters are being learned in some other layer, which are also a vector indexed by "time". It's a bit more complicated because they are looking at differences in parameter values between iterations (residuals) instead of just the parameter being learned. Someone correct me if I'm wrong.
Ok thx!