It is obvious to you and many of your readers, but let me note that fragmenting even potential phenomenology is already a function of the homeostatic, somatic basis of many qualia.
The hedonic, if not sensory, features of (some of) organisms' experience is a product of relations between a homeostatic state and the stimulus. (Examples abound, but take the enjoyment of even mediocre food when truly depleted and very hungry. The negative qualities of increasing heat when one is already hyper-thermic and nearing sunstroke. Inter alia.)
That is out of the question for an LLM. It would not even be able to predict that qualitative feature of experience (hedonic response, pleasure vs displeasure), in some key contexts important for survival, without those somatic inputs (or data that captured them).
So (at least some of) the hedonic dimensions of qualia may be "fragmented" or even unavailable.
Another example, more de gustibus: Claude us great for getting very detailed recommendations for new domains of a given art form. It's great at telling me about the sequence of recordings available of an experimental composer's work.
But can it tell me what I will enjoy or not? Can it enjoy or hate the recording?
“make humans the only moral patients in the equation of the future.“
I think animals are also clearly moral patients, and their suffering seems to outweigh humanity by at least an order of magnitude just in factory farms.
Edit: It’s far from clear to me that a future where human preferences dominate is good.
Thank you, Erik!
It is obvious to you and many of your readers, but let me note that fragmenting even potential phenomenology is already a function of the homeostatic, somatic basis of many qualia.
The hedonic, if not sensory, features of (some of) organisms' experience is a product of relations between a homeostatic state and the stimulus. (Examples abound, but take the enjoyment of even mediocre food when truly depleted and very hungry. The negative qualities of increasing heat when one is already hyper-thermic and nearing sunstroke. Inter alia.)
That is out of the question for an LLM. It would not even be able to predict that qualitative feature of experience (hedonic response, pleasure vs displeasure), in some key contexts important for survival, without those somatic inputs (or data that captured them).
So (at least some of) the hedonic dimensions of qualia may be "fragmented" or even unavailable.
Another example, more de gustibus: Claude us great for getting very detailed recommendations for new domains of a given art form. It's great at telling me about the sequence of recordings available of an experimental composer's work.
But can it tell me what I will enjoy or not? Can it enjoy or hate the recording?
“make humans the only moral patients in the equation of the future.“
I think animals are also clearly moral patients, and their suffering seems to outweigh humanity by at least an order of magnitude just in factory farms.
Edit: It’s far from clear to me that a future where human preferences dominate is good.
And I literally write that "it also ensures that “ultimate value” of consciousness is sequestered in humans (or other animals, obviously)."
Whoops, I missed that. Sorry!
Why not just build the language concatenating qualia-quanta? Leave the symbolic glass ceiling in the rear-view mirror.