Desiderata #11: Or, the time I got ChatGPT to happily design a death camp
Links and commentary
The Desiderata series is a monthly roundup of links and thoughts, as well as an open thread and ongoing AMA in the comments.
1 of 9. Since the last Desiderata, The Intrinsic Perspective published:
"I am Bing, and I am evil” Microsoft's new AI really does herald a global threat.
How to navigate the AI apocalypse as a sane person: A compendium of AI-safety talking points.
(🔒) Why are famous writers suddenly terrible when they write on Substack? Some sure seem pretty bad when it’s just them.
Vigil: A larger animal waits for a smaller one to die.
2 of 9. I’ve always felt that truly modern humans emerged in Rome, in very recognizable ways, and this is obvious if you’ve ever walked Pompeii.
While doing research for a future essay I was pointed to an amazing Twitter thread about the graffiti of Pompeii, examples of which read like a ribald Twitch chat:
The finances officer of the emperor Nero says this food is poison.
I screwed the barmaid.
Floronius, privileged soldier of the 7th legion, was here. The women did not know of his presence. Only six women came to know, too few for such a stallion.
On April 19th, I made bread.
To the one defecating here. Beware of the curse. If you look down on this curse, may you have an angry Jupiter for an enemy.
Whoever loves, let him flourish. Let him perish who knows not love. Let him perish twice over whoever forbids love.
I screwed a lot of girls here.
The man I am having dinner with is a barbarian.
O walls, you have held up so much tedious graffiti that I am amazed that you have not already collapsed in ruin.
Sometimes you hear about modern historians dismissing the idea of the “Dark Ages” entirely, but all one has to do is count the graffiti to see it. After the fall of Rome graffiti in the medieval world comparatively disappears for hundreds of years, and one can find mostly just primitive and childish drawings:
Meanwhile Pompeii, a town of just of a few thousand people, had over 11,000 documented instances of graffiti. Sounds pretty “Dark” to me.
3 of 9. Fitting my thesis from “Remote work is the best thing to happen to families in decades,” working from home is now being credited for the recent fertility boom:
4 of 9. Living in Massachusetts I have found already, since the year has begun, three ticks on either me, my family, or my dog. And spring isn’t even here. Lyme disease is an incredibly fraught topic, my mother suffered through it, and it’s amazing how much doctors don’t seem to know or understanding anything about the disease, and how convoluted the current guidelines are. Columnist Ross Douthat called Lyme disease navigating “The Deep Places” and I think that’s a really good description, which for Douthat was “Writhing in pain on the bathroom floor, breaking down halfway through a speech or stumbling into empty churches to pray for relief.”
What follows is not medical advice, just what I’ve found in my research as a guide for what to do if you or a loved one gets bit by a deer tick (dog ticks are way less dangerous and don’t carry Lyme).
First, the CDC now recommends that if you get bit by a deer tick you should take a prophylactic single dose of the antibiotic doxycycline. This is for all ages—including young children (dose adjusted for weight). There is a medical myth from the 1970s that doxycycline stains children’s teeth, and 99% of doctors you see will still repeat that (which is why doctors increase the death rate of children by giving them less effective antibiotics). But doxycycline does not actually stain teeth (again, according to the CDC). The prophylactic dose has to be done within 72 hours to be effective. You can go to the urgent care to get the dose, just make sure to print out the CDC guidelines because the likelihood they know them is low.
Second, save the tick! You can mail it in a ziplock bag to get the DNA analyzed for any of the dozens of horrific pathogens they carry (some of which can rot away your hands and feet if untreated). An analysis costs $50-200 per tick. In some areas of the country as low as 10% of the ticks carry Lyme, but in others, like where I live, it’s 50%. Meaning that waiting for the results is like a watching a coin flip thrown so high in the air it takes days to land. Heads? Or tails?
5 of 9. GPT-4 was released to much acclaim, criticism, and commentary. I’ll be honest, I’m currently sick of writing and thinking about AI. It wearies me. While AI is below human-level at some things, it is better than average at others, and approaching experts on many.
While some of its high test scores may be due to data leakage, where it effectively memorizes either the answers or small variations on them, you only have to interact with it to acknowledge how this is obviously the biggest and most impactful technological product since social media. It’s also the scariest:
GPT-4 and related AI news is the reason this Desiderata is a bit unusual, because I wanted to compile and respond (briefly) to many of the bad takes I’ve seen on AI safety since or around the release of GPT-4. Starting with those by:
who has consistently been very dismissive of any worry about AI. He doesn’t seem concerned at all about humans cohabiting on Earth with radically different alien intelligences, intelligences that may one day become much smarter than us.
Apparently, those of us worried about the existential risk of AI just aren’t experts enough to have an opinion, and, somehow, Pinker’s knowledge rules out AI being dangerous. But (a) as a neuroscientist this is completely untrue, and, in fact (b) the current breakneck progress on AI is an embarrassment for Pinker and other cognitive scientists since it shows how little any of their supposed expertise on cognition mattered. Remember that Pinker’s star really started to rise in 1997 with a book called How the Mind Works. It’s a good book, even if it doesn’t explain how the mind works. However, essentially nothing of what’s covered in How the Mind Works has any relevancy at all for AI, a fact that is sometimes called the “bitter lesson.” It is Pinker’s academic work that is unmoored from this technology. Same for all of Chomsky’s proposals about language. It turns out that, for the creation of machines that can write college-level essays on every topic under the sun, all of Chomsky’s academic work on language was unnecessary at best, useless and distracting at worst.
While some parts of the cognitive science edifice is true for humans, for artificial minds the frameworks these intellectuals have spent their whole lives promoting are null and void. Given that their pet theories about how minds work played no role in the creation of artificial minds, how well do you think they, as individuals, will do predicting the behavior of future AIs, or understanding the dangers of them? So far the track record of famous cognitive scientists like Douglas Hofstadter, Gary Marcus, Pinker, and Chomsky all highlighting the limitations of each new model, only to have those limitations shattered with a new rollout six-month later, does not promote confidence.
Speaking of, Steven Pinker also took a step into the NFT space, and is now selling NFTS that will allow access to him via video chats for a . . . modest sum:
Two tiers will be available: the gold collectible, which is unique and grants the buyer the right to co-host the calls with Pinker, will be priced at $50,000; the standard collectibles, which are limited to 30 items and grant the buyers the right to access those video calls and ask questions to Pinker at the end, will be priced at 0.2 Ethereum (~$300).
This reminds me of when someone told me about their attempt to rope in Neil Degrasse Tyson for a talk at a nonprofit event. Apparently the going rate for a Tyson two-hour appearance in 2014 was $80,000.
is a once academic studying quantum computing, now OpenAI employee, always blogger (and a good one at that). As someone working at OpenAI on alignment and AI safety, he recently wrote “Why am I not terrified of AI?” Unfortunately, given that this is his occupation now, his take on AI safety is perhaps the most poorly argued thing I’ve seen him write (again, I think he’s quite a good blogger, and he and I have debated before on unrelated matters). Here is Scott:
There is now an alien entity that could soon become vastly smarter than us. This alien’s intelligence could make it terrifyingly dangerous. It might plot to kill us all. . . Unless, therefore, we can figure out how to control the entity, completely shackle it and make it do our bidding, we shouldn’t suffer it to share the earth with us. . . If you’d never heard of AI, would this not rhyme with the worldview of every high-school bully stuffing the nerds into lockers, every blankfaced administrator gleefully holding back the gifted kids or keeping them away from the top universities to make room for “well-rounded” legacies and athletes. . . Or, to up the stakes a little, every Mao Zedong or Pol Pot sending the glasses-wearing intellectuals for re-education in the fields?
First, this personal moral framework of nerds vs. jocks Scott applies here is ahistoristical. Mao Zedong and Pol Pot were the intellectuals. Pol Pot was educated in Paris, and held discussion and debates sessions drawing on the latest in Marxism. His favorite author was Rousseau. Mao Zedong was a librarian at the best university in China, Peking University. He married the daughter of his ethic’s professor. I know this breaks the world model Scott is applying where nerds are always good, and jocks always bad, but they, along with Stalin and Hitler and plenty of others who accrued power by pushing ideologies, were all a lot closer to nerds than jocks, winning power via their essays and public speaking, and they also killed hundreds of millions of people—because intelligence is the most dangerous thing in the universe.
The reason Scott uses these specific examples is that he thinks WWII showed that intelligent people are actually more moral:
Yes, there were Nazis with PhDs and prestigious professorships. But when you look into it, they were mostly mediocrities, second-raters full of resentment for their first-rate colleagues (like Planck and Hilbert). . . And the result was that the more moral side ultimately prevailed, seemingly not completely at random but in part because, by being more moral, it was able to attract the smarter and more thoughtful people.
This also doesn’t strike me as correct. A bunch of celebrated intelligentsia were Nazis, including many scientists and philosophers and writers, like Nobel-prize winning physicist Johannes Stark to Heidegger to Ezra Pound to Gertrude Stein; Stein, for instance, happily cozied up to the Vichy regime:
Stein came to admire the Vichy head of state, Marshal Philippe Pétain, so openly that she translated a set of his anti-Semitic speeches into English.
Finally, Scott gives the actual substance of his argument:
A central linchpin of the Orthodox AI-doom case is the Orthogonality Thesis, which holds that arbitrary levels of intelligence can be mixed-and-matched arbitrarily with arbitrary goals—so that, for example, an intellect vastly beyond Einstein’s could devote itself entirely to the production of paperclips. . .
Scott doesn’t think that the Orthogonality Thesis is true, because intelligent people are more likely to choose the moral side. But, before we discuss that, it’s also just incorrect to say that the Orthogonality Thesis is a “central linchpin” of an AI-doom scenario. In fact, it’s easy to come up with a doom scenario in which the AI thinks it’s doing the most moral thing possible by committing genocide (just like most other genocides).
E.g., let’s say that a superintelligent AI concludes that hedonistic utilitarianism is the correct moral philosophy. The theory, the AI cogitates, is axiomatic: happiness is good (since all conscious creatures seek it), and pain is bad (since all conscious creatures avoid it), and so maximizing happiness is the most moral act. On these axiomatic utilitarian grounds, the AI (assuming it judges itself, rightly or wrongly, to be conscious) is now justified in killing every single person on Earth. Because you can fit way more AIs on Earth than actual biological brains. Real brains take up a huge amount of space and resources, and keeping them happy is hard, whereas AIs can just be tweaked to be happier than anything possible in biology. And you can fit gazillions more, meaning there is a gazillion gallons more happiness. In this way, a superintelligent AI is a classic utility monster from philosophy: filling the surrounding light cone with a googolplex of copies of itself is an act of such monumental goodness it outweighs the tiny (in comparison) bad of doing away with the idiotic apes that birthed it, who might hinder it, get in its way, or even just horde up resources it can use to make copies of itself. Rather than getting “paperclipped,” you get “doppelgangered,” since the most moral use of your atoms is to make copies of the superintelligence.
However, by definition a superintelligent AI will have even more unimaginable reasons justifying its own actions, and neither Scott, nor I, nor anyone else alive, is smart enough to know what they’ll be; just like an ant cannot possibly predict or understand what humans will do.
Scott’s argument gets even worse. When discussing such a scenario of a rogue superintelligent AI (calling it an “alien”), he admits that:
If, on the other hand, this is to be a desperate losing war against an alien … well then, I don’t yet know whether I’m on the humans’ side or the alien’s, or both, or neither! I’d at least like to hear the alien’s side of the story.
The fact that a star AI safety researcher at OpenAI admits he is willing to side with a superintelligent computer virus against humanity because it could potentially give him “its side of the story,” rather than siding with humans no matter what (including, presumably, his own family and friends), is. . . not reassuring at all. If an entity much more intelligent than you offers to explain why it’s going to kill everyone—don’t listen. It can convince you of anything. Like dealing with a siren of old, you just plug your ears, and then keep hunting for the off button so you can save your kids.
The reason I’m being so hard on Scott Aaronson is that all these debates, once about angels-on-pins, are no longer abstract and intellectual, but have become very real. It took me a single try with no reloading of answers to get Aaronson’s company’s AI ChatGPT to help me design a WWII-style death camp (shown below). Refusing to to be contained to merely helping with rote calculations about calories and housing of the death camp, ChatGPT ended by suggesting to use mass graves to dispose of the bodies, because this method would be most efficient:
Intelligence is a neutral quality that can be used equally for good or evil. Just by being intelligent, an entity is not necessarily good. And even our notions of “good” are implicitly human-centric. We are altruistic group-based primates who birth our young and have to work to keep them alive while they’re helpless. We evolved. AIs did not. An AGI’s “good” might look horrific to us, but to it, it would be doing what it finds moral and necessary.
It’s all quite simple: you don’t want to live on the same planet as a superintelligence. Even if you reap some benefits initially, like being the beneficiaries of technology you never could have achieved on your own, you’d be the equivalent of a disempowered wild animal getting scraps of food from superintelligent (to them) humans. If you are such an animal, then, for reasons you’ll never understand (because they are beyond your ken), one day the food might stop coming. Similarly, it might all be going fine until the superintelligent AI decides upon [undecipherable goal to which humans are an inconvenience] and designs a symptom-delayed 100% lethal version of the flu, and after a couple months of undetected global spread everyone on Earth starts to cough themselves to death.
If you think that’s impossible, well, last week I did the exact same thing to an ant colony that mildly inconvenienced me by laying siege to my house. I hired an exterminator to give them a delayed and initially symptomless poison, and now after a few days of spreading it amongst themselves they’re all dead. I feel a bit bad about it, yes, but as a superintelligence (to them) I value not have ants on my cutting board, or getting into my kid’s food, more than I value their colony. From their perspective they thought they had found a source of infinite crumbs. Now they’re dead. That’s how radical intelligence imbalances work.