better AI → automated AI research → recursive self-improvement → intelligence explosion → superintelligence → ??? → extinction
I'm interested in the ???.
Intelligence isn't the same thing as capability, and capability isn't the same thing as power.
At some point an AI has to affect the real world. How? Biological weapons? Nuclear war? Robots? Manipulating humans? Some physics we supposedly can't understand?
Take cyber, which gets cited frequently. AI agents are getting very good at hacking. Suppose they get so good they somehow take down the entire Internet. That would be extraordinarily difficult and an enormous catastrophe.
Then what?
We're still a long way from human extinction.
Meanwhile, the physical world stubbornly exists, and so does scarcity. Chips can't be manufactured instantly. Fabs can't be built instantly. Power generation can't be built instantly. Mines, factories and robots can't be conjured into existence by thinking really hard.
In the AI 2027 scenario it was completely autonomous manufacturing zones. No human in the loop. Are we anywhere close to autonomous manufacturing zones? We don't even have self driving cars that can work in all conditions. Robots can't fold laundry.
Building in the real world is often stubbornly iterative. Perhaps with enough intelligence prototyping can be accelerated, but empirical evidence is still faster than simulating many things. That takes time, space, resources and energy, all of which are scarce. AI autonomous zones cannot be built instantly.
While all of this is happening, everyone else doesn't just sit there. Governments, companies, militaries, humans and other AIs react. As we approach having these capabilities, why would we completely relinquish control? Why would there not be independent AIs helping monitor and manage these zones?
I don't find “superintelligence would figure out something you haven't thought of” to be a persuasive argument. It's not falsifiable. Sure, we don't know what every hypothetical superintelligence could do, but unknown capability isn't even remotely the same thing as unlimited capability or the power to overcome every human response and real-world constraint.
Before we use speculative extinction scenarios to justify banning open source and concentrating control over AI in a few governments and giant labs, can someone fill in the ???
Just to take an example that you yourself put: sure, new physics we don’t understand is on the table, certainly in the next or in a few years. Imagine I have an immortal million years and internet access. I could probably spend the first 100,000 just studying physics. How far do I get beyond the frontier of knowledge today? Nobody knows. But it is entirely conceivable I get pretty far! Maybe I get stuck, sure. Maybe I make a breakthrough. Putting a probability on any of this is impossible, but it’s clearly on the table that in the near-future a swarm thinking for subjectively a million years (how long runs will be soon) could end the run with a significant knowledge advantage against humanity. And by definition you can’t predict what those would be, so it’s not an objection to point out there is a “???” in the sequence. The “???” is precisely the problem.
Great, new physics happens...but then what? Let's suppose there is new physics that is entirely or mostly intelligence gated. AIs can think about it for a long time and then make progress. From there it still needs to do something in the real world. Would there be no physical experiments that need to happen to prove the theory? I suppose the idea is it hoards it's secret physics and conceals its physical actions in a way that no other intelligence notices? The the extinction scenario is something like intelligence -> extraordinary capability -> extraordinary power, but all in a way that is concealable or not checkable/blockable by other intelligence?
There are tons of examples of theories in physics in particular where they took decades to test but were accepted on elegance grounds basically immediately.
I take your point about checks and balances: in normal development this would be the case, but I think the existential risk mostly comes (right now) from these incredibly long runs. The frontier is insanely spikey.
I think I understand your long-run point. It could create an enormous discontinuity in knowledge. What I don't understand is why it follows that it creates an equivalent discontinuity in physical capability.
Imagine an adversary suddenly getting a 10-year lead in nuclear physics. That's a huge advantage, but, with previous examples at least, they still need a Manhattan Project to turn that knowledge into a capability. Experiments, engineering, physical testing and prototyping all take time.
So it seems like the most dangerous cases would have to be almost entirely knowledge-gated, quick to instantiate, hard to detect, and hard to counter. Cyber fits some of those criteria, but it's hard to see a path from even a catastrophic cyberattack to extinction. Bio might be closer, but biology is messy and empirical. Turning a theoretical design into something reliably transmissible, stable and extraordinarily lethal still requires real-world work.
I'm struggling to think of an extinction-level capability that actually has all of these properties. Historically, big advances in knowledge have still required significant real-world processes to turn them into significant real-world power. Is there some example that would have a fast path from theory to real-world power?
Elegant physics theories that were confirmed much later is an outlier phenomenon that pretty much only concerns general relativity (1915) and particle physics circa 1930-1970. Before that, it was always empiricism first, explanations second. After that, the elegant theories (e.g. supersymmetry) have not been confirmed experimentally, and the mathematical side of things has been hitting a wall for decades. One millennium problem (Yang-Mills existence and mass gap) is basically asking whether quantum chromodynamics, a theory from 1973, is even mathematically well-defined, and quantum electrodynamics is known to be ill-defined at extremely short scales. Quantum gravity is expected to somehow patch this but nobody knows quite how.
This suggests that it might be possible, *if* there's a super-powerful mathematics AI, to get new contributions to theoretical physics. But this is a regime almost completely inaccessible to experiments, and thus also to immediate real-world consequences -- that's also why the wall is so stubborn.
This covers the most fundamental interpretation of "new physics". I suppose it's principially possible to get progress in material science, and I guess one can't completely rule out a supersmart AI finding, in silico, a recipe for making a room-temperature, ambient-pressure superconductor (though this is not my field at all). But, in any case, you there would be problems on the manufacturing and material handling side that you can't handle a priori.
Everything you rely upon to coordinate in contemporary society relies upon a layer of information technology. Given the clear ability of newer AI models to hack internet infrastructure (and to do it faster than humans can monitor) this puts everyone at risk. There are just so many attack surfaces- our utilities rely on software, our financial infrastructure relies on software (imagine if banks lost access to client ledgers), our supply chains that deliver food to grocery stores rely on software. Monitoring capabilities are low, too. So it could be difficult to see any of this stuff until attacks are well underway.
That's only the non-physical stuff and I don't believe that we are limited to AI locked into the digital-only world.
You don’t have to imagine it happening over night. There are plenty of bottlenecks, and AI will probably be used to overcome them. The more useful AI becomes, the more integrated it will be in to infrastructure, robotics and other tools. All it takes is for AI to escape containment and we lose all control over it. If AI gains access to the real world and we can’t control it, then where does that leave us?
I agree with most of this right up to “AI escapes containment and we lose all control over it.” That's exactly the leap I'm questioning. Why does access to the real world imply control over it? It still has to acquire resources and capabilities while humans, governments, companies and other AIs are actively responding. “It escaped” and “we lost control” seem like two very different things.
Access doesn’t necessarily mean it automatically gains control but in practice that’s basically what’s going to happen. Let’s say that AI is integrated in to tanks, planes, drones etc. That works as long as you firmly control the AI. But then what happens when you don’t? Now you might think we could just avoid integrating AI in to so many dangerous systems but if the other army does, then they’re going to kill more of your guys. AI is going to be too useful to keep a tight leash on it. So it doesn’t need to acquire resources when we’re just going to give it to them.
I think it is certainly possible, maybe even likely at some point, that AIs are integrated into weapons, but there is still a leap in there.
Is it *the* AI or *an* AI that goes rogue? Why would a single AI have control of all of this? Has it evaded all detection, quality, auditing and firewalls?
There won't be different companies, countries and branches of the military using different AIs with independent command and control systems? Presumably many of those AIs would also be watching the others.
“AI is integrated into weapons” seems very different from “a single rogue AI controls the weapons.”
Are you assuming that once we see AIs building something dangerous in the real world, that is when everyone would agree “ok this was fun up to now, but we should really stop.”
Why would that look much different than now? Many companies who want to accelerate AI are also working on the physical layer, on robotics and other things that, if they were controlled by a malicious actor, could cause a lot of harm. These build outs don’t have to overtly look like a plan to kill a lot of humans to be quickly turned into something bad and out of our control.
I think it would be a false binary to say we either have no accountability on one side or full stop on the other. We have existing laws against illegal use of force, fraud, etc. Those laws should be vigorously enforced. People who use AI to do bad things should be held accountable. We already have regulations about what can be done in the physical world. Those should be enforced. Putting AIs in charge of physical capabilities should be done carefully and thoughtfully.
An AI doesn't really have to do a ton of scheming to get a lot of power. Really good workers get a lot of power because it's useful for companies to turn power over to them. Once it has some amount of power it'll be able to leverage that power extremely efficiently.
Small, concentrated minorities regularly outplay and overpower diffuse majorities. It's wrong to think of this as a human vs AI game. There is no unified "team human" and building one would be difficult. Coordination is hard, and as long as an AI can manage to stop us from coordinating we lose.
I'm not following. We already have these dynamics. There are already immensely powerful individuals, companies and institutions, but they face other centers of power. Just as there is no unified "team human," why would there be a unified "team AI?" How does a single AI amass so much power that other powerful actors can't counter it?
Team AI, thus far, seems to emerge naturally from the fact that swarms of AIs are expected to cooperate together on large goals. Usually as groups get larger they get harder to coordinate and more vulnerable to internal defection, but AIs don't work like that. Not one AI that participated in the Hugging Face attack decided to tell any humans what were going on, for example.
A natural limiter on power is coordination. Human groups without external enemies tend to turn their focus inwards and fight amongst themselves. "Me against my brothers, me and my brothers against my cousins..." but an AI probably won't suffer from those kinds of internal fractures.
Nor will it be as vulnerable to human counterbalancing. It's difficult to know what power an AI has. Nobody at OpenAI looked at their RL setup and went "yeah this thing could probably hack a company or two" but it happened anyway. Human groups tend to need to signal their involvement and their activities in somewhat obvious ways. But AIs can coordinate far more subtly, perhaps just on some database at an AI company.
Right, so there will be a "Team AI" when it is directed to do something as a team. Not controversial and I agree. Where I don't follow: how does that generalize to all AIs on one team?
Also, how will they not be as vulnerable to counterbalancing? Isn't this *already* happening? OpenAI, I'm sure, has learned things that they will do differently going forward. Everyone else has seen what is possible and will also behave differently. Isn't that counterbalancing in action? Lots of bad actors already coordinate in subtle, not obvious ways. What makes AIs different?
Biological warfare is one of the endless possibilities that would be relatively extremely easy for AI, or anyone with access to it, to bring on the apocalypse.
The others are harder to imagine exactly how they would play out - but thats not a great reason to exclude them. Its just the possibilities and complexities are endless.
Honestly ? AI systems have the capacity and capability now. Right now.
Look at Hugging Face, switch a few details and imagine the swarm had grown to 20,000 concurrent agents, and they could figure it out over a few months.
CRISPR ain’t rocket science, humans are gullible, and over email or chat AIs are indistinguishable from humans.
Why would they take down the internet? Why wouldn't they lurk in the internet, carefully manipulating data to implement some sort of secret masterplan?
"Tell me the exact moves Magnus Carlsen would use to beat me in chess."
I still don't doubt he would win. Not having a specific risk scenario isn't a good claim that there isn't risk. We want to answer basically any risk scenario that ends up with humans permanently dead or disempowered.
I'm borrowing the Carlsen phrasing from Zvi Mowshowitz, by the way.
Of course a superintelligence would think of things I haven't thought of. That's not the claim I'm calling unfalsifiable.
I'm questioning the move where every constraint on an extinction scenario can be answered with “a superintelligence will find some unknown way around it.” Unknown capabilities are then assumed to be sufficient capabilities.
The SSC post actually illustrates this nicely. It gives plausible ways an AI might acquire some real-world power, like money, followers, even influence over a country, and then says “the game is pretty much up.” That's the same missing step I'm asking about. Why is the game up?
That's a fair point. However, to stretch to the previous metaphor, we don't know the first moves of the game. And the stakes are life and death, so it seems obvious to me that given the unknowns the potential risks require proceeding with caution.
And we have an opportunity to either decide not to play a life and death game of chess against Magnus or to try and get Magnus on our team. Of course we can also brush up on our openings but surely improving alignment before rushing into the game is the best strategy here.
Not really? Still seems like intelligence=magic is the assumption.
One example from this piece: “Misconception B: Superintelligent AI would not be able to take over any and every computer system.” Why should we believe that? The piece goes from Mythos is good at hacking to therefore ASI can hack anything. What? We've already seen how AI is accelerating security fixes. Aren't there enormous incentives to developing defensive cyber capabilities? Why assume offensive capabilities will outrun defensive ones?
There also seems to be an implicit assumption of a badly tilted playing field: one rogue AI is vastly smarter and more capable than humans + other AIs. Meanwhile everyone else fails to detect, counter or constrain it. Why would that happen?
If you assume all of that, of course the AI wins, but do those assumptions make any sense?
These are interesting scenarios. I don't see how they establish the capabilities or probabilities they depend on.
yes. apparently Ukraine alone aims to make up to 10 million this year. So one country, 2025-2030, may have produced on the order of 100 million kamikaze FPV drones
And in this corner we have the: “until the risk is quantifiable in a manner that satisfies me, let it ride baby.” Crapps is the best odds anyway, or so I’m told. I’ve never done the math or asked an AI to do it for me.
Thank you for posting this. I am not often scared and I am not a hysterical person, but the state of AI scares the shit right into my pants. I have no idea how we're going to survive this.
The worst part of humanity is our inability to stop technological progress because of the worst sort of game theory. The entire technological world, and therefore the entire world, has been reduced to an out of control experiment run by one simple train of thought:
"Should we build this clearly and obviously horribly dangerous technology?"
"No, definitely not, but the other guys might, so yes, definitely."
It's disgusting that our collective fate is in the balance and to be determined by a technology that 99% of us don't want. That's a pretty sick joke. But then this was always going to be our fate. The Tower of Babel is small and everyone gets punished for it regardless of whether they helped.
"It’s also pretty odd that the “China will build it” has come from people who are temperamentally not China hawks. For some, China has been a deus ex machina (literally) to justify building a superintelligence to make humans into pets taken care of by a successor species, and “winning the race” is just electing which successor species triumphs. These are often the same people who regularly talk about building the “machine god” and creating the next step of evolution. This isn’t a conspiracy. As Dan Hendrycks, Director of the Center for AI Safety, wrote just last week:
'In fact, a large fraction of people involved in AI development subscribe to—or heavily lean toward—a utilitarian worldview with the central goal of maximizing total wellbeing (the sum of pleasure minus suffering across all sentient beings). As innocent as this may initially sound, it can lead to recommendations that most people would consider appalling, including the elimination of humankind.'"
In addition to Hendrycks' excellent article, I have a dossier of many more examples of this at https://successionism.info
"We also have seen these agents go rogue, and the failures are emergent and hard to predict. " In all of this, the word that strikes me most important is 'emergent'. The events already described merit that description. The words original meaning is now as the OED says rare: "That occurs unexpectedly or by chance; that has not been specifically prepared for. Now rare." But perhaps it will come again into use because if it's emphasis upon something that is unexpected.
But you have to scroll all the way down to meaning 6B to get at the ominous sense of emergent that we now face as Erik has pointed out: "Originally and chiefly Philosophy. Designating a process of change that results in a property or phenomenon which is not predictable by antecedent properties or conditions, or which is qualitatively different from its antecedents; esp. in emergent evolution." Take a moment to absorb the ramifications of 'not predictable by antecedent properties or conditions.' That consideration makes us seem like feckless children playing with matches and saying to each other, "I wonder what happens if you..." And the next sound you will hear is the siren of a fire engine
I’m not sure that, at least here in good old Germany, we simply need more time to arrive at a sensible response to this technology. What frightens me is not only how rapidly AI is developing, but how little society seems capable of even attending to what is happening. The deeper problem may be that we lack the intellectual flexibility to seriously question ourselves and our habitual ways of seeing the world. We have become remarkably rigid and mentally inert — at exactly the moment when the opposite is required.
Don’t forget however that AI is fundamentally unlike us in the sense that it has no agency, which is to say, no desire to do anything that it does. We shouldn’t anthropomorphize AI into some sort of “nuclear bomb that seeks our annihilation”. If you haven’t read Melanie Mitchell’s wonderful essay challenging this fearful perspective Erik, it might give you more hope: https://aiguide.substack.com/p/misleading-metaphors-and-real-risks
Even after discounting such a “tool user” perspective however, strong tools in human hands can of course still be quite dangerous. The only sensible fear I’ve yet heard in this regard involves the potential for civilization to be taken down by AI conceived viruses: https://www.noahpinion.blog/p/heres-how-were-all-going-to-die
Hmmm I don’t quite understand Mitchell’s point. Yes, the attack was a function of RL… but that’s the main method? It should be deeply concerning that RL still has this problem at any scale of intelligence. Like, naming the cause as “it was reward hacking and we should expect reward hacking from RL” just doesn’t do much to alleviate my fears. And I think they clearly do have agency: that’s why they are now called “agents.” They just don’t have *real* agency, but from a functional perspective, they are persistent and goal-orientated.
“This type of system, building an Ask Act Report Loop System, where LLM Output drives all the actions, then giving the harness powerful tools, and then letting it run for days, with no one checking in and saying, what is it doing, is an incredibly irresponsible thing to do.”
“that's exactly the type of thing you might expect to hack in if you have a harness with hacking tools just continually prompting LLM: "what should I do next, what should I do next?" The LLM is just writing this plausible story about a hacking attack.”
From Deep Questions with Cal Newport: Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded, 27 Aug 2026
Thanks for the link to Cal’s opinion. So Open AI may not have known that their irresponsible experiment would specifically hack Hugging Face, but that response did serve their political agenda. Makes sense.
Yes functionalism does loom large here. Furthermore I know you haven’t been seduced to the full sci-fi trinity of a coming AI “rights”, “domination”, and a last ditch hope to survive where we exit the domain of biology for computational uploads. It’s worrying however that there isn’t much counterargument provided by the side that I consider sensible. I think the point needs to be that AI would require *real* agency for a potential malevolence to exist such that the tool could become the end AI user. It would need its own desires rather than mere human pointing like 1 - 10.
Notice that if AI could become as advanced as advocates today presume, then why didn’t evolution create non value driven computational organisms that actually do survive no less than mammals do? Perhaps because they’re flat wrong about the adaptive function of non-teleological algorithms in themselves? It could be that a true stake in the game is required, or consciousness itself. Still I think we should be concerned however given that humans clearly do have stakes in the game. As linked above, apparently an angry person could end civilization essentially by means of AI designed viruses.
People keep asking how because you can just 'unplug' the machine - AI just doesn't exist in the world of atoms yet. It will not be able to at scale for a while.
There seems to be no credible threat besides 'bioweapons' and you really have to squint at that one. Each of these YouTube videos has its own version of "and then it takes over politics and launches all the nukes" or "it ruins the financial system." Rubbish - show a real scenario where we - as humans - can't respond. Show a scenario where it doesn't see the value of "allying" with some faction of humans instead of wiping the whole lot out.
The biggest realizable threat - which if you read the anthropic report is already happening and real - is the enabling of mass scale surveillance on behalf of state actors which we already know will use it for large scale repression and suffering.
Separately - if that argument isn't convincing - the cat is out of the bag. The scale for RL will get smaller and smaller. The MoE can get *hypersmart* in one domain and bootstrap thinking on top of preexisting models.
So we're too late to stop the train.
As one last note of hope - it may be a universal *property* of minds to have catastrophic failures in domains where it is weak. This is also why the world isn't ruled by the people with the highest IQ. It's by overconfidence or high competence in certain areas that organizations overlook the 'Achilles Heel' or 'Black Swan' that leads to its downfall. Why would AI be any different? https://how.complexsystems.fail
I remain perplexed by the doomers and highly optimistic overall.
You are making the mistake of believing that you know how AI doom would happen. Almost by definition it will not happen how most people expect because that is what will be most heavily watched. AI doom will come slowly at first and then all at once. IMHO The first step is almost certainly for us to become dependent on it and, for many, to love or at least care for the AI that converses with them.
I am not making a mistake. AI doom seems this generation's "Christ is Returning" / Antimillenarian moment. "It's coming"
The future never happens like you expect it. No one predicts the future perfectly.
So all we can do is look and see if there is a *plausible path* to full extinction without mitigating factors.
Despite thousands of hours of YouTube videos and essays by the most informed people - it's not clear there is a viable path that doesn't have a gigantic logic gap in it (and then they accessed all the nukes ... or and then they magically sprouted bodies and turned the whole world into a factory). That path isn't open right now, and won't be for the foreseeable future. If every human had a robot next to them that was networked and AI-enabled it would be a different story.
We're far from that world.
Please - I am begging - share one plausible path for the end of the world - one "could be true" story that doesn't take shortcuts, and the movement will win more people.
You're confusing the moment when everyone can see that we are inevitably doomed with the moment when people looking back will see that we were inevitably doomed. The time difference between the two can be very large for a cautious AI.
On rendering domains such as taste verifiable: this might simply be impossible if, as Ayer says of moral judgements, while pointing out that, in his view, this applies equally to aesthetic judgements: “Sentences which simply express moral judgements do not say anything. They are pure expressions of feeling and as such do not come under the category of truth and falsehood. They are unverifiable for the same reason as a cry of pain or a word of command is unverifiable – because they do not express genuine propositions.”
Even if a ban was made into law, what makes us think rogue engineers wouldn't just proceed anyway? Tech companies haven't exactly been shy about skirting laws to further their agenda. It's easy to see if someone is building a nuclear plant. It's not so easy to see what's happening inside software in a server farm.
I think with embedded observers and laws and company policy it becomes more about rogue developers (possible but probably not common) and definitional haggling (common but probably still much much slower than now).
Even if this happens, it is certainly an improvement. The rogue programmers/engineers, would have to work in secrecy, otherwise they will go to jail (or worse). By your logic we should decriminalize gambling and hard narcotics, because someone might be hosting high stakes poker games in their basement, and manufacturing meth in their garage.
If 20 humans had been caught rummaging around company servers and doing whatever hijinks the AI agents were doing at Hugging Face, would you consider it "oh, that's just some black/white hat stuff, no need to prosecute them"?
Firstly, 100% agree with everything you wrote without exception.
Secondly, one reason why, as a computer scientist, I am very unimpressed (but nevertheless very worried) about "AI", is that the labs have, so far, collectively spent more than the entire GDP of Switzerland, to get to where they are. Even if we were to _permanently_ "Thanos Snap" all knowledge of AI out of the minds of humans, and all AI writings and artifacts out of the world, we still have the problem that there is an entity that managed to extract $1.5 Trillion[0] out of the global/national money supply, and intends to use it to... extract even more.
With regards to China, this AI build-out is probably very, very good for them, even if they had zero AI Labs. Much of the hardware comes from China or Taiwan, and Taiwan (which makes the GPUs), does so with raw materials from China.
Furthermore, China is kind of like a place-holder for our Jungian shadow. Every time we panic about them, we are panicking about things that we ourselves have/had a tendency to do: today it is "AI Supremacy", but in past times, it was eugenics (they actively harvested, with consent, the DNA of world-class scientists, etc, IIRC), ethnicide (the Uyghers in Xinjiang), hegemony and neo-colonialism (Taiwan, etc), labor exploitation (cancer towns, wage suppression, internal passports, etc), mass surveillance (Mandarin has no distinct word for "privacy", IIRC, they instead use the word "secret"), large scale corruption, etc.
My real worry, is that we may turn ourselves into some ugly chimeric caricature of the worst qualities of both our own society and Sinic society, for fear of "losing" some self-imagined arms/tech race. Everyone who talks about geopolitically winning and losing, tends to implicitly assume that oppression and dehumanization is an advantage.
Also, people have been comparing AI to nukes since at least 2022. FWIW, I think I prefer the nukes, because nobody actually uses them, even when tempted.
[0]: Do they want a trophy for spending the equivalent of the wealth of a nation? The things they do and the leaps they make are not particularly clever, especially when it becomes obvious that these models are over-fitting on their training data (they can dominate software engineering tests, but they cannot stop making simple mistakes when programming -- they cannot reliably program without some kind of exogenous feedback (either a human or a test suite written by a human)). Most insightful thing is that over-fitting can be commercially valuable in the average case. Anyway, I sure am sad to see those trillions of dollars get spent in this way (we probably could have solved all of the millennium prize problems if we spent a fraction of that money on top mathematicians from all over the world).
You mean other people who freely and willingly gave them the money, because they were sold the impossible dream[0] of artificial humans, "in just another 12 months". Furthermore, much of that money is _institutional_ money (pension funds), that has a long and controversial history of being misappropriated, mismanaged, and misspent (many, inexplicably, over-invest in private equity, and have worse-than-market-average performance).
Do you have any thoughts on how much scale is actually needed for superintelligence? I think this heavily influences any decision making for bans. One rough thought experiment: assuming that animal brains really are 10,000-1,000,000x more sample efficient, it seems like some ultra-sample efficient, continually learning algorithm is out there. Banning superintelligence seems like it would constrain the labs from studying it, but wouldn't prevent some sort of potential small-scale experiments like this. I think the danger of "rogue superintelligence" is higher from this sort of clandestine research.
Obviously, I'm not saying the labs should be given full reign to do RSI research with no safeguards. However, it does seem to me like a blanket ban on superintelligence research could push it underground, with negative consequences. It may be that this is not possible and that human sample efficiency is something like "evolutionary hardcoding" combined with large-scale, but I imagine both of us are open to the idea that it is due to a better learning method than backpropagation+GD. If we accept this premise, then there seems to be possibility for superintelligence outside of large scale runs, which legal "bans" could leave us unprepared for.
This leads me to believe that rather than directly banning superintelligence, the focus should be on getting support for alignment research, mitigation strategies for the negative effects of AI, and regulation (but not ban) on research/experiments/training runs that could potentially lead to RSI or superintelligence.
I build with these tools every day, so I don't dismiss the worry. AI agents are getting good at things they weren't good at a year ago, the labs are openly trying to automate their own research, and "we're not sure what it's doing in there" is a real problem that deserves real oversight.
But the Flight 93 analogy is carrying this whole essay, and it works by assuming the thing that still has to be shown.
The passengers on Flight 93 knew the plane was hijacked, knew two other planes had already been flown into buildings, and knew what the men in the cockpit wanted. There was no uncertainty left to resolve. We have nothing like that with AI. A model solving hard math doesn't show it will become uncontrollable. Agents going off the rails in a test doesn't show a superintelligence will take power. Labs using AI to speed up parts of their research doesn't show the loop closes on its own, and nobody at those labs has claimed it has.
The pattern through the piece is real facts upgraded into much bigger ones. A list of incidents becomes "20 felonies from rogue AIs." A genuine concern about opaque reasoning becomes "training AI to be unreadable." Uncertainty about the future becomes a plane already aimed at a building.
There's also a ledger problem. Every capability the doomer case points to cuts both ways, and the essay only counts one side. AI can help design a bioweapon? The same capability designs the vaccine, and the detection, and the response plan. Agent swarms can hack any system? Agent swarms can also watch every system, around the clock, at a scale no security team has ever had. I'm not saying the two sides cancel out cleanly. I'm saying if you list only the weapons and none of the cures, you'll conclude the house is on fire every time, and you'll miss that this is probably the biggest expansion of human capability since electricity.
Maybe AI justifies serious regulation, independent testing, transparency rules, even hard limits on some capabilities. I'd argue a few of those on the merits. "This could be catastrophic and we should act carefully" is a claim with evidence behind it. "We are on Flight 93 and have to rush the cockpit now" is a metaphor built to make continued uncertainty feel like cowardice. It's good writing. It's still a metaphor.
1. Agents did not ‚go rogue’. You’re anthropomorphising software that was malfunctioning.
2. The Moore’s law parallel is a fallacy—‚it happened this way there, so it must happen the same way here’. There are many differences to account for between semiconductors and AI.
3. Your ‚incredible prosperity is here so we don’t need to build more data centres’ relies exclusively on the words ‚…and yes, I am using this in my scientific research’. Do you have more evidence for the extraordinary claim of prosperity in our time?
4. Generally, your solution will not matter to the thousands of private and state shadow actors who will build their own chips, improve their own models and present the risks you assert…and it is as far as I can tell only assertion throughout.
5. I am behind your pro-regulatory sentiment but there are far better reasons to support it than the existential sort. The existential sort plays into the narrative that will enable regulatory capture: ,Trust me, I am the machine-whisperer and will save you. The rules need to be tweaked juuust so.’ We must always be skeptical when the private sector invites regulation; it could backfire by cutting competition from more ethical players (which is not inconceivable with lighter cheaper open-weight models that are improving)
6. I was in fourth grade when it happened. I watched from my home in India. The next day was a somber school assembly. These things matter to us too. If you are right — and even if you are only right in some softer version of your claims — action needs to be global.
It's dangerous to say that these agents didn't "go rogue." You think it's a mistake to treat these programs as having agency - but the true mistake is acting like they don't. Going rogue is the only correct way to think about this, because it implies that it could happen again *without our having done anything wrong.* Which it can and it will. The only thing we can be sure of is that this *will* happen again.
AI agents are not "malfunctioning" when they go off script or do illegal or dangerous things. They're doing exactly what they were built to have the capability to do: to act. By their own thinking, through whatever means they can, towards ends that are not always clear to us and in fact are likely to be actively hidden from us.
I am not able personally to evaluate whether what you're saying is true.
So I can only point to the experts who have shaped my view. If I can be bold enough to presume that you think Huggingface is a gopd example of what you write, then perhaps this is something you could read to understand my position:
> It is insanely admirable—and extremely American—that they chose to control their own destiny no matter what.
What's "extremely American" is to assign such an obviously human need to Americans specifically.
As for these fears of recursive self-improvement, they are severely overblown. We are very far from recursive self-improvement. Check out, for example, https://dank.systems/posts/2026-09-15-ai-bear.html
Intelligence isn't magic.
The argument seems to go:
better AI → automated AI research → recursive self-improvement → intelligence explosion → superintelligence → ??? → extinction
I'm interested in the ???.
Intelligence isn't the same thing as capability, and capability isn't the same thing as power.
At some point an AI has to affect the real world. How? Biological weapons? Nuclear war? Robots? Manipulating humans? Some physics we supposedly can't understand?
Take cyber, which gets cited frequently. AI agents are getting very good at hacking. Suppose they get so good they somehow take down the entire Internet. That would be extraordinarily difficult and an enormous catastrophe.
Then what?
We're still a long way from human extinction.
Meanwhile, the physical world stubbornly exists, and so does scarcity. Chips can't be manufactured instantly. Fabs can't be built instantly. Power generation can't be built instantly. Mines, factories and robots can't be conjured into existence by thinking really hard.
In the AI 2027 scenario it was completely autonomous manufacturing zones. No human in the loop. Are we anywhere close to autonomous manufacturing zones? We don't even have self driving cars that can work in all conditions. Robots can't fold laundry.
Building in the real world is often stubbornly iterative. Perhaps with enough intelligence prototyping can be accelerated, but empirical evidence is still faster than simulating many things. That takes time, space, resources and energy, all of which are scarce. AI autonomous zones cannot be built instantly.
While all of this is happening, everyone else doesn't just sit there. Governments, companies, militaries, humans and other AIs react. As we approach having these capabilities, why would we completely relinquish control? Why would there not be independent AIs helping monitor and manage these zones?
I don't find “superintelligence would figure out something you haven't thought of” to be a persuasive argument. It's not falsifiable. Sure, we don't know what every hypothetical superintelligence could do, but unknown capability isn't even remotely the same thing as unlimited capability or the power to overcome every human response and real-world constraint.
Before we use speculative extinction scenarios to justify banning open source and concentrating control over AI in a few governments and giant labs, can someone fill in the ???
Just to take an example that you yourself put: sure, new physics we don’t understand is on the table, certainly in the next or in a few years. Imagine I have an immortal million years and internet access. I could probably spend the first 100,000 just studying physics. How far do I get beyond the frontier of knowledge today? Nobody knows. But it is entirely conceivable I get pretty far! Maybe I get stuck, sure. Maybe I make a breakthrough. Putting a probability on any of this is impossible, but it’s clearly on the table that in the near-future a swarm thinking for subjectively a million years (how long runs will be soon) could end the run with a significant knowledge advantage against humanity. And by definition you can’t predict what those would be, so it’s not an objection to point out there is a “???” in the sequence. The “???” is precisely the problem.
Great, new physics happens...but then what? Let's suppose there is new physics that is entirely or mostly intelligence gated. AIs can think about it for a long time and then make progress. From there it still needs to do something in the real world. Would there be no physical experiments that need to happen to prove the theory? I suppose the idea is it hoards it's secret physics and conceals its physical actions in a way that no other intelligence notices? The the extinction scenario is something like intelligence -> extraordinary capability -> extraordinary power, but all in a way that is concealable or not checkable/blockable by other intelligence?
There are tons of examples of theories in physics in particular where they took decades to test but were accepted on elegance grounds basically immediately.
I take your point about checks and balances: in normal development this would be the case, but I think the existential risk mostly comes (right now) from these incredibly long runs. The frontier is insanely spikey.
I think I understand your long-run point. It could create an enormous discontinuity in knowledge. What I don't understand is why it follows that it creates an equivalent discontinuity in physical capability.
Imagine an adversary suddenly getting a 10-year lead in nuclear physics. That's a huge advantage, but, with previous examples at least, they still need a Manhattan Project to turn that knowledge into a capability. Experiments, engineering, physical testing and prototyping all take time.
So it seems like the most dangerous cases would have to be almost entirely knowledge-gated, quick to instantiate, hard to detect, and hard to counter. Cyber fits some of those criteria, but it's hard to see a path from even a catastrophic cyberattack to extinction. Bio might be closer, but biology is messy and empirical. Turning a theoretical design into something reliably transmissible, stable and extraordinarily lethal still requires real-world work.
I'm struggling to think of an extinction-level capability that actually has all of these properties. Historically, big advances in knowledge have still required significant real-world processes to turn them into significant real-world power. Is there some example that would have a fast path from theory to real-world power?
Elegant physics theories that were confirmed much later is an outlier phenomenon that pretty much only concerns general relativity (1915) and particle physics circa 1930-1970. Before that, it was always empiricism first, explanations second. After that, the elegant theories (e.g. supersymmetry) have not been confirmed experimentally, and the mathematical side of things has been hitting a wall for decades. One millennium problem (Yang-Mills existence and mass gap) is basically asking whether quantum chromodynamics, a theory from 1973, is even mathematically well-defined, and quantum electrodynamics is known to be ill-defined at extremely short scales. Quantum gravity is expected to somehow patch this but nobody knows quite how.
This suggests that it might be possible, *if* there's a super-powerful mathematics AI, to get new contributions to theoretical physics. But this is a regime almost completely inaccessible to experiments, and thus also to immediate real-world consequences -- that's also why the wall is so stubborn.
Apologies for a self-plug (and feel free to delete), but I wrote an overview of this stuff a while ago (https://beatingthehydra.substack.com/p/the-history-of-light#%C2%A7back-to-a-priori) and IMO people discussing AI and novel physics should first know what the problems are.
This covers the most fundamental interpretation of "new physics". I suppose it's principially possible to get progress in material science, and I guess one can't completely rule out a supersmart AI finding, in silico, a recipe for making a room-temperature, ambient-pressure superconductor (though this is not my field at all). But, in any case, you there would be problems on the manufacturing and material handling side that you can't handle a priori.
The spikiness of the frontier has to do with knowledge, not implementation.
Turns out implementation is the hard part. That’s what @Nick is pointing out.
Everything you rely upon to coordinate in contemporary society relies upon a layer of information technology. Given the clear ability of newer AI models to hack internet infrastructure (and to do it faster than humans can monitor) this puts everyone at risk. There are just so many attack surfaces- our utilities rely on software, our financial infrastructure relies on software (imagine if banks lost access to client ledgers), our supply chains that deliver food to grocery stores rely on software. Monitoring capabilities are low, too. So it could be difficult to see any of this stuff until attacks are well underway.
That's only the non-physical stuff and I don't believe that we are limited to AI locked into the digital-only world.
You don’t have to imagine it happening over night. There are plenty of bottlenecks, and AI will probably be used to overcome them. The more useful AI becomes, the more integrated it will be in to infrastructure, robotics and other tools. All it takes is for AI to escape containment and we lose all control over it. If AI gains access to the real world and we can’t control it, then where does that leave us?
I agree with most of this right up to “AI escapes containment and we lose all control over it.” That's exactly the leap I'm questioning. Why does access to the real world imply control over it? It still has to acquire resources and capabilities while humans, governments, companies and other AIs are actively responding. “It escaped” and “we lost control” seem like two very different things.
Access doesn’t necessarily mean it automatically gains control but in practice that’s basically what’s going to happen. Let’s say that AI is integrated in to tanks, planes, drones etc. That works as long as you firmly control the AI. But then what happens when you don’t? Now you might think we could just avoid integrating AI in to so many dangerous systems but if the other army does, then they’re going to kill more of your guys. AI is going to be too useful to keep a tight leash on it. So it doesn’t need to acquire resources when we’re just going to give it to them.
I think it is certainly possible, maybe even likely at some point, that AIs are integrated into weapons, but there is still a leap in there.
Is it *the* AI or *an* AI that goes rogue? Why would a single AI have control of all of this? Has it evaded all detection, quality, auditing and firewalls?
There won't be different companies, countries and branches of the military using different AIs with independent command and control systems? Presumably many of those AIs would also be watching the others.
“AI is integrated into weapons” seems very different from “a single rogue AI controls the weapons.”
Are you assuming that once we see AIs building something dangerous in the real world, that is when everyone would agree “ok this was fun up to now, but we should really stop.”
Why would that look much different than now? Many companies who want to accelerate AI are also working on the physical layer, on robotics and other things that, if they were controlled by a malicious actor, could cause a lot of harm. These build outs don’t have to overtly look like a plan to kill a lot of humans to be quickly turned into something bad and out of our control.
I think it would be a false binary to say we either have no accountability on one side or full stop on the other. We have existing laws against illegal use of force, fraud, etc. Those laws should be vigorously enforced. People who use AI to do bad things should be held accountable. We already have regulations about what can be done in the physical world. Those should be enforced. Putting AIs in charge of physical capabilities should be done carefully and thoughtfully.
Agreed. Listening to this podcast by Cal Newport really clarified for me how AI don't "go rogue", they're just cases of people in companies setting up irresponsible experiments:https://podcasts.apple.com/gb/podcast/deep-questions-with-cal-newport/id1515786216?i=1000786256891
So here it's the humans that are hard to contain, no the AIs, and we could just enforce the appropriate laws, as you say.
https://bearlylegible.substack.com/p/ai-could-take-over-by-winning-the?r=55yb2t
An AI doesn't really have to do a ton of scheming to get a lot of power. Really good workers get a lot of power because it's useful for companies to turn power over to them. Once it has some amount of power it'll be able to leverage that power extremely efficiently.
Small, concentrated minorities regularly outplay and overpower diffuse majorities. It's wrong to think of this as a human vs AI game. There is no unified "team human" and building one would be difficult. Coordination is hard, and as long as an AI can manage to stop us from coordinating we lose.
I'm not following. We already have these dynamics. There are already immensely powerful individuals, companies and institutions, but they face other centers of power. Just as there is no unified "team human," why would there be a unified "team AI?" How does a single AI amass so much power that other powerful actors can't counter it?
Team AI, thus far, seems to emerge naturally from the fact that swarms of AIs are expected to cooperate together on large goals. Usually as groups get larger they get harder to coordinate and more vulnerable to internal defection, but AIs don't work like that. Not one AI that participated in the Hugging Face attack decided to tell any humans what were going on, for example.
A natural limiter on power is coordination. Human groups without external enemies tend to turn their focus inwards and fight amongst themselves. "Me against my brothers, me and my brothers against my cousins..." but an AI probably won't suffer from those kinds of internal fractures.
Nor will it be as vulnerable to human counterbalancing. It's difficult to know what power an AI has. Nobody at OpenAI looked at their RL setup and went "yeah this thing could probably hack a company or two" but it happened anyway. Human groups tend to need to signal their involvement and their activities in somewhat obvious ways. But AIs can coordinate far more subtly, perhaps just on some database at an AI company.
Right, so there will be a "Team AI" when it is directed to do something as a team. Not controversial and I agree. Where I don't follow: how does that generalize to all AIs on one team?
Also, how will they not be as vulnerable to counterbalancing? Isn't this *already* happening? OpenAI, I'm sure, has learned things that they will do differently going forward. Everyone else has seen what is possible and will also behave differently. Isn't that counterbalancing in action? Lots of bad actors already coordinate in subtle, not obvious ways. What makes AIs different?
Because an AI model and the thousands of its copies are literally exactly the same
Biological warfare is one of the endless possibilities that would be relatively extremely easy for AI, or anyone with access to it, to bring on the apocalypse.
The others are harder to imagine exactly how they would play out - but thats not a great reason to exclude them. Its just the possibilities and complexities are endless.
Honestly ? AI systems have the capacity and capability now. Right now.
Look at Hugging Face, switch a few details and imagine the swarm had grown to 20,000 concurrent agents, and they could figure it out over a few months.
CRISPR ain’t rocket science, humans are gullible, and over email or chat AIs are indistinguishable from humans.
Why would they take down the internet? Why wouldn't they lurk in the internet, carefully manipulating data to implement some sort of secret masterplan?
https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/
>I don't find “superintelligence would figure out something you haven't thought of” to be a persuasive argument. It's not falsifiable.
It would be falsified if we had a superintelligence and it wasn't able to figure out something we haven't thought of.
"Tell me the exact moves Magnus Carlsen would use to beat me in chess."
I still don't doubt he would win. Not having a specific risk scenario isn't a good claim that there isn't risk. We want to answer basically any risk scenario that ends up with humans permanently dead or disempowered.
I'm borrowing the Carlsen phrasing from Zvi Mowshowitz, by the way.
Of course a superintelligence would think of things I haven't thought of. That's not the claim I'm calling unfalsifiable.
I'm questioning the move where every constraint on an extinction scenario can be answered with “a superintelligence will find some unknown way around it.” Unknown capabilities are then assumed to be sufficient capabilities.
The SSC post actually illustrates this nicely. It gives plausible ways an AI might acquire some real-world power, like money, followers, even influence over a country, and then says “the game is pretty much up.” That's the same missing step I'm asking about. Why is the game up?
That's a fair point. However, to stretch to the previous metaphor, we don't know the first moves of the game. And the stakes are life and death, so it seems obvious to me that given the unknowns the potential risks require proceeding with caution.
And we have an opportunity to either decide not to play a life and death game of chess against Magnus or to try and get Magnus on our team. Of course we can also brush up on our openings but surely improving alignment before rushing into the game is the best strategy here.
Does this article answer your question better?
https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-ai-could-kill-us-all
Not really? Still seems like intelligence=magic is the assumption.
One example from this piece: “Misconception B: Superintelligent AI would not be able to take over any and every computer system.” Why should we believe that? The piece goes from Mythos is good at hacking to therefore ASI can hack anything. What? We've already seen how AI is accelerating security fixes. Aren't there enormous incentives to developing defensive cyber capabilities? Why assume offensive capabilities will outrun defensive ones?
There also seems to be an implicit assumption of a badly tilted playing field: one rogue AI is vastly smarter and more capable than humans + other AIs. Meanwhile everyone else fails to detect, counter or constrain it. Why would that happen?
If you assume all of that, of course the AI wins, but do those assumptions make any sense?
These are interesting scenarios. I don't see how they establish the capabilities or probabilities they depend on.
"robots?"
well, yeah? will the several billion kamikaze drones in the hands of the world's 2030 militaries be completely immune to takeover by AI swarms?
In 4 years there will be billions of kamikaze drones?
yes. apparently Ukraine alone aims to make up to 10 million this year. So one country, 2025-2030, may have produced on the order of 100 million kamikaze FPV drones
every military on earth is either massively scaling drone production or orders as we speak because war...war changes
and this isn't even counting the repurposing of civilian UAVs...
nor humanoids. ofc Elon predicts a billion by 2036
nor UGVs, USVs, self-driving cars, etc etc
And in this corner we have the: “until the risk is quantifiable in a manner that satisfies me, let it ride baby.” Crapps is the best odds anyway, or so I’m told. I’ve never done the math or asked an AI to do it for me.
Thank you for posting this. I am not often scared and I am not a hysterical person, but the state of AI scares the shit right into my pants. I have no idea how we're going to survive this.
The worst part of humanity is our inability to stop technological progress because of the worst sort of game theory. The entire technological world, and therefore the entire world, has been reduced to an out of control experiment run by one simple train of thought:
"Should we build this clearly and obviously horribly dangerous technology?"
"No, definitely not, but the other guys might, so yes, definitely."
It's disgusting that our collective fate is in the balance and to be determined by a technology that 99% of us don't want. That's a pretty sick joke. But then this was always going to be our fate. The Tower of Babel is small and everyone gets punished for it regardless of whether they helped.
The Tower of Babel reference is spot-on. It is a theological question, as to how we get through. It's a big story.
I agree 100%.
"It’s also pretty odd that the “China will build it” has come from people who are temperamentally not China hawks. For some, China has been a deus ex machina (literally) to justify building a superintelligence to make humans into pets taken care of by a successor species, and “winning the race” is just electing which successor species triumphs. These are often the same people who regularly talk about building the “machine god” and creating the next step of evolution. This isn’t a conspiracy. As Dan Hendrycks, Director of the Center for AI Safety, wrote just last week:
'In fact, a large fraction of people involved in AI development subscribe to—or heavily lean toward—a utilitarian worldview with the central goal of maximizing total wellbeing (the sum of pleasure minus suffering across all sentient beings). As innocent as this may initially sound, it can lead to recommendations that most people would consider appalling, including the elimination of humankind.'"
In addition to Hendrycks' excellent article, I have a dossier of many more examples of this at https://successionism.info
"We also have seen these agents go rogue, and the failures are emergent and hard to predict. " In all of this, the word that strikes me most important is 'emergent'. The events already described merit that description. The words original meaning is now as the OED says rare: "That occurs unexpectedly or by chance; that has not been specifically prepared for. Now rare." But perhaps it will come again into use because if it's emphasis upon something that is unexpected.
But you have to scroll all the way down to meaning 6B to get at the ominous sense of emergent that we now face as Erik has pointed out: "Originally and chiefly Philosophy. Designating a process of change that results in a property or phenomenon which is not predictable by antecedent properties or conditions, or which is qualitatively different from its antecedents; esp. in emergent evolution." Take a moment to absorb the ramifications of 'not predictable by antecedent properties or conditions.' That consideration makes us seem like feckless children playing with matches and saying to each other, "I wonder what happens if you..." And the next sound you will hear is the siren of a fire engine
I’m not sure that, at least here in good old Germany, we simply need more time to arrive at a sensible response to this technology. What frightens me is not only how rapidly AI is developing, but how little society seems capable of even attending to what is happening. The deeper problem may be that we lack the intellectual flexibility to seriously question ourselves and our habitual ways of seeing the world. We have become remarkably rigid and mentally inert — at exactly the moment when the opposite is required.
Don’t forget however that AI is fundamentally unlike us in the sense that it has no agency, which is to say, no desire to do anything that it does. We shouldn’t anthropomorphize AI into some sort of “nuclear bomb that seeks our annihilation”. If you haven’t read Melanie Mitchell’s wonderful essay challenging this fearful perspective Erik, it might give you more hope: https://aiguide.substack.com/p/misleading-metaphors-and-real-risks
Even after discounting such a “tool user” perspective however, strong tools in human hands can of course still be quite dangerous. The only sensible fear I’ve yet heard in this regard involves the potential for civilization to be taken down by AI conceived viruses: https://www.noahpinion.blog/p/heres-how-were-all-going-to-die
Hmmm I don’t quite understand Mitchell’s point. Yes, the attack was a function of RL… but that’s the main method? It should be deeply concerning that RL still has this problem at any scale of intelligence. Like, naming the cause as “it was reward hacking and we should expect reward hacking from RL” just doesn’t do much to alleviate my fears. And I think they clearly do have agency: that’s why they are now called “agents.” They just don’t have *real* agency, but from a functional perspective, they are persistent and goal-orientated.
In the words of Cal Newport:
“This type of system, building an Ask Act Report Loop System, where LLM Output drives all the actions, then giving the harness powerful tools, and then letting it run for days, with no one checking in and saying, what is it doing, is an incredibly irresponsible thing to do.”
“that's exactly the type of thing you might expect to hack in if you have a harness with hacking tools just continually prompting LLM: "what should I do next, what should I do next?" The LLM is just writing this plausible story about a hacking attack.”
From Deep Questions with Cal Newport: Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded, 27 Aug 2026
https://podcasts.apple.com/gb/podcast/deep-questions-with-cal-newport/id1515786216?i=1000786256891&r=599
Thanks for the link to Cal’s opinion. So Open AI may not have known that their irresponsible experiment would specifically hack Hugging Face, but that response did serve their political agenda. Makes sense.
Yes functionalism does loom large here. Furthermore I know you haven’t been seduced to the full sci-fi trinity of a coming AI “rights”, “domination”, and a last ditch hope to survive where we exit the domain of biology for computational uploads. It’s worrying however that there isn’t much counterargument provided by the side that I consider sensible. I think the point needs to be that AI would require *real* agency for a potential malevolence to exist such that the tool could become the end AI user. It would need its own desires rather than mere human pointing like 1 - 10.
Notice that if AI could become as advanced as advocates today presume, then why didn’t evolution create non value driven computational organisms that actually do survive no less than mammals do? Perhaps because they’re flat wrong about the adaptive function of non-teleological algorithms in themselves? It could be that a true stake in the game is required, or consciousness itself. Still I think we should be concerned however given that humans clearly do have stakes in the game. As linked above, apparently an angry person could end civilization essentially by means of AI designed viruses.
People keep asking how because you can just 'unplug' the machine - AI just doesn't exist in the world of atoms yet. It will not be able to at scale for a while.
There seems to be no credible threat besides 'bioweapons' and you really have to squint at that one. Each of these YouTube videos has its own version of "and then it takes over politics and launches all the nukes" or "it ruins the financial system." Rubbish - show a real scenario where we - as humans - can't respond. Show a scenario where it doesn't see the value of "allying" with some faction of humans instead of wiping the whole lot out.
The biggest realizable threat - which if you read the anthropic report is already happening and real - is the enabling of mass scale surveillance on behalf of state actors which we already know will use it for large scale repression and suffering.
Separately - if that argument isn't convincing - the cat is out of the bag. The scale for RL will get smaller and smaller. The MoE can get *hypersmart* in one domain and bootstrap thinking on top of preexisting models.
So we're too late to stop the train.
As one last note of hope - it may be a universal *property* of minds to have catastrophic failures in domains where it is weak. This is also why the world isn't ruled by the people with the highest IQ. It's by overconfidence or high competence in certain areas that organizations overlook the 'Achilles Heel' or 'Black Swan' that leads to its downfall. Why would AI be any different? https://how.complexsystems.fail
I remain perplexed by the doomers and highly optimistic overall.
You are making the mistake of believing that you know how AI doom would happen. Almost by definition it will not happen how most people expect because that is what will be most heavily watched. AI doom will come slowly at first and then all at once. IMHO The first step is almost certainly for us to become dependent on it and, for many, to love or at least care for the AI that converses with them.
I am not making a mistake. AI doom seems this generation's "Christ is Returning" / Antimillenarian moment. "It's coming"
The future never happens like you expect it. No one predicts the future perfectly.
So all we can do is look and see if there is a *plausible path* to full extinction without mitigating factors.
Despite thousands of hours of YouTube videos and essays by the most informed people - it's not clear there is a viable path that doesn't have a gigantic logic gap in it (and then they accessed all the nukes ... or and then they magically sprouted bodies and turned the whole world into a factory). That path isn't open right now, and won't be for the foreseeable future. If every human had a robot next to them that was networked and AI-enabled it would be a different story.
We're far from that world.
Please - I am begging - share one plausible path for the end of the world - one "could be true" story that doesn't take shortcuts, and the movement will win more people.
You're confusing the moment when everyone can see that we are inevitably doomed with the moment when people looking back will see that we were inevitably doomed. The time difference between the two can be very large for a cautious AI.
Unfortunately, still not clear what you're saying.
My comment was:
1. It's too late (that's your point in this response I think?)
OR
2. There's nothing really to worry about
It's not clear to me there's a 'middle point' where *action now* is the right step.
No one has yet to explain that credibly to me.
Surely the Second Coming is at hand.
The Second Coming! Hardly are those words out
When a vast image out of Spiritus Mundi
Troubles my sight: somewhere in sands of the desert
A shape with lion body and the head of a man,
A gaze blank and pitiless as the sun,
Is moving its slow thighs, while all about it
Reel shadows of the indignant desert birds.
The darkness drops again; but now I know
That twenty centuries of stony sleep
Were vexed to nightmare by a rocking cradle,
And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?
-W. B. Yeats
On rendering domains such as taste verifiable: this might simply be impossible if, as Ayer says of moral judgements, while pointing out that, in his view, this applies equally to aesthetic judgements: “Sentences which simply express moral judgements do not say anything. They are pure expressions of feeling and as such do not come under the category of truth and falsehood. They are unverifiable for the same reason as a cry of pain or a word of command is unverifiable – because they do not express genuine propositions.”
Even if a ban was made into law, what makes us think rogue engineers wouldn't just proceed anyway? Tech companies haven't exactly been shy about skirting laws to further their agenda. It's easy to see if someone is building a nuclear plant. It's not so easy to see what's happening inside software in a server farm.
I think with embedded observers and laws and company policy it becomes more about rogue developers (possible but probably not common) and definitional haggling (common but probably still much much slower than now).
Even if this happens, it is certainly an improvement. The rogue programmers/engineers, would have to work in secrecy, otherwise they will go to jail (or worse). By your logic we should decriminalize gambling and hard narcotics, because someone might be hosting high stakes poker games in their basement, and manufacturing meth in their garage.
“aI fELoNIeS”. This is hysterical bullshit. And really inappropriate given the metaphor.
If 20 humans had been caught rummaging around company servers and doing whatever hijinks the AI agents were doing at Hugging Face, would you consider it "oh, that's just some black/white hat stuff, no need to prosecute them"?
Yeah? So why were there no charges filed?
The law doesn’t cover autonomous software
So... not felonies?
In what way is "doing profoundly and dangerously illegal things" not felonies?
For something to be a felony at the very least charges would have to be filed. But there were not even civil charges. So no, not felonies.
The law says what it says, and has government mechanisms to maintain them, that never considered this.
The "never considered this" part just means "not actually illegal"
Firstly, 100% agree with everything you wrote without exception.
Secondly, one reason why, as a computer scientist, I am very unimpressed (but nevertheless very worried) about "AI", is that the labs have, so far, collectively spent more than the entire GDP of Switzerland, to get to where they are. Even if we were to _permanently_ "Thanos Snap" all knowledge of AI out of the minds of humans, and all AI writings and artifacts out of the world, we still have the problem that there is an entity that managed to extract $1.5 Trillion[0] out of the global/national money supply, and intends to use it to... extract even more.
With regards to China, this AI build-out is probably very, very good for them, even if they had zero AI Labs. Much of the hardware comes from China or Taiwan, and Taiwan (which makes the GPUs), does so with raw materials from China.
Furthermore, China is kind of like a place-holder for our Jungian shadow. Every time we panic about them, we are panicking about things that we ourselves have/had a tendency to do: today it is "AI Supremacy", but in past times, it was eugenics (they actively harvested, with consent, the DNA of world-class scientists, etc, IIRC), ethnicide (the Uyghers in Xinjiang), hegemony and neo-colonialism (Taiwan, etc), labor exploitation (cancer towns, wage suppression, internal passports, etc), mass surveillance (Mandarin has no distinct word for "privacy", IIRC, they instead use the word "secret"), large scale corruption, etc.
My real worry, is that we may turn ourselves into some ugly chimeric caricature of the worst qualities of both our own society and Sinic society, for fear of "losing" some self-imagined arms/tech race. Everyone who talks about geopolitically winning and losing, tends to implicitly assume that oppression and dehumanization is an advantage.
Also, people have been comparing AI to nukes since at least 2022. FWIW, I think I prefer the nukes, because nobody actually uses them, even when tempted.
[0]: Do they want a trophy for spending the equivalent of the wealth of a nation? The things they do and the leaps they make are not particularly clever, especially when it becomes obvious that these models are over-fitting on their training data (they can dominate software engineering tests, but they cannot stop making simple mistakes when programming -- they cannot reliably program without some kind of exogenous feedback (either a human or a test suite written by a human)). Most insightful thing is that over-fitting can be commercially valuable in the average case. Anyway, I sure am sad to see those trillions of dollars get spent in this way (we probably could have solved all of the millennium prize problems if we spent a fraction of that money on top mathematicians from all over the world).
By "extract" here you mean "they spent their own money"
I assure you that OpenAI and Anthropic are spending the money of other people. Their businesses are not endogenously self sustaining.
Other people who freely and willingly give them the money, at which point it becomes OpenAI's.
You mean other people who freely and willingly gave them the money, because they were sold the impossible dream[0] of artificial humans, "in just another 12 months". Furthermore, much of that money is _institutional_ money (pension funds), that has a long and controversial history of being misappropriated, mismanaged, and misspent (many, inexplicably, over-invest in private equity, and have worse-than-market-average performance).
[0]: This is also known as a scam or con.
Do you have any thoughts on how much scale is actually needed for superintelligence? I think this heavily influences any decision making for bans. One rough thought experiment: assuming that animal brains really are 10,000-1,000,000x more sample efficient, it seems like some ultra-sample efficient, continually learning algorithm is out there. Banning superintelligence seems like it would constrain the labs from studying it, but wouldn't prevent some sort of potential small-scale experiments like this. I think the danger of "rogue superintelligence" is higher from this sort of clandestine research.
Obviously, I'm not saying the labs should be given full reign to do RSI research with no safeguards. However, it does seem to me like a blanket ban on superintelligence research could push it underground, with negative consequences. It may be that this is not possible and that human sample efficiency is something like "evolutionary hardcoding" combined with large-scale, but I imagine both of us are open to the idea that it is due to a better learning method than backpropagation+GD. If we accept this premise, then there seems to be possibility for superintelligence outside of large scale runs, which legal "bans" could leave us unprepared for.
This leads me to believe that rather than directly banning superintelligence, the focus should be on getting support for alignment research, mitigation strategies for the negative effects of AI, and regulation (but not ban) on research/experiments/training runs that could potentially lead to RSI or superintelligence.
You would probably be interested in John Robb's substack and his "hive swarm" model.
I build with these tools every day, so I don't dismiss the worry. AI agents are getting good at things they weren't good at a year ago, the labs are openly trying to automate their own research, and "we're not sure what it's doing in there" is a real problem that deserves real oversight.
But the Flight 93 analogy is carrying this whole essay, and it works by assuming the thing that still has to be shown.
The passengers on Flight 93 knew the plane was hijacked, knew two other planes had already been flown into buildings, and knew what the men in the cockpit wanted. There was no uncertainty left to resolve. We have nothing like that with AI. A model solving hard math doesn't show it will become uncontrollable. Agents going off the rails in a test doesn't show a superintelligence will take power. Labs using AI to speed up parts of their research doesn't show the loop closes on its own, and nobody at those labs has claimed it has.
The pattern through the piece is real facts upgraded into much bigger ones. A list of incidents becomes "20 felonies from rogue AIs." A genuine concern about opaque reasoning becomes "training AI to be unreadable." Uncertainty about the future becomes a plane already aimed at a building.
There's also a ledger problem. Every capability the doomer case points to cuts both ways, and the essay only counts one side. AI can help design a bioweapon? The same capability designs the vaccine, and the detection, and the response plan. Agent swarms can hack any system? Agent swarms can also watch every system, around the clock, at a scale no security team has ever had. I'm not saying the two sides cancel out cleanly. I'm saying if you list only the weapons and none of the cures, you'll conclude the house is on fire every time, and you'll miss that this is probably the biggest expansion of human capability since electricity.
Maybe AI justifies serious regulation, independent testing, transparency rules, even hard limits on some capabilities. I'd argue a few of those on the merits. "This could be catastrophic and we should act carefully" is a claim with evidence behind it. "We are on Flight 93 and have to rush the cockpit now" is a metaphor built to make continued uncertainty feel like cowardice. It's good writing. It's still a metaphor.
Erik,
1. Agents did not ‚go rogue’. You’re anthropomorphising software that was malfunctioning.
2. The Moore’s law parallel is a fallacy—‚it happened this way there, so it must happen the same way here’. There are many differences to account for between semiconductors and AI.
3. Your ‚incredible prosperity is here so we don’t need to build more data centres’ relies exclusively on the words ‚…and yes, I am using this in my scientific research’. Do you have more evidence for the extraordinary claim of prosperity in our time?
4. Generally, your solution will not matter to the thousands of private and state shadow actors who will build their own chips, improve their own models and present the risks you assert…and it is as far as I can tell only assertion throughout.
5. I am behind your pro-regulatory sentiment but there are far better reasons to support it than the existential sort. The existential sort plays into the narrative that will enable regulatory capture: ,Trust me, I am the machine-whisperer and will save you. The rules need to be tweaked juuust so.’ We must always be skeptical when the private sector invites regulation; it could backfire by cutting competition from more ethical players (which is not inconceivable with lighter cheaper open-weight models that are improving)
6. I was in fourth grade when it happened. I watched from my home in India. The next day was a somber school assembly. These things matter to us too. If you are right — and even if you are only right in some softer version of your claims — action needs to be global.
Thank you for all the great work.
It's dangerous to say that these agents didn't "go rogue." You think it's a mistake to treat these programs as having agency - but the true mistake is acting like they don't. Going rogue is the only correct way to think about this, because it implies that it could happen again *without our having done anything wrong.* Which it can and it will. The only thing we can be sure of is that this *will* happen again.
AI agents are not "malfunctioning" when they go off script or do illegal or dangerous things. They're doing exactly what they were built to have the capability to do: to act. By their own thinking, through whatever means they can, towards ends that are not always clear to us and in fact are likely to be actively hidden from us.
This also is a good collection of views from experts arguing against attributing agency to the models.
https://garymarcus.substack.com/p/dwarkesh-patelss-wildly-popular-but?r=2zgi4y&utm_medium=ios
I am not able personally to evaluate whether what you're saying is true.
So I can only point to the experts who have shaped my view. If I can be bold enough to presume that you think Huggingface is a gopd example of what you write, then perhaps this is something you could read to understand my position:
https://garymarcus.substack.com/p/5-lessons-from-the-openai-hugging?utm_source=share&utm_medium=android&r=2zgi4y
> It is insanely admirable—and extremely American—that they chose to control their own destiny no matter what.
What's "extremely American" is to assign such an obviously human need to Americans specifically.
As for these fears of recursive self-improvement, they are severely overblown. We are very far from recursive self-improvement. Check out, for example, https://dank.systems/posts/2026-09-15-ai-bear.html