Superintelligence is impossible

There are no free lunches

At long last, serious arguments that the risk of superintelligent AI has been overblown are gaining some traction.¹ One particularly good article is “The Myth of Superhuman AI” by Kevin Kelly. He was also on Sam Harris’ podcast, speaking about the same issues. More recently, there was an essay by famed science fiction author Ted Chiang "Why Computers Won’t Make Themselves Smarter” in The New Yorker, in which Chiang argues, relatively simplistically:

For example, there are plenty of people who have I.Q.s of 130, and there’s a smaller number of people who have I.Q.s of 160. None of them have been able to increase the intelligence of someone with an I.Q. of 70 to 100, which is implied to be an easier task. None of them can even increase the intelligence of animals, whose intelligence is considered to be too low to be measured by I.Q. tests. 

I don’t really buy this sort of argument, but it’s nice to see some doubt around this issue. An increase of skepticism is important because there’s been a great deal of intellectual resources devoted to what is, in the end, a hypothetical risk. Of course, for getting people to seriously contemplate existential threats is something thinkers like Nick Bostrom deserve serious credit for, as well as credit for crafting evocative thought experiments. But I agree with Kevin Kelly and Ted Chiang that the risk of this particular existential threat has been radically overblown, and, more importantly, I think the reasons why the claims are overblown are themselves interesting and evocative.

First, some definitions. Superintelligence refers to some entity compared to which humans would seem like children, at minimum, or ants, at maximum. The worrying runaway threat is that something just a bit smarter than us might be able to build something smarter than it, etc, so the intelligence ratchet from viewing humans as children to viewing humans as ants would occur exponentially quickly. While such an entity could possibly adopt a beneficent attitude toward humanity, since it’s also incomprehensible and uncontrollable, its long-term goals would probably be orthogonal to ours in such a way that “eliminates our maps” as it were. Regardless, even just our fates being so far outside of our control constitutes an existential threat, especially if we can’t predict what the AI will do when we ask it for something (this is called the alignment problem).

Certainly it’s an interesting hypothetical scenario. But all existential threats are not equal. Some are so vanishingly small in their probabilities (rogue planets, like in the film Melancholia) that it would be madness to devote a lot of time to worrying about them. And at this point my impression from the tremors in the world wide web is that the threat of superintelligent AI is being ranked by a significant number of thinkers in the same category as the threat of nuclear war. Here’s Elon Musk, speaking at MIT back in 2014:

“I think we should be very careful about artificial intelligence. If I had to guess at what our biggest existential threat is, it’s probably that.”

Consider that statement. Because the obvious answer to the question of the biggest existential threat, which people have been giving for decades, is the threat of nuclear annihilation. After all, the risk of civilizational reset brought about by a global nuclear war is totally concrete and has almost happened on multiple occasions. Whereas the risk of AI is totally hypothetical, has a host of underlying and unproven assumptions, and has never even gotten close to happening.

To deserve its title as leading or even top-tier existential threat, the arguments for the risk of superintelligent AI have to be incredibly strong. Kevin Kelly in his article does a good job listing five pretty sensible objections (wording is his):

1. Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept.

2. Humans do not have general purpose minds, and neither will AIs.

3. Emulation of human thinking in other media will be constrained by cost.

4. Dimensions of intelligence are not infinite.

5. Intelligences are only one factor in progress.

While he lists five separate objections, I think most actually spring naturally from using a specific framework to think about intelligence. Here, I’d like to explore where I think the fundamental disagreement between believers and non-believers in superintelligence lies. Roughly, there are two frameworks: either an evolutionary one (non-believers) or a deistic one (believers).

Note that I don’t mean to imply that the deistic view is wrong merely because I’ve labeled it as deistic; plenty of really wonderful thinkers have had this framework and they aren’t adopting it for trivial reasons. Rather it’s deistic in that it assumes that intelligence is effectively a single variable (or dimension) in which rocks lie at one end of the spectrum and god-like AI lies at the other end. It’s “deistic” because it assumes that omniscience, or something indistinguishable to it from the human perspective, is possible.

All of the points Kevin Kelly makes actually stem from him eschewing the deistic understanding of intelligence in favor of the evolutionary. Which framework is correct? The demarcation between the two frameworks begins with an image, a scale laid out in Nick Bostrom’s book Superintelligence:

This is the deistic frame in a nutshell: there’s some continuum from mice up to Einstein up to near-omniscient recursively self-improved AI.

While Kelly immediately objects to this by saying that there is no agreed upon definition of general intelligence, I think actually there are two broad options. One is to define intelligence based on human intelligence, which can be captured (at least statistically) with IQ. The Intelligence Quotient holds up well across life, is heritable, and is predictive for all the things we associate intelligence with. Essentially, we can use the difference between low-intelligence people and high-intelligence people as a yardstick (this is how Chiang is arguing as well).

But clearly rating a superintelligence using an IQ test is useless. We wouldn’t use Einstein as a metric for a real superintelligence of the variety Bostrom is worried about. The danger isn’t just from an AI with an exceptionally high IQ (since it’s not like high IQ people run the world anyways). Rather, the danger comes from the possibility of a runaway process of a learning algorithm that creates an AI with a god-like IQ, something different in kind. To examine is that’s possible we need a more abstract and generalizable notion of intelligence than IQ tests.

A mathematical definition of intelligence is actually given by Legg and Hutter in their paper “Universal intelligence: a definition of machine intelligence.” Taking their point broadly, the intelligence of an agent is the sum of the performance of that agent on all possible problems, weighted by the simplicity of those problems (simple problems are worth more). A superintelligent entity would then be something that scores extremely high on this scale.

So universal intelligence is at least describable. Interestingly, Einstein scores pretty low on this metric. In fact, every human would score pretty low on this metric. This is because the space of all problems is mostly stuff that human beings are really bad at, like picking out the same two color pixels on a TV screen (and three pixels triads, and so on). This scale or metric for intelligence fits perfectly with the deistic framework for intelligence, making it imaginable or at least definable that intelligence is a dial that can be turned up indefinitely.

In contrast, let’s now introduce the evolutionary framework. In this framework, intelligence is really about adapting to some niche portion of the massive state space of all possible problems. Its inspiration is On the Origin of Species, the closing of which I’ll quote in full because it’s such a wonderful paragraph:

It is interesting to contemplate a tangled bank, clothed with many plants of many kinds, with birds singing on the bushes, with various insects flitting about, and with worms crawling through the damp earth, and to reflect that these elaborately constructed forms, so different from each other, and dependent upon each other in so complex a manner, have all been produced by laws acting around us. These laws, taken in the largest sense, being Growth with reproduction; Inheritance which is almost implied by reproduction; Variability from the indirect and direct action of the conditions of life, and from use and disuse; a Ratio of Increase so high as to lead to a Struggle for Life, and as a consequence to Natural Selection, entailing Divergence of Character and the Extinction of less improved forms. Thus, from the war of nature, from famine and death, the most exalted object which we are capable of conceiving, namely, the production of the higher animals, directly follows. There is grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone circling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being evolved.

In the evolutionary framework different intelligences live together on a tangled bank (although here the bank is pretty abstract). This is pretty much the way our artificial intelligences live together right now. One AI to fly the plane, another AI to filter spam from email, and so on.

The evolutionary framework of intelligence doesn’t really allow for a superintelligence. To draw on something specific to support the evolutionary framework, consider the “No Free Lunch” theorem. Taken from the abstract of the paper by Wolpert and Macready that defines the theorem:

“A number of “no free lunch” (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over another class.”

The proof, which there are multiple versions of in the field of machine learning, search, and optimization, is highly relevant to the field of intelligence research. No Free Lunch means that no single model, or optimization, works best for all circumstances.

However, I don’t want to rely directly on the math or the specific proof itself, as these kinds of things are always limited in their universality by their assumptions. The “No Free Lunch” theorems are true, but they are also in the context of the entirety of possibility space (without weighing these spaces by the likelihood of their existence). I think the No Free Lunch theorem should instead be thought of as a concrete example of the more generalizable idea. Some sort of broader “No Free Lunch principle” (rather than theorem), that applies generally across evolution, intelligence, learning, engineering, and function.

First, an argument by analogy: consider the absurdity of designing a “superorganism” that has higher fitness in all possible environments than the highest fitness of any extant organism in any of those environments. While such an entity is definable, it is not constructible. It would need to be more temperature resistant than deep-vent bacteria, more hardy than a tardigrade, need to double faster than bacteria, hunt better than a pack of orcas, and, well, you get the idea. While it might be definable in terms of a fitness optimization problem (just wave your hands about precisely how it is optimizing for fitness), there isn’t any actual such thing as a universal organism that has a high fitness in all possible, or even just likely-here-on-Earth, environments. Again, a superorganism that is highly fit in every environment is definable, but not possible.

This particular free lunch is impossible in evolution because there’s no way to increase an organism’s fitness in regard to all niches or contexts. In fact, it’s probably the case that increasing fitness in regards to any particular environment necessarily decreases fitness in regards to other environments. While environments on Earth aren’t uniform in their likelihood, they are incredibly variable and they do change quickly. We can make a general claim that the more changeable an environment, and the more the likelihood is spread out over diverse environments, the more generalized No Free Lunch principle is likely to apply.

In fact, I think it’s precisely this No Free Lunch principle that actually generates the endless forms of evolution. A mutation may increase the fitness of an organism with regard to the current environment, but it will often, perhaps always, decrease fitness in regard to other environments. Since the environment is always changing, this will forever cause a warping and shifting of where organisms are located in the landscape of possible phenotypes. The No Free Lunch principle is why there’s no end state to evolution. And since there are never any free lunches, evolution is the one game nobody can stop playing. It’s the game that lasts forever.

This same sort of reasoning about fitness (in regards to the environment) applies also to intelligence (in regards to a set of problems). While a superintelligence is definable, it’s not possible. There’s just no one brain, or one neural network, that plays perfect Go, can pick the two matching pixels of color out of the noise of a TV screen in an instant, move with agility, come up with a funny story, bluff in poker, factor immensely large numbers quickly, and plan world domination.

For my PhD I studied Integrated Information Theory under Giulio Tononi, trying to come up with mathematical ways to measure when when a system breaks down into subsystems. It’s hard to make big integrated systems, let alone those that can perform multiple functions, and I’d posit there is no integrated constructible system that can perform all functions perfectly.

The No Free Lunch principle shows up in intelligence because whenever you’re adapting a neural network or an organism to perform in some way you’re implicitly handicapping it in some other way. The infamous difficulty of getting a neural network trained on one task to perform well on another task is implicit proof of this.

​I recently saw a practical example of the No Free Lunch principle as specifically concerns AI, in a talk given at Yhouse in New York City by Julian Togelius, who studies intelligence by designing AIs that play computer games. Here’s a figure from a paper of his where they researched how different controllers (AIs) performed across a range of different computer games.

Basically, no controller (read, simple AI) does well across the full selection of games (and the games aren’t actually even that different). It’s not like they are trying directly to build superintelligence, but it’s a telling example of how the No Free Lunch principle crops up even in what seem like small domains of expertise.

Even if there were a general intelligence that did okay across a very broad domain of problems, it would be outcompeted by specialists willing to sacrifice their abilities in some domains to maximize abilities in others. In fact, this is precisely what’s happening with artificial neural networks and human beings right now. It’s the generalists who are being replaced.

If the No Free Lunch principle applies to intelligence this in turn indicates there will forever be endless forms of intelligence. Changes to the neural architecture of a network or a brain may help with the current problems at hand, but this will necessarily decrease its capability of solving other types of problems. Overall, a tangled bank will emerge.

Superintelligence would be an end to evolution, but in truth evolution never stops. It’s the one game you can’t help but play.

1. This article was originally published in July 2017 on a now defunct personal blog. I have tried to update it occasionally to cite new information. There has been a lot of progress in deep learning since then, and early hints of AGI. I still stand by this generalized version of the NFL Principle and think true superintelligence is impossible and would contradict, broadly, the lessons from evolutionary theory. However, it is undoubtably true that, for instance, now individual agents can do very well across multiple tasks without the kind of switching between modules I described, and there has been massive progress in natural language processors in particular. I now do think we will see human-equivalent “general” AI in my lifetime, although I doubt it will have the genie-like properties of superintelligence Bostrom and others think it will.