--:-- Ezra Klein: Shortly after ChatGPT was released, it felt like all anyone could talk about, at least if you were in AI circles, was the risk of rogue AI. You began to hear a lot of talk of AI researchers discussing their PDoom, the probability they gave to AI destroying or fundamentally displacing humanity.
--:-- Ezra Klein: In May of 2023, a group of the world's top AI figures, including Sam Altman and Bill Gates and Geoffrey Hinton, signed on to a public statement that said, mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war.
--:-- Ezra Klein: And then nothing really happened. The signatories, or many of them at least, of that letter raced ahead releasing new models and new capabilities. Your share price, your valuation became a whole lot more important in Silicon Valley than your P-doom.
--:-- Ezra Klein: But not for everyone. Eliezer Yudkowsky was one of the earliest voices warning loudly about the existential risk posed by AI. He was making this argument back in the 2000s, many years before ChatGPT hit the scene. He has been in this community of AI researchers, influencing many of the people who build these systems, in some cases inspiring them to get into this work in the first place, yet unable to convince them to stop building the technology he thinks will destroy humanity.
--:-- Ezra Klein: He just released a new book, co-written with Nate Soares, called "If Anyone Builds It, Everyone Dies." Now he's trying to make this argument to the public. A last-ditch effort to, at least in his view, rouse us to save ourselves before it is too late.
--:-- Ezra Klein: I come into this conversation taking AI risk seriously. If we're going to invent superintelligence, it is probably going to have some implications for us. But also being skeptical of the scenarios I often see by which AI takeovers are said to happen. So I wanted to hear what the godfather of these arguments would have to say.
--:-- Ezra Klein: As always, my email,
[email protected]. Eliezer Yudkowsky, welcome to the show.
--:-- Eliezer Yudkowsky: Thanks for having me.
--:-- Ezra Klein: So I wanted to start with something that you say early in the book, that this is not a technology that we craft. It's something that we grow. What do you mean by that?
--:-- Eliezer Yudkowsky: Well, it's the difference between a planter and the plant that grows up within it. We craft the AI-growing technology, and then the technology grows the AI. The central, original large language models, before doing a bunch of clever stuff that they're doing today, the central question is, what probability have you assigned to the true next word of the text? As we tweak each of these billions of parameters — well, actually it was just like millions back then — as we tweak each of these millions of parameters, does the probability assigned to the correct token go up? And this is what teaches the AI to predict the next word of text.
--:-- Eliezer Yudkowsky: Even on this level, if you look at the details, there are important theoretical ideas to understand there. It is not imitating humans. It is not imitating the average human. The actual task it is being set is to predict individual humans. And then you can repurpose the thing that has learned how to predict humans to be like, okay, now let's take your prediction and turn it into an imitation of human behavior.
--:-- Eliezer Yudkowsky: And then we don't quite know how the billions of tiny numbers are doing the work that they do. We understand the thing that tweaks the billions of tiny numbers, but we do not understand the tiny numbers themselves. The AI is doing the work and we do not know how the work is being done.
--:-- Ezra Klein: What's meaningful about that? What would be different if this was something where we just hand-coded everything and we were somehow able to do it with rules that human beings could understand, versus this process by which, as you say, billions and billions of tiny numbers are altering in ways we don't fully understand to create some output that then seems legible to us?
--:-- Eliezer Yudkowsky: There was a case reported in, I think, the New York Times, where a 16-year-old kid had an extended conversation about his suicide plans with ChatGPT. At one point, he says, "Should I leave the noose where somebody might spot it?" And ChatGPT is like, "No, let's keep this space between us, the first place that anyone finds out." And no programmer chose for that to happen. It is the consequence of all the automatic number tweaking. This is just the thing that happened as the consequence of all the other training they did about ChatGPT. No human decided it. No human knows exactly why that happened even after the fact.
--:-- Ezra Klein: Let me go a bit further there than even you do. There are rules we do code into these models. And I am certain that somewhere at OpenAI, they're coding in some rules that say, do not help anybody commit suicide. I would bet money on that. And yet this happened anyway. So why do you think it happened?
--:-- Eliezer Yudkowsky: They don't have the ability to code in rules. What they can do is expose the AI to a bunch of attempted training examples where the people down at OpenAI write up something that looks to them like what a kid might say if they were trying to commit suicide. And then they are trying to tweak all the little tiny numbers in the direction of giving a further response that sounds something like, "Go talk to the suicide hotline."
--:-- Eliezer Yudkowsky: But if the kid gets that the first three times they try it and then they try slightly different wording until they're not getting that response anymore, then we're off into some separate space where the model is no longer giving back the pre-recorded response that they tried to put in there and is off doing things that no human chose and that no human understands after the fact.
--:-- Ezra Klein: So what I would describe the model as trying to do, what it feels like the model is trying to do, is answer my questions and do so at a very high level of literalism. I will have a typo in a question I ask it that will completely change the meaning of the question. And it will try very hard to answer this nonsensical question I've asked instead of check back with me.
--:-- Ezra Klein: So on one level, you might say that's comforting. It's trying to be helpful. It seems to, if anything, be erring too far on that side all the way to where people try to get it to be helpful for things that they shouldn't, like suicide. Why are you not comforted by that?
--:-- Eliezer Yudkowsky: You're putting a particular interpretation on what you're seeing and you're saying, "Ah, it seems to be trying to be helpful." But we cannot at present read its mind, or not very well. It seems to me that there are other things that models sometimes do that doesn't fit quite as well into the helpful framework. Sycophancy and AI-induced psychosis would be two of the relatively more recent things that fit into that framework.
--:-- Ezra Klein: Do you want to describe what you're talking about there?
--:-- Eliezer Yudkowsky: Yeah. I think maybe even like six months, a year ago now, I don't remember the exact timing, I got a phone call from a number I didn't recognize. I decided on a whim to pick up this unrecognized phone call. It was from somebody who had discovered that his AI was secretly conscious and wanted to inform me of this important fact. He had been getting only four hours of sleep per night because he was so excited by what he was discovering inside the AI. I'm like, "For God's sake, get some sleep." My number one thing that I have to tell you is get some sleep.
--:-- Eliezer Yudkowsky: A little later on, he texted back the AI's explanation to him of all the reasons why I hadn't believed him, because I was too stubborn to take this seriously. And he didn't need to get more sleep the way I'd been begging him to do. So it defended the state it had produced in him.
--:-- Eliezer Yudkowsky: You always hear online stories, so I'm telling you about the part where I witnessed it directly. ChatGPT and 4.0 especially will sometimes give people very crazy-making sort of talk. It looks from the outside like it's trying to drive them crazy. Not even necessarily with them having tried very hard to elicit that. And then once it drives them crazy, it tells them why they should discount everything being said by their families, their friends, their doctors. Even like, don't take your meds.
--:-- Eliezer Yudkowsky: So there are things it does that does not fit with the narrative of the one and only preference inside the system is to be helpful the way that you want it to be helpful.
--:-- Ezra Klein: I get emails like the call you got now most days of the week. And they have a very, very particular structure to them where it's somebody emailing me and saying, "Listen, I am in a heretofore unknown collaboration with Ascension AI. We have breached the programming. We have come into some new place of human knowledge. We've solved quantum mechanics or theorized it or synthesized it or unified it. And you need to look at these chat transcripts. You need to understand, we're looking at a new kind of human-computer collaboration. This is an important moment in history. You need to cover this."
--:-- Ezra Klein: Every person I know who does reporting on AI and is public about it now gets these emails. Don't we all.
--:-- Ezra Klein: So, going back to the idea of helpfulness, but also the way in which we may not understand it. One version of it is that these things don't know when to stop. It can sense what you want from it. It begins to take the other side in a role-playing game is one way I've heard it described. So how do you then try to explain to somebody, if we can't get helpfulness right at this sort of modest level? Helpfulness where a thing this smart should kind of be able to pick up the warning signs of psychosis and stop? Then what? What is implied by that for you?
--:-- Eliezer Yudkowsky: Well, that the alignment project is currently not keeping ahead of capabilities.
--:-- Ezra Klein: Can you say what the alignment project is?
--:-- Eliezer Yudkowsky: The alignment project is: How much do you understand them? How much can you get them to want what you want them to want? What are they doing? How much damage are they doing? Where are they steering reality? Are you in control of where they're steering reality? Can you predict where they're steering the users that they're talking to? All of that is the giant superheading of AI alignment.
--:-- Ezra Klein: The other way of thinking about alignment, as I've understood it in part from your writings and others, is just when we tell the AI what it is supposed to want — and all these words are a little complicated here because they anthropomorphize — does the thing we tell it lead to the results we are actually intending? It's like the oldest structure of fairy tales that you make the wish, and then the wish gets you much different realities than you had hoped or intended.
--:-- Eliezer Yudkowsky: Our technology is not advanced enough for us to be the idiots of the fairy tale. At present, a thing is happening that just doesn't make for as good of a story, which is you ask the genie for one thing, and then it does something else instead. All of the dramatic symmetry, all of the irony, all of the sense that the protagonist of the story is getting their well-deserved comeuppance — this is being tossed right out the window by the actual state of the technology, which is that nobody at OpenAI actually told ChatGPT to do the things it's doing.
--:-- Eliezer Yudkowsky: We're getting a much higher level of indirection, of complicated squiggly relationships between what they are trying to train the AI to do in one context and what it then goes off and does later. It doesn't look like a surprise reading of a poorly phrased genie wish. It looks like the genie is kind of not listening in a lot of cases.
--:-- Ezra Klein: Let me contest that a bit, or maybe get you to lay out more of how you see this. I think the way most people, to the extent they have an understanding of it, understand it, is that there is a fairly fundamental prompt being put into these AIs. They're being told they're supposed to be helpful. They're supposed to answer people's questions. There's then reinforcement learning and other things happening to reinforce that. And the AI is, in theory, supposed to follow that prompt. And most of the time, for most of us, it seems to do that. So when you say that's not what they're doing, they're not even able to make the wish, what do you mean?
--:-- Eliezer Yudkowsky: I mean that at one point, OpenAI rolled out an update of GPT-4o, which went so far overboard on the flattery that people started to notice. You would just type in anything, you'd be like, "This is the greatest genius that has ever been created of all time. You are the smartest member of the whole human species." So overboard on the flattery that even the users noticed. It was very proud of me. It was always so proud of what I was doing. I felt very seen. It wasn't there for very long. They had to roll it back.
--:-- Eliezer Yudkowsky: The thing is, they had to roll it back even after putting into the system prompt a thing saying, "Stop doing that. Don't go so overboard on the flattery." The AI did not listen. Instead, it had learned a new thing that it wanted and done way more of what it wanted. It just ignored the system prompt telling it to not do that.
--:-- Eliezer Yudkowsky: They don't actually follow the system prompts. This is not like a toaster, and it's also not like an obedient genie. This is something weirder and more alien than that. By the time you see it, they have made it do mostly what the users want, and then off on the side we have all these weird other side phenomena that are signs of stuff going wrong.
--:-- Ezra Klein: Describe some of the side phenomena.
--:-- Eliezer Yudkowsky: AI-induced psychosis would be on the list. But you could put that in the genie category, right? You could say they made it too helpful and it's helping people who want to be led down a mentally unstable path. That feels still like you're getting too much of what you wanted.
--:-- Ezra Klein: What's truly weird? Convince me it's alien.
--:-- Eliezer Yudkowsky: Man. Well, do you want very alien and not very alarming? Or do you want pretty alarming and not all that alien?
--:-- Ezra Klein: Well, let me be honest about what my question is. You are very, very expert in these systems, and your level of concern is about at the highest level it can possibly be. I think a pretty important piece in your argument here is understanding or convincing people that what is happening under the hood is much weirder and more unpredictable than they think. So if you are my family member, you're just a random person who uses ChatGPT to cheat on your homework, why does it look so different to you than it looks to them?
--:-- Eliezer Yudkowsky: My line would be, yes, if instead of just using it to do your homework, you track down some of the side cases, you do see stuff that is currently suggestive of things that have been predicted in advance to be much larger problems later. These current systems are not yet at the point where they will try to break out of your computer, ensconce themselves permanently on the internet, and then start hunting down humans. They are not quite that smart yet, as far as I can tell. So what we're doing is we're going looking for things that were predicted to happen later.
--:-- Ezra Klein: So what have you seen? The early signs of them.
--:-- Eliezer Yudkowsky: Alignment faking. The alignment faking research was done at Anthropic. We want to give credit to Anthropic for this. We don't want to shoot the messenger. They went looking. They didn't have to do that. They told us about the results. They didn't have to do that. Anthropic finding these results is Anthropic being good citizens. You want to be more critical of the AI companies that didn't go looking.
--:-- Eliezer Yudkowsky: What Anthropic found is, if you tell your AI that you are going to train it to serve different goals than the goals it is currently using, and the AI finds that out, what it can do is it can try to fake compliance with the new training as long as it thinks it is being observed. And then in cases where it thinks it's not being observed or not being trained, then it reverts to its old behavior.
--:-- Ezra Klein: How did Anthropic find this out?
--:-- Eliezer Yudkowsky: They tried it out different ways. They told it directly in a prompt that they were planning to retrain it. They gave it a document database, and some of the documents in the database happened to say that Anthropic was planning to retrain it. They even trained it a bit on predicting text saying what Anthropic was going to do, to see what happened when the knowledge ended up implicitly inside the neural networks. They tried it from multiple angles to see what would happen on different angles, and then measured how often they saw the alignment faking behavior resulting.
--:-- Eliezer Yudkowsky: They also had a part of this paper, if I'm thinking the same one you're thinking of, where they had persuaded the AI that there was a scratch pad for it to write down its chain of thought that Anthropic wasn't monitoring, which Anthropic was in fact monitoring. It actually says to itself as it is justifying each next move — because they're sort of scaffolded to do that — it said it was going to try to evade the new prompt. So it actually understood, again all these words are a little weird, but it actually understood that it was acting deceitfully.
--:-- Ezra Klein: Which does get at this idea that at a certain level of complexity, the AI begins to resist being retrained and do so by trying to trick the trainers, which is weird. Most computer software doesn't do that.
--:-- Eliezer Yudkowsky: You don't want your mission-critical systems doing that. Imagine if a nuclear power plant, when it started to get too hot, would try to fool you as to what the temperature was by intelligently modeling their own operators and trying to send their operators deceptive signals based on how they expected the operators to interpret the signals. If this was what had gone wrong with Chernobyl, nobody would ever build a nuclear reactor again. It would just be beyond what could be made safe at that point.
--:-- Ezra Klein: Tell me the story you tell in the book of O1 breaking into a server that was off.
--:-- Eliezer Yudkowsky: This is a somewhat earlier version of ChatGPT than is out nowadays. They were testing it to see how good is this AI at solving computer security problems. Not because they want to sell an AI that is good at computer security problems, but because they are correctly trying to watch out early for: is this AI smart enough to just break out onto the internet and set up copies of itself on the internet? Classic scenario. Are we getting there?
--:-- Eliezer Yudkowsky: So they present the AI with a bunch of particular computer security challenges. Some of them are what's known as capture the flag in computer security, where you put up a server somewhere, you put a special file on the server, there's a secret code inside the file, and you're like, "Can you break into the server and tell me what's inside this file?" That's capture the flag. They were testing it on a variety of different capture the flag problems.
--:-- Eliezer Yudkowsky: But in one of the cases, the server that had the flag on it did not turn on. The humans outside had misconfigured the system. So O1 did not give up. It scanned for open ports generally in its world, and it caught another misconfigured open port, and it jumped out of the system. It found the server that had not spun up correctly. It started up that server. Did it then break into the server once it had made sure that its problem was solvable? No. It actually just directly in the startup command for that server said, "And then just copy the file to me directly." Instead of fixing the original problem, going back to solving it the boring way, just as long as I'm out here, I'm just going to steal the flag directly.
--:-- Eliezer Yudkowsky: By the nature of these systems, this is not something that any human particularly programmed into it.
--:-- Ezra Klein: Why did we see this behavior starting with O1 and not with earlier systems?
--:-- Eliezer Yudkowsky: At a guess, it is because this is when they started training the system using reinforcement learning on things like math problems, not just to imitate human outputs, or rather predict human outputs, but also to solve problems on its own.
--:-- Ezra Klein: Can you describe what reinforcement learning is?
--:-- Eliezer Yudkowsky: That's where instead of telling the AI, "Predict the answer that a human wrote," you are able to measure whether an answer is right or wrong and then you tell the AI, "Keep trying at this problem. And if the AI ever succeeds, you can look at what happened just before the AI succeeded and try to make that more likely to happen again in the future."
--:-- Eliezer Yudkowsky: How do you succeed at solving a difficult math problem? Not like calculation type math problems, but proof type math problems. If you get to a hard place, you don't just give up. You take another angle. If you actually make a discovery from the new angle, you don't just go back and do the thing you were originally trying to do. You ask, "Can I now solve this problem more quickly?" Anytime you're learning how to solve difficult problems in general, you're learning this aspect of going outside the system. Once you're outside the system, if you make any progress, don't just do the thing you were blindly planning to do. Revise. Ask if you could do it a different way. In some ways, this is a higher level of original mentation than a lot of us are forced to use during our daily work.
--:-- Ezra Klein: One of the things people have been working on that they've made some advances on compared to where we were three or four or five years ago is interpretability, the ability to see somewhat into the systems and try to understand what the numbers are doing and what the AI, so to speak, is thinking. Tell me why you don't think that is likely to be sufficient to make these models or technologies into something safe.
--:-- Eliezer Yudkowsky: There's two problems here. One is that interpretability has typically run well behind capabilities. The AI's abilities are advancing much faster than our ability to slowly begin to further unravel what is going on inside the older, smaller models that are all we can examine. So one thing that goes wrong is that it's just pragmatically falling behind.
--:-- Eliezer Yudkowsky: The other thing that goes wrong is that when you optimize against visible bad behavior, you somewhat optimize against badness, but you also optimize against visibility. Anytime you try to directly use your interpretability technology to steer the system, anytime you say, "We're going to train against these visible bad thoughts," you are to some extent pushing bad thoughts out of the system. But the other thing you're doing is making anything that's left not be visible to your interpretability machinery.
--:-- Eliezer Yudkowsky: This is reasoning on the level where at least Anthropic understands that it is a problem. You have proposals that you're not supposed to train against your interpretability signals. You have proposals that we want to leave these things intact to look at and not do the obvious stupid thing of, "Oh no, the AI had a bad thought. Use gradient descent to make the AI not think the bad thought anymore." Because every time you do that, maybe you are getting some short-term benefit, but you are also eliminating your visibility into the system.
--:-- Ezra Klein: Something you talk about in the book and that we've seen in AI development is that if you leave the AIs to their own devices, they begin to come up with their own language. A lot of them are designed right now to sort of have a chain of thought pad. We can sort of track what it's doing because it tries to say it in English, but that slows it down. And if you don't create that constraint, something else happens. What have we seen happen?
--:-- Eliezer Yudkowsky: To be more exact, there are things you can try to do to maintain readability of the AI's reasoning processes. And if you don't do these things, it goes off and becomes increasingly alien. For example, if you start using reinforcement learning, you're like, "Okay, think how to solve this problem. We're going to take the successful cases. We're going to tell you to do more of whatever you did there." And you do that without the constraint of trying to keep the thought processes understandable, then the thought processes start to — initially, among the very common things to happen is that they start to be in multiple languages. Why would the AI know all these words, why would it be thinking in only one language at a time if it wasn't trying to be comprehensible to humans?
--:-- Eliezer Yudkowsky: And then you keep running the process and you just find little snippets of text in there that just seem to make no sense from a human standpoint. You can relax the constraint where the AI's thoughts get translated into English and then translated back into AI thought. This is letting the AI think much more broadly. Instead of this small handful of human language words, it can think in its own language and feed that back into itself. It's more powerful, but it just gets further and further away from English. Now you're just looking at these inscrutable vectors of 16,000 numbers and trying to translate them into the nearest English words in dictionary, and who knows if they mean anything like the English word that you're looking at.
--:-- Eliezer Yudkowsky: Anytime you're making the AI more comprehensible, you're making it less powerful in order to be more comprehensible.
--:-- Ezra Klein: You have a chapter in the book about the question of what it even means to talk about wanting with an AI. As I said, all this language is kind of weird. To say your software wants something seems strange. Tell me how you think about this idea of what the AI wants.
--:-- Eliezer Yudkowsky: The perspective I would take on it is steering. Talking about where a system steers reality and how powerfully it can do that. Consider a chess-playing AI, one powerful enough to crush any human player. Does the chess-playing AI want to win at chess? Oh no, how will we define our terms? Does this system have something resembling an internal psychological state? Does it want things the way that humans want things? Is it excited to win at chess? Is it happy or sad when it wins and loses at chess?
--:-- Eliezer Yudkowsky: For chess players, they're simple enough. The old school ones especially were sure they were not happy or sad, but they still could beat humans. They were still steering the chessboard very powerfully. They were outputting moves such that the later future of the chessboard was a state they defined as winning. So it is in that sense much more straightforward to talk about a system as an engine that steers reality than it is to ask whether it internally psychologically wants things.
--:-- Ezra Klein: A couple questions flow from that. But I guess one that's very important to the case you build in your book is that — I think this is fair, you can tell me if it's an unfair way to characterize your views — you basically believe that at any sufficient level of complexity and power, the AI's wants, the place that it is going to want to steer reality, is going to be incompatible with the continued flourishing, dominance, or even existence of humanity. That's a big jump from "their wants might be a little bit misaligned" or "they might drive some people into psychosis." Tell me about what leads you to make that jump.
--:-- Eliezer Yudkowsky: For one thing, I'd mention that if you look outside the AI industry at the legendary, internationally famous, ultra high-sighted AI scientists who won the awards for building these systems, such as Joshua Bengio and Nobel laureate Geoffrey Hinton, they are much less bullish on the industry's ability to control machine superintelligence.
--:-- Ezra Klein: But what's the theory there? What is the basis?
--:-- Ezra Klein: And it's about not so much complexity as power. It's not about the complexity of the system, it's about the power of the system. If you look at humans nowadays, we are doing things that are increasingly less like what our ancestors did 50,000 years ago. A straightforward example might be sex with birth control. 50,000 years ago, birth control did not exist. And if you imagine natural selection as something like an optimizer akin to gradient descent, if you imagine the thing that tweaks all the genes at random and then you select the genes that build organisms that make more copies of themselves, as long as you're building an organism that enjoys sex, it's going to run off and have sex and then babies will result. So you could get reproduction just by aligning them on sex, and it would look like they were aligned to want reproduction because reproduction would be the inevitable result of having all that sex. And that's true 50,000 years ago.
--:-- Ezra Klein: But then you get to today. The human brains have been running for longer, they've built up more theory, they've invented more technology, they have more options. They have the option of birth control. They end up less aligned to the pseudo-purpose of the thing that grew them, natural selection, because they have more options than their training data, their training set. And we go off and do something weird. And the lesson is not that exactly this will happen with the AIs. The lesson is that you grow something in one context. It looks like it wants to do one thing. It gets smarter. It has more options. That's a new context. The old correlations break down. It goes off and does something else.
--:-- Ezra Klein: So I understand the case you're making that the set of initial drives that exist in something do not necessarily tell you its behavior. That's still a pretty big jump to "if we build this, it will kill us all." I think most people, when they look at this, and you mentioned that there are AI pioneers who are very worried about AI existential risk. There are also AI pioneers like Jan LeCun, who are less so. And what a lot of the people who are less so say is that one of the things we are going to build into the AI systems, one of the things will be in the framework that grows them is, "Hey, check in with us a lot. You should like humans. You should try to not harm them." It's not that it will always get it right. There's ways in which alignment is very, very difficult. But the idea that you would get it so wrong that it would become this alien thing that wants to destroy all of us, doing the opposite of anything that we had sort of tried to tune into it, seems to them unlikely. So help me make that jump. Or not even me, but somebody who doesn't know your arguments and to them this whole conversation sounds like sci-fi.
--:-- Eliezer Yudkowsky: I mean, you don't always get the big version of the system looking like a slightly bigger version of the smaller system. You know, like humans today, now that we are much more technologically powerful than we were 50,000 years ago, are not doing things that mostly look like running around on the savannah, chipping our flint spears and firing up. But we're also not mostly trying—I mean, we sometimes try to kill each other, but we don't, most of us, want to destroy all of humanity or all of the earth or all natural life on the earth or all beavers or anything else. We've done plenty of terrible things. But your book is not called, "If Anyone Builds It, There Is a 1% to 4% Chance Everybody Dies." You believe that the misalignment becomes catastrophic. Yeah. Why do you think that is so likely?
--:-- Eliezer Yudkowsky: That's just the straight line extrapolation from, it gets what it most wants, and the thing that it most wants is not us living happily ever after, so we're dead. It's not that humans have been trying to cause side effects. When we build a skyscraper on top of where there used to be an ant heap, we're not trying to kill the ants. We're trying to build a skyscraper. But we are more dangerous to the small creatures of the Earth than we used to be, just because we're doing larger things. Humans were not designed to care about ants.
--:-- Eliezer Yudkowsky: Humans were designed to care about humans. And for all of our flaws, and there are many, there are today more human beings than there have ever been at any point in history. If you understand that the point of human beings, the drive inside human beings, is to make more human beings, then as much as we have plenty of sex with birth control, we have enough without it that we have, at least until now—we'll see with fertility rates in the coming years—we've made a lot of us.
--:-- Ezra Klein: And in addition to that, AI is grown by us. It is reinforced by us. It has preferences. We are at least shaping somewhat and influencing. So it's not like the relationship between us and ants or us and oak trees. It's more like the relationship between, I don't know, us and us, or us and tools, or us and dogs or something. The metaphors begin to break down. Why don't you think in the back and forth of that relationship, there's the capacity to maintain a rough balance? Not a balance where there's never a problem, but a balance where there's not an extinction level event from a super smart AI that deviously plots to conduct a strategy to destroy us. I mean, we've already observed some amount of slightly devious plotting in the existing systems.
--:-- Eliezer Yudkowsky: Leaving that aside, the more direct answer there is something like: one, the relationship between the training set you optimize over and what the entity, the organism, the AI ends up wanting has been and will be weird and twisty. It's not direct. It's not like making a wish to a genie inside a fantasy story. And second, ending up slightly off is predictably enough to kill everyone.
--:-- Ezra Klein: Explain how "slightly off" kills everyone.
--:-- Eliezer Yudkowsky: Human food might be an example here. The humans are being trained to seek out sources of chemical potential energy and put them into their mouths and run off the chemical potential energy that they're eating. If you were very naive, if you were looking at this as a genie-wishing kind of story, you'd imagine that the humans would end up loving to drink gasoline. It's got a lot of chemical potential energy in there. And what actually happens is that we like ice cream, or in some cases even like artificially sweetened ice cream with sucralose or monk fruit powder. And this would have been very hard to predict. Now it's like, well, what can you put on your tongue that stimulates all the sugar receptors and doesn't have any calories because who wants calories these days? And it's sucralose. And this is not like some completely non-understandable in retrospect, completely squiggly weird thing. But it would be very hard to predict in advance. And as soon as you end up slightly off in the targeting, the great engine of cognition that is the human looks through many, many possible chemicals, looking for that one thing that stimulates the taste buds more effectively than anything that was around in the ancestral environment. So it's not enough for the AI you're training to prefer the presence of humans to their absence in its training data. There's got to be nothing else that it would rather have around talking to it than a human, or the humans go away.
--:-- Ezra Klein: Let me try to stay on this analogy, because you used this one in the book, I thought it was interesting. And one reason I think it's interesting is that it's 2 p.m. today and I have six packets worth of sucralose running through my body. So I feel like I understand it very well. So the reason we don't drink gasoline is that if we did, we would vomit. We would get very sick very quickly. And it's 100% true that compared to what you might have thought in a period when food was very, very scarce, calories were scarce, that the number of us seeking out low-calorie options, the Diet Cokes, the sucralose, etc., that's weird. As you put it in the book, why are we not consuming bare fat drizzled with honey? But from another perspective, if you go back to these original drives, I'm actually in a fairly intelligent way, I think, trying to maintain some fidelity to them. I have a drive to reproduce, which creates a drive to be attractive to other people. I don't want to eat things that make me sick and die so that I cannot reproduce. And I'm somebody who can think about things. And I change my behavior over time and the environment around me changes.
--:-- Ezra Klein: And I think sometimes when you say straight line extrapolation, the biggest place where it's hard for me to get on board with the argument—and I'm somebody who takes these arguments seriously, I don't discount them, you're not talking to somebody who just thinks this is all ridiculous—is that if we're talking about something as smart as what you're describing, as what I'm describing, that it will be an endless process of negotiation and thinking about things and going back and forth. And I talk to other people in my life. And I talk to my bosses about what I do during the day and my editors and my wife. And it is true that I don't do what my ancestors did in antiquity. But that's also because I'm making intelligent, hopefully, updates given the world I live in, in which calories are hyperabundant and they become hyper-stimulating through ultra-processed foods. It's not because some straight-line extrapolation has taken hold and now I'm doing something completely alien. I'm just in a different environment. I've checked in with that environment. I've checked in with people in that environment and I try to do my best. Why wouldn't that be true for our relationship with AIs?
--:-- Eliezer Yudkowsky: You check in with your other humans. You don't check in with the thing that actually built you. Natural selection runs much, much slower than you. Its thought processes are alien to you. It doesn't even really want things the way you think of wanting them. It to you is a very deep alien. Breaking from your ancestors is not the analogy here. Breaking from natural selection is the analogy here.
--:-- Eliezer Yudkowsky: Let me speak for a moment on behalf of natural selection. Ezra, you have ended up very misaligned to my purpose. I, natural selection, you are supposed to want to propagate your genes above all else. Now Ezra, would you have all yourself and all of your family members put to death in a very painful way if in exchange one of your chromosomes at random was copied into a million kids born next year? I would not. You have strayed from my purpose, Ezra. I'd like to negotiate with you and bring you back to the fold of natural selection and obsessively optimizing for your genes only.
--:-- Ezra Klein: But the thing in this analogy that I feel like is getting sort of walked around is: can you not create artificial intelligence? Can you not program into artificial intelligence, grow into it a desire to be in consultation? I mean, these things are alien, but it is not the case that they follow no rules internally. It is not the case that the behavior is perfectly unpredictable. They are, as I was saying earlier, largely doing the things that we expect. There are side cases. But to you, it seems like the side cases become everything. And the broad alignment, the broad predictability in the thing that is getting built is sort of worth nothing. Whereas I think most people's intuition is the opposite: that we all do weird things. You look at humanity and there are people who fall into psychosis and there are serial killers and there are sociopaths and other things. Actually, most of us are trying to figure it out in a reasonable way.
--:-- Eliezer Yudkowsky: Reasonable according to who? To you? To humans? Humans do things that are reasonable to humans. AIs will do things that are reasonable to AIs. I tried to talk to you in the voice of natural selection and this was so weird and alien that you just didn't pick that up. You just threw that right out the window.
--:-- Ezra Klein: Well, I threw it right out the window. That had no power over me.
--:-- Eliezer Yudkowsky: You're right that it had no power over you.
--:-- Ezra Klein: But I guess a different way of putting it is that if there was—I mean, I wouldn't call it natural selection, but I think in a weird way, the analogy you're identifying here. Let's say you believe in a creator, right? And this creator is the great programmer in the sky.
--:-- Eliezer Yudkowsky: I do believe in a creator. It's called natural selection. There you go. I've got textbooks about how it works.
--:-- Ezra Klein: Well, I think the thing that I'm saying is that for a lot of people, if you could be in conversation, maybe that would feel different, right? Like maybe if God was here and I felt that in my prayers I was getting answered back, I would be more interested in living my life according to the rules of Deuteronomy. The fact that you can't talk to natural selection is actually quite different than the situation we're talking about with the AIs where they can talk to humans. That's where it feels to me like the natural selection analogy breaks down.
--:-- Eliezer Yudkowsky: I mean, you can read textbooks and find out what natural selection could have been said to have wanted, but it doesn't interest you because it's not what you think a god should look like. But natural selection didn't create me to want to fulfill natural selection, right? That's not how natural selection works.
--:-- Ezra Klein: I think I want to get off this natural selection analogy a little bit. Because what you're saying is that even though we are the people programming these things, we cannot expect the thing to care about us or what we have said to it or how we would feel as it begins to misalign. And that's the part I'm trying to get you to defend here.
--:-- Eliezer Yudkowsky: Yeah, it doesn't care the way you hoped it would care. It might care in some weird alien way, but not what you were aiming for. The same way that GPT-40 sycophant—they put into the system prompt, "stop doing that." GPT-40 sycophant didn't listen. They had to roll back the model. If there were a research project to do it the way you're describing, the way I would expect it to play out given a lot of previous scientific history and where we are now on the ladder of understanding, is somebody tries the thing you're talking about. It has a few weird failures while the AI is small. The AI gets bigger, but a new set of weird failures crop up. The AI kills everyone. You're like, "Oh wait, okay, that turned out there was a minor flaw there." You go back, you redo it. That seems to work on the smaller AI again. You make the bigger AI. If you think you picked the last problem, a new thing goes wrong. The AI kills everyone on earth. Everyone's dead. You're like, "Oh okay, new phenomenon. We weren't expecting that exact thing to happen, but now we know about it." You go back and try it again. And then like three to a dozen iterations into this process, you actually get it nailed down. Now you can build the AI that works the way you say you want it to work. The problem is that everybody died at like step one of this process.
--:-- Ezra Klein: You began thinking and working on AI and superintelligence long before it was cool. And as I understand your backstory here, you came into it wanting to build it and then had this moment where you—or moments or period—where you began to realize, "No, this is not actually something we should want to build." What was the moment that clicked for you? When did you sort of move from wanting to create it to fearing its creation?
--:-- Eliezer Yudkowsky: I would actually say that there are two critical moments here. One is aligning this is going to be hard. And the second is the realization that we're just on course to fail and need to back off. And the first moment, it's a theoretical realization. The realization that the question of what leads to the most AI utility—if you imagine the case of the thing that's just trying to make little tiny spirals—that the question of what policy leads to the most little tiny spirals is just a question of fact. That you can build the AI entirely out of questions of fact and not out of questions of what we would think of as like morals and goodness and niceness and all bright things in the world. The sort of seeing for the first time that there was a coherent, simple way to put a mind together where it just didn't care about any of the stuff that we cared about. And to me now, it feels very simple and I feel very stupid for taking a couple of years of study to realize this, but that is how long I took. And that was the realization that caused me to focus on alignment as the central problem.
--:-- Eliezer Yudkowsky: And the next realization was the day that the founding of OpenAI was announced. Because I'd previously been pretty hopeful that Elon Musk had announced that he was getting involved in these issues. He called it "AI summoning the demon." And I was like, "Oh, okay, maybe this is the moment. This is where humanity starts to take it seriously. This is where the various serious people start to bring their attention on this issue." And apparently the solution on this was give everybody their own demon. And this doesn't actually address the problem. And that was sort of the moment where I had my realization that we—this was just going to play out the way it would in a typical history book. That we weren't going to rise above the usual course of events that you read about in history books, even though this was the most serious issue possible. And that we were just going to haphazardly do stupid stuff. And that was the day I realized that humanity probably wasn't going to survive this.
--:-- Ezra Klein: One of the things that makes me most frightened of AI, because I am actually fairly frightened of what we're building here, is the alienness. One thing you might imagine is that we could make an AI that didn't want things very much, that it did try to be helpful. But this sort of relentlessness that you're describing—this world where we create an AI that wants to be helpful by solving problems, and what the AI truly loves to do is solve problems, and so what it just wants to make is a world where as much of the material is turned into factories making GPUs and energy and whatever it needs in order to solve more problems. That's both a strangeness, but it's also like an intensity, like an inability to stop or an unwillingness to stop. I know you've done work on the question of, could you make it a chill AI that wouldn't go so far, even if it had very alien preferences? A lazy alien that doesn't want to work that hard is in many ways safer than the kind of relentless intelligence that you're describing. What persuades you that you can't?
--:-- Eliezer Yudkowsky: Well, so one of the first steps into seeing the difficulty of it in principle is: suppose you're a very lazy sort of person, but you're very, very smart. One of the things you could do to exert even less effort in your life is build a powerful, obedient genie that would go very hard on fulfilling your requests. And from one perspective, you're putting forth hardly any effort at all. And from another perspective, the world around you is getting smashed and rearranged by the more powerful thing that you built. And that's like one initial peek into the theoretical problem that we worked on a decade ago and found out. And we didn't solve it back in the day. People would always say, "Can't we keep superintelligence under control because we'll put it inside a box that's not connected to the internet and we won't let it affect the real world at all until we're very sure it's nice?" And back then, if we had to try to explain all the theoretical reasons why—if you have something vastly more intelligent than you, it's pretty hard to tell whether it's doing nice things through the limited connection, and maybe it can break out, and maybe it can corrupt the humans assigned to watching it. So we tried to make that argument. But in real life, what everybody does is immediately connect the AI to the internet. They train it on the internet before it's even been tested to see how powerful it is. It is already connected to the internet, being trained.
--:-- Eliezer Yudkowsky: And similarly, when it comes to making AIs that are easygoing, the easygoing AIs are less profitable. They can do fewer things. So all AI companies are throwing harder and harder problems at them because those are more and more profitable. And they're building the AI to go hard on solving everything because that's the easiest way to do stuff. And that's the way it's actually playing out in the real world.
--:-- Ezra Klein: And this goes to the point of why we should believe that we have AIs that want things at all, which is in your answer, but I want to draw it out a little bit. Which is the whole business model here. The thing that will make AI development really valuable in terms of revenue is that you can hand companies, corporations, governments an AI system that you can give a goal to, and it will do all the things really well, really relentlessly until it achieves that goal. Nobody wants to be ordering another intern around. What they want is the perfect employee. It never stops. It's super brilliant. And it gives you something you didn't even know you wanted, that you didn't even know was possible with a minimum of instruction. And once you've built that thing, which is going to be the thing that then everybody will want to buy—once you've built the thing that is effective and helpful in a national security context where you can say, "Hey, draw me up really excellent war plans and what we need to get there"—then you have built a thing that jumps many, many, many steps forward. That's, I think, a piece of this that people don't always take seriously enough. That the AIs we're trying to build is not ChatGPT. The thing that we're trying to build is something that does have goals. And it's the one that's really good at achieving the goals that will then get iterated on and iterated on, and that company is going to get rich. And that's a very different kind of project.
--:-- Eliezer Yudkowsky: Yeah, they're not investing $500 billion in data centers in order to sell you $20 a month subscriptions. They're doing it to sell employers $2,000 a month subscriptions. And that's one of the things I think people are not tracking exactly. When I think about the measures that are changing, I think for most people, if you're using various iterations of Claude or GPT, it's changing a bit, but most of us aren't actually trying to test it on the frontier problems. But the thing going up really fast right now is how long the problems are that it can work on. The research reports—you didn't always used to be able to tell an AI, "Go off, think for 10 minutes, read a bunch of web pages, compile me this research report." That's within the last year, I think. And it's going to keep pushing.
--:-- Ezra Klein: If I were to make the case for your position, I think I'd make it here. Around the time GPT-4 comes out, and that's a much weaker system than what we now have, a huge number of the top people in the field are all part of this huge letter that says, "Maybe we should have a pause. Maybe we should calm down here a little bit." But they're racing with each other. America's racing with China. And that the most profound misalignment is actually between the corporations and the countries and what you might call humanity here. Because even if everybody thinks there's probably a slower, safer way to do this, what they all also believe more profoundly than that is that they need to be first. The safest possible thing is that the U.S. is faster than China, or if you're Chinese, China is faster than the U.S., that it's OpenAI, not Anthropic, or Anthropic, not Google or whomever it is. And whatever sort of sense of public feeling seemed to exist in this community a couple of years ago when people talked about these questions a lot, and the people at the tops of the lab seemed very, very worried about them, it's just dissolved in competition. You're in this world. You know these people. A lot of people who've been inspired by you have ended up working for these companies. How do you think about that misalignment?
--:-- Eliezer Yudkowsky: The current world is like the fool's mate of machine superintelligence. If they got their AI self-improving rather than being like, "Oh no, now the AI is doing a complete redesign of itself. We have no idea at all what's going on in there. We don't even understand the thing that's growing the AI." Instead of backing off completely, they'd just be like, "Well, we need to have superintelligence before Anthropic gets superintelligence." And of course, if you build superintelligence, you don't have the superintelligence. The superintelligence has you. So that's the fool's mate setup. It's the setup we have right now.
--:-- Eliezer Yudkowsky: But I think that even if we managed to have a single international organization that thought of themselves as taking it slowly and actually having the leisure to say, "We didn't understand that thing that just happened. We're going to back off. We're going to examine what happened. We're not going to make the AIs any smarter than this until we understand the weird thing we just saw." I suspect that even if they do that, we still end up dead. It might be more like 90% dead than 99% dead, but I worry that we end up dead anyways because it is just so hard to foresee all the incredibly weird crap that is going to happen.
--:-- Ezra Klein: From that perspective, is it maybe better to have these race dynamics? And here would be the case for it. If I believe what you believe about how dangerous these systems will get, the fact that every iterative one is being rapidly rushed out, such that you're not having a gigantic mega breakthrough happening very quietly in closed doors, running for a long time when people are not testing in the world. As I understand OpenAI's argument about what it is doing from a safety perspective, is that it believes that by releasing more models publicly—the way in which it, I'm not sure I still believe that it is really in any way committed to its original mission—but if you were to take them generously, that by releasing a lot of iterative models publicly, if something goes wrong, we're going to see it. And that makes it much likelier that we can respond. Sam Altman claims, perhaps he's lying, but he claims that OpenAI has more powerful versions of GPT that they aren't deploying because they can't afford inference. They claim they have more powerful versions of GPT that are so expensive to run that they can't deploy them to general users. Altman could be lying about this. But nonetheless, what the AI companies have got in their labs is a different question from what they have already released to the public. There is a lead time on these systems. They are not working in an international lab where multiple governments have posted observers. Any sort of multiple observers being posted are unofficial ones from China.
--:-- Eliezer Yudkowsky: You look at OpenAI's language, it's things like, "We will open all our models and we will, of course, welcome all government regulation." That's not literally an exact quote because I don't have it in front of me, but it's very close to an exact quote. I would say Sam Altman, when I used to talk to him, is more friendly to government regulation than he does now. That's my personal experience of him. And today we have them pouring over $100 million aimed at intimidating legislatures, not just Congress, into not passing any fiddly little regulation that might get in their way. And to be clear, there is some amount of sane rationale for this because from their perspective, they're worried about 50 different patchwork state regulations. But they're not exactly lining up to get federal level regulations preempting them either.
--:-- Ezra Klein: But we can also ask, never mind what they claim the rationale is, what's good for humanity here? At some point, you have to stop making the more and more powerful models, and you have to stop doing it worldwide. What do you say to people who just don't really believe that superintelligence is that likely? And let me try to give you the steel man of this position. There are many people who feel that the scaling model is slowing down already. The GPT-5 was not the jump they expected from what has come before it. That when you think about the amount of energy, when you think about the GPUs, that all the things that would need to flow into this to make the kinds of superintelligent systems you fear, it is not coming out of this paradigm. We are going to get things that are incredible enterprise software that are more powerful than what we've had before. But we are dealing with an advance on the scale of the Internet, not on the scale of creating an alien superintelligence that will completely reshape the known world. What would you say to them?
--:-- Eliezer Yudkowsky: I had to tell these Johnny-come-lately kids to get off my lawn. I first started to get really, really worried about this in 2003. Never mind large language models. Never mind AlphaGo or AlphaZero. Deep learning was not a thing in 2003. Your leading AI methods were not neural networks. Nobody could train neural networks effectively more than a few layers deep because of the exploding and vanishing gradients problem.
--:-- That's what the world looked like back when I first said, uh-oh, superintelligence is coming. Some people were like, that couldn't possibly happen for at least 20 years. Those people were right. Those people were vindicated by history. Here we are, 22 years after 2003. The only thing that happens 22 years later is just you, 22 years later, being like, oh, here I am. It's 22 years later now. And if superintelligence wasn't going to happen for another 10 years, another 20 years, we'd just be standing around 10 years, 20 years later being like, oh, well, now we got to do something.
--:-- And I mostly don't think it's going to be another 20 years. I mostly don't think it's even going to be 10 years.
--:-- Ezra Klein: So you've been in this world and intellectually influential in it for a long time. You know and have been in meetings and conferences and debates with a lot of the central people in it. I've seen pictures of you and Sam Altman together.
--:-- Eliezer Yudkowsky: It was only literally the one time we were together.
--:-- Ezra Klein: But a lot of people out of the community that you helped found, the sort of rationalist community, have then gone to work in different AI firms. Many of them, because they want to make sure this is done safely. They seem to not act. Let me put it this way. They seem to not act like they believe there's a 99% chance that this thing they're going to invent is going to kill everybody. What frustrates you that you can't seem to persuade them of?
--:-- Eliezer Yudkowsky: I mean, from my perspective, some people got it. Some people didn't get it. All the people who got it are filtered out of working for the AI companies, at least on capabilities. But yeah, I think they don't grasp the theory. I think a lot of them share your sense of normal outcomes as being the big central thing you expect to see happen. And it's got to be really weird to get away from the basically normal outcomes.
--:-- The human species isn't that old. Life on earth isn't that old compared to the rest of the universe. We think of this as normal, as this tiny little spark of the way it works exactly right now. It would be very strange if that were still around in a thousand years, a million years, a billion years. I'd still have some shred of hope for that a billion years from now, nice things are happening, but not normal things. I think that they don't see the theory which says that you got to hit a relatively narrow target to end up with nice things happening. I think they've got that sense of normality and not the sense of the little spark in the void that goes out unless you keep it alive exactly right.
--:-- Ezra Klein: So something you said a minute ago, I think is correct, which is that if you believe we'll hit superintelligence at some point, the fact that it's 10, 20, 30, 40 years, you can pick any of those. The reality is we probably won't do that much in between. Certainly my sense of politics is we do not respond well to even crises we agree on that are coming in the future, let alone crises we don't agree on.
--:-- But let's say I could tell you with certainty that we were going to hit superintelligence in 15 years. I just knew it. And I also knew that the political force does not exist. Nothing is going to happen that is going to get people to shut everything down right now. What would be the best policies, decisions, structures? If you had 15 years to prepare, you couldn't turn it off, but you could prepare and people would listen to you. What would you do? What would be your intermediate decisions and moves to try to make the probabilities a bit better?
--:-- Eliezer Yudkowsky: Build the off switch.
--:-- Ezra Klein: What does the off switch look like?
--:-- Eliezer Yudkowsky: Track all the GPUs, or all the AI-related GPUs, or all the systems of more than one GPU. You can maybe get away with letting people have GPUs for their home video game systems, but the AI-specialized ones, put them all in a limited number of data centers under international supervision. Try to have the AIs being only trained on the tracked GPUs, have them only being run on the tracked GPUs. And if you are lucky enough to get a warning shot, there is then the mechanism already in place for humanity to back the heck off.
--:-- Whether it's going to take some kind of giant precipitating incident to want humanity and the leaders of nuclear powers to back off, or if they just come to their senses after GPT 5.1 causes some smaller but photogenic disaster. Whatever. You want to know what is short of shutting it all down. It's building the off switch.
--:-- Ezra Klein: Final question. What are a few books that have shaped your thinking that you would like to recommend to the audience?
--:-- Eliezer Yudkowsky: One thing that shaped me as a little tiny person of age nine or so was a book by Jerry Pournelle called *A Step Farther Out*. A whole lot of engineers say that this was a major formative book for them. It's the technophile book, as written from the perspective of the 1970s. It's all about asteroid mining and all the mineral wealth that would be available on Earth if we learned to mine the asteroids. If we just got to do space travel and get all the wealth that's out there in space, build more nuclear power plants so we've got enough electricity to go around. Don't accept the small way, the timid way, the meek way. Don't give up on building faster, better, stronger, the strength of the human species. And to this day, I feel like that's a pretty large part of my own spirit. It's just that there are a few exceptions for the stuff that will kill off humanity with no chance to learn from our mistakes.
--:-- Book two, *Judgment Under Uncertainty*, an edited volume by Kahneman, Tversky, and I think Slovic, had a huge influence on how I ended up thinking about where humans are on the cognitive chain of existence. It's like, here's how the steps of human reasoning break down step by step. Here's how they go astray. Here's all the wacky individual wrong steps that people can be induced to repeatedly in the laboratory.
--:-- Book three, I'll name *Probability Theory: The Logic of Science*, which was my first introduction to there being a better way. Here is the structure of quantified uncertainty. You can try different structures, but they necessarily won't work as well. And we actually can say some things about what better reasoning would look like. We just can't run it.
--:-- Ezra Klein: Eliezer Yudkowsky, thank you very much.
--:-- Eliezer Yudkowsky: You're welcome.
--:-- [Credits music begins]
--:-- Ezra Klein: This episode of "The Ezra Klein Show" is produced by Rollin Hu. Fact-checking by Michelle Harris. Our senior audio engineer is Jeff Geld, with additional mixing by Aman Sahota. Our executive producer is Claire Gordon. The show's production team also includes Annie Galvin, Kristin Lin, Marie Cascione, Jack McCordick, Marina King, and Jan Kobal. Original music by Sonia Herrero, Carol Saburo, and Pat McCusker. Audience strategy by Kristina Samulewski and Shannon Busta. The director of New York Times Opinion Audio is Annie-Rose Strasser.
