Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history

Leopold Aschenbrenner: Today I'm chatting with my friend Leopold Aschenbrenner. He grew up in Germany, graduated valedictorian of Columbia when he was 19, and then he had a very interesting gap year, which we'll talk about. Then he was on the OpenAI super alignment team, may it rest in peace. Now he,

Dwarkesh Patel: with some anchor investments from Patrick and John Collison and Daniel Gross and Nat Friedman, is launching an investment firm. So Leopold, I know you're off to a slow start, but life is long and I wouldn't worry about it too much.

Leopold Aschenbrenner: You'll make up for it in due time. But thanks for coming on the podcast. Thank you. You know, I first discovered your podcast when your best episode had, you know, like a couple hundred views. And so it's just been it's been amazing to follow your trajectory. And it's a delight to be on.

Dwarkesh Patel: Yeah, yeah. Well, I think in the Shelter in Trenton episode, I mentioned that a lot of the things I've learned about AI, I've learned from talking with them. And the third part of this triumvirate, probably the most significant in terms of the things that I've learned about AI has been you. We'll get all this stuff on the record now. Great. Okay, first thing I had to get on record. Tell me about the trillion dollar cluster. But by the way, I should mention, so the context of this podcast is today there's you're releasing a series called

Leopold Aschenbrenner: Situational Awareness. We're going to get into it. First question about that is tell me about the trillion dollar cluster. Yeah, so unlike basically most things that have come out of Silicon Valley recently, AI is kind of this industrial process. The next model doesn't just require some code. It's building a giant new cluster. Now it's building giant new power plants. Pretty soon it's going to be building giant new fabs. And, you know, since ChatGPT, this kind of extraordinary sort of techno capital acceleration has been set into motion. I mean, basically, you know, exactly a year ago today, you know, NVIDIA had their first kind of blockbuster earnings call, right? Where it like went out 25% after hours and everyone was like, oh my God, AI, it's a thing. You know, I mean, I think within a year, you know, NVIDIA data center revenue has gone from like, you know, a few billion a quarter to like, you know, 20, 25 billion a quarter now. And, you know, continuing to go up, like, you know, big tech CapEx is skyrocketing. And, you know, It's funny because there's this crazy scramble going on, but in some sense, it's just the continuation of straight lines on a graph. There's this long run trend, basically almost a decade of training compute of the largest AI systems growing by about half an order of magnitude, 0.5 booms a year. And you can just kind of play that forward, right? So GPT-4, rumored or reported to have finished pre-training in 2022. The sort of cluster size there was rumored to be about 25,000 H100s, sorry, A100s on semi-analysis. That's roughly, if you do the math on that, it's maybe like a $500 million cluster. It's very roughly 10 megawatts. And just play that forward, half a new year. So then 2024, that's a cluster that's 100 megawatts. That's like 100,000 H100 equivalents. That's costs in the billions. Play it forward two more years. 2026, that's a cluster that's a gigawatt. That's sort of a large nuclear reactor size. It's like the power of the Hoover Dam. That costs tens of billions of dollars. That's like a million H100 equivalents. You know, 2028, that's a cluster that's 10 gigawatts, right? That's more power than kind of like most U.S. states. That's, you know, like 10 million H100s equivalents, you know, costs hundreds of billions of dollars. And then 2030, trillion dollar cluster, 100 gigawatts, over 20% of U.S. electricity production, you know, 100 million H100 equivalents. And that's just the training cluster, right? That's like the one largest training cluster. And then there's more inference GPUs as well, right? Most of, you know, once there's products, most of them are going to be inference GPUs. And so... You know, U.S. power production has barely grown for like, you know, decades. And now we're really in for a ride. So, I mean, when I had Zuck on the podcast, he was claiming not a plateau per se, but that AI progress would be bottlenecked by specifically this constraint on energy and specifically like, oh, gigawatt data centers are going to build another Three Gorges Dam or something. I know that there's companies, according to public reports, who are planning things on the scale of a gigawatt data center

Dwarkesh Patel: 10 gigawatt data center, who's going to be able to build that? I mean, the 100 gigawatt center, like a state, are you going to pump that into one physical data center? How is this going to be possible?

Leopold Aschenbrenner: Yeah.

Dwarkesh Patel: What is Zuck missing?

Leopold Aschenbrenner: I mean, you know, I don't know. I think the 10 gigawatt, you know, like six months ago, you know, 10 gigawatts was the talk of the town. I mean, I think, I feel like now, you know, people have moved on. You know, 10 gigawatts is happening. I mean, I don't know. There's the information report on OpenAI and Microsoft planning a $100 billion cluster. So, you know, you've got to, you know. Is that the gigawatt or is that the 10 gigawatt? I mean, I don't know. But if you try to map out how expensive would the 10 gigawatt cluster be, that's maybe a couple hundred billion. So it's sort of on that scale. And they're planning it. They're working on it. you know, it's not just sort of my crazy take. I mean, AMD, AMD, I think forecasted a $400 billion AI accelerator market by 27. You know, I think, I think it's, you know, and AI accelerators are only part of the expenditures. It's sort of, you know, I think sort of a trillion dollars of sort of like total AI investment by 2027 is sort of like, we're very much in track on. I think the trillion dollar cluster is going to take a bit more sort of acceleration. But, you know, we saw how much sort of chat GPT unleashed, right? And so like every generation, you know, the models are going to be kind of crazy and people, it's going to shift the Overton window. And then, you know, obviously the revenue comes in, right? So these are forward-looking investments. The question is, do they pay off, right? And so if we sort of estimated the, you know, the GPT-4 cluster at around 500 million, by the way, that's sort of a common mistake people make is they say, you know, people say like $100 million for GPT-4, but that's just the rental price, right? They're like, ah, you rent the cluster for three months. But it's,

Dwarkesh Patel: you know,

Leopold Aschenbrenner: if you're building the biggest cluster, you got to like, you got to build the whole cluster. You got to pay for the whole cluster. You can't just rent it for three months. But I mean, really, you know, once you're trying to get into the sort of hundreds of billion, Eventually, you've got to get to $100 billion a year revenue. I think this is where it gets really interesting for the big tech companies because their revenues are on order hundreds of billions. It's like $10 billion, fine, and it'll pay off the 2024 size training cluster. But really, when big tech, it'll be gangbusters, it's $100 billion a year. The question is how feasible is $100 billion a year from AI revenue? Yeah. You know, it's a lot more than right now, but I think, you know, if you sort of believe in the trajectory of the AI systems as I do, and which we'll probably talk about, it's not that crazy, right? So there's, I think there's like 300 million, you know, ish Microsoft Office subscribers, right? And so they have Copilot now, and I don't know what they're selling it for, but, you know, suppose you sold some sort of AI add-on for 100 bucks a month, and you sold that to, you know, a third of Microsoft Office subscribers subscribed to that. That'd be 100 billion right there. $100 a month is a lot. That's a lot, yeah. It's a lot, it's a lot. For a third of our office subscribers? Yeah, but for the average knowledge worker, it's like a few hours of productivity a month. You have to be expecting pretty lame AI progress to not hit some few hours of productivity a month. Okay, sure. So let's assume all this. What happens in the next few years in terms of what is the one gigawatt training,

Dwarkesh Patel: the AI that's trained on the one gigawatt data center? What can it do the one on the 10 gigawatt data center? Just map out the next few years of AI progress for me.

Leopold Aschenbrenner: Yeah, I think probably the sort of 10 gigawatt-ish range is sort of my best guess for when you get the sort of true AGI. I mean, yeah, I think it's sort of like one gigawatt data center. And again, I think actually compute is overrated and we're going to talk about that, but we will talk about compute right now. So, you know, I think sort of 25, 26, we're going to get models that are, you know, basically smarter than most college graduates. I think sort of the practice, a lot of the economic usefulness, I think really depends on sort of, you know, sort of on hobbling. Basically it's, you know, the models are kind of, you know, they're smart, but they're limited, right? You know, there's this chat bot, you know, and things like being able to use a computer, things like being able to do kind of like a Gen Tech long horizon tasks. And then I think by 27, 28, you know, if you extrapolate the trends and, you know, we'll talk about that more later and I talk about it in the series, I think we hit, you know, basically, you know, like as smart as the smartest experts. I think the unhobbling trajectory kind of points to, you know, you know, it looks much more like an agent than a chatbot and much more, almost like basically like a drop-in remote worker, right? So it's not like, I think basically, I mean, I think this is the sort of question on the economic returns. I think a lot of the, a lot of the intermediate AI systems could be really useful, but you know, it actually just takes a lot of schlep to integrate them, right? Like GPT-4, you know, whatever, 4.5, you know, probably there's a lot you could do with them in a business use case, but you know, you really got to change your workflows to make them useful. And it's just like, there's a lot of, you know, it's a very Tyler Cowen-esque take. It just takes a long time to diffuse, you know, it's like, You know, we're an SF and so we missed that or whatever. But I think in some sense, you know, the way a lot of these systems want to be integrated is you kind of get this sort of sonic boom where it's, you know, the sort of intermediate systems could have done it, but it would have taken Schlapp. And before you do the Schlapp to integrate them, you get much more powerful systems, much more powerful systems that are sort of unhobbled. And so they're this agent and there's drop in remote worker. And, you know, and then you're kind of interacting with them like a coworker, right? You know, You can do Zoom calls with them and you're slacking them and you're like, ah, can you do this project? And then they go off and they go away for a week and write a first draft and get feedback on them and run tests on their code. And then they come back and you see it and you tell them a little bit more things. And that'll be much easier to integrate. And so it might be that actually you need a bit of overkill to make the sort of transition easy and to really harvest the gains. What do you mean by the overkill? Overkill on the model capabilities? Yeah, yeah. So basically the intermediate models could do it, but it would take a lot of schlap. I see. And so then, you know, they're like, actually, it's just the drop-in remote worker kind of AGI that can automate, you know, cognitive tasks that actually just ends up kind of like, you know, basically it's your like, you know, the intermediate models would have made the software engineer more productive, but, you know, will the software engineer adopted? And then, you know, 27 model is, well, you know, you just don't need the software engineer. You can literally interact with it like a software engineer and it'll do the work of a software engineer.

Dwarkesh Patel: So the last episode I did was with John Schulman. Yeah. And I was asking about basically this. And one of the questions I asked is, we have these models that have been coming out in the last year, and none of them seem to have significantly surpassed GPT-4, and certainly not in the agentic way in which they are interacting with as a coworker. You know, they'll brag that they got a few extra points on MMLU or something. And even GPT-4, oh, it's cool that they can talk like Scarlett Johansson or something, but like... And honestly, I'm going to use that. Oh, I guess not anymore. Not anymore. Okay, but the whole co-worker thing. So this is going to be a run-on question, but you can address it in any order. But It makes sense to me why they'd be good at answering questions. They have a bunch of data about how to complete Wikipedia text or whatever. Where is the equivalent training data that enables it to understand what's going on in the Zoom call? How does this connect with what they were talking about in the Slack? What is the cohesive project that they're going after based on all this context that I have? Where is that training data coming from?

Leopold Aschenbrenner: Yeah, so I think a really key question for AI progress in the next few years is, how hard is it to unlock the test time compute overhang? So right now, GPT-4 answers a question. And it can do a few hundred tokens of chain of thought. And that's already a huge improvement. This is a big unhobbling. Before, answer a math question. It's just shotgun. And if you try to answer a math question by saying the first thing that came to mind, you wouldn't be very good. So GPT-4 thinks for a few hundred tokens. And, you know, if I thought for a few hundred, you know, if I think at like 100 tokens a minute and I thought... You think at much more than 100 tokens a minute. I don't know. If I thought for like 100 tokens a minute, you know, it's like what GP4 does. Maybe it's like, you know, it's equivalent to me thinking for three minutes or whatever, right? Um... You know, suppose GPT-4 could think for millions of tokens, right? That's sort of plus four rooms, plus four orders of magnitude on test time compute, just like on one problem. It can't do it right now. It kind of gets stuck, right? Like write some code, even if, you know, you can do a little bit of iterative debugging, but eventually it just kind of like, it can't, it kind of gets stuck in something. It can't correct its errors and so on. And, you know, In a sense, there's this big overhang. In other areas of ML, there's this great paper on AlphaGo where you can trade off train time and test time compute. And if you can use four rooms more test time compute, that's almost like a three and a half room bigger model. Just because, again, if 100 tokens a minute, a few million tokens, that's a few months of working time. There's a lot more you can do in a few months of working time than right now. So the question is, how hard is it to unlock that? And I think the, you know, the sort of short timelines AI world is if it's not that hard. And the reason that might not be that hard is that, you know, there's only really a few extra tokens you need to learn, right? You need to kind of learn the error correction tokens, the tokens where you're like, ah, I think I made a mistake. Let me think about that again. You need to learn the kind of planning tokens. That's kind of like, I'm going to start by making a plan. Here's my plan of attack. And then I'm going to write a draft and I'm going to like, now I'm going to critique my draft. I'm going to think about it. And so it's not things that models can do right now, but the question is, how hard is that? And in some sense, also, there's sort of two paths to agents, right? When Cholto was on your podcast, he talked about kind of scaling leading to more nines of reliability. And so that's one path. I think the other path is a sort of like unhobbling path where it needs to learn this kind of like system two process. And if it can learn this sort of system two process, it can just use kind of millions of tokens and think for them and be cohesive and be coherent. One analogy, so when you drive, here's an analogy. When you drive, right? Okay, you're driving. And most of the time you're kind of on autopilot, right? You're just kind of driving and you're doing well. But sometimes you hit a weird construction zone or a weird intersection. And then I sometimes am like, my passenger seat, my girlfriend, I'm kind of like, ah, be quiet for a moment. I need to figure out what's going on, right? And that's sort of like, you go from autopilot to the system two is jumping in and you're thinking about how to do it. And so the scaling is improving that system one autopilot. And I think it's sort of, it's the brute force way to get to kind of agents. You just improve that system. But if you can get that system to working, then, you know, I think you could like quite quickly jump, you know, to sort of this like more gentrified, you know, test time, compute overhang is unlocked. What's the reason to think that this is an easy win in the sense that, oh, you just get the, there's like some loss function that easily enables you to train it to enable the system to thinking. Yeah. There's not a lot of animals that have system to thinking, you know, it like took a long time for evolution to give us system to thinking. The free training is like, listen, I get it. You got like trillions of tokens of internet techs. I get that. Like, yeah, you like match that and you get all these, all this free training capabilities. What's the reason to think that this is an easy and hobbling? Yeah, so, okay, a bunch of things. First of all, free training is magical, right? And it gave us this huge advantage for models of general intelligence, because you could just predict the next token, but predicting the next token, I mean, it's sort of a common misconception, but what it does is it lets this model learn these incredibly rich representations, right? Like these sort of representation learning properties are the magic of deep learning. You have these models, and instead of learning just kind of like whatever statistical artifacts or whatever, it learns sort of these models of the world. That's also why they can generalize, because it learned the right representations. You pre-train these models and you have this raw bundle of capabilities that's really useful, this almost unformed raw mass. The unhobbling we've done over GPT-2 to GPT-4 was you took this raw mass and then you RLH-hafted it into a really good chatbot. That was a huge win. In the original, I think, ConstructGPT paper, RLHF versus non-RLHF model, it's like 100x model size win on human preference rating. It started to be able to do simple chain of thought and so on. But you still have this advantage of all these raw capabilities, and I think there's still a huge amount that you're not doing with them. And by the way, I think this pre-training advantage is also the difference to robotics, where I think robotics... I think people used to say it was a hardware problem, but I think the hardware stuff is getting solved. But the thing we have right now is you don't have this huge advantage of being able to bootstrap yourself with pre-training. You don't have all this unsupervised learning you can do. You have to start right away with the RL self-play and so on. So now the question is, why might some of this on hobbling and RL and so on work? And again, there's sort of this advantage of bootstrapping, right? So your Twitter bio is being pre-trained, right? But you're actually not being pre-trained anymore. You're not being pre-trained anymore. You were pre-trained in grade school and high school. At some point, you transition to being able to learn by yourself. right um you weren't able to do that in elementary school um i don't know middle school probably high school is maybe when it sort of started you need some guidance um you know college you know if you're smart you can kind of teach yourself and then sort of models are just starting to enter that regime right and so it's sort of like it's a little bit probably a little bit more scaling um and then you got to figure out what goes on top and it won't be trivial right so a lot of um a lot of deep learning is sort of like, you know, it sort of seems very obvious in retrospect. And there's sort of some obvious cluster of ideas, right? There's sort of some kind of like thing that seems a little dumb, but that just kind of works. But there's a lot of details you have to get right. So I'm not saying this, you know, we're going to get this, you know, next month or whatever. I think it's going to take a while to like really figure out the details. A while for you is like half a year or something.

Dwarkesh Patel: Well, I think, I don't know. That makes a month, six months. Between six months and three years, you know?

Leopold Aschenbrenner: But, you know, I think it's possible. And I think there's... I think it's also very related to the issue of the data wall. One intuition on the learning by yourself is pre-training is the words are flying by. It's like the teacher is lecturing to you. The models, the words are flying by. They're just getting a little bit from it. But that's sort of not what you do when you learn from yourself, right? When you learn by yourself, you know, so you're reading a dense math textbook, you're not just kind of like skimming through it once. You wouldn't learn that much from it. I mean, some word cells just skim through, reread and reread the math textbook, and then they memorize, you know, like you just repeated the data, then they memorize. What you do is you kind of like you read a page, kind of think about it. You have some internal monologue going on. You have a conversation with a study buddy. You try a practice problem. You know, you fail a bunch of times. At some point it clicks. Then you're like, this made sense. Then you read a few more pages. And so we've kind of bootstrapped our way to being able to do that now with models. We're just starting to be able to do that. And then the question is being able to read it, think about it, try problems. And the question is, all this sort of self-play synthetic data RL is kind of making that thing work. So basically, translating in context. Right now, there's in-context learning, super sample efficient. In the Gemini paper, it just learns a language in context. And then you have pre-training, not at all sample efficient. But what humans do is they do in-context learning. You read a book. You think about it until eventually it clicks. But then you somehow distill that back into the weights. And in some sense, that's sort of like what RL is trying to do. And RL is super finicky, but when RL works, RL is kind of magical because it's sort of the best possible data for the model. It's like when you try a practice problem and then you fail, and at some point you kind of figure it out in a way that makes sense to you, that's sort of like the best possible data for you because it's like the way you would have solved the problem. And that's what RL is, rather than just you kind of read how somebody else solved the problem and doesn't initially click. Sure. Yeah. By the way, if that take sounds familiar, because it was part of the question I asked John Shulman, that goes to illustrate the thing I said in the intro, where a bunch of the things I've learned about AI, just like, we do these dinners before the interviews,

Dwarkesh Patel: and me, Shulto, and a couple, I'm like, what should I ask John Shulman? What should I ask Dario? Okay, suppose this is the way things go and we get these unhobblings. Yeah.

Leopold Aschenbrenner: Well, and the scaling, right? So it's like you have this baseline, just enormous force of scaling, right? Where it's like GPT-2 to GPT-4. You know, GPT-2, it could kind of like, it was amazing, right? It could string together plausible sentences. But, you know, it could barely do anything. It was kind of like preschooler. And then GPT-4 is, you know, it's writing code. It like, you know, can do hard math. And so it's sort of like smart high school. And so this big jump, and, you know, and sort of the essay series I go through and kind of count the orders of magnitude of compute scale up, of algorithmic progress. But scaling alone, you know, sort of by 27, 28 is going to do another kind of preschool to high school jump on top of GPT-4. And so that will already be just like at a per token level, just incredibly smart. They'll get you some more reliability. And then you add these unhobblings that make it look much less like a chatbot, more like this agent, like a drop-in remote worker. And, you know, that's when things really get going. Okay.

Speaker 2: Yeah.

Leopold Aschenbrenner: I want to ask more questions about this. I think, yeah, let's zoom out. Okay. So suppose you're right about this. And I guess you, this is because of the 2027 cluster, we've got 10 gigawatt, 2027, 10 gigawatts. 28 is the 10 gigawatt.

Dwarkesh Patel: Okay.

Leopold Aschenbrenner: Maybe it'll be pulled forward.

Dwarkesh Patel: Okay, sure.

Leopold Aschenbrenner: Something. Yeah.

Dwarkesh Patel: And so I guess that's like 5.5 level by 2027, like whatever that's called, right? What does the world look like at that point? You have these remote workers who can replace people.

Leopold Aschenbrenner: what is the reaction to that in terms of the economy, politics, geopolitics? Yeah. So, you know, I think 2023 was kind of a really interesting year to experience as somebody who was like, you know, really following the eye stuff where, you know, before that, what were you doing in 2023? I mean, open AI. Oh yeah. Yeah. And, and, uh, and, um, you know, it kind of went, you know, I mean, you know, I was, I was been thinking about this and, you know, like talking to a lot of people, you know, in the years before, and it was this kind of weird thing, you know, you almost didn't want to talk about AI or AGI and it was kind of a dirty word. Right. And then 2023, you know, people saw chat GPT for the first time and saw GPT-4 and it just like exploded. Right. It triggered this kind of like, you know, you know, a huge sort of capital expenditures from all these firms and, and, and, you know, the explosion in revenue from NVIDIA and so on. And, um, Things have been quiet since then, but the next thing has been in the oven. And I expect every generation, these G-forces to intensify. It's like people see the models. People haven't counted them, so they're going to be surprised and it'll be kind of crazy. And then revenue is going to accelerate. Suppose you do hit the 10 billion end of this year. Suppose it just continues on this doubling trajectory of every six months of revenue doubling. It's like you're not actually that far from 100 billion. Maybe that's like 26%. And so, you know, at some point, you know, like, you know, sort of what happened to Nvidia is gonna happen to big tech, you know, like their stocks, you know, that's gonna explode. And I mean, I think a lot more people are going to feel it. Right. I mean, I think the, um, I think 2023 was the sort of moment for me where it went from kind of AGI as a sort of theoretical abstract thing. And you'd make the models to like, I see it, I feel it. And like, I see the path, I see where it's going. I like, I think I can see the cluster where it's strained on, like the rough combination of algorithms, the people, like how it's happening. And I think, you know, most of the world is not, you know, most of the people feel it are like right here,

Dwarkesh Patel: you know? Right.

Leopold Aschenbrenner: Uh, but, but, you know, I think a lot more of the world is going to start feeling it. Um. And I think that's going to start being kind of intense.

Dwarkesh Patel: Okay. So right now, who feels that you can, you go on Twitter and there's these GPT rapper companies like, whoa, GPT-4 is going to change our business.

Leopold Aschenbrenner: I'm so bearish on the rapper companies, right? Because like, they're the ones that are going to be like, the rapper companies are betting on stagnation, right? The rapper companies are betting like, you have these intermediate models and it takes so much time to integrate them. And I'm kind of like, I'm really bearish because I'm like, we're just going to Sonic Boom you, you know, and we're going to get the unhopped ones. We're going to get the drop-in remote worker. And then, you know, your stuff is not going to matter.

Dwarkesh Patel: Okay, sure, sure. So that's done. Now, SF is paying attention now, or this crowd here is paying attention. Who is going to be paying attention in 2026, 2027? And presumably these are years in which hundreds of billions of CapEx is being spent on AI.

Leopold Aschenbrenner: I mean, I think the national security state is going to be starting to pay a lot of attention. And I hope we get to talk about that. Okay, let's talk about it now. We'll talk about it now. What is the sort of political reaction immediately? Yeah. And even like internationally,

Dwarkesh Patel: like what people see, like right now, I don't know if like Xi Jinping like reads the news and sees like, oh my God, like MMLU score on that. What are you doing about this comrade? So what happens when he sees a remote replacement and it has $100 billion in revenue? There's a lot of businesses that have $100 billion in revenue and people aren't staying up all night talking about it.

Leopold Aschenbrenner: I think the question is, when does the CCP and when does the sort of American national security establishment realize that superintelligence is going to be absolutely decisive for national power, right? And this is where, you know, the sort of intelligence explosion stuff comes in, which, you know, we should also talk about later. You know, it's sort of like, you know, you have AGI, you have this sort of drop-in remote worker that can replace, you know, you or me, at least that sort of remote jobs, you know, cognitive jobs. And then, you know... I think fairly quickly. By default, you turn the crank one or two more times, and then you get a thing that's smarter than humans. But I think even more than just turning the crank a few more times, I think one of the first jobs to be automated is going to be that of an AI researcher engineer. And if you can automate AI research... I think things can start going very fast. Right now, there's already this trend of half an order of magnitude a year of algorithmic progress. At this point, you're going to have GPU fleets in the tens of millions for inference or more, and you're going to be able to run 100 million human equivalents of these automated AI researchers. And if you can do that, you know, you can maybe do, you know, a decade's worth of sort of ML research progress in a year. You know, you get some sort of 10x speed up. And if you can do that, I think you can make the jump to kind of like AI that is vastly smarter than humans, you know, within a year, a couple years. And then, you know, that broadens, right? So you have this sort of initial acceleration of AI research. That broadens to, like, you apply R&D to a bunch of other fields of technology. And the sort of, like, extremes, you know, at this point, you have, like, a billion just super intelligent researchers, engineers, technicians, everything. You're superbly competent at all the things. You know, they're going to figure out robotics. We talked about it being a software problem. Well, you know, you have a billion of... super smart, smarter than the smartest human researchers, AI researchers on your cluster at some point during the intelligence explosion, they're going to be able to figure out robotics. And then, again, that expands. And I think if you play this picture forward, I think it is fairly unlike any other technology in that... I think a couple years of lead could be utterly decisive in, say, military competition. If you look at Gulf War I, the Western Coalition forces, they had a 100 to 1 kill ratio. They had better sensors on their tanks. They had more precision missiles, like GPS. They had stealth. They had maybe 20, 30 years of technological lead. They just completely crushed them. superintelligence applied to sort of broad fields of R&D, and then, you know, the sort of industrial explosion as well. You have the robots, you're just making lots of material. You know, I think that could compress, I mean, basically compress kind of like a century worth of technological progress into less than a decade. And that means that, you know, a couple years could mean a sort of Gulf War I style, like, you know, advantage in military affairs. Including a decisive advantage that even preempts nukes. Suppose, how do you find the stealth in nuclear submarines? Right now, that's a problem of you have sensors, you have the software to attack where they are. You can do that, you can find them. You have millions or billions of mosquito-like sized drones, and they take out the nuclear submarines, they take out the mobile launchers, they take out the other nukes. Anyway, so I think enormously destabilizing, enormously important for national power. And at some point, I think people are going to realize that. Not yet, but they will. And when they will, I think there will be sort of you know, I don't think it'll just be the sort of AI researchers in charge. And, you know, I think on the, you know, the CCP is going to, you know, have sort of an all-out effort to like infiltrate American AI labs, right? You know, like billions of dollars, thousands of people, you know, full force of the sort of, you know, Ministry of State Security. CCP is going to try to, you know, like outbuild us, right? Like they, you know, their, you know, power in China, you know, like the electric grid, you know, they added a US's, you know, a complete, like, they added as much power in the last decade as the entire US electric grid. The 100 gigawatt cluster, at least the 100 gigawatts is going to be a lot easier for them to get. By this point, I think it's going to be an extremely intense international competition.

Dwarkesh Patel: In this picture, One thing I'm uncertain about is whether it's more like what you say, where it's more of an implosion of you have developed an AGI and then you make it

Leopold Aschenbrenner: into an AI researcher. And for a while, a year or something, you're only using this ability to make hundreds of millions of other AI researchers. And then like the thing that comes out of this, really frenetic process is a super intelligence. And then that goes out in the world and is developing robotics and helping you take over other countries and whatever.

Dwarkesh Patel: I mean,

Leopold Aschenbrenner: I think it's a little bit more, you know, it's a little bit more kind of like, you know, it's not like, you know, on and off. It's a little bit more gradual, but it's sort of like, it's an explosion that starts narrowly. It can do cognitive jobs, you know, isri use for cognitive jobs is make the eye better like solve robotics you know and as as as you know you solve robotics now you can do r&d and you know like biology and other technology um you know initially you start with the factory workers you know they're wearing the glasses and the airpods you know and the ai is instructing them right because you know you kind of make any worker into a skilled technician and then you have the robots come in and anyway so it sort of expands this process expands metas ray-bans are a compliment to their llama well you know it's like Whatever, like the fabs in the US, the constraint to skilled workers, right? Even if you don't have robots yet, you have the cognitive superintelligence and you can kind of make them all into skilled workers immediately. But that's a very brief period. Robots will come soon. Sure, okay. Okay, so suppose this is actually how the tech progresses. In the United States, maybe because these companies are already experiencing hundreds of billions of dollars of AI revenue, And at this point, companies are borrowing hundreds of billions or more in the corporate debt markets. But why is a CCP bureaucrat, some 60-year-old guy, he looks at this and he's like, oh, it's like Copilot has gotten better now. Why are they now... I mean, this is much more than Copilot has gotten better now. I mean, at this point, the tracking won't be... Like, yeah, because to shift the production of an entire country to dislocate energy that is otherwise being used for consumer goods or something and to make that all feed into the data centers. Part of this whole story is you realize the super intelligence is coming soon, right? And I guess you realize it. Maybe I realize it.

Dwarkesh Patel: I'm not sure how much I realize it.

Leopold Aschenbrenner: But will the national security apparatus in the United States and will the CCP realize it? Yeah. I mean, look, I think in some sense, this is a really key question. Um, I think we have sort of a few more years of mid game basically, and where you have a few more 2023s and that just starts updating more and more people. And, you know, I think, you know, the trend lines, you know, will become clear. Um, you know, I think, I think you will see some amount of the sort of COVID dynamic, right? You know, like COVID was like, you know, February, February of 2020, you know, it's like, honestly feels a lot like today, you know, where it's like, you know, it feels like this utterly crazy thing has happened is about, you know, is impending is coming. Um, you kind of see the exponential and yet most of the world just doesn't realize, right. The mayor of New York is like, go out to the shows. And this is just, you know, like Asian racism or whatever, you know? And, and, um, But, you know, at some point, the exponential, like, you know, at some point, people saw it. And then, you know, like, just kind of crazy radical reactions came. Right. Okay. So, by the way, what were you doing during COVID? Or when, like, February? Like, freshman, sophomore, what?

Speaker 2: Junior.

Leopold Aschenbrenner: But still, like, what were you, like, 17-year-old junior or something? Yeah. And then you bought, like, did you short the market or something?

Dwarkesh Patel: Yeah, yeah, yeah. Did you sell at the right time? Yeah. Okay. Yeah, so there will be, like, a March 2020 moment. The thing that was COVID, but here. Now, then you can, like, make the analogy that you make in the series that this will then... cause the reaction of, like, we gotta do the Manhattan Project for America here. I wonder what the politics of this will be like,

Leopold Aschenbrenner: because the difference here is it's not just, like, we need the bomb to beat the Nazis. It's we're building this thing that's making all our energy prices rise a bunch, and it's automating a bunch of our jobs.

Dwarkesh Patel: And the climate change stuff, like, people are gonna be like, oh, my God, it's making climate change worse. And it's helping big tech. Like, Politically this doesn't seem like a dynamic where the national security apparatus or the guy the president is like We have to step on the gas here and like make sure America wins Yeah,

Leopold Aschenbrenner: I mean again I think a lot of this really depends on sort of how much people are feeling it how much people are seeing it You know, I think there's a thing where you know kind of basically our generation right we're kind of so used to kind of You know, basically peace and like, you know the world, you know American hegemony and nothing matters and but you know, the sort of like extremely intense and these extraordinary things happening in the world and like intense international competition is like very much the historical norm. Like in some sense, it's like, you know, sort of this, there's this sort of 20 year, very unique period, but like, you know, the history of the world is like, you know, uh, you know, like in world war two, right. It was like 50% of GDP went to, you know, like, you know, war production, the U S borrowed over 60% of GDP, you know, in, in, in, you know, I think Germany and Japan over a hundred percent world war one, you know, UK, Japan, sorry, UK, France, Germany, all borrowed over a hundred percent of GDP. Um, and, um, you know, I think the sort of much more was on the line, right? Like, you know, and, you know, people talk about world war one being so destructive and, you know, like 20 million Soviet soldiers dying and like 20% of Poland. But, you know, that was just the sort of like that happened all the time, right? You know, like seven years war, you know, like whatever, 20, 30% of Prussia died, you know, like 30 years war, you know, like, uh, I think like, you know, up to 50% of like large swath of Germany died. Um, and, um, you know, I think the question is, will these sort of like, will people see that the stakes here are really, really high and that basically sort of like history is actually back? And I think, you know, I think the American national security state thinks very seriously about stuff like this. They think very seriously about competition with China. I think China very much thinks of itself on this as our historical mission and rejuvenation of the Chinese nation. They think a lot about national power. They think a lot about like the world order. And then, you know, I think there's a real question on timing. Do they start taking this seriously when the intelligence explosion is already happening quite late? Or do they start taking this seriously two years earlier? And that matters a lot for how things play out. But at some point, they will. And at some point, they will realize that this will be sort of utterly decisive for... not just kind of like some proxy war somewhere, but whether liberal democracy can continue to thrive, whether the CCP will continue existing. And I think that will activate sort of forces that we haven't seen in a long time. The great power conflict thing definitely seems compelling. I think just all kinds of different things seem much more likely when you think from a historical perspective, when you zoom out beyond the liberal democracy that we've been living in, had the pleasure to live in America, let's say 80 years. Including dictatorships, including obviously war, famine, whatever. I was reading the Gulag Archipelago,

Dwarkesh Patel: and one of the chapters begins with Solzhenitsyn saying, if you would have told a Russian citizen under the Tsars that because of all these new technologies, we wouldn't see some great Russian revival or becomes a great power and the citizens are... made wealthy, but instead what you would see is tens of millions of Soviet citizens tortured by millions of beasts in the worst possible ways, and that this is what would be the result of the 20th century, they wouldn't have believed you. They'd have called you a slanderer.

Leopold Aschenbrenner: Yeah, and you know, the, um... you know, the possibilities for dictatorship with superintelligence are sort of even crazier, right? I think, you know, imagine you have a perfectly loyal military and security force, right? That's it. No more rebellions, right? No more popular uprisings, you know, perfectly loyal. You know, you have, you know, perfect lie detection. You know, you have surveillance of everybody. You know, you can perfectly figure out who's the dissenter, weed them out. You know, no Gorbachev would have ever risen to power who had some doubts about the system. You know, no military coup would have ever happened. And I think you, I mean... I think there's a real way in which part of why things have worked out is that you know, ideas can evolve. And, you know, there's sort of like some sense in which sort of time heals a lot of wounds and time, you know, solves, you know, a lot of debates and a lot of people had really strong convictions. But, you know, a lot of those have been overturned by time because there has been this continued pluralism and evolution. I think there's a way in which kind of like, you know, if you take a CCP-like approach to kind of like truth, truth is what the party says. And you supercharge that with superintelligence. I think there's a way in which that could just be like locked in and trying for, you know, a long time. And I think the possibilities are pretty terrifying. Yeah. you know, your point about, you know, history and, and, and sort of like living in America for the past eight years, you know, um, I think this is one of the things I sort of took away from growing up in Germany is a lot of this stuff feels more visceral, right? Like, you know, my, my mother grew up in the former East, my father in the former West, they like met shortly after the wall fell, right? Like the end of the cold war was this sort of extremely pivotal moment for me because it's, you know, it's the reason I exist. Right. And then, you know, growing up in Berlin and, you know, the former wall, you know, um, My great grandmother, who is still alive, is very important in my life. She was born in 34, you know, grew up, you know, during the Nazi era, during, you know, all that, you know, then World War Two, you know, like saw the firebombing of Dresden from the sort of, you know, country cottage or whatever, where they as kids were, you know, then and then, you know, then spends most of her life in sort of the East German communist dictatorship. She'd tell me about, you know, in like 54, when there's like the popular uprising, you know, and Soviet tanks came in, you know, her husband was telling her to get home really quickly, you know, get off off the streets. you know, had a, had a son who, who tried to, you know, ride a motorcycle across, across the iron curtain and then was put in a Stasi prison for a while. Um, you know, and then finally, you know, when she's almost 60, you know, it was the first time she lives in, you know, a free country, um, and, and, and a wealthy country. And, um, you know, when I was a kid, she was, um, she, the thing she always really didn't want me to do was like get involved in politics. Cause like joining a political party was just, you know, it was a very bad connotations for her. Um, anyway, and she, and she sort of raised me when I was young, you know? And so, um, it, It doesn't feel that long ago. It feels very close.

Dwarkesh Patel: Yeah. So I wonder when we're talking today about the CCP, listen, the people in China who will be doing their version of the project will be AI researchers who are somewhat westernized, who interact with, either got educated in the West or have colleagues in the West. Are they going to...

Leopold Aschenbrenner: sign up for the CCP project that's going to hand over control to Xi Jinping? What's your sense on... I mean, fundamentally, they're just people, right? Can't you convince them about the dangers of superintelligence or something? Will they be in charge, though? I mean, this is also the case, you know, in the US or whatever. This is sort of like rapidly depreciating influence of the lab employees. Like right now, the sort of AI lab employees have so much power, right? Over this, you know, like you saw this November event, so much power, right? But both, I mean, both are going to get automated and they're going to lose all their power. And it'll just be, you know, kind of like a few people in charge with their sort of armies of automated eyes. But also, yeah. you know, it's sort of like the politicians and the generals and the sort of national security state, you know, a lot of, you know, I mean, there's sort of, this is the sort of some of these classic scenes from, you know, the Oppenheimer movies, you know, the scientists built it and then it was kind of, you know, and the bomb was shipped away and it was out of their hands. Um, you know, I actually, yeah, I think, I actually think it's good for like lab employees to be aware of this. It's like, you have a lot of power now, but, um, you know, um, uh, maybe not for that long and, you know, use it wisely. Um, Yeah, I do. I do think they would benefit from some more, you know, organs of representative democracy. What do you mean? Oh, I mean, you know, in the sort of the in the opening board events, you know, employee power was exercised in a very sort of direct democracy way. And I feel like that's how some of how that went about. You know, I think it really highlighted the benefits of representative democracy and having some deliberative organs. Interesting. Yeah.

Dwarkesh Patel: Let's go back to the $100 billion revenue, whatever, and so these companies are deploying, trying to build clusters that are this big. Where are they building it? Because if you say it's the amount of energy that would be required for a small or medium-sized U.S. state, is it then Colorado gets no power and it's happening in the United States, or is it happening somewhere else?

Leopold Aschenbrenner: Oh, I mean, I think that, I mean, in some sense, this is the thing that I always find funny is, you know, you talk about Colorado gets new power. You know, the easy way to get the power would be like, you know, displace less economically useful stuff. You know, it's like, whatever, buy up the aluminum smelting plant. And, you know, that has to get a lot. And, you know, we're going to replace it with with the data center because that's important. I mean, that's not actually happening because a lot of these power contracts are really sort of long term locked in. You know, there's obviously people don't like things like this. And so it seems like in practice what it's requiring, at least right now, is building new power. That might change. And I think that's when things get really interesting, when it's like, no, we're just dedicating all of the power to the AGI. So right now it's building new power. 10 gigawatt, I think quite doable. It's like a few percent of US natural gas production. When you have the 10 gigawatt chaining cluster, you have a lot more inference. So that starts getting more. I think 100 gigawatt, that starts getting pretty wild. Again, it's like over 20% of US electricity production. I think it's pretty doable, especially if you're willing to go for natural gas. I do think it is incredibly important that these clusters are in the United States.

Dwarkesh Patel: Why does it matter it's in the US?

Leopold Aschenbrenner: Look, I think there are some people who are trying to build clusters elsewhere. There's a lot of free-flowing Middle Eastern money that's trying to build clusters elsewhere. I think this comes back to the national security question we talked about earlier. Would you do the Manhattan Project in the UAE? I think basically putting the clusters, I think you can put them in the US, you can put them in ally democracies, but I think once you put them in dictatorships, authoritarian dictatorships, you create this irreversible security risk. clusters there, much easier for them to exfiltrate the weights. You know, they can like literally steal the AGI, the superintelligence. It's like they got a copy of the, you know, of the of the atomic bomb, you know, and they just got direct replica of that. And it makes it much easier to them. I mean, we're ties to China. You can ship that to China. So that's a huge risk. Another thing is they can just seize the compute, right? Like maybe right now they just think of this. I mean, in general, I think people, you know, I think the issue here is people are thinking of this as a, you know, chat GPT, big tech product clusters. But I think the cluster is being planned now, you know, three to five years out, like it will be the like AGI super intelligence clusters. And so anyway, so like when things get hot, you know, they might just seize the compute. And I don't know, supposedly put like, you know, 25% of the compute capacity in these sort of Middle Eastern dictatorships. Well, they seize that. And now it's sort of a ratio of compute of three to one. And, you know, We still have some more, but even only 25% of compute there, I think it starts getting pretty hairy. I think three to one is not that great of a ratio. You can do a lot with that amount of compute. And then look, even if they don't actually do this, even if they don't actually seize the compute, even if they actually don't steal the weights, there's just a lot of implicit leverage you get. They get the seat at the AGI table. And I don't know why we're giving authoritarian dictatorships the seat at the AGI table. Okay, so there's going to be a lot of compute in the Middle East if these deals go through.

Dwarkesh Patel: First of all, is it just like every single big tech company is just trying to figure out what they're going to do?

Leopold Aschenbrenner: Not everyone, some. Okay, okay. I guess there's reports, I think Microsoft? Yeah, yeah, yeah. Which we'll get into. So UAE gets a bunch of compute because we're building the clusters there.

Speaker 2: Yeah.

Leopold Aschenbrenner: And why? So let's say they have 25%. Why does a compute ratio matter?

Dwarkesh Patel: If it's about them being able to kick off the intelligence explosion, isn't it just some threshold where you have 100 million AI researchers or you don't?

Leopold Aschenbrenner: I mean, you can do a lot with, you know, 33 million extremely smart scientists. And, you know, and again, a lot of the stuff, you know, so first of all, it's like, you know, that might be enough to build the crazy bioweapons, right? And then you're in a situation where like now, wow, we've just like, they stole the weights, they seized the compute. Now they can make, you know, they can build these crazy new WMDs that, you know, will be possible super intelligence. And now you've just kind of like proliferated the stuff and, you know, it'll be really powerful. And also, I mean, I think, you know, 3x on compute isn't actually that much. I think a thing I worry a lot about is I think the riskiest situation is if we're in some sort of really tight neck, feverish international struggle. If we're really close with the CCP and we're months apart. I think the situation we want to be in, we could be in if we played our cards right, is a little bit more like the US building the atomic bomb versus the German project. Way behind, years behind. And if we have that, I think we just have so much more wiggle room to get safety right. We're going to be building like, you know, there's going to be these crazy new WMDs, you know, things that completely undermine, you know, nuclear deterrence, you know, intense competition. And that's so much easier to deal with if, you know, you're like, you know, it's not just, you know, you don't have somebody right on your tails. You got to go, go, go. You got to go maximum speed. You have no wiggle room. You're worried that at any time they can overtake you. I mean, they can also just try to outbuild you, right? Like they might literally win, like China might literally win. Um, if they can steal the weights, cause they can, they can outbuild you. Um, and they maybe have less caution, uh, both, you know, good and bad caution, you know, kind of like whatever unreasonable regulations we have. Um, or you're just in this really tight race. And I think is that sort of like, if you're in this really tight race, this sort of fever struggle, I think that's when sort of there's the greatest peril of self-destruction.

Dwarkesh Patel: So then presumably the companies that are trying to build clusters in the Middle East realize this. Is it just that it's impossible to do this in America? And if you want American companies to do this at all, then you do it in the Middle East or not at all? And then you're just like, I'm trying to build a three gorgeous dam cluster.

Leopold Aschenbrenner: I mean, there's a few reasons. One of them is just like people aren't thinking about this as the AGI superintelligence cluster. They're just like, ah, you know, like cool clusters for my, you know, for my chat GBC.

Dwarkesh Patel: So they're building, and the plans right now are clusters, which are ones that are, like, because if you're doing ones for inference, presumably you could, like, spread them out across the country or something. But the ones they're building, they realize we're going to do one training run in this thing we're building.

Leopold Aschenbrenner: I just think it's harder to distinguish between inference and training compute. And so people can claim it's training compute, but I think they might realize that actually, you know, this is going to be useful for, sorry, they might say it's inference compute, and actually it's useful for training compute too.

Dwarkesh Patel: Because it's synthetic data and things like that?

Leopold Aschenbrenner: Yeah, the future of training, you know, like RL looks a lot like inference, for example, right? Or you just kind of like end up connecting them, you know, in time, you know, it's like raw material, you know,

Dwarkesh Patel: it's like,

Leopold Aschenbrenner: you know, it's placing your uranium refinement facilities there. Sure. Anyway, so a few reasons, right? One is just like, they don't think about this as the AGX cluster. Another is just like easy mining from the Middle East, right? Another one is like, you know, people saying, some people think that, you know, you can't do it in the US. And, yeah. You know, I think we actually face a sort of real system competition here because, again, some people think it's only autocracies that can do this, that can kind of like top down, mobilize the sort of industrial capacity, the power, you know, get the stuff done fast. And again, this is the sort of thing, you know, we haven't faced in a while. But, you know, during the Cold War, like we really there was this sort of intense system competition, right? Like East West Germany was this, right? Like West Germany, kind of like liberal democratic capitalism versus kind of, you know, communist state planned. And, you know, now it's obvious that the sort of, you know, the the The free world would win. But, you know, even even as late as like 61, you know, Paul Samuelson was predicting the Soviet Union would would outgrow the United States because they were able to sort of mobilize industry better. And so, yeah, there's some people who, you know, shitpost about loving America by day. They're betting against America. They're betting against the liberal order. And I think I basically just think it's a bad bet. And the reason I think it's a bad bet is I think this stuff is just really possible in the US. And so to make it possible in the US, there's some amount that we have to get our act together, right? So I think there's basically two paths to doing it in the US. One is you just got to be willing to do natural gas. And there's ample natural gas, right? You put your cluster in West Texas. You put it in Southwest Pennsylvania by the Mercell shale. 10 gigawatt cluster, super easy. 100 gigawatt cluster, also pretty doable. I think natural gas production in the United States has almost doubled in a decade. If you do that one more time over the next seven years or whatever, you could power multiple trillion-dollar data centers. But the issue there is a lot of people have made these climate commitments. They're not just government. It's actually the private companies themselves, the Microsoft, the Amazons, and so on. They have these climate commitments, so they won't do natural gas. I admire the climate commitments, but I think at some point, the national interest and national security is more important. The other path is like, you know, you can do this sort of green energy mega projects, right? You do the solar and the batteries and the, you know, the SMRs and geothermal. But if we want to do that, there needs to be sort of a sort of broad deregulatory push, right? So like you can't have permitting take a decade, right? So you got to reform FERC. You got to like have, you know, blanket NEPA exemptions for this stuff. You know, there's like inane state level regulations, you know, that are like, yeah, you could build, you know, you can build the solar panels and batteries next to your data center, but it'll still take years because, you know, You actually have to hook it up to the state electrical grid. And you have to use governmental powers to create rights of way to have multiple clusters and connect them and have thick cables, basically. And so look, ideally we do both. Ideally we do natural gas and the broad deregulatory agenda. I think we have to do at least one. And then I think this possible stuff is just possible in the United States. Yeah. I think a good analogy for this, by the way, before the conversation I was reading, there's a good book about World War II industrial mobilization in the United States called Freedom's Forge. And... I guess when we think back on that period, especially if you're from,

Dwarkesh Patel: if you read like the Patrick Collison Fast and the progress study stuff, it's like, you had state capacity back then and people just got shit done, but now it's a clusterfuck. Wasn't at all the case. No, so it was really interesting. So you have people who are from the Detroit auto industry side, like Knudsen, who are running mobilization for the United States. And they were extremely competent. But then at the same time, you had labor organization agitation, which is actually very analogous to the climate pledges and climate change concern we have today,

Leopold Aschenbrenner: where they would have these strikes while literally into 1941, that would cost millions of man hours worth of time when we're trying to make tens of millions,

Dwarkesh Patel: sorry, tens of thousands of planes a month or something. And they would just debilitate factories for, you know, trivial, like, pennies-on-the-dollar kind of concessions from capital.

Leopold Aschenbrenner: And it was concerns that, oh, the auto companies are trying to use the pretext of a potential war to actually

Dwarkesh Patel: prevent paying labor the money it deserves. And so what climate change is today, like, you think, ah, fuck, America's fucked. Like, we're not going to be able to build this shit.

Leopold Aschenbrenner: Like, if you look at NEPA or something. But I didn't realize how debilitating labor was in, like, World War II. Well, it wasn't just that, right? It was just, you know, before, at the... you know, sort of like 39 or whatever, the American military was in total shambles, right? You read about it and it reads a little bit like, you know, the German military today, right? It's like, you know, military expenditures, I think, were less than 2% of GDP. You know, all the European countries had gone, even in peacetime, you know, like above 10% of GDP, sort of this like rapid mobilization. There's nothing, you know, like we're making kind of like no planes. There's no military contracts. Everything had been starved during the Great Depression. But there was this latent capacity. And, you know, at some point, the United States got their act together. I mean, the thing I'll say is I think, you know, the supply is sort of the other way around too, to basically to China, right? And I think sometimes people are, you know, they kind of count them out a little bit and they're like the export controls and so on. And, you know, they're able to make seven nanometer chips now. I think there's a question of like, how many could they make? But, you know, I think there's at least a possibility that they're going to be able to mature that ability and make a lot of seven nanometer chips. And there's a lot of latent industrial capacity in China and they are able to like, you know, build a lot of power fast. And maybe that isn't activated for AI yet. But at some point, you know, the same way the United States and like, you know, a lot of people in the U.S. and the United States government is going to wake up, you know, at some point the CCP is going to wake up.

Dwarkesh Patel: Yeah. Okay. Going back to the question of presumably companies, are they blind to the fact that there's going to be some sort of, well, okay, so they realize that there's going, they realize scaling is a thing, right? Obviously their whole plans are continued. non-scaling and so they understand that we're going to be in 2028 building the 10 gigawatt data centers and at this point that the people who can keep up are big tech just potentially at like the edge of their capabilities yeah then sovereign wealth fund funded things yeah and also big major countries like america china whatever So what's their plan? If you look at these AI labs, what's their plan given this landscape? Do they not want the leverage of being in the United States?

Leopold Aschenbrenner: I mean, I think I don't know. I think I mean, one thing the Middle East does offer is capital, but it's like America has plenty of capital. Right. It's like, you know, we have trillion dollar companies like what are these Middle Eastern states? They're kind of like trillion dollar oil companies. We have trillion dollar companies and we have very deep financial markets. And it's like, you know, Microsoft could issue hundreds of billions of dollars of bonds and they can pay for these clusters. I mean, look, I think another argument being made, and I think it's worth taking seriously, is an argument that, look, if we don't work with the UAE or with these Middle Eastern countries, they're just going to go to China, right? And so, you know, they're going to build data centers. They're going to pour money into AI regardless. And if we don't work with them, you know, they'll just support China. And look, I mean, I think... I think there's some merit to the argument in the sense that I think we should be doing basically benefit sharing with them. I think we should talk about this later, but I think basically on the road to AGI, there should be two tiers of coalitions. There should be this narrow coalition of democracies that's the coalition that's developing AGI, and then there should be a broader coalition where we go to other countries, including dictatorships, and we're willing to offer them we're willing to offer them some of the benefits of the AI, some of the sharing. So it's like, look, if the UAE wants to use AI products, if they want to run meta recommendation engines, if they want to run the last generation models, that's fine. I think by default, they just wouldn't have had this seat at the AGI table. And so it's like, yeah, they have some money, but a lot of people have money. And the only reason they're getting this sort of core seat at the AGI table, the only reason we're giving, these dictators will have this enormous amount of leverage. over this extremely national security relevant technology is because we're kind of getting them excited and offering it to them. You know, I think the other, yeah.

Dwarkesh Patel: Who, like, who specifically is doing this? Like, just the companies who are going there to fundraise or, like, this is, the AGI is happening and you can fund it or you can't.

Leopold Aschenbrenner: It's been reported that, you know, Sam is trying to raise, you know, $7 trillion or whatever for a chip project. And, you know, it's unclear how many of the clusters will be there and so on. But it's, you know, definitely, definitely stuff is happening. I mean, look, I think another reason I'm a little bit, at least, suspicious of this argument of, like, look, if the U.S. doesn't work with them, they'll go to China, is... I've heard from multiple people, and this wasn't from my time at OpenAI and I haven't seen the memo, but I have heard from multiple people that at some point several years ago, OpenAI leadership had laid out a plan to fund and sell AGI by starting a bidding war between the governments of the United States, China, and Russia. It's surprising to me that they're willing to sell AGI to the Chinese and Russian governments, but also there's something that feels a bit eerily familiar about starting this bidding war and then playing them off each other. you don't do this china will do it so anyway interesting okay so that's pretty

Dwarkesh Patel: fucked up but given that that's okay so suppose that you're right about we ended up in this place because we got at one the way one of our friends put it is that the middle east has like no other place in the world billions of dollars or trillions

Leopold Aschenbrenner: of dollars up for persuasion and

Dwarkesh Patel: But so let's say you're right that you shouldn't have gotten them excited about AGI in the first place But now we're in a place where they are excited about AGI.

Leopold Aschenbrenner: Yeah, and they're like fuck We want us to have GPT-5. We're gonna be off building superintelligence. This atom surpiece thing doesn't work for us and if you're in this place, I

Dwarkesh Patel: Don't they already have the leverage?

Leopold Aschenbrenner: I think the UAE on its own is not competitive. They're already export controlled. You're not actually supposed to ship NVIDIA chips over there. It's not like they have any of the leading AI labs. It's like they have money, but it's actually hard to just translate money into like- But the other things you've been saying about laying out your vision is very much there's this almost

Dwarkesh Patel: industrial process of you put in the compute and then you put in the algorithms. You add that up and you get AGI on the other end. If it's something more like that, then the case for somebody being able to catch up rapidly seems more compelling than if it's some bespoke.

Leopold Aschenbrenner: Well, well, if they can steal the algorithms and if they can steal the weights. And that's really where sort of, I mean, we should talk about this. This is really important. And I think, you know,

Dwarkesh Patel: So, like, right now, how easy would it be for an actor to steal the things that are, like, not the things that are released about Scarlett Johansson's voice, but the RL things you're talking about, the unhobblings?

Leopold Aschenbrenner: I mean, extremely easy, right? You know, DeepMind even, like, they don't make a claim that it's hard, right? DeepMind put out their, like, whatever, Frontier Safety something, and they like lay out security levels and they, you know, security level zero to four and four is the one resistant to state actors. And they say, we're at level zero, right? And then, you know, I mean, just recently, there's like an indictment of a guy who just like stole the code, a bunch of like really important AI code and went to China with it. And, you know, all he had to do to steal the code was that, you know, copy the code and put it into Apple notes and then export it as PDF. And that got past their monitoring, right? And, you know, Google is the best security of any of the AI labs probably because they have the, you know, the Google infrastructure. I mean, I think, I don't know, roughly I would think of this as like, you know, security of a startup, right? And like, what does security of a startup look like, right? You know, it's not that good. It's easy to steal. So even if that's the case. Yeah. A lot of your posts was making the argument that,

Dwarkesh Patel: you know, why are we gonna get the intelligence explosion? Because if we have somebody with the intuition of an Alec Radford to be able to come up with all these ideas, that intuition is extremely valuable and you scale that up. But if it's a matter of these, if it's just in the code, That, like, if it's just the intuition, then that's not gonna be just in the code, right? And also because of expert controls, these countries are gonna have slightly different hardware. You're gonna have to make different trade-offs and probably rewrite things to be able to be compatible with that. including all these things is it just a matter of getting the right pen drive and you plug it into the gigawatt data center next to the three gorges dam and then

Leopold Aschenbrenner: you're off to the races i mean like there's a few different things right so one one threat model is just stealing the weights themselves and the weights one is sort of particularly insane right because they can just like steal the literal like end product right just like make a replica of the atomic bomb and then they're just like ready to go and um you know i think that one just is you know extremely important around the time we have agi and super intelligence right because it's um you know china can build a big cluster default we'd have a big lead right because we have the better scientists but we make the super intelligence they just steal it they're off to the races weights are a little bit less important right now um because you know who cares if they steal the gpd4 weights right like whatever um and um so you know we still have to get started on weight security now because you know look if we think agi by 27 you know this stuff is going to take a while And it's not just going to be like, oh, we do some access control. If you actually want to be resistant to sort of Chinese espionage, it needs to be much more intense. The thing, though, that I think people aren't paying enough attention to is the secrets, as you say. And... I think the compute stuff is sexy. We talk about it. But I think people underrate the secrets because I think the half an order of magnitude a year, just by default, sort of algorithmic progress, that's huge. If we have a few-year lead by default, that's 10, 30x, 100x bigger cluster if we protected them. And then there's this additional layer of the data wall. We have to get through the data wall. That means we actually have to figure out some sort of basic new paradigm, sort of the AlphaGo step two, right? AlphaGo step one is learns from human imitation. AlphaGo step two is the sort of self-play RL. And everyone's working on that right now. And maybe we're going to crack it. And, you know, if China can't steal that, then they're stuck. If they can't steal it, they're off to the races. But whatever that thing is... Is it like literally, I can write down on the back of a napkin?

Dwarkesh Patel: Because if it's that easy, then why is it that hard for them to figure it out? And if it's more about the intuitions, then don't you just have to hire Alec Bradford? Like, what are you copying down?

Leopold Aschenbrenner: Well, I think there's a few layers to this, right? So I think at the top is kind of like sort of the... you know, fundamental approach, right? And sort of like, I don't know, on pre-training, it might be, you know, like, you know, unsupervised learning, next token protection, train on the entire internet. You actually get a lot of juice out of that already. That one's very quick to communicate. Then there's like, there's a lot of details that matter. And you were talking about this earlier, right? It's like probably the way that thing people are going to figure out is going to be like somewhat obvious or there's going to be some kind of like clear, you know, not that complicated thing that'll work, but there's going to be a lot of details to getting that right.

Dwarkesh Patel: But if that's true, then again, why are we even,

Leopold Aschenbrenner: why do we think that getting state-level security in these startups will prevent China from catching up? If it's just like, oh, we know some sort of self-play RL will be required to get past the data wall.

Dwarkesh Patel: And if it's as easy as you say in some fundamental sense,

Leopold Aschenbrenner: but it's going to be solved by 2027, you say,

Dwarkesh Patel: right? So it's not that hard.

Leopold Aschenbrenner: I just think, you know, the US and the sort of, I mean, all the leading AI labs in the United States and they have this huge lead. I mean, by default, you know, China actually has some good LLMs. Why do they have good LLMs? They're just using the sort of open source code, right? You know, Lama or whatever. And so the, I think people really underrate the sort of, both the sort of divergence on algorithmic progress than the lead the US would have by default. Because all this stuff was published until recently, right? Like, chinchilla scaling laws were published. There's a bunch of MOE papers. There's transformers. And all that stuff was published. And so that's why open source is good. That's why China can make some good models. That stuff is now, I mean, at least they're not publishing it anymore. And if we actually kept it secret, it would be this huge edge. Um, to, to your point about sort of like some tacit knowledge now, like Bradford, you know, there's, there's another layer at the bottom that is something about like, you know, large scale engineering work to make these big training ones work. I think that is a little bit more tacit knowledge. So I think that, but I think China will be able to figure that out. That's like sort of engineering slap. They're going to figure out how to figure that out, but not how to get the RL thing working. Um, I mean, look, I don't know. Germany during World War II, they went down the wrong path. They did heavy water, and that was wrong. And there's an amazing anecdote in the making of the atomic bomb on this. So secrecy is actually one of the most contentious issues early on as well. And part of it was sort of you know, Zillard or whatever really thought, you know, this sort of nuclear chain reaction was possible. Um, and so an atomic bomb was possible and he went around and he was like, ah, this is going to be of enormous strategic importance, military importance. Um, and a lot of people didn't believe it or they're kind of like, well, maybe this is possible, but you know, I'm going to act as though it's not possible. And, you know, science should be open and all these things. And, um, Anyway, and so these early days, so there had been some sort of incorrect measurements made on graphite as a moderator and that Germany had. And so they thought graphite was not going to work. We have to do heavy water. But then Fermi made some new measurements on graphite, and they indicated that graphite would work. This is really important. Um, and then, you know, Zillard kind of assaulted Fermi with the kind of another secrecy appeal. And Fermi was just kind of, he was pissed off, you know, at a temper tantrum, you know, he was like, he thought it was absurd, you know, like, come on, this is crazy. Um, but you know, you know, Zillard persisted. I think they roped in another guy, Pegram, and then Fermi didn't publish it. And, you know, that was just in time because Fermi not publishing it meant that the Nazis didn't figure out graphite would work. They went down this path of heavy water and that was the wrong path. That was why this, you know, this is a key reason why the German project didn't work out. They were kind of way behind. And, you know, I think we face a similar situation on are we are we just going to instantly leak the sort of how do we get past the data wall? What's the next paradigm? Or are we not?

Dwarkesh Patel: so and the reason this would matter is if there's like being one year ahead would be a huge advantage in the world where it's like you deploy ai over time and they're just like ah they're gonna catch up anyway i i mean i interviewed richard rhodes the guy who wrote making an atomic bomb yeah and one of the anecdotes he had was when so they realized america had the bomb obviously we dropped it in japan yeah

Leopold Aschenbrenner: And Beria goes, the guy who ran the NKVD, which is a famously ruthless guy, just evil. And he goes to,

Dwarkesh Patel: I forgot the name of the guy, the Soviet scientist who was running their version of the Manhattan Project, who says, comrade, you will get us the American bomb. And the guy says, well, listen, their implosion device actually is not optimal. We should make it a different way. And Beria says, no, you will get us the American bomb or your family will be camp dust. But the thing that's, relevant about that anecdote is actually the Soviets would have had a better bomb if they hadn't copied the American design, at least initially. And which suggests that often in history, this is something that's not just for the Manhattan Project, but there's this pattern of parallel invention where, because the tech tree implies that the certain thing is next, in this case, a soft play, RL, whatever, then people are just like working on that. And like, people are going to figure out around the same time, there's not, there's not going to be that much gap in who gets it first. Um, wasn't like famously the bunch of people were invented something like the light bulb around the same time and so forth. Yeah. So, but is it just that like, yeah, that might be true, but it'll,

Leopold Aschenbrenner: with the one year or the six months or whatever, two years makes all the difference. I don't know if it'll be two years, though. This is a really difficult high school competition math problems. We were at a few percent, couldn't solve anything. Now it's solved. And that was at the normal pace of AI progress. You didn't have a billion super intelligent researchers. So a year is a huge difference. And then particularly after super intelligence, once this is applied to lots of elements of R&D, once you get the industrial explosion with the robots and so on, I think a couple of years might be decades worth of technological progress. Again, it's like Gulf War I, right? 20, 30 years of technological lead, totally decisive. I think it really matters. The other reason it really matters is, suppose they steal the weight, suppose they steal the algorithms, and they're close on our tails. Suppose we still pull out ahead, right? We're a little bit faster, we're three months ahead. I think the sort of, like, world in which we're really neck and neck, you know, we only have a three-month lead, are incredibly dangerous, right? And we're in this, like, feverish struggle where, like, if they get ahead, they get to dominate, you know, sort of, maybe they'd get a decisive advantage. They're building clusters like crazy. They're willing to throw all caution to the wind. We have to keep up. There's some crazy new WMDs popping up. And then we're going to be in the situation where it's, like, you know, crazy new military technology, crazy new WMDs, you know, like, deterrence and mutually disturbed instruction, like, keeps changing, you know, every few weeks, and it's a completely unstable, volatile situation. That is incredibly dangerous. So I think both from just the technologies are dangerous from the alignment point of view. I think it might be really important during the intelligence explosion to have this sort of six-month wiggle room to be like, look, we're going to dedicate more compute to alignment during this period because we have to get it right. We're feeling uneasy about how it's going. So I think in some sense, one of the most important inputs to whether we will kind of destroy ourselves or whether we will get through this just incredibly crazy period is whether we have that buffer. Why?

Dwarkesh Patel: So before we go further object level in this, I think it's very much worth noting. Yeah. that almost nobody, at least nobody I talk to, thinks about the geopolitical implications of AI. And I think I have some object level disagreements I will get into, or at least things I want to iron out. I may not disagree in the end. But the basic premise that obviously if you keep scaling and obviously if people realize that this is where intelligence is headed, it's not just going to be like the same old world where like what model are we deploying tomorrow and what is the latest? Like people on Twitter are like, oh, the GPT-4 is going to shake your expectations or whatever. You know, COVID is really interesting because before a year or something, when March 2020 hit, we, it became clear to the world, like, presidents,

Leopold Aschenbrenner: CEOs, media, average person, there's other things happening in the world right now, but the main thing we as a world are dealing with right now is COVID. Soon on AGI. Yeah. This is the quiet period. You want to go on vacation. Maybe now is the last time you can have some kids. My girlfriend sometimes complains that when I'm off doing work or whatever, she's like, I'm not spending time with her. She threatens to replace me with GPT-6 or whatever. And I'm like, GPT-6 will also be too busy doing AI research. Yeah.

Dwarkesh Patel: Okay, anyway, so what's the answer to the question of why aren't other people talking national security?

Leopold Aschenbrenner: I made this mistake with COVID, right? So I, you know, February of 2020, and I, you know, I thought just it was going to sweep the world and all the hospitals would collapse and it'd be crazy. And then, you know, and then it'd be over. Um, and a lot of people thought this kind of the beginning of COVID, they shut down their offices a month or whatever. I think the thing I just really didn't price in was the societal reaction. Right. And, and within weeks, you know, Congress spent over 10% of GDP on like COVID measures, right? The entire country was shut down. That was crazy. Um, and so I don't know, I didn't price it in with COVID sufficiently. Um, I don't know. Why do people underrate it? I mean, I think there's a sort of way in which being kind of in the trenches actually kind of, I think, gives you a less clear picture of the trend lines. You actually have to zoom out that much, only like a few years, right? But you're in the trenches. You're trying to get the next model to work. There's always something that's hard. For example, you might underrate algorithmic progress because you're like, ah, things are hard right now or data wall or whatever. But you zoom out just a few years and you actually try to count up how much algorithmic progress made in the last few years. And it's enormous. But I also just don't think people think about this stuff. I think smart people really underrate espionage. Right. And, you know, I think part of the security issue is I think people don't realize like how intense state level espionage can be. Right. Like, you know, you know, this Israeli company had had software that could just zero click hack any iPhone. Right. They just put in your number and then it's just like straight download of everything. Right. Like the United States infiltrated an air gapped atomic weapons program. Right. Wild. You know, like. Yeah. You know, the, you know, you know, intelligence agencies have just stockpiles of zero days, you know. when things get really hot, you know, I don't know, maybe we'll send special forces, right? To like, you know, get, go to the data center or something that's, you know, or, you know, I mean, China does this, they threaten people's families, right? And they're like, look, if you don't cooperate, if you don't give us the intel, um, uh, there's a, there's a good book, you know, along the lines of the, the Gulag, you know, the, um, inside the aquarium, um, which is by a Soviet GRU defector. Um, GRU was like military intelligence. Ilya recommended this book to me. And, you know, I think reading that, I was just kind of like shocked at how intense sort of state-level espionage is. The whole book was about like, they go to these European countries and they try to like get all the technology and recruit all these people to get the technology. I mean, yeah, maybe one anecdote, you know, so when, so the spy, you know, this eventual defector, you know, so he's being trained, he goes to the kind of GRU spy academy. And so then to graduate from the spy academy, sort of before you're sent abroad, you kind of had to pass a test to show that you can do this. And the test was, you know, you had to, in Moscow, recruit a Soviet scientist and recruit them to give you information, sort of like you would do in the foreign country. But of course, for whomever you recruited, the penalty for giving away sort of secret information was death. And so to graduate from the Soviet spy, this GRU spy academy, you had to condemn a countryman to death. States do this stuff.

Dwarkesh Patel: I started reading the book because I saw it in the series, and I was actually wondering, the fact that you use this anecdote, and then you're like,

Leopold Aschenbrenner: a book recommended by Ilya,

Dwarkesh Patel: is this some sort of Easter egg?

Leopold Aschenbrenner: We'll leave that for an exercise for the reader. Okay, so... The beatings will continue until the morale improves. So, suppose that we live in the world in which these secrets are locked down,

Dwarkesh Patel: but China still realizes that this progress is happening in America. So, in that world, Especially if they realize, and I guess it's a very interesting open question.

Leopold Aschenbrenner: I mean, the secrets probably won't be locked down. Okay. We're probably going to live in the bad world.

Dwarkesh Patel: Yeah.

Leopold Aschenbrenner: It's going to be really bad. Why are you so confident that they won't be locked down? I mean, I'm not confident that it won't be locked down, but I think it's just, it's not happening.

Dwarkesh Patel: And so tomorrow, the lab leaders get the message that,

Leopold Aschenbrenner: how hard, like, what do they have to do? They get the more security guards? They like air gap the, what do they do? So again, I think basically it's, you know, I think people, there's kind of like two reactions there, which is like, it's, you know, we're already secure, you know, not. And there's, you know, fatalism, it's impossible. And I think the thing you need to do is you kind of got to stay ahead of the curve of basically how EGI appeals to the CCP, right? So like right now, you've got to be resistant to kind of like normal economic espionage, right? They're not right. I mean, I probably wouldn't be talking about the stuff that the labs were right. Cause I wouldn't want to wake them up more, the CCP, but they're not, you know, this is like, this stuff is like really trivial for them to do right now. I mean, it's also, anyway, so they're not resistant to that. I think it would be possible for private company to be resistant to it. Right. So, you know, both of us have, you know, friends in the kind of like quantitative trading world. Right. And, and, you know, I think actually those secrets are shaped kind of similarly where it's like, you know, you know, they've said, you know, yeah, if I got on a call for an hour with somebody from a competitive firm, I could, most of our alpha would be gone. And that's sort of like, that's the like list of details of like really how to make an hour. You're gonna have to worry about that pretty soon. You're gonna have to worry about that pretty soon. Yeah, yeah, yeah. Well, anyway, and so all the alpha could be gone. But in fact, they're off a persists, right. And, you know, often, often for many years and decades. And so this doesn't seem to happen. And so I think there's like, you know, I think there's a lot you could go if you went from kind of current startup security, you know, you just got to look through the window and you can look at the slides, you know, it's, it's, it's kind of like, you know, you know, good private sector security hedge funds, you know, the way Google treats, you know, customer data or whatever. Um, that'd be good right now. The issue is, you know, basically the CCP will also get more AGI build and At some point, we're going to face kind of the full force of, you know, the Ministry of State Security. And again, you're talking about smart people underwriting espionage and sort of insane capabilities of states. I mean, this stuff is wild, right? You know, they can get like, you know, there's papers about, you know, you can find out the location of like where you are on a video game map just from sounds, right? Like states can do a lot with like electromagnetic emanations, you know, like, you know, at some point, like you got to be working from a scuff, like your cluster needs to be air gapped and basically be a military base. It's like, you know, you need to have, you know, intense kind of security clearance procedures for employees. You know, they have to be like, you know, all this shit is monitored, you know, they're, you know, they basically have security guards, you know, it's, you know, you, you can't use any kind of like, you know, other dependencies. It's all gotta be like intensely vetted and, you know, all your hardware has to be intensely vetted. Um, and, You know, I think basically if they actually really face the full force of state-level espionage, I don't really think this is a thing private companies can do. Both, I mean, empirically, right? Like, you know, Microsoft recently had executives' emails hacked by Russian hackers and, you know, government emails they've hosted hacked by government actors. But also... You know, it's basically, there's just a lot of stuff that only kind of, you know, the people behind the security currencies know and only they deal with. And so, you know, I think to actually kind of resist the sort of full force of espionage, you're going to need the government. Anyway, so I think basically we could do it by always being ahead of the curve. I think we're just going to always be behind the curve. And I think, you know, maybe unless we get the sort of government project.

Dwarkesh Patel: Okay, so going back to the naive perspective of we're very much coming at this from there's going to be a race and the CCP, we must win. Yeah. Um, and listen, I understand, like, bad people are in charge of the Chinese government, like, the CCP and everything.

Leopold Aschenbrenner: Um, but just stepping back in a sort of galactic perspective, humanity is developing AGI. And do we want to come at this from the perspective of, we need to feed China to this?

Dwarkesh Patel: Our super intelligent Jupiter brain descendants won't know who China, like... China will be something like distant memory that they have, America too. So shouldn't it be a more, the initial approach just come to them like, listen, this is super intelligence. This is something like we come from a cooperative perspective. Why immediately sort of rush into it from a hawkish competitive perspective?

Leopold Aschenbrenner: I mean, look, I mean, one thing I want to say is like a lot of the stuff I talk about in the series is, you know, is sort of primarily, you know, descriptive, right? And so I think that on the China stuff, it's like, you know, Yeah, in some ideal world, you know, we, you know, it's just all, you know, merry-go-round and cooperation. But again, it's sort of, I think, I think people wake up to AGI. I think the issue particular on sort of like, can we make a deal? Can we make an international treaty? I think it really relates to sort of what is the stability of sort of international arms control agreements, right? And so we did very successful arms control on nuclear weapons in the 80s, right? And the reason it was successful is because the sort of new equilibrium was stable, right? So you take, go down from, you know, whatever, 60,000 nukes to 10,000 nukes. When you have 10,000 nukes, basically breakout doesn't matter that much. Suppose the other guy now tried to make 20,000 nukes. Well, it's like, who cares? It's still mutually assured destruction. Suppose a rogue state went from zero nukes to one nukes. It's like, who cares? We still have way more nukes than you. I mean, it's still not ideal for destabilization, but it'd be very different if the arms control agreement had been zero nukes. Because if it had been zero nukes, then it's just like one rogue state makes one nuke. The whole thing is destabilized. Breakout is very easy. you know, your adversary state starts making nukes. And so basically when you're going to sort of like very low levels of arms or when you're going to kind of, in your sort of very dynamic technological situation, arms control is really tough because breakout is easy. You know, there's, I mean, there's some other sort of stories about this in sort of like 1920s, 1930s. You know, it's like, you know, all the European states have done disarmament and Germany was kind of did this like crash program to build the Luftwaffe. And that was able to like massively destabilize things because not that, you know, they were the first, they were able to like pretty easily build kind of a modern, you know, air force because the others didn't really have one. And that, you know, that really destabilized things. And so I think the issue with EGI and super intelligence is the explosiveness of it, right? So if you have an intelligence explosion, if you're able to go from kind of EGI to super intelligence, if that super intelligence is decisive, like either, you know, like a year after, because you've developed some crazy new WMD or because you have some like, you know, super hacking ability that lets you, you kind of, you know, completely deactivate the sort of enemy arsenal. Um, that means like, suppose, suppose you're trying to like put in a break, you know, like we both, we're both going to like cooperate and we're going to go slower, you know, on the cusp of AGI or whatever, you're always going to be such an enormous incentive to kind of race ahead to break out. And we're just going to do the intelligence explosion. If we can get three months ahead, we win. Um, I think that makes it basically, I think any sort of arms control agreement that comes as a situation where it's close, very unstable. That's really interesting. This is very analogous to, um, kind of a debate I had with Rose on the podcast, where he argued for nuclear disarmament. But if some country tries to break out and starts developing nuclear weapons,

Dwarkesh Patel: the six months or whatever that you would get is enough to get international consensus and invade the country and prevent them from getting nukes. And I thought that was sort of... That's not a stable equilibrium.

Leopold Aschenbrenner: It just seems really tough, yeah. Um... On this, right?

Dwarkesh Patel: So maybe it's a bit easier because you have AGI and so you can monitor the other person's cluster or something. Data centers, you can see them from space, actually. You can see the energy draw they're getting. There's a lot of things, as you were saying, there's a lot of ways to get information from an environment if you're really dedicated. And also because unlike a nuke, the data centers are Nukes, you have obviously the submarines, planes, you have bunkers, mountains, whatever. You have them so many different places. A data center, your 100 gigawatt data center, we can blow that shit up if we're concerned, right?

Leopold Aschenbrenner: Like just some cruise missile or something. But it's very vulnerable to sabotage. That gets to the sort of insane vulnerability of this period post-superintelligence. So you have the intelligence explosion. You have these vastly superhuman things on your cluster. But you haven't done the industrial explosion yet. You don't have your robots yet. You haven't covered the desert in robot factories yet. And that is the sort of crazy moment where, say the United States is ahead, the CCP is somewhat behind. There's actually an enormous incentive for a first strike, right? Because if they can take out your data center, they know you're about to have just this command and decisive lead. They know if we can just take out this data center, then we can stop it. And they might get desperate. And so I think basically we're going to get into a position. It's actually, I think it's going to be pretty hard to defend early on. I think we're basically going to be in a position where we're protecting data centers with like the threat of nuclear retaliation. It's like, maybe it sounds kind of crazy though, you know. Is this the inverse of the Eleazar? We got to... the data centers in New Orleans. Nuclear deterrence for data centers. I mean, this is Berlin, you know, in the late 50s, early 60s. Both Eisenhower and Kennedy multiple times kind of made the threat of full-on nuclear war against the Soviets if they tried to encroach on West Berlin. It's sort of insane. It's kind of insane that that went well. But basically, I think that's going to be the only option for the data centers. It's a terrible option. This whole scheme is terrible, right? Like, being in this, like, neck-and-neck race sort of at this point is terrible. And, you know, it's also, you know, I think I have some uncertainty basically on how easy that decisive advantage will be. I'm pretty confident that if you have super intelligence, you have two years, you have the robots, you're able to get that 30 year lead. Look, then you're in this like go for one situation. You have your like, you know, millions or billions of like mosquito sized drones that can just take it out. I think there's even a possibility you can kind of get a decisive advantage earlier. So, you know, there's these stories, you know, about these as well, about, you know, like colonization and like the sort of 1500s where it was, you know, these like a few hundred kind of Spaniards were able to like topple the Aztec empire, you know, a couple, I think a couple other empires as well. You know, each of these had a few million people and it was not like godlike technological advantage. It was some technological advantage. It was, I mean, it was some amount of disease and then it was kind of like cunning strategic play. And so I think there's a possibility that even sort of early on, You haven't gone through the full industrial explosion yet. You have super intelligence, but you're able to manipulate the imposing generals, claim you're allying with them. Then you have some crazy new bioweapons. Maybe there's even some way to pretty easily get a paradigm that deactivates enemy nukes. Anyway, so I think this stuff could get pretty wild. Here's what I think we should do. I really don't want this volatile period. And so a deal with China would be nice. It's going to be really tough if you're in this unstable equilibrium. I think, basically, we want to get in a position where it is clear that the United States, that a sort of coalition of democratic allies will win. It's clear the United States is clear to China. That will require having locked down the secrets. That will require having built the 100 gigawatt cluster in the United States and having done the natural gas and doing what's necessary. And then when it is clear that the democratic coalition is well ahead, then you go to China and then you offer them a deal. And China will know they're going to win. This is going to be they're very scared of what's going to happen. We're going to know we're going to win, but we're also very scared of what's going to happen because we really want to avoid this kind of like breakneck race right at the end and where things could really go awry. And so then we offer them a deal. I think there's an incentive to come to the table. I think there's a sort of more stable arrangement you can do. It's a sort of an Atoms for Peace arrangement. And we're like, look, we're going to respect you. We're not going to use super intelligence against you. You can do what you want. You're going to get your slice of the galaxy. We're gonna benefit share with you. We're gonna have some like compute agreement where it's like there's some ratio of compute that you're allowed to have and that's like enforced with her like opposing AIs or whatever. And we're just not gonna do, we're just not gonna do this kind of like volatile sort of WMD arms race to the death. And sort of it's like a new world order that's US led, that's sort of democratic led, but that respects China, lets them do what they want. Okay, there's so much to, there's so much there.

Dwarkesh Patel: First on the galaxies thing. I think it's just a funny anecdote, so I kind of want to tell it. We were at an event, and I'm respecting Chatham House rules here. I'm not revealing anything about it, but we were talking to somebody, or Leopold was talking to somebody influential. Afterwards, that person asked the group, Leopold told me that he wants, he's not going to spend any money on consumption until he's ready to buy galaxies. And he goes, the guy goes,

Leopold Aschenbrenner: I honestly don't know if he meant galaxies like the brand of private plane galaxy or the physical galaxies. And there was an actual debate. He went away to the restroom and there was an actual debate among people who are very influential about, he can't have meant galaxies.

Dwarkesh Patel: And the other people who knew you better would be like, no, he means galaxies. I mean the galaxies.

Leopold Aschenbrenner: I mean, I think it'll be interesting. I mean, I, yeah, I think there's a, I mean, there's two ways to buy the galaxies. One is like at some point, you know, it's like post-superintelligence, you know, there's some crazy- But by the way,

Dwarkesh Patel: I love, okay, so what happens is he's otherwise,

Leopold Aschenbrenner: I'm laughing my ass off. I'm not even saying anything. People are like gaffing this debate. And then Leopold comes back and somebody's like, oh, Leopold, we're having this debate about whether you meant you want to buy the galaxy or you want to buy the other thing. And Leopold assumes they must mean not the priority of playing the galaxy versus

Dwarkesh Patel: the actual galaxy,

Leopold Aschenbrenner: but do you want to buy the property rights of the galaxy or actually just send out the probes right now?

Speaker 4: Exactly, exactly.

Dwarkesh Patel: Oh, my God. All right. Back to China. There's a whole bunch of things I could ask about that plan,

Leopold Aschenbrenner: about whether you're going to get credible, promised, you will get some part of galaxies, whether they care about that. I mean, you have AIs to help you enforce stuff.

Dwarkesh Patel: Okay, sure.

Leopold Aschenbrenner: We'll leave that aside.

Dwarkesh Patel: That's a different rabbit hole.

Leopold Aschenbrenner: The thing I want to ask is... But it has to be the thing we need. The only way this is possible is if we lock it down.

Dwarkesh Patel: I see.

Leopold Aschenbrenner: If we don't lock it down, we are in this fever struggle greatest peril mankind will have ever seen.

Dwarkesh Patel: So, but given the fact that during this period,

Leopold Aschenbrenner: instead of just taking their chances and they don't really understand how this AI governance scheme is going to work, where they're going to check, whether we actually get the galaxies.

Dwarkesh Patel: Yeah, yeah. The data centers, they can't be built underground. They have to be built above ground. Taiwan is right off the coast of us. They need the chips from there. Why aren't we just going to invade? Listen, we don't want like worst case scenario is they win the super intelligence, which they're on track to do anyways. Wouldn't this instigate them to either invade Taiwan or blow up the data center in Arizona or something like that?

Leopold Aschenbrenner: Yeah, I mean, look, I mean, you talked about the data center one, and then, you know, you probably have to, like, threaten nuclear retaliation to protect that. They might also just blow it up. There's also maybe ways they can do it without sort of attribution, right? Like you pay... Stuxnet. Stuxnet. Yeah,

Speaker 2: I mean, look, this is,

Leopold Aschenbrenner: I mean, this is part of, we'll talk about this later, but, you know, I think, look, I think we need to be working on the Stuxnet for the Chinese project. But the... But by the way,

Dwarkesh Patel: for the audience...

Leopold Aschenbrenner: I mean, Taiwan, the Taiwan thing, the, you know, you know, I talk about, you know, AGI by, you know, 27 or whatever. Do you know about the, like, terrible 20s? No. Okay. Well, I mean, sort of in this sort of Taiwan watcher circles, people often talk about like the late 2020s as like maximum period of risk for Taiwan because sort of like, you know, military modernization cycles and basically extreme fiscal tightening on the military budget in the United States over the last decade or two has meant that sort of we're in this kind of like, you know, trough in the late 20s of like, you know, basically overall naval capacity is And, you know, that's sort of when China is saying they want to be ready. So it's already kind of like it's kind of pitching, you know, some sort of like, you know, parallel timeline there. Yeah, look, it looks appealing to invade Taiwan. I mean, maybe not because they, you know, basically remote cut off of the chips. And so then it doesn't mean they get the chips, but it just means they they, you know, it's just it's, you know, the machines are deactivated. But look, I mean, imagine if during the Cold War, you know, all of the world's uranium deposits had been in Berlin. You know, and Berlin was already, I mean, almost multiple times when it was causing nuclear war. So, um, God help us all.

Dwarkesh Patel: Well, the Groves had a plan after the war that the plan was that America would go around the world and... getting the rights to every single uranium deposit because they didn't realize how

Leopold Aschenbrenner: much uranium there was in the world and they thought this was the thing that was feasible not realizing of course that there's like huge deposits in the soviet union itself right um um okay east germany too there's a there's always there's a lot of east german workers who kind of got screwed oh interesting got cancer yeah

Dwarkesh Patel: Okay, so the framing we've been talking about, that we've been assuming, and I'm not sure I buy yet, is that the United States, this is our leverage, this is our data center, China is the competitor. Right now, obviously, that's not the way things are progressing. Private companies control these AIs. They're deploying them. It's a market-based thing. Why will it be the case that it's like the United States has this leverage or is doing this thing versus China is doing this thing?

Leopold Aschenbrenner: Yeah, I mean, look, on the project, there's sort of descriptive and prescriptive claims or sort of normative positive claims. I think the main thing I'm trying to say is, look, we're at these SF parties or whatever, and I think people talk about AGI, and they're always just talking about the private AI labs. And I think I just really want to challenge that assumption. It just seems like... seems pretty likely to me, you know, as we've talked about, for reasons we've talked about, that looks like the national security state is going to get involved. And, you know, I think there's a lot of ways this could look like, right? Is it like nationalization? Is it a public-private partnership? Is it a kind of defense-contra-elector-like relationship? Is it a sort of government project that soaks up all the people? And so there's a spectrum there. But I think people are just vastly underrating the chances of this more or less looking like a government project. And look, I mean, look, if... You know, it's sort of like, you know, do you think, do you think like we all have literal, like, you know, when we have like literal super intelligence on our cluster, right? And it's like, you know, you have a hundred billion, they're like, sorry, you have a billion like super intelligent scientists that they can like hack everything. They can like stuxnet the Chinese data centers. You know, they're starting to build the robo armies. You know, you like, you really think there'll be like a private company and the government would be like, oh my God, what is going on? You know, like... Yeah.

Dwarkesh Patel: Suppose there's no China. Suppose there's people like Iran, North Korea, who theoretically at some point will be able to do super intelligence, but they're not on our heels or don't have the ability to be on our heels. In that world, are you advocating for the national project or do you prefer the private path forward?

Leopold Aschenbrenner: Yeah, so I mean, two responses to this. One is, I mean, you still have like Russia, you still have these other countries. You know, you've got to have Russia-proof security, right? It's like you can't just have Russia steal all your stuff. And like maybe their clusters aren't going to be as big, but like they're still going to be able to make the crazy bioweapons and the, you know, the mosquito-sized drone swarm, you know, and so on. And... So, I mean, I think the security component is just actually a pretty large component of the project in the sense of, like, I currently do not see another way where we don't kind of, like, instantly proliferate this to everybody. And so, yeah, so I think it's sort of like you still have to deal with Russia, you know, Iran, North Korea, and, you know, like, you know, Saudi and Iran are going to be trying to get it because they want to screw each other. And, you know, Pakistan and India, because they want to screw each other. There's like this enormous destabilization still. That said, look, I agree with you. If, you know, if, you know, by some somehow things are shaking out differently. I'm like, you know, AGI would have been in 2005, you know, sort of like unparalleled in American hegemony. I think there would have been more scope for. less government involvement. But again, you know, as we were talking about earlier, I think that would have been sort of this, like, very unique moment in history. And I think basically, you know, almost all other moments in history, there would have been this sort of great power competitor.

Dwarkesh Patel: So, okay, so let's get into this debate. So, my position here is, if you look at the people who were involved in the Manhattan Project itself, many of them regretted their participation, as you said. Now, we can infer from that that we should sort of start off with a cautious approach to the nationalized ASI project. Then you might say, well, listen, obviously...

Leopold Aschenbrenner: Did they regret their participation because of the project or because of the technology itself? I think people will regret it, but I think it's about the nature of the technology and it's not about the project.

Dwarkesh Patel: I think they also probably had a sense that different decisions would have been made if it wasn't some concerted effort that everybody had agreed to participate in. That if it wasn't in the context of this, we need to race to beat Germany and Japan, you might not develop. So that's the technology part, but also like you wouldn't actually like hit them.

Leopold Aschenbrenner: You know, it's like the sort of the destructive potential, the sort of military potential. It's not it's not because of the project. It is because of the technology. And that will unfold regardless. But I think this underrates the power of modeling. Imagine you go through the 20th century in a decade. It's just the sort of, yes, great technological progress.

Dwarkesh Patel: Let's just actually run to that example. Suppose there was some reason that the 20th century would be run through in one decade. Do you think the cause of that should have been, the technologies that happened through the 20th century shouldn't have been privatized? that it should have been a more sort of concerted government-led project?

Leopold Aschenbrenner: You know, look, there is a history of just dual-use technologies, right? And so I think AI, in some sense, is going to be dual-use in the same way. And so there's going to be lots of civilian uses of it, right? Like nuclear energy itself, right? There's the government project developed the military angle of it. And then it was like, then the government worked with private companies. There's a sort of real flourishing of nuclear energy until the environmentalists stopped it. You know, planes, right? Like Boeing. Actually, you know, the Manhattan Project wasn't the biggest defense R&D project during World War II. It was the B-29 bomber, right? Because they needed the bomber that had long enough range to reach Japan to destroy their cities. And then, you know, Boeing made some, Boeing made that, Boeing made the B-47, made the B-52, you know, the plane the U.S. military uses today. And then they used that technology later on to destroy to, you know, build the 707 and sort of the... But what does later on mean in this context? Because in the other... Like, I get what it means after a war to privatize. But if you have... The government has ASI. Maybe just let me back up and explain my concern. Yeah.

Dwarkesh Patel: So you have the only institution in our society which has a monopoly on violence. Yes. And then we're gonna give it some, in a way that's not broadly deployed, access to the ASI.

Leopold Aschenbrenner: The counterfactual, and this maybe sounds silly, but listen, we're gonna go through higher and higher levels of intelligence. Private companies will be required by regulation to increase their security,

Dwarkesh Patel: but they'll still be private companies and they'll deploy this and they're gonna release the AGI. Now McDonald's and JP Morgan and some random startup are now more effective organizations because they have a bunch of AGI workers. And it'll be sort of like the Industrial Revolution in the sense that the benefits were widely diffused. If you don't end up in a situation like that, then the... I mean, even backing up, like, what is it we're trying to... Why do we want to win against China? We want to win against China because we don't want a top-down authoritarian system to win. Now, if the way to beat that is that the most important technology that humanity will have has to be controlled by a top-down government, what was the point? So let's run our cards with privatization. That's the way we get to the classic liberal market-based system we want for the ESIs.

Leopold Aschenbrenner: All right, so a lot of talk about here. Maybe I'll start a bit about actually looking at what the private world would look like. And I think this is part of where the there's no alternative comes from. And then let's look at what the government project looks like, what checks and balances look like, and so on. All right, private world. I mean, first of all, OK, so a lot of people right now talk about open source. And I think there's this sort of misconception that AGI development is going to be like, oh, it's going to be some beautiful, decentralized thing and some giddy community of coders who gets to collaborate on it. That's not how it's going to look like, right? hundred billion dollar trillion dollar cluster it's not going to be that many people that have it the algorithms you know it's like right now open source is kind of good because people just use the stuff that was published and so they basically you know the algorithms were published or you know as mistral they just kind of like leave deep mind and you know take all the secrets with them and they just kind of replicate it um um but that's not going to continue being the case and so you know the sort of like open source alternative i mean also people say stuff like you know 1026 flops it'll be in my phone or you know it's No, it won't. You know, it's like Moore's law is really slow. I mean, AI chips are getting better, but like, you know, the a hundred billion dollar computer will not cost, you know, like a thousand dollars, you know, within your lifetime or whatever, aside from me. So anyway, so it's going to be, it's going to be like two or three, you know, big players on the private world. And so look, a few things. So first of all, You know, you talk about the sort of like, you know, enormous power that sort of super intelligence will have and that the government will have. I think it's pretty plausible that the alternative world is that like one AI company has that power. Right. And it's basically, if we're talking about lead, you know, it's like, what, I don't know, open AI has a six month lead. And then, you know, so then you're not talking, you're talking about basically, you know, the most powerful weapon ever. And it's, you know, you're kind of making this like radical bet on like a private company CEO is the benevolent dictator. No, no, not necessarily. Any other thing that's privatized, we don't count on that being benevolent.

Dwarkesh Patel: Look, think of, for example, somebody who manufactures industrial fertilizer. The person with this factory, if they went back to an ancient civilization, they could blow up Rome. It could probably blow up Washington, D.C. And I think in their series, you talk about Tyler Cowen's phrase of muddling through. And I think even with privatization, people sort of underrate that there are actually a lot of private actors who have the ability to like... There's a lot of people who control the water supply or whatever. And we can count on cooperation and market-based incentives to basically keep a balance of power.

Leopold Aschenbrenner: I get that things are proceeding really fast. But we have a lot of historical evidence that this is the thing that works best. So look, I mean... I mean, what do we do with nukes, right? The way we keep the sort of nukes in check is not like, you know, a sort of beefed up Second Amendment where like each state has their own like little nuclear arsenal and like, you know, Dario and Sam have their own little nuclear arsenal. No, no, it's like it's institutions, it's constitutions, it's laws, it's courts. And so I don't actually, I'm not sure that this, you know, I'm not sure that the sort of balance of power analogy holds. In fact, you know, sort of the government having the biggest guns was sort of like an enormous civilizational achievement, right? Like Landfrieden in the sort of Holy Roman Empire, right? You know, if somebody from the town over kind of committed a crime on you, you know, you didn't kind of start a sort of a, you know, a big battle between the two towns. No, you take it to a court of the Holy Roman Empire and they would decide. And it's a big achievement. Now, the thing about, you know, the industrial fertilizer, I think the key difference is kind of speed and often defense balance issues, right? So it's like 20th century and, you know, 10 years and a few years. that is an incredibly scary period. And it is incredibly scary, you know, cause it's, you know, you're going through just this sort of enormous array of destructive technology and the sort of like enormous amount of like, you know, basically military event. I mean, you would have gone from, you know, kind of like, you know, you know, bayonets and horses to kind of like tank armies and fighter jets in like a couple of years. And then from, you know, like, you know, and then to like, you know, nukes and, you know, ICBMs and stuff, you know, and just like in a matter of years. And so, um, It is sort of that speed that creates, I think basically the way I think about it is there's going to be this initial just incredibly volatile, incredibly dangerous period. And somehow we have to make it through that. And that's going to be incredibly challenging. That's where you need the kind of government project. If you can make it through that, then you kind of go to like, you know, now we can, now, you know, the situation has been stabilized. You know, we don't face this imminent national security threat. You know, it's like, yes, there were kind of WMDs that came along the way, but either we've managed to kind of like have a sort of stable offense defense balance, right? Like I think bioweapons initially are a huge issue, right? Like an attacker can just create like a thousand different synthetic viruses and spread them. And it's going to be really hard for you to make a defense against each. But maybe at some point you figure out the universal defense against every possible virus. And then you're in a stable situation again on the offense-defense balance. Or you do the thing you do with planes, where there's certain capabilities that the private sector isn't allowed to have. And you've figured out what's going on, restrict those. And then you can let the civilian uses run free. So I'm skeptical of this because... Well, sorry, I mean, the other important thing is so I talked about the sort of, you know, maybe it's like it's it's a you know, it's you know, it's one company with all this power. And I think it's like I think it is unprecedented because it's like the industrial fertilizer guy cannot overthrow the U.S. government. I think it's quite plausible that like the AI company with super intelligence can

Dwarkesh Patel: overthrow the multiple companies.

Leopold Aschenbrenner: Right. And I buy that one of them could be ahead. So it's not obvious that it'll be multiple. I think it's, again, if there's like a six-month lead, maybe there's two or three. I agree. But if there's two or three, then what you have is just like a crazy race between these two or three companies. It's like whatever, Demis and Sam, they're just like, I don't want to let the other one win. And they're both developing their nuclear arsenals in the robot. It's just also like, come on, the government is not going to let these people. Are they going to let, is Dario going to be the one developing the kind of super hacking Stuxnet and deploying against the Chinese data center? The other issue, though, is it won't just... If it's two or three, it won't just be two or three. There'll be two or three, and it'll be China and Russia and North Korea. Because in the private lab world, there's no way they'll have security that is good enough.

Dwarkesh Patel: I think we're also assuming that somehow if you nationalize it, like, the security just... Especially in the world where this stuff is priced in by the CCP, that now you've, like, got it nailed down. And I'm not sure why we would expect that to be the case.

Leopold Aschenbrenner: But on this... The government's the only one who does this stuff.

Dwarkesh Patel: So... If it's not Sam or Dario, we don't trust them to be a benevolent dictator or whatever.

Leopold Aschenbrenner: But here we're counting on... Because you can cause a coup, the same capabilities are going to be true of the government project,

Dwarkesh Patel: right? And so the modal president in 2025, Donald Trump, will be the person that you don't trust Sam or Dario to have these capabilities. And why... okay, I agree that I'm worried if Sam or Dario have a one-year lead on ASI in that world, then I'm concerned about this being privatized. But in that exact same world, I'm very concerned about Donald Trump having the capability. And potentially, if we're living in a world where the takeoff is slower than you anticipate, in that world, I'm like, very much I want the private companies. So in no part of this matrix is it obviously true that the government-led project is better than the private project.

Leopold Aschenbrenner: Let's talk about the government project a little bit and checks and balances. In some sense, I think my argument is a sort of Birkin argument, which is like American checks and balances have held for, you know, over 200 years and through crazy technological revolutions. You know, the U.S. military could kill like every civilian in the United States.

Dwarkesh Patel: You're going to make that argument. The private public balance of power has held for hundreds of years.

Leopold Aschenbrenner: But yeah, why has it held? Because the government has the biggest guns. and has never before has a single CEO or a random nonprofit board had the ability to launch nukes. And so again, it's like, you know, what is the track record of the government checks and balances versus the track record of the private company checks and balances? Well, the iLab, you know, like first stress test, you know, went really badly, you know, that didn't really work, you know? I mean, even worse in the sort of private company world. So it's both like, it is not just the two, it is like the two private companies and the CCP, and they just like instantly have all the shit. And then it's, you know, they probably won't have good enough internal control. So it's like, not just like the random CEO, but it's like, you know, rogue employees that can kind of like use these super intelligences to do whatever they want. And this won't be true of the government? Like the rogue employees won't exist on the project? Well, the government actually has decades of experience and actually really cares about this stuff. I mean, it's like they deal with nukes. They deal with really powerful technology. And this is the stuff that the national security state cares about. Let's talk about the government checks and balances a little bit. So what are checks and balances in the government world? First of all, I think it's actually quite important that you have some amount of international coalition. And I talked about these sort of two tiers before. Basically, I think the inner tier is a sort of model on the Quebec Agreement, right? This was like Churchill and Roosevelt. They kind of agreed secretly, we're going to pull our efforts on nukes, but we're not going to use them against each other, and we're not going to use them against anyone else with their consent. And I think basically, look, bring in bringing the UK, they have DeepMind, bringing in the kind of like Southeast Asian states who have the chip supply chain, bringing in some more of kind of like NATO, close democratic allies for talent and industrial resources. And you have this sort of like, you know, so you have those checks and balances in terms of like more international countries at the table. Sorry, somewhat... separately, but then you have the sort of second tier of coalitions, which is the sort of atoms for peace thing, where you go to a bunch of countries, including like the UAE, and you're like, look, we're going to basically like, you know, there's a deal similar to like the NPT stuff where it's like, you're not allowed to like do the crazy military stuff, but we're going to share the civilian applications. We're in fact going to help you and share the benefits and, you know, sort of kind of like this new sort of post superintelligence world order. All right, U.S. checks and balances, right? So obviously Congress is going to have to be involved, right? Appropriate in trillions of dollars. I think probably ideally you have Congress needs to kind of like confirm whoever's running this. So you have Congress, you have like different factions of the government, you have the courts. I expect the First Amendment to continue being really important. And maybe that, I think that sounds kind of crazy to people, but I actually think, again, I think these are like institutions that have withheld this test of time in a really sort of powerful way. You know, eventually, you know, this is why, honestly, alignment is important is like, you know, the AIs, you program the AIs to follow the constitution. And it's like, you know, why does the military work? It's like generals, you know, are not allowed to follow unlawful orders. They're not allowed to follow unconstitutional orders. You have the same thing for the AIs.

Dwarkesh Patel: So what's wrong with this argument where you say, listen, maybe you have a point in the world where we have extremely fast takeoff. It's like one year from AGI to ASI. Yeah.

Leopold Aschenbrenner: And then you have the like eight years after ASI where you have this like extraordinary explosion of technological progress.

Dwarkesh Patel: Maybe you have a point. We don't know. You have these arguments. We'll like get into the weeds on them about why that's a more likely world. But like maybe that's not the world we live in. And in the other world, I'm very on the side of making sure that these things are privately held. Now, when you nationalize, that's a one-way function. You can't go back. Why not wait until we have more evidence on which of those worlds we live in? I think rushing on the nationalization might be a bad idea while we're not sure. Okay, I'll let you respond to that first.

Leopold Aschenbrenner: I mean, I don't, I don't expect us to nationalize tomorrow. If anything, I expect it to be kind of with COVID where it's like kind of too late. Like ideally you nationalize it early enough to like actually lock stuff down. It'll probably be kind of chaotic. And like, you know, you're going to be trying to like do this crash program to lock stuff down and it'll be kind of late. It'll be kind of clear what's happening. We're not going to nationalize when it's not clear what's happening.

Dwarkesh Patel: I think the whole historically these institutions have held up well. First of all, they've actually almost broken a bunch of times. This is similar to the argument that some people who are saying that we shouldn't be that concerned about nuclear war say, where it's like, listen, we have the nuke for 80 years. And we've been fine so far, so the risk must be low. And then the answer to that is no, actually, it is a really high risk. And the reason we've avoided it is people have gone through a lot of effort to make sure that this thing doesn't happen. I don't think that giving government ASI without knowing what that implies is going through a lot of effort. And I think the base rate, like you can talk about America. I think America is very exceptional, not just in terms of dictatorship, but in terms of every other country in history has had a complete drawdown of wealth because of war, revolution, and something. America is very unique in not having that. And the historical base rate, we're talking about great power competition. I think that has a really big... That's something we haven't been thinking about the last 80 years, but it's really big. Dictatorship is also something that is just the default state of mankind. And I think relying on... institutions which, in an ASI world, like, there's a... Fundamentally, right now, if the government tried to overthrow, there's a... It's much harder if you don't have the ASI, right? Like, there's people who have AR-15s, and I... There's, like, things that make it harder.

Leopold Aschenbrenner: The government could crush the AR-15s. No, I think it'd actually be pretty hard. The reason it was Vietnam and Afghanistan were pretty hard. They just nuked the whole country.

Dwarkesh Patel: Yeah, yeah, I agree, but, like, I'm... They could. I agree.

Leopold Aschenbrenner: I mean, similar with ASI. Um, yeah, I think it's just, like, easier if you have what you were talking about.

Dwarkesh Patel: But there are institutions,

Leopold Aschenbrenner: there are constitutions, there are legal restraints, there are courts, there are checks and balances. The crazy bet is the bet which are, like, private company CEOs.

Dwarkesh Patel: The same thing, by the way,

Leopold Aschenbrenner: isn't the same thing true of nukes where we have these institutional agreements

Dwarkesh Patel: about nonproliferation and whatever,

Leopold Aschenbrenner: and we're still very concerned about that being broken and somebody getting nukes and, like, you should stay up at night worrying about that. That's a precarious situation. But ASI is going to be a really precarious situation as well. And like, given how precarious nukes are, we've done pretty well. And so what does privatization in this world even mean? I mean, I think the other thing is- Like what happens after? I mean, the other thing, you know, because we're talking about like whether the government project is good or not. And it's like, I have very mixed feelings about this as well. Again, I think my primary argument is like, you know, if you're at the point where this thing has like vastly superhuman hacking capabilities, if you're at the point where this thing can develop, you know, bioweapons, you know, like in crazy bioweapons, ones that are like targeted, you know, can kill everybody but the Han Chinese or, you know, that, you know, would wipe out, you know, entire countries where you're talking about like building robo-armors, you're talking about kind of like drone swarms that are, you know, again, the mosquito-sized drones that could take it out, you know, the United States national security state is going to be intimately involved with this. And I think, again, a lot of what I think is the government project looks like, it is basically a joint venture between the cloud providers, between some of the labs and the government. And so I think there is no world in which the government isn't intimately involved in this crazy period. At the very least, basically, the intelligence agencies need to be running security for these labs. So they're already kind of like they're controlling everything. They're controlling access to everything. Then they're going to be like, probably again, if we're in this really volatile international situation, a lot of the initial applications, it'll suck. It's not what I want to use ASI for. We'll be trying to somehow stabilize this crazy situation. Somehow we need to prevent proliferation of some crazy new WMDs and the undermining of mutually assured destruction to North Korea and Russia and China. I basically think your world... you know, I think there's much more spectrum than you're acknowledging here. And I think basically the world in which it's private labs is like extremely heavy government involvement. And really what we're debating is like, you know, what form of government project, but it is going to look much more like, you know, the national security state than anything. It does look like, like a startup as it is right now. And I think the, yeah.

Dwarkesh Patel: Look, I think something like that makes sense. I would be if it's like the Manhattan Project, then I'm very worried where it's like this is part of the U.S. military, where if it's more like, listen, you got to talk to Jake Sullivan before you run the next training.

Leopold Aschenbrenner: It's like Lockheed Martin. Skunkworth's part of the U.S. military. It's like they call the shots.

Dwarkesh Patel: Yeah, I don't think that's great. I think that's bad. I think it would be bad if that happened with ASI.

Leopold Aschenbrenner: And what is the scenario? What is the alternative?

Dwarkesh Patel: Okay, so it's closer to my end of the spectrum where, yeah, you do have to talk to Jake Sullivan before you can launch the next training cluster. But there's many companies who are still going for it. And the government will be intimately involved in the security. But the three different companies are trying to... Yeah, what do you mean? Launching, okay.

Leopold Aschenbrenner: Dart is activating the Chinese data centers.

Dwarkesh Patel: I think this is similar to the story you could tell about, there's a lot of companies, like literally big tech right now. I think Satya, if he wanted to, he probably could get his engineers, like what are the zero days in Windows? And how do we get, infiltrate the president's computer so that we can- Maybe shut down. No, no, no. Right now I'm saying Satya could do that, right? Because he knows the zero days.

Leopold Aschenbrenner: Maybe shut down. What do you mean? Government wouldn't let them do that.

Dwarkesh Patel: Yeah, I think there's a story you could tell where they could pull off a coup or whatever. But I think there's multiple AI companies.

Leopold Aschenbrenner: They could not pull off a coup. They could not pull off, come on. Okay, fine, fine, fine. I agree. I'm just saying something closer to,

Dwarkesh Patel: so what's wrong with the scenario where the government is, there's multiple companies going for it, but the AI is still broadly deployed and alignment works in the sense that you can make sure that it's not, the system level prompt is you can't help people make bioweapons or something. Sure, sure, sure. But these are still broadly deployed, so that... I mean, I expect the AIs to be broadly deployed.

Leopold Aschenbrenner: I mean, first of all, yeah.

Dwarkesh Patel: Even if it's a government project? Yeah,

Leopold Aschenbrenner: I mean, look, I think, first of all, like, I think the metas of the world, you know, open sourcing their AIs, you know, that are two years behind or whatever, yeah, super valuable role. They're going to, like, you know... And so there's going to be some question of, like, either the offense-defense balance is fine, and so, like, even if they open source two-year-old AIs, it's fine, or it's, like, there's some restrictions on the most extreme dual-use capabilities, like, you know, you don't let private companies sell kind of crazy weapons. And that's great. And that will help with the diffusion. And after the government project, there's going to be this initial tense period, hopefully that's stabilized. And then look, yeah, like Boeing, they're going to go out and they're going to do all the flourishing civilian applications. And like nuclear energy, the civilian applications will have their day. I think part of my argument here is that- And how does that proceed, right?

Dwarkesh Patel: Because in the other world, there's existing stocks of capital that are worth- Yeah, the clusters,

Leopold Aschenbrenner: there'll still be Google clusters.

Dwarkesh Patel: And so Google, because they got the contract from the government, they'll be the ones that control the ASI. But why are they trading with anybody else?

Leopold Aschenbrenner: It'll also be Microsoft and Amazon. It'll be the same companies that would be doing it anyway. But in this world, they're just contracting with the government or their DPA for all their compute goes to the government.

Dwarkesh Patel: It's very natural. It's sort of how other UUS techs work. After you get the ASI and we're building the robot armies and building fusion reactors or whatever. Only the government will get to build robot armies. Yeah, now I'm worried. Or, like, the fusion reactors and stuff. It's what we do with nukes.

Leopold Aschenbrenner: It's the same situation we have today. Because if you already have the Royal Armies and everything,

Dwarkesh Patel: like, the existing society doesn't have some leverage where it makes sense for the government to... But they don't have that today. Yeah, in the sense that they have a lot of capital that the government wants and there's other things.

Leopold Aschenbrenner: Why was Boeing privatized after? Government has the biggest guns. And the way we regulate is institutions, constitutions, legal restraints.

Dwarkesh Patel: Okay, so tell me what privatization looks like in the ESI world afterwards.

Leopold Aschenbrenner: afterwards like the boeing example right it's like you have this government who gets it like the google microsoft and who are they selling it to like they already

Dwarkesh Patel: have the robo factory it's like why are they selling it to us like they already have they don't need like our this is chum change in the asi world because we didn't get like the ai the asi broadly deployed through throughout this takeoff so we don't have the robot like we don't have like the fusion reactors and whatever advanced decades of advanced science that you're talking about so like it just what are they trading with us for

Leopold Aschenbrenner: Trading with whom for?

Dwarkesh Patel: Everybody who was not part of the project. They've got that technology that's decades ahead.

Leopold Aschenbrenner: Yeah, I mean, look, that's a whole other issue of how does economic distribution work or whatever. I don't know. That'll be rough.

Dwarkesh Patel: Yeah.

Leopold Aschenbrenner: I'm just saying, I don't, I don't, basically I'm kind of like, I don't see the alternative. The alternative is you like overturn a 500 year civilizational achievement of Lundfleet and you basically instantly leak the stuff to the CCP and either you like barely scrape out ahead. Um, and, but you're in this fever struggle, you're like proliferating crazy WMDs. It's just like enormously dangerous situation, enormously dangerous on alignment. Cause you're in this kind of like crazy race at the end and you don't have the ability to like take six months to get alignment. Right. Um, the alternative is, um, you know, alternative is like you aren't actually bundling your efforts to kind of like win the race against the authoritarian powers. You know, yeah. And so, you know, I don't like it. You know, I wish I wish the thing we use the ASI for is to like, you know, cure the diseases and do all the good in the world. But it is my prediction that sort of like by the in the end game, what will be at stake will not just be kind of cool products, but what will be at stake is like whether liberal democracy survives, like whether the CCP survives, like what the world order for the next century will be. And when that is at stake, forces will be activated that are sort of way beyond what we're talking about now. And like, you know, in the sort of like crazy race at the end, like the sort of national security implications will be the most important, you know, sort of like, you know, World War II. It's like, yeah, you know, nuclear energy had its day. But in the initial kind of period when, you know, when this technology was first discovered, you had to stabilize the situation, you had to get nukes, you had to do it right. And then the civilian applications had their day.

Dwarkesh Patel: I think a closer analogy to what this is, because nuclear, I agree that nuclear energy is a thing that happens later on and it's like dual use in that way. But it's something that happened like literally a decade after nuclear weapons were developed.

Speaker 4: Yeah.

Dwarkesh Patel: Whereas with AI, like the immediately all the applications are unlocked and it's closer to literal. I mean, this is an analogy people actually make in the context of AGI is like assume your society had 100 million more John Wayne Neumanns. Yeah. And I don't think, like, if that was literally what happened, if tomorrow you just have 100 million more of them, the approach would have been, well, some of them will convert to ISIS, and we need to, like, be really careful about that. And then, like, oh, you know, like, what if a bunch of them are born in China? And then, like, if we get to nationalize the John von Neumanns, I'm like, no, I think it'll be generally a good thing. And I'd be concerned about one power getting, like, all the John von Neumanns.

Leopold Aschenbrenner: I mean, I think the issue is the sort of, like, bottling up in this sort of intensely short period of time. Like, this enormous sort of, like, you know... unfolding of technological progress of an industrial explosion. And I think we do worry about the 100 million John Mnuchin. It's like rise of China. Why are we worried about the rise of China? Because it's like 100 billion people, and they're able to do a lot of industry and do a lot of technology. And but it's just like, you know, the rise of China times like, you know, 100, because not just 101 billion people, it's like a billion, super intelligent, crazy, you know, crazy things. So and and in like, in a very short period.

Dwarkesh Patel: Let's talk practically because if the goal is we need to beat China, part of that is protecting... I mean,

Leopold Aschenbrenner: that's one of the goals, right? Yeah, I agree.

Dwarkesh Patel: One of the goals is to beat China.

Leopold Aschenbrenner: And also just manage this incredibly crazy scary period. Right. So part of that is making sure we're not leaking algorithmic secrets to them. Yep. Part of that is... Building the trillion dollar cluster. Huh? I mean, building the trillion dollar cluster, right?

Dwarkesh Patel: Yeah,

Leopold Aschenbrenner: but isn't your whole point that Microsoft can release corporate bonds that are... I think Microsoft can do the hundreds of billions dollar cluster. I think the trillion dollar cluster is closer to a national effort. I thought that your earlier point was that American capital markets are deep and so forth. They're good. They're pretty good. I mean, I think it's possible it's private. It's possible. But it's going to be like, you know. By the way, at this point, we have AGI that's rapidly accelerating productivity. I think the trillion dollar cluster is going to be planned before the AGI happens. I think it's sort of like you get the AGI on the 10 gigawatt cluster. Maybe you have one more year where you're kind of doing some final and hobbling to fully unlock it. Then you have the intelligence explosion. And meanwhile, the trillion dollar cluster is almost finished. And then you do your super intelligence on your trillion dollar cluster, or you run it on your trillion dollar cluster. And by the way, you have not just your trillion dollar cluster, but hundreds of millions of GPUs on inference clusters everywhere. And this isn't resolved. In this world, I think private companies have the capital and can raise capital to do it. The thing you will need the government force to do it fast.

Dwarkesh Patel: I was just about to ask, wouldn't it be the... We know private companies are on track to be able to do this and beat China if

Leopold Aschenbrenner: they're unhindered by climate pledges or whatever. Well, that's part of what I'm saying.

Dwarkesh Patel: So if that's the case,

Leopold Aschenbrenner: and if it really matters that we beat China, there's all kinds of practical difficulties of like, will the AI researchers actually join the AI effort? If they do, there's going to be three different teams at least who are currently doing pre-training on different... different companies. Now who decides, at some point you're gonna have like YOLO the hyper parameters of the trillion

Dwarkesh Patel: dollar cluster.

Leopold Aschenbrenner: Who decides that? Just like merging extremely complicated research and development processes across very different organizations. This is somehow supposed to speed up America against the Chinese? Why don't we just let it? Brain and DeepMind merged, and it was a little messy, but it was fine. It was pretty messy, and it was also the same company, and also much earlier on in the process. Pretty similar, right? Different code bases and lots of different infrastructure and different teams. And it wasn't the smoothest of all processes, but DeepMind is doing, I think, very well.

Dwarkesh Patel: I mean, look, you give the example of COVID and the COVID example is like, listen, we woke up to it.

Leopold Aschenbrenner: Maybe it was late, but then we had deployed all this money. And COVID response to government was a clusterfuck. And like the only part of it that was worked is I agree Warp Street was like enabled by the government. It was literally just giving the permission that you can actually do. Well, no, it was also taking, making like the big contract commitments or whatever. But I agree. But it was like fundamentally, it was like a private sector led effort. That was the only part of COVID that worked. I mean, I think, I think, again, I think the project will look closer to operation warp speed and it's not even, I mean, I think, I think you'll have all the companies involved in the government project. I'm not that sold that merging is that difficult. You know, you have one and you select one code base and you know, you, you run free training on like GPUs with, you know, one code base, and then you do the sort of second RL step on the, you know, the other code base with TPUs. And I think it's fine. Um, um, I mean, to the topic of like, will people sign up for it? They wouldn't sign up for it today. I think this would be kind of crazy to people. Um, but also, you know, I mean, this is part of the like secrets thing, you know, people gather at parties or whatever, you know, you know, this, um, you know, I don't think anyone has really gotten up in front of these people and been like, look, you know, the thing you're building is the most important thing for like the national security of the United States for like, whether, you know, like, you know, the free world will have another century ahead of it. Like this is this thing you're doing is really important, like for your country, for democracy. Um, And, you know, don't talk about the secrets. And it's not just about, you know, DeepMind or whatever. It's about, you know, these really important things. And so, you know, I don't know. Like, again, we're talking about the Manhattan Project, right? This stuff was really contentious initially. But, you know, at some point it was, like, clear that this stuff was coming. It was clear that there was, like, sort of a real sort of, like, exigency on the military and national security front. And, you know, I think a lot of people come around. On the like, whether it'll be competent, I agree. I mean, this is again, where it's like a lot of the stuff is more like predictive in the sense, I think this is like reasonably likely. And I think not enough people are thinking about it. You know, like a lot of people think about like AI lab politics or whatever, but like nobody has a plan for the project. You know, it's like, you know, like- Should I make you more pessimistic about it? And like, we don't have a plan for it. We need to do it very soon because AGI is upon us. Yeah. Then fuck, the only capable, competent technical institutions capable of making AI right now are private companies. Yeah, and they're going to continue to play that leading role. It'll be a sort of a partnership basically. But, you know, the other thing is like, you know, again, we talked about World War Two and, you know, American unpreparedness, the beginning of World War Two is complete, you know, complete shambles. Right. And so there is a sort of like very company. I think America has a very deep bench of just like incredibly competent managerial talent. You know, I think, you know, there's a lot of really dedicated people. And, you know, I think basically a sort of operational warp speed, public private partnership, something like that, you know, is sort of what I imagine it would look like. Yeah, I mean, recruiting the talent is an interesting question because the same sort of thing

Dwarkesh Patel: where initially for the Manhattan Project, you had to convince people we've got to beat the Nazis and you've got to get on board. I think a lot of them maybe regretted how much they accelerated the bomb. And I think this is generally a thing of war where...

Leopold Aschenbrenner: I mean, I think they're also wrong to regret it, but... Yeah, why? What's the reason for regretting it? I think there's a world in which you don't have... The way in which nuclear weapons were developed after the war was pretty explosive

Dwarkesh Patel: because there was a precedent that you actually can use nuclear weapons. Then because of the race that was set up, you immediately go to the H-bomb. I mean,

Leopold Aschenbrenner: I think my view is, again, and this is related to the view on AI and maybe some of our disagreement, is that was inevitable. Of course, there was this World War. And then, obviously, there was the Cold War right after. Of course, the military angle of this would be pursued with ferocious intensity. And I don't really think there's a world in which that doesn't happen, where it's like, ah, we're all not going to build nukes. And also, nukes went really well. I think that could have gone terribly. Again, I mean, this sort of I think this is like not physically possible with nukes, this sort of pocket nukes for everybody. But I think sort of like WMDs that are sort of proliferated and democratized and like all the countries have it like the U.S. leading on nukes and then sort of like building this new world order that was kind of U.S. led or at least sort of like a few great powers and a nonproliferation regime for nukes, a partnership and a deal that's like, look, no military sort of application of nuclear technology, but we're going to help you with the civilian technology. We're going to enforce safety norms on the rest of the world. That worked. It worked. And it could have gone so much worse.

Dwarkesh Patel: I mean,

Leopold Aschenbrenner: I say this a bit in the piece, but it's like actually the A-bomb, you know, like the A-bomb in Hiroshima and Nagasaki was just like, you know, the firebombing. I think the thing that really changed the game was like the super, you know, the H-bombs and ICBMs. And then I think that's really when it took it to like a whole new level.

Dwarkesh Patel: I think part of me thinks when you say we will tell the people that for the free world to survive, we need to pursue this project. It sounds similar to World War II is... So, World War II is a sad story, obviously, in the fact that it happened, but also, like, the victory is sad in the sense that Britain goes in to protect Poland, and at the end, the USSR, which is...

Leopold Aschenbrenner: you know, as your family knows, is incredibly brutal, ends up occupying half of Europe. And the, like, part of, like, we're protecting the free world, that's why we've got to rush the AI. And, like, if we end up with the American AI Leviathan, I think there's a world where we look back on this where it has the same sort of

Dwarkesh Patel: twisted irony that Britain going into World War II had about trying to protect Poland.

Leopold Aschenbrenner: Look, I mean, I think there's going to be a lot of unfortunate things that happen. I'm just like, I'm just hoping we make it through. I mean, to the point of it's like, I really don't think the pitch will only be the sort of like, you know, the race. I think the race will be sort of a backdrop to it. I think the sort of general like, look, it's important that democracy shape this technology. We can't just like leak this stuff to, you know, North Korea is going to be important. I think also for the just safety, including alignment, including the sort of like creation of new WMDs, um, I'm not currently sold. There's another path, right? So it's like, if you just have the breakneck grace, both internationally, because you're just instantly leaking all this stuff, including the weights and just, you know, the commercial race, you know, Demis and Dario and, and, and Sam, you know, just kind of like, they all want to be first. Um, And then it's incredibly rough for safety. And then you say, OK, safety regulation. But it's sort of like the safety regulation that people talk about. It's like, oh, well, NIST. And they take years. And they figure out what the expert consensus is. And then they write some guidelines.

Dwarkesh Patel: Isn't that what's going to happen to the project as well?

Leopold Aschenbrenner: I mean, I think the sort of alignment angle during the intelligence explosion, it's not a process of like years of bureaucracy and then you can kind of write some standards. I think it looks much more like basically a war and like you have a fog of war. It's like, look, it's like, is it safe to do the next OOM? And it's like, ah, we're like three OOMs into the intelligence explosion. We don't really understand what's going on anymore. You know, the... you know, like a bunch of our like generalization scaling curves are like kind of looking not great. You know, some of our like automated AI researchers that are doing alignment are saying it's fine, but we don't quite trust them. In this test, you know, the like, the eyes started doing naughty things and ah, but then we like hammered it out and then it was fine. And like, ah, should we go ahead? Should we take, you know, another six months? Also, by the way, you know, like China just stole the weights or we, you know, they're about to like deploy the Romo army. Like, what do we do? I think it's this, I think it is this crazy situation. And, you know, basically you, you are relying much more on kind of like a sane chain of command than you are on sort of some like, you know, deliberative regulatory scheme. I wish you had, you were able to do the deliberative regulatory scheme. And this is the thing about the private companies too. I don't think, you know, they all claim they're going to do safety, but I think it's really rough when you're in the commercial race and they're startups, you know, and startups, startups are startups. You know, I think they're not fit to handle WMDs.

Dwarkesh Patel: Yeah, I'm coming closer to your position, but part of me also, so with the responsible scaling policies, I was told by people who are advancing that, that the way to think about this, because they know I'm like a libertarian type of person. And the way they approached me about it was that fundamentally, this is a way to protect market-based development of AGI in the sense that if you didn't have this at all, then you would have this sort of misuse and then it would have to be nationalized. And the RSPs are a way to make sure that through this deployment, you can still have a market-based order, but then there's these safeguards that make sure that things don't go off the rails. And I wonder if the...

Leopold Aschenbrenner: It seems like your story seems self-consistent, but it does feel...

Dwarkesh Patel: I know this was never your position, so I'm not looping you into this, but a sort of Martin Bailey, almost, in the sense of...

Leopold Aschenbrenner: Well, look, here's what I think about RSP-type stuff or sort of safety regulation that's happening now. I think they're important for helping us figure out what world we're in and flashing the warning signs when we're close, right? And so... The story we've been telling is sort of what I think the modal version of this decade is. But I think there's lots of ways it could be wrong. We should talk about the data wall more. Again, I think there's a world where the stuff stagnates. There's a world where we don't have AGI. And so the RSP thing is preserving the optionality. Let's see how this stuff goes. But we need to be prepared. If the red lights start flashing, if we're getting the automated eye researcher, then it's crunch time. And then it's time to go. I think, okay, I can be on the same page on that, that we should have a very, very strong prior on pursuing in a market-based way, unless you're right about what the explosion looks like,

Dwarkesh Patel: the intelligence explosion. And so don't move yet, but in that world where it really does seem like Alec Radford can be automated, and that is the only bottleneck to getting TSI. Okay, I think we can leave it at that. I can, yeah, I'm somewhat of the way there.

Leopold Aschenbrenner: Okay, okay. I hope it goes well. It's going to be very stressful. And again, right now is the chill time. Enjoy your vacation while it lasts.

Dwarkesh Patel: It's funny to look out over, just like this is San Francisco. Yeah, yeah, yeah.

Leopold Aschenbrenner: Opening eyes right there, you know, anthropics there. I mean, again, this is kind of like, you know, it's like you guys have this enormous power over how it's going to go for the next couple of years. And that power is depreciating. Yeah. Who is you guys? Like, you know, people at labs. Yeah, yeah, yeah. But it is a sort of crazy world. And you're talking about like, you know, I feel like you talk about like, oh, maybe they'll nationalize too soon. It's like, you know, almost nobody like really like feels it, sees what's happening. And it's I think this is the thing that I find stressful about all this stuff is like, look, maybe I'm wrong. Like, if I'm right, we're in this crazy situation where there's like, you know, like a few hundred guys that are like paying attention. And it's it's daunting. I went to Washington a few months ago, and I was talking to some people who are doing AI policy stuff there.

Dwarkesh Patel: And I was asking them how likely they think nationalization is. And they said, oh, you know, like... it's really hard to nationalize stuff. It's been a long time since we've done it. There's these very specific procedural constraints on what kinds of things can be nationalized.

Leopold Aschenbrenner: And then I was asked, well, like ASI. So that means because there's, there's constraints at a defense production act or whatever, that that won't be nationalized. There's the Supreme court would overturn that.

Dwarkesh Patel: And they're like, yeah, I guess that would be nationalized.

Leopold Aschenbrenner: That's the short summary of my post or my view on the project. Uh,

Dwarkesh Patel: But before we go further on the AI stuff, let's just back up. We began the conversation, I think people will be confused. You graduated valedictorian of Columbia when you were 19. So you got to college when you were 15. So you were in Germany, then you got to college at 15. How the fuck did that happen?

Leopold Aschenbrenner: I really wanted out of Germany. I, you know, I went to kind of a German public school. It was not a good environment for me. And, you know, in what sense?

Dwarkesh Patel: It's just like no peers that are.

Leopold Aschenbrenner: Yeah, look, I mean, it wasn't. Yeah, it was, you know. I mean, there's also just a sense in which sort of like there's this particular sort of German cultural sense. I think in the US, you know, there's all these like amazing high schools and like sort of an appreciation of excellence. And in Germany, there's really this sort of like Paul Poppy syndrome almost, right? Where it's, you know, you're the curious kid in class and you want to learn more instead of the teacher being like, ah, that's great. They're like, they kind of resent you for it. And they're like trying to crush you. Yeah. I mean, there's also like there's no kind of like elite universities for undergraduate, which is kind of crazy. So, you know, the sort of, you know, there's sort of like basically like the meritocracy was kind of crushed in Germany at some point. Also, I mean, there's a sort of incredible sense of, you know, complacency, you know, across the board. I mean, one of the things that always puzzles me is like, you know. even just going to a U.S. college was this kind of like radical act.

Dwarkesh Patel: And like,

Leopold Aschenbrenner: you know, it doesn't seem radical to anyone here because it's like, ah, this is obviously the thing you do and you can go to Columbia, you go to Columbia. But it's, you know, it is very unusual. And it's it's it's wild to me because it's like, you know, this is where stuff is happening. You can get so much of a better education. And, you know, like America is where, you know, it's where where where where all the stuff is and people don't do it. And so, you Yeah, anyway, so I, you know, I don't know, I skipped a few grades and, you know, I think at the time it seemed very normal to me to kind of like go to college at 15 and come to America. I think, you know, now one of my sisters is now like turning 15, you know, and so then I, you know, and I look at her and I'm like, now I understand how my mother reacted to it. And as you get to college, you were presumably the only 15-year-old there. Yeah, yeah. So it was just like normal for you to be a 15-year-old? Like what was the initial years like? It felt so normal at the time. So again, it's like now I understand why my mother's worried. I worked on my parents for a while. Eventually I persuaded them. No, but yeah, it felt very normal at the time. And it was great. It was also great because, you know, I actually really like college, right? And in some sense, it sort of came at the right time for me where, you know, I mean, I, you know, for example, I really appreciated the sort of like liberal arts education and, you know, like the core curriculum and reading sort of core works of political philosophy and literature. You did what, econ and... I mean, my majors were math and statistics and economics. But, you know, Columbia is a sort of pretty heavy core curriculum in liberal arts education. And honestly, like, you know, I shouldn't have done all the majors. I should have just I mean, the best courses were sort of the courses where it's like there's some amazing professor and it's some history class. And I mean, that's that's honestly the thing I would recommend people spend their time on in college. Was there one professor or class that stood out that way? I mean, there's a class by Richard Betts on war, peace, and strategy. Adam Too is obviously fantastic and has written very riveting books. You should have him on the podcast, by the way.

Dwarkesh Patel: I tried. I think you tried for me.

Leopold Aschenbrenner: Yeah, you got to get him on the pod, man. Oh, it'd be so good. Okay, so then in a couple of years, we were talking to Tyler Cowen recently,

Dwarkesh Patel: and he said that the way he first encountered you was you wrote this paper on economic growth and existential risk. And he said, when I read it, I couldn't believe that a 17-year-old had written it. I thought if this was an MIT dissertation, I'd be impressed.

Leopold Aschenbrenner: So you were like... How did you go from, I guess you were a junior then, you're writing pretty novel economic papers. Why did you get interested in this kind of thing and what was the process to get into that? I don't know. I get interested in things. In some sense, it feels very natural to me. I get excited about a thing. I read about it. I immerse myself. I think I can learn information very quickly and understand it. I think to the paper, I think one actual... at least for the way I work, I feel like sort of moments of peak productivity matter much more than sort of average productivity. I think there's some jobs, you know, like COO or something, you know, like average productivity really matters. But I think there's sort of, I often feel like I have periods of like, you know, there's some, there's a couple months where there's sort of nephrolessence and I'm like, you know, and the other times I'm sort of computing stuff in the background. And at some point, you know, like writing the series, this is also kind of similar. And it's just like you write it and it's like, it's really flowing. And that's sort of what ends up mattering.

Dwarkesh Patel: I think even for CEOs, it might be the case that the peak productivity is very important.

Leopold Aschenbrenner: There's one of our, following Chatham House rules, one of our friends in a group chat has pointed out how many famous CEOs and

Dwarkesh Patel: founders have been bipolar manic, which is very much the peak, like the call option on your productivity is the most important thing. And you get it by just increasing the volatility through bipolar. Um, okay. So that's interesting. And so you get interested in economics first. First of all, why economics? Like you could read about anything at this move. Like you, if you wanted, you know, you could, you kind of got a slow start on them. All right. You wasted all these years on econ. There's an alternate world where you're like on the super alignment team at 17 instead of 21 or whatever it was.

Leopold Aschenbrenner: Um, um, I mean, in some sense, I'm still doing economics, right? You know, what is straight lines on a graph? I'm looking at the log-log plots and, like, figuring out what the trends are and, like, thinking about the feedback loops and equilibrium arms control dynamics. And, you know, I think it is a sort of a way of thinking that I find very useful. And, yeah. you know, like what, you know, Dario and Ilya seeing scaling early. In some sense, that is a sort of very economic way of thinking. Also, the sort of physics, kind of like empirical physics. You know, a lot of them are physicists. I think the economists usually can't code well enough, and that's their issue. But I think it's that sort of way of thinking. I mean, the other thing is, you know, I thought there were sort of... You know, I thought of a lot of the sort of like core ideas of economics I thought were just beautiful. And, you know, in some sense, I feel like I was a little duped, you know, where it's like actually econ academia is kind of decadent now. You know, I think that, you know, for example, the paper I wrote, you know, it's sort of I think the takeaway, you know, it's a long paper. It's 100 pages of math or whatever. I think the core takeaway I can, you know, kind of. give the core intuition for in like you know 30 seconds and it makes sense and it's and it's like you don't actually need the math yeah i think that's the sort of the best pieces of economics are like that where you do the work but you do the work um to kind of uncover insights that weren't obvious to you before once once you've done the work it's like some sort of like mechanism falls out of it that like makes a lot of crisp intuitive sense that like explains some facts about the world that you can then use in arguments i think you know i think A lot of econ 101 like this, and it's great. A lot of econ in the 50s and the 60s was like this. And Chad Jones' papers are often like this. I really like Chad Jones' papers for this. I think, why did I ultimately not pursue econ academia? A number of reasons. One of them was Tyler Cowen. You know, he kind of took me aside and he was kind of like, look, I think you're one of the top young economists I've ever met. But also, you should probably not go to grad school.

Dwarkesh Patel: Oh, interesting. Really? I didn't realize that.

Leopold Aschenbrenner: Well, yeah. And it was it was good because he kind of introduced me to the, you know, I don't know, like the Twitter weirdos or just like, you know, and I think the takeaway from that was kind of, you know, got to move out last one more time. Wait, Tyler introduced you to the Twitter weirdos? A little bit, yeah. Or just kind of like the sort of brought... Like the 60-year-old,

Dwarkesh Patel: the old economist to introduce you to that Twitter? Yeah, well,

Leopold Aschenbrenner: you know, I had been... So I went from Germany, you know, completely, you know, on the periphery to kind of like, you know, a U.S. elite institution and sort of got some vibe of like sort of, you know, meritocratic, elite, you know, US society. And then sort of, yeah, basically this sort of like, there is a sort of directory then to being like, look, I, you know, find the true American spirit. I got to come out here. But the other reason I didn't become an economist was because, or at least econ academia, was because I think sort of econ academia has become a bit decadent. And maybe it's just ideas getting harder to find, and the beautiful, simple things have been discovered. What are econ papers these days? It's like 200 pages of empirical analyses on what happened when Wisconsin bought 100,000 more textbooks on educational outcomes. And I'm really happy that work happened. I think it's important work, but I think it is not uncovering these fundamental insights and mechanisms in society. Or even the theory work is kind of like, here's a really complicated model. And the model spits out, if the Fed does x, then y happens. And you have no idea why that happened, because it's like gazillion parameters, and they're all calibrated in some way, and it's some computer simulation. And you have no idea about the validity. So I think the most important insights are the ones where you have to do a lot of work to get them. But then there's this crisp intuition. Yeah, the P versus NP of .. That's really interesting.

Dwarkesh Patel: So just going back to your time in college. Yeah. You say that peak productivity kind of explains this paper and things. But the valedictorian that's getting straight A's or whatever is very much average productivity phenomenon, right?

Leopold Aschenbrenner: There's one award for the highest GPA, which I won. But the valedictorian is like among the people which have the highest GPA and then like selected by faculty or something.

Dwarkesh Patel: Okay, yeah. So it's not just peak productivity.

Leopold Aschenbrenner: I generally just love this stuff. I was curious, and I thought it was really interesting, and I loved learning about it, and it made sense to me, and it was very natural. I think I'm not... I think one of my faults is I'm not that good at eating glass or whatever. I think there's some people who are very good at it. I think the moments of pre-productivity come when I'm just really excited and engaged and love it. If you take the right courses, that's what you get in college.

Dwarkesh Patel: It's the Bruce Banner code in Avengers. I'm always angry.

Speaker 4: I'm always excited.

Dwarkesh Patel: I'm always curious. That's why I'm always pre-productivity.

Leopold Aschenbrenner: So it's interesting, by the way, when you were in college, I was also in college. I think you were, despite being a year younger than me, I think you're ahead in college than me, or at least maybe two years ahead.

Dwarkesh Patel: And we met around this time.

Speaker 2: Yeah, yeah, yeah.

Dwarkesh Patel: We also met, I think, through the Tyler Cowen universe. And it's very insane how small the world is. Did I reach out to you? I must have. About when I had a couple of videos and they had a couple hundred views or something.

Leopold Aschenbrenner: It's a small world. I mean, this is the crazy thing about the AI world, right? It's kind of like, it's the same few people at the kind of SF parties and they're the ones, you know, running the models at DeepMind and OpenAI and Anthropic and... I mean, I think some other friends of ours have mentioned this who are now later in their career and very successful that they actually met all the people who are also kind of very successful in Silicon Valley now, like when they're in their, before their 20s or when they're really 20s. I mean, look, I actually think, and why is it a small world? I mean, I think one of the things is some amount of like, you know, some sort of agency. And I think in a funny way, this is a thing I sort of took away from the sort of Germany experience where it was, I mean, look, it was crushing. I really didn't like it. And it was like, it was such an unusual move to kind of skip grades and such an unusual move to come to the United States. And, you know, a lot of these things I did were kind of unusual moves. And, you know, there's some amount where like, just like just trying to do it and then it was fine and it worked um that kind of reinforced like you know you don't you don't just have to kind of conform to what the overturn window is you can just kind of try to do the thing the thing that seems right to you um and like you know most people can be wrong i don't know things like that and i think that was kind of a you know valuable kind of like

Dwarkesh Patel: early experience that was sort of formative okay so after college what did you do

Leopold Aschenbrenner: I did econ research for a little bit in Oxford and stuff, and then I worked at Future Fund.

Dwarkesh Patel: So tell me about it.

Leopold Aschenbrenner: Future Fund was a foundation that was funded by Sam Bankman-Fried. We were our own thing. We were based in the Bay. At the time, this was in early 22, It was this incredibly exciting opportunity. It was basically like a startup foundation, which doesn't come along that often. We thought we'd be able to give away billions of dollars. We thought we'd be able to remake how philanthropy is done from first principles. Thought we'd be able to have this great impact. The causes we focused on were biosecurity, AI, finding exceptional talent, and putting them to work on hard problems. And a lot of the stuff we did, I was really excited about. Academics, who would usually take six months, would send us emails like, ah, this is great. This is so quick and straightforward. In general, I feel like I've often find that with a little bit of encouragement, a little bit of empowerment, removing excuses, making the process easy, you can get people to do great things. I think on the future front, the other thing is... context for people who might not realize, not only were you guys planning on deploying billions of dollars, but it was a team of four people. Yeah, yeah, yeah. So you at 18 are on a team of four people that is in charge of deploying billions of dollars.

Dwarkesh Patel: Yeah.

Leopold Aschenbrenner: I mean, just, I mean, yeah, Future Fund, you know, the... Yeah, I mean, so that was sort of the heyday, right? And then obviously, you know, when in sort of, you know, November of 22, you know, it was kind of revealed that Sam was this, you know, giant fraud. And from one day to the next, you know, the whole thing collapsed. That was just really tough. I mean, you know, obviously, it was devastating. It was devastating, obviously, for the people at their money in FTX, you know. Closer to home, you know, all that, you know, all these grantees, you know, we wanted to help them and we thought they were doing amazing projects. And so but instead of helping them, we ended up saddling them with like a giant problem. You know, personally, it was, you know, it was a startup. Right. And so I, you know, I'd worked 70 hour weeks every week for basically a year on this to kind of build this up. You know, we're a tiny team. And then from one day to the next, it was all gone and not just gone. It was associated with this giant fraud. And so, you know, that was incredibly tough. Yeah. Yeah.

Dwarkesh Patel: And were there any signs early on that SBF was...

Leopold Aschenbrenner: Yeah, and look, obviously I didn't know he was a fraud and the whole, you know, I would have never worked with him if I did, you know? And, you know, we weren't, you know, we were a separate thing. We weren't working with the business. I mean, I think, I do think there are some takeaways for me. I think one takeaway was, you know, I think there's a, I had this tendency, I think people in general have this tendency to kind of like, you know, give successful CEOs a pass on their behavior because, you know, they're successful CEOs and that's how they are and that's just successful CEO things. And, you know, I didn't know Sam Bankman-Fried was a fraud, but I knew SBF. And I knew he was extremely risk-taking. I knew he was narcissistic. He didn't tolerate disagreement well. By the end, he and I just didn't get along well. And I think the reason for that was there were some biosecurity grants he really liked because they were cool and flashy. And at some point, I'd run the numbers. And it didn't really seem that cost-effective. And I pointed that out. And he was pretty unhappy about that. And so I knew his character. Um, and I think, you know, I feel like one takeaway for me was, um, was, um, you know, like, I think it's really worth paying attention to people's character and including like people you work for and successful CEOs. Um, and, um, you know, that can save you a lot of pain down the line.

Dwarkesh Patel: Okay, so after that, FTX implodes and you're out. And then you got into, you went into OpenAI, the super alignment team had just started.

Leopold Aschenbrenner: I think you were like part of the initial team. And so what was the original idea? What was compelling about that for you to join? Yeah, totally. So what was the goal of the super alignment team? The alignment team at OpenAI and other labs several years ago had done basic research, and they developed RLHF, reinforcement learning from human feedback. And that ended up being a really successful technique for controlling current generation of AI models. What we were trying to do was basically kind of be the basic research bet to figure out what is the successor to RLHF. And the reason we needed that is, you know, basically, you know, RLHF probably won't scale to superhuman systems. RLHF relies on sort of human raters who kind of thumbs up, thumbs down, you know, like the model said something, it looks fine, looks good to me. At some point, you know, the superhuman models, the superintelligence, it's going to write, you know, a million lines of, you know, crazy complex code. You don't know at all what's going on anymore. And so how do you kind of steer and control these systems? How do you add side constraints? You know, the reason I joined was... I thought this was an important problem, and I thought it was just a really solvable problem, right? I thought this was basically, you know, there's, I think there's a, I still do, I mean, even more so do, I think there's a lot of just really promising sort of ML research on alignment, on sort of aligning superhuman systems. And maybe we should talk about that a bit more later. But so, and then- It was so solvable, you solved it in a year. It's all over now. It's been quite some time. I wanted to do this really ambitious effort on alignment and Elliot was backing it and I liked a lot of the people there. I was really excited. I think there was a lot of people on alignment. There's always a lot of people making hay about it. I appreciate people highlighting the importance of the problem. I was just really into, let's just try to solve it. Let's do the ambitious effort. Let's do the Operation Warp Speed for solving alignment. It seemed like an amazing opportunity to do so.

Dwarkesh Patel: Okay, and now basically the team doesn't exist. I think the head of it has left. Both heads of it have left, Jan and Ilya. That's been the news of the last week. What happened? Why did the thing break down?

Leopold Aschenbrenner: I think OpenAI sort of decided to take things in a somewhat different direction. Meaning what? I mean, that super alignment isn't the best way to frame the... No, I mean, look, obviously, sort of after the November board events, you know, there were personnel changes. I think Ilya leaving was just incredibly tragic for OpenAI. And, you know, I think some amount of reprioritization, I think some amount of, you know... I mean, there's been some reporting on the super alignment compute commitment. You know, there's this 20% compute commitment as part of, you know, how a lot of people were recruited. You know, it's like, we're going to do this ambitious effort on alignment. And, you know... some amount of not keeping that and deciding to go in a different direction.

Dwarkesh Patel: Okay, so now Jan has left, Ilya has left. So this team itself has dissolved, but you were the sort of first person who left or was forced to leave.

Leopold Aschenbrenner: The information reported that you were fired for leaking. What happened? Is this accurate? Yeah. Look, why don't I tell you what they claim I leaked, and you can tell me what you think. Yeah, so OpenAI did claim to employees that I was fired for leaking. And I and others have sort of pushed them to say what the leak is. And so here's their response in full. Sometime last year, I had written a sort of brainstorming document on preparedness, on safety and security measures we need in the future on the path to AGI. And I shared that with three external researchers for feedback. So that's it. That's the leak. I think for context, it was totally normal at OpenAI at the time to share sort of safety ideas with external researchers for feedback. It happened all the time. The doc was sort of my ideas before I shared it. I reviewed it for anything sensitive. The internal version had a reference to a future cluster, but I redacted that for the external copy. There's a link in there to some slides of mine, internal slides. But that was a dead link to the external people I shared it with. The slides weren't shared with them. And so obviously I... I pressed them to sort of tell me, what is the confidential information in this document? And what they came back with was a line in the doc about planning for AGI by 27, 28, and not setting timelines for preparedness. I wrote this doc a couple months after the super alignment announcement. We had put out this sort of four-year planning horizon. I didn't think that planning horizon was sensitive. You know, it's, it's the sort of thing Sam says publicly all the time. Um, I think sort of John said a couple of weeks ago. Um, anyway, so that's it. That's it. So that's pretty thin for, uh, if, if the cause was leaking, that's pretty thin. Was there anything else to it? Yeah, I mean, so that was the leaking claim. I mean, you can say a bit more about sort of what happened in the firing. Sure, yeah. So one thing was last year I had written a memo, internal memo, about opening eye security. I thought it was egregiously insufficient. I thought it wasn't sufficient to protect the theft of model weights or key algorithmic secrets from foreign actors. So I wrote this memo. I shared it with a few colleagues, a couple members of leadership who sort of mostly said it was helpful. But then, you know, a couple of weeks later, a sort of major security incident occurred. And that prompted me to share the memo with a couple of members of the board. And so after I did that, you know, days later, it was made very clear to me that leadership was very unhappy with me having shared this memo with the board. You know, apparently the board had hassled leadership about security. And then I got sort of an official HR warning for this memo, you know, for sharing it with the board. The HR person told me it was racist to worry about CCP espionage, and they said it was sort of unconstructive. And look, I think I probably wasn't at my most diplomatic. I definitely could have been more politically savvy, but I thought it was a really, really important issue. And the security incident had made me really worried. Anyway, and so I guess the reason I bring this up is when I was fired, it was sort of made very explicit that the security memo is a major reason for my being fired. I think it was something like, the reason that this is a firing and not a warning is because of the security memo. But you were sharing it with the board. the warning I'd gotten for the security memo. Anyway, and I mean, some other, you know, what might also be helpful context is the sort of questions they asked me when they fired me. So, you know, this was a bit over a month ago. I was pulled aside for a chat with a lawyer, you know, that quickly turned very adversarial. And, you know, the questions were all about my views on AI progress, on AGI, on the level of security appropriate for AGI, on, you know, whether government should be involved in AGI, on whether I and Superalignment were loyal to the company, on what I was up to during the OpenAI board events, things like that. And then they chatted to a couple of my colleagues, and then they came back and told me I was fired. And they'd gone through all of my digital artifacts from the time at OpenAI, messages, docs, and that's when they found the leak. Yeah, and so anyway, so the main claim they made was this leaking allegation. That's what they told employees, the security memo. There's a couple other allegations they threw in. One thing they said was that I was unforthcoming during the investigation because I didn't initially remember who I had shared the doc with, the preparedness brainstorming doc, only that I had spoken to some external researchers about these ideas. And look, the doc was over six months old. I'd spent a day on it. It was a Google Doc I shared with my open AI email. It wasn't a screenshot or anything I was trying to hide. It simply didn't stick because it was such a non-issue. And then they also claimed that I was engaging on policy in a way that they didn't like. And so what they cited there was that I had spoken to a couple of external researchers, somebody got a think tank about my view that AGI would become a government project, as we discussed. You know, in fact, I was speaking to lots of sort of people in the field about that at the time. I thought it was a really important thing to think about. Anyway, and so they found, you know, they found a DM that I'd written to like a friendly colleague, you know, five or six months ago where I relayed this and, you know, they cited that. And, you know, I had thought it was well within open eye norms to kind of talk about high level issues on the future of AGI with external people in the field. So anyway, so that's what they alleged, that's what happened. I've spoken to kind of a few dozen former colleagues about this since. I think the sort of universal reaction is kind of like, that's insane. I was sort of surprised as well. I had been promoted just a few months before. I think Ilya's comment for the promotion case at the time was something like, Leopold's amazing, we're lucky to have him. But look, I mean, I think the thing I understand and I think in some sense is reasonable is like, you know, I think I ruffled some feathers and, you know, I think I was probably kind of annoying at times. You know, it's like security stuff and I kind of like repeatedly raised that and maybe not always in the most diplomatic way. You know, I didn't sign the employee letter during the board events, you know, despite pressure to do so.

Dwarkesh Patel: And you were one of eight people or something?

Leopold Aschenbrenner: Not that many people. I think the two senior most people who didn't sign were Andre and Jan, who have both since left. On the letter, by the way, by the time on Monday morning when that letter was going around, I think probably it was appropriate for the board to resign. I think they'd lost too much credibility and trust with the employees. But I thought the letter had a bunch of issues. I mean, I think one of them was it just didn't call for an independent board. I think it's sort of like basics of corporate governance to have an independent board. Anyway, you know, it's other things, you know, in sort of other discussions, I pressed leadership for sort of opening eye to abide by its public commitments. I raised a bunch of tough questions about whether it was consistent with the OpenAI mission and consistent with the national interest to sort of partner with authoritarian dictatorships to build the core infrastructure for AGI. So, look, it's a free country, right? That's what I love about this country. We talked about it. And so they have no obligation to keep me on staff. And I think in some sense, I think it would have been perfectly reasonable for them to come to me and say, look, we're taking the company in a different direction. We disagree with your point of view. You know, we don't trust you enough to sort of toe the company line anymore. And, you know, thank you so much for your work at OpenAI, but I think it's time to part ways. I think that would have made sense. I think, you know, we did start sort of materially diverging on sort of views on important issues. I'd come in very excited and aligned with OpenAI, but that sort of changed over time. And... So look, I think there would have been a very amicable way to part ways. And I think it's a bit of a shame that this is the way it went down. All that being said, I think I really want to emphasize there's just a lot of really incredible people at OpenAI. And it was an incredible privilege to work with them. And overall, I'm just extremely grateful for my time there.

Dwarkesh Patel: When you left, now there's been reporting about an NDA that former employees have to sign in order to have access to their vested equity. Did you sign such NDAs?

Leopold Aschenbrenner: No. My situation was a little different in that I was basically right before my cliff. But then they still offered me the equity, but I didn't want to sign the non-disparagement. Freedom is priceless.

Dwarkesh Patel: And how much was the equity?

Leopold Aschenbrenner: Close to a million dollars. Okay.

Dwarkesh Patel: So it was definitely a thing you and others were aware of that this is like a choice that OpenAI is explicitly offering you. Yeah. And presumably the person on OpenAI's staff knew that we're offering them equity, but they had to sign this NDA that has these conditions that you can't, for example, give the kind of statements about your thoughts on AGI and OpenAI that you're giving on this podcast right now.

Leopold Aschenbrenner: Look, I don't know what the whole situation is. I certainly think sort of vested equity is pretty rough if you're conditioning that onto an NDA. It might be a somewhat different situation if it's a sort of severance agreement.

Dwarkesh Patel: Right. But an OpenAI employee who had signed it presumably could not give the podcast that you're giving today.

Leopold Aschenbrenner: Quite plausibly not.

Speaker 2: Yeah. I don't know.

Dwarkesh Patel: Okay, so analyzing the situation here, I guess if you were to... Yeah, the board thing is really tough because if you were trying to defend them, you would say, well... Listen, you were just kind of going outside the regular chain of command, and maybe there's a point there. Although, the way in which the person from HR thinks that you have an adversarial relationship with... Or you're supposed to have an adversarial relationship with the board, where to give the board some information which is relevant to... whether OpenAI is fulfilling its mission and whether it can do that in a better way is part of the leak, as if the board is, that is supposed to ensure that OpenAI is following its mission as some sort of external actor.

Leopold Aschenbrenner: That seems pretty... I mean, I think, I mean, to be clear, the leak allegation was just that sort of document I chose for feedback. This is just sort of a separate thing that they cited and they said, I wouldn't have been fired if not for the security memo.

Dwarkesh Patel: They said you wouldn't have been fired.

Leopold Aschenbrenner: They said the reason this is a firing and not a warning is because of the warning you had gotten for the security memo.

Dwarkesh Patel: Oh. Before you left, the incidents with the board happened where Sam was fired and then rehired a CEO, and now he's on the board. Now, Ilya and Jan, who are the heads of the super alignment team, and Ilya, who is a co-founder of OpenAI, obviously the most significant in terms of stature member of OpenAI from a research perspective, they've left, seems like, especially with regards to Super Alignment stuff and just generally with OpenAI, a lot of this sort of personnel drama has happened over the last few months. What's going on?

Leopold Aschenbrenner: Yeah, there's a lot of drama about OpenAI. Yeah, so why is there so much drama? I think there would be a lot less drama if all OpenAI claimed to be was building ChatGPT or building business software. I think where a lot of the drama comes from is OpenAI really believes they're building AGI. And it's not just a claim they make for marketing purposes. Whatever. There's this report that Sam is raising $7 trillion for chips. And it's like, that stuff only makes sense if you really believe in AGI. And so I think what gets people sometimes is sort of the cognitive dissonance between sort of really believing in AGI, but then sort of not taking some of the other implications seriously. You know, this is going to be incredibly powerful technology, both for good and for bad. And that implicates really important issues, like the national security issues we spoke about. Like, you know, are you protecting the secrets from the CCP? Like, you know, does America control the core AGI infrastructure? Or does it, you know, a Middle Eastern dictator control the core AGI infrastructure? And then, I mean, I think the thing that... you know, really gets people is the sort of tendency to kind of then make commitments and sort of like, you know, they say they take these issues really seriously, they make big commitments on them, but then sort of frequently don't follow through, right? So, you know, again, as mentioned, there's this commitment around superalignment compute, you know, sort of 20% of compute for this long-term safety research effort. And I think, you know, you and I could have a totally reasonable debate about what is the appropriate level of compute for superalignment. But that's not really the issue. The issue is that this commitment was made and it was used to recruit people and it was very public and it was made because there's a recognition that there would always be something more urgent than a long-term safety research effort, like some new product or whatever. But then in fact, they just really didn't keep the commitment. And so there was always something more urgent than long-term safety research. I mean, I think another example of this is when I raised these issues about security, you know, they would tell me, you know, security is our number one priority. But then, you know, invariably when it came time to sort of invest serious resources, when it came time to make trade-offs, to sort of take some pretty basic measures, security would not be prioritized. And so, yeah, I think it's the cognitive dissonance and I think it's the sort of unreliability that causes a bunch of the drama.

Dwarkesh Patel: So let's zoom out, talk about a big part of the story and also a big motivation of the way in which we must proceed with regards to geopolitics and everything is that once you have the AGI, pretty soon after, you proceed to ASI, superintelligence, because you have these AGIs which can function as researchers into further AI progress.

Leopold Aschenbrenner: And within a matter of years, maybe less, you go to something that is like super intelligence. And then from there, then you do, according to your story, do all this research and development into robotics and pocket nukes and whatever

Dwarkesh Patel: other crazy shit. Yeah. I'm skeptical of this story for many reasons. At a high level,

Leopold Aschenbrenner: it's not clear to me this input-output model of research is how things actually happen in research.

Dwarkesh Patel: We can look at economy-wide, right? Patrick Hollis and others have made this point that compared to 100 years ago, we have 100x more researchers in the world. It's not like progress is happening 100x faster.

Leopold Aschenbrenner: So it's clearly not the case that you can just pump in more population into research and you get higher research on the other end. I don't know why it would be different for the AI researchers themselves. Okay, great. So this is getting into some good stuff. I have a classic disagreement I have with Patrick and others. All right, so, you know, obviously inputs matter, right? So it's like the United States produces a lot more scientific and technological progress than, you know, Lichtenstein, right? Or Switzerland. And even if I made, you know, Patrick Collison, dictator of like Lichtenstein or Switzerland, and Patrick Collison was able to implement his like, you know, utopia of ideal institutions, keeping the town full fix, he's not able to like do some crazy high school immigration thing or like, you know, whatever, some like crazy genetic breeding scheme or whatever he wants to do. Keeping the talent pool fixed, but amazing institutions. I claim that still, even if you made Patrick Collison dictator of Switzerland, maybe you get some factor, but Switzerland is not going to be able to out-compete the United States in scientific and technological powers. Obviously, magnitudes matter.

Dwarkesh Patel: No, actually, I'm not sure I agree with this.

Leopold Aschenbrenner: There's been many examples in history where you have small groups of people who are part of, like, Bell Labs or Skunk Works or something. There's a couple hundred researchers. OpenAI, right? A couple hundred researchers. They do make... Highly selected, though, right? It's like saying, you know,

Speaker 2: the 500,000...

Leopold Aschenbrenner: That's part of why Patrick Collison as a dictator is going to do a good job of this. Well, yes, if you can highly select all the best AI researchers in the world, you might only need a few hundred. But that's the talent pool. It's like you have the 300 best AI researchers in the world.

Dwarkesh Patel: But there has been. It's not the case that from 100 years to now, there haven't been. The population has increased massively. In fact, you would expect the density of talent to have increased in the sense that malnutrition and other kinds of poverty, whatever, that have debilitated past talent at the same sort of level is no longer debilitated. Yeah, yeah.

Leopold Aschenbrenner: To the 100x point, right? So I don't know if it's 100x. I think it's easy to inflate these things. Probably at least 10x. And so people are sometimes like, ah, you know, like, you know, come on. Ideas haven't gotten that much harder to find. You know, why would you have needed this 10x increase in research effort? Whereas to me, I think this is an extremely natural story. And why is it a natural story? It's a straight line on a log-log plot. This is sort of a deep learning researcher's dream, right? What is this log-log plot? On the x-axis, you have log cumulative research effort. On the y-axis, you have log GDP, or ooms of algorithmic progress, or log transistors per square inch, or in the sort of experience curve for solar, kind of like whatever the log of the price for a gigawatt of solar. And it's extremely natural for that to be a straight line. You know, this is sort of a class. Yeah, it's a classic. And, you know, it's basically the first thing is very easy. Then basically, you know, you have to have log increments of cumulative research effort to find the next thing. And so, you know, in some sense, I think there's a natural story. Now, one objection kind of people then make is like, oh, you know, isn't it suspicious, right? That like ideas, you know, well, we increased research effort 10x and ideas also just got 10x harder to find. And so it perfectly, you So there, I say, it's an equilibrium. It's an adagious equilibrium. So it's like, isn't it a coincidence that supply equals demand and the market clears? And the same thing here. So how much ideas have gotten harder to find is a function of how much progress you've made. And then what the overall growth rate has been is a function of how much ideas have gotten harder to find. in ratio to how much you've been able to increase research effort. What is the growth of log cumulative research effort? So in some sense, I think the story is fairly natural. And you see this not just economy-wide, you see it in experience curve for all sorts of individual technologies. So I think there's some process like this. I think it's totally plausible that institutions have gotten worse by some factor. Obviously, there's some sort of exponent of diminishing returns on more people. So serial time is better than just parallelizing. But still, I think it's like, clearly, inputs matter.

Dwarkesh Patel: Yeah, I agree. But if the coefficient of how fast they diminish as you grow the input is high enough, then in the abstract, the fact that inputs matter isn't that relevant. OK, so I mean, we're talking a very high level, but just like take it down to the actual concrete thing here. OpenAI has a staff of at most low hundreds who are directly involved in the

Leopold Aschenbrenner: algorithmic progress in future models. If it was really the case that you could just arbitrarily scale this number and you could have much faster algorithmic progress,

Dwarkesh Patel: and that would result in much better AIs for OpenAI basically, then it's not clear why OpenAI doesn't just go out and hire every single person with 150 IQ, of which there are hundreds of thousands in the world. My story there is there's transaction costs to managing all these people that don't just go away if you have a bunch of AIs, that these tasks aren't easy to parallelize. And I think you, I'm not sure how you would explain the fact of like, why does an open AI go on a recruiting binge of every single genius in the world?

Leopold Aschenbrenner: Okay, great. So let's talk about the OpenAI example and let's talk about the automated AI researchers. So, I mean, the OpenAI case, I mean, just, you know, just kind of like look at the inflation of like AI researcher salaries over the last year. I mean, I think like, I don't know, I don't know what it is, you know, 4X, 5X, it's kind of crazy. So they're clearly really trying to recruit the best AI researchers in the world. And, you know, I don't know, it's, they do find the best AI researchers in the world. And I think my response to your thing is like, you know, almost all of these 150 IQ people, you know, if you just hire them tomorrow, they wouldn't be good AI researchers. They But they're willing to make investments that take years to pan out. The data centers they're buying right now will come online in 2026 or something. Why wouldn't they be able to make every 150 IQ person? Some of them won't work out. Some of them won't have the traits we like. But some of them by 2026 will be amazing AI researchers. Why aren't they making that bet? Yeah. And so sometimes this happens, right? Like smart physicists have been really good at AI research. You know, it's like all the anthropic co-founders. But like if you talk to, I had Daria on the podcast,

Dwarkesh Patel: they have this very careful policy of like, we're not going to just hire arbitrarily. We're going to be extremely selective.

Leopold Aschenbrenner: Training is not as easily scalable, right? So training is very hard. You know, if you just hired, you know, 100,000 people, it's like, I mean, you couldn't train them all. It'd be really hard to train them all. You know, you wouldn't be doing any AI research. Like, you know, there's huge costs to bringing on a new person training them. This is very different with AIs, right? And I think this is, it's really important to talk about the sort of like advantages the AIs will have. So it's like, you know, training, right? It's like, what does it take to be an Alec Radford? You know, we need to be in a really good engineer, right? The AIs, they're going to be an amazing engineer. They're going to be amazing at coding. You can just train them to do that. They need to have... you know, not just be a good engineer, but have really good research intuitions and like really understand deep learning. And this is stuff that, you know, I like Radford or, you know, somebody like him has acquired over years of research over just like being deeply immersed in deep learning, having tried lots of things himself and failed. Um, the eyes, you know, they're going to be able to read every research paper I've written every experiment ever run at the lab, you know, like gain the intuitions from all of this. They're going to be able to learn in parallel from all of each other's experiments, you know, experiences. Um, you know, I don't know what else, you know, it's like, what does it take to be an Alec Redford? Well, there's a, there's a sort of cultural acclimation aspect of it, right? You know, if you hire somebody new and there's like politicking, maybe they don't fit in. Well, in the AI case, you just make replicas, right? There's a like motivation aspect for it, right? So it's like, you know, Alec, you know, But I could just duplicate Alec Radford. And before I run every experiment, I have him spend a decade's worth of human time double checking the code and thinking really carefully about it. I mean, first of all, I don't have that many Alec Radfords. And he wouldn't care. And he would not be motivated. But the AI, they can just be like, look, I have 100 million of you guys. I'm just going to put you on, just really making sure this code is correct. There are no bugs. This experiment was thought through. Every hyperparameter is correct. Final thing I'll say is the 100 million human equivalent AI researchers, that is just a way to visualize it. So that doesn't mean you're going to have literally 100 million copies. So there's trade-offs you can make between serial speed and in parallel. So you might make the trade-off is, look, we're going to run them at 10x, 100x serial speed. It's going to result in fewer tokens overall because of sort of inherent trade-offs. But then we have, I don't know what the numbers would be, but then we have 100,000 of them running at 100x human speed and thinking, and there's other things you can do on coordination. They can kind of like share latent space, attend to each other's contacts. There's basically this huge range of possibilities of things you can do. The 100 million thing is more, I mean, another illustration of this is, if you kind of, I run the math in my series, and it's basically 27, 28, you have this automated AI researcher you're going to be able to generate an entire internet's worth of tokens every single day. So it's clearly sort of a huge amount of intellectual work that you can do. I think the analogous thing there is today we generate more patents in a year than during the actual physics revolution in the early 20th century that we're generating across like half a century or something. And are you making more physics progress in a year today?

Dwarkesh Patel: So yeah, you're going to generate all these tokens. Are you generating as much codified knowledge as humanity has been able to generate in the internet tokens are

Leopold Aschenbrenner: usually final output, right? A lot of these tokens, if we talk, we talked about the unhobbling, right. And I think, I think of a kind of like, you know, a GPTN token is sort of like one token of my internal monologue. Right. And so that's how I do this math on human equivalents. You know, it's like a hundred tokens a minute and then, you know, humans working for X hours and you know, what is the, what is the equivalent there? I think this goes back to something we were talking about earlier, where why haven't we seen the huge revenues from... People often ask this question, that if you took GPT-4 back 10 years and you showed people this,

Dwarkesh Patel: they think this is already automated, half the jobs. And so there's a sort of modus ponens, modus tollens here, where part of the explanation is like, oh, it's just on the verge, you need to do these unhobblings. And part of that is probably true, but there is another lesson to learn there, which is that just looking at face value at a set of abilities, there's probably more sort of hobblings that you don't realize that are hidden behind the scenes. I think the same will be true of the AGI that you have running as AI researchers.

Leopold Aschenbrenner: I basically agree, right? I think my story here is like, I think there's going to be some long tail, right? And so maybe it's like 26, 27. You're like the proto-automated engineer, and it's really good at engineering. It doesn't have the research intuition yet. You don't quite know how to put them to work. But even the underlying pace of AI progress is already so fast, right? In three years from not being able to do any kind of math at all to now crushing these math competitions. And so you have the initial thing in 26, 27. Maybe it's an automated research engineer. It speeds you up by 2x. You go through a lot more progress in that year. By the end of the year, you've figured out the remaining unhobblings. You've got a smarter model. And maybe then that thing, or maybe it's two years. And that thing really can automate 100%. And again, they don't need to be doing everything. They don't need to be making coffee. Maybe there's a bunch of tacit knowledge in a bunch of other fields. But AI researchers at AI labs really know the job of an AI researcher. And in some sense, there's lots of clear metrics. It's all virtual. There's code. It's things you can kind of develop and train for.

Dwarkesh Patel: So, I mean, another thing is how do you actually manage a million AI researchers? Humans, the sort of comparative ability we have that we've been especially trained for is

Leopold Aschenbrenner: like working in teams. And despite this fact, we have for thousands of years we've been learning about how we work together in groups. And despite this, management is a clusterfuck, right? It's like most companies are badly managed. It's really hard to do this stuff.

Dwarkesh Patel: for AIs, the sort of like, we talk about AGI, but it'll be some bespoke set of abilities, some of which will be higher than humans, some of which will be at human level. And so it'll be some bundle and we'll need to figure out how to... put these bundles together with their human overseers, with the equipment and everything. And the idea that as soon as you get the bundle, you'll figure out how to just shove millions of them together and manage them, I'm just very skeptical of. Like, any other revolution... technological revolution in history has been very piecemeal, much more piecemeal than you would expect on paper. If you just thought about what is the industrial revolution?

Leopold Aschenbrenner: Well, we dig up coal that powers the steam engines. You use the steam engines to run these railroads that helps us get more coal out. And there's sort of like Factorio story you can tell where in like six hours, you can be pumping thousands of times more coal. But in real life, it takes centuries often, right?

Dwarkesh Patel: In fact, the electrification, there's this famous study about how to initially to electrify factories. It was decades after electricity to change from the pulleys and water wheel based system that we had for steam engines to one that works with more spread out electrical motors and everything. I think this will be the same kind of thing. It might take like decades to actually get millions of AI researchers to work together.

Leopold Aschenbrenner: Okay, great. This is great. Okay, so a few responses to that. First of all, I mean, I totally agree with the kind of like real-world bottlenecks type of thing. I think this is sort of, you know, I think it's easy to underrate. You know, basically what we're doing is we're removing the labor constraint. We automate labor and we like kind of explode technology. But, you know, there's still lots of other bottlenecks in the world. And so I think this is part of why the story is it kind of like starts pretty narrow at the thing where you don't have these bottlenecks. And then only over time, as we let it, it kind of expands to sort of broader areas. This is part of why I think it's like initially this sort of AI research explosion, right? It's like AI research doesn't run into these real world bottlenecks. It doesn't require, you know, like plow a field or dig up coal. It's just, you're just doing AI research. The other thing, you know, the other thing about- I love how like in your model,

Dwarkesh Patel: the AI research, it's not complicated like about flipping a burger.

Leopold Aschenbrenner: It's just AI research. I mean, this is because people make these arguments of like, oh, you know, AGI won't do anything because it can't flip a burger. I'm like, yeah, it won't be able to flip a burger, but it's going to be able to do algorithmic progress, you know? And then when it does algorithmic progress, it'll figure out how to flip a burger, you know? And then we'll have the burger flipping robot. You know, look, sorry, the other thing is about, you know, again, these are the sort of quantities are lower bound, right? So it's like, this is just like, we can definitely run 100 million of these. probably what will happen is one of the first things we're gonna try to figure out is how to like, again, run like, you know, translate quantity into quality, right? And so it's like, even at the baseline rate of progress, you're like quickly getting smarter and smarter systems, right? If we said it was like, you know, four years between the preschooler and the high schooler, right? So I think, you know, pretty quickly, you know, there's probably some like simple algorithmic changes you find, you know, if instead of one Alec Bradford, you have a hundred, you know, you don't even need a hundred million. And then you get even smarter systems. And now these systems are capable of creative, complicated behavior you don't understand. Maybe there's some way to use all this test time compute in a more unified way rather than all these parallel copies. And so they won't just be quantitatively superhuman. They'll pretty quickly become qualitatively superhuman. You're a high school student, you're trying to wrap your mind around standard physics, and then there's some super smart professor who is like, quantum physics, it all makes sense to him, and you're just like, what is going on? I think pretty quickly, you enter that regime, just given even the underlying pace of AI progress, but even more quickly than that, because you have this accelerated force of now this automated AI research.

Dwarkesh Patel: I agree that over time, you would... I'm not denying that ASI is a thing that's possible. The time is just not that much, you know? I'm just like, how is this happening in a year? Like, you've... Okay, first of all...

Leopold Aschenbrenner: So I think the story is sort of like, basically, I think it's a little bit more continuous. You know, like, I think already, you know, like I talked about, you know, 25, 26, you're basically going to have models as good as a college graduate. And, you know... I don't know where the unhobbling is going to be, but I think it's possible that even then you have kind of the proto-automated engineer. So I think there is a bit of like a smear, kind of an AGI smear or whatever, where it's like there's sort of unhobblings that you're missing. There's kind of like ways of connecting them you're missing. There's like some level of intelligence you're missing. But then at some point you are going to get the thing that is like 100% automated Alec Radford. Once you have that, you know, things really take off, I think.

Dwarkesh Patel: Yeah. Okay. So let's go back to the end hobblings. Yeah. Is there, we're going to get a bunch of models by the end of the year. Is there something, let's suppose we didn't get some capacity by the end of the year. Yeah.

Leopold Aschenbrenner: Is there some such capacity which lacking would suggest that AI progress is going to take longer than you are projecting? Yeah, I mean, I think there's two kind of key things. There's the un-hobbling and there's the data wall, right? I think we should talk about the data wall for a moment. I think the data wall is, you know, even though kind of like all this stuff has been about, you know, crazy AI progress, I think the data wall is actually sort of underrated. I think there's like a real scenario where we're just stagnant. You know, because we've been running this tailwind of just like, it's really easy to bootstrap and you just do unsupervised learning, next token prediction, it learns these amazing world models and, like bam you know great model and you just got to buy some more compute you know do some simple efficiency changes um you know and and again like so much of deep learning all these like big gains on efficiency have been like pretty dumb things right like you know you add a normalization layer you know um you know you fix the scaling laws you know these already have been huge things let alone kind of like obvious ways in which these models aren't good yet anyway so data wall big deal um you know i don't know some like put some numbers on this. You do Common Crawl online as 30 trillion tokens. Lama3 trained on 15 trillion tokens. So you're basically already using all the data. And then you can get somewhat further by repeating it. So there's an academic paper by Boaz Barak and some others that does scaling laws for this. And they're basically like, yeah, you can repeat it sometime. After 16 times of repetition, returns basically go to zero. You're just completely screwed. And so, I don't know, say you can get another 10X on data from, say like LAMA3 and GPT-4, you know, LAMA3 is already kind of like at the limit of all the data. You know, maybe you can get 10X more by repeating data. You know, I don't know. Maybe that's like at most 100X better model than GPT-4, which is like, you know, 100X effective compute from GPT-4 is, you know, not that much. You know, if you do half an order of magnitude a year of compute, half an order of magnitude a year of algorithmic progress, you know, that's kind of like two years from GPT-4. So, you know, GPT-4 finished pre-training in 22, you know, 24. Yeah. So I think one thing that really matters, I think we won't quite know by end of the year, but 25, 26, are we cracking the data wall?

Dwarkesh Patel: Okay, so suppose we had three orders of magnitude less data in common crawl on the internet than we just happen to have now. And for decades, the internet, other things, we've been rapidly increasing the stock of data that humanity has. Is it your view that for contingent reasons, we just happen to have enough data to train models that are just powerful enough at 4.5 level where they can kick off the self-play RL loop? Or is it just that if it had been three rooms higher, then progress would have been slightly faster? In that world, we would have been looking back at how hard it would have been to kick off the RL explosion with just 4.5, but we would have figured it out. And then so in this world, we would have gotten to GPT-3 and then we'd have to kick off some sort of RL explosion. But we would have still figured it out. We didn't just like luck out on the amount of data we happen to have in the world.

Leopold Aschenbrenner: I mean, 3 ohms is pretty rough, right? Like, 3 ohms, if less data means, like, 6 ohms smaller model. 6 ohms, like, less compute model and chill scaling laws. You know, it's basically capping out at, like, GPT-2. So I think that would be really rough. I think you do make an interesting point about the contingency. I guess earlier, we were talking about when in the human trajectory are you able to learn from yourself. And so if we go with that analogy, again, if you'd only gotten the preschooler model, it can't learn from itself. If you'd only gotten the elementary schooler model, it can't learn from itself. And maybe GPT-4, smart high school is really where it starts. Ideally, you have a somewhat better model, and then it really is able to learn from itself or learn by itself. So yeah, I think there's an interesting, I mean, I think maybe one UMLess data, I would be more iffy, but maybe still doable. Yeah, I think it would feel chiller if we had, you know, like one or two...

Dwarkesh Patel: It would be an interesting exercise to get probability distributions of HEI contingent done across, like, rooms of data. Okay. The thing that makes me skeptical of this story is that the things... It totally makes sense why pre-training works so well. These other things, they're stories of, in principle, why they ought to work. Like, humans can learn this way and so on. And maybe they're true. But I worry that a lot of this case is based on sort of first principles evaluation of how learning happens, that fundamentally we don't understand how humans learn and maybe there's some key thing we're missing. On the sort of sample efficiency, yeah, humans actually, maybe there's, you say, well, the fact that these things are way less sample efficient in terms of learning than humans are suggests that there's a lot of room for improvement. Another perspective is that we are just on the wrong path altogether, right? That's why there's a sample inefficient when it comes to pre-trading. So yeah, I'm just like, there's a lot of like first principles arguments stacked on top of each other where you get these unhopplings and then you get to AGI. Then you, because of these reasons why you can stack all these things on top of each other, you get to ASI. And I'm worried that there's too many steps of this sort of first principles thinking.

Leopold Aschenbrenner: I mean, we'll see, right? I mean, on the sort of sample efficiency thing, again, sort of first principles, but I think, again, there's this clear sort of missing middle. And so, you know, and sort of like, you know, people hadn't been trying. Now people are really trying, you know? And so it's sort of, you know, I think often, again, in deep learning, something like the obvious thing works, and there's a lot of details to get right. So it might take some time, but it's now where people are really trying. So I think we get a lot of signal in the next couple of years. You know... On hobbling, I mean, what is the signal on hobbling that I think would be interesting? I think the question is basically like, are you making progress on this test time compute thing, right? Like, is this thing able to think longer horizon than just a couple hundred tokens, right? That was unlocked by chain of thought.

Dwarkesh Patel: And on that point in particular, the many people who have longer timelines that have come on the podcast have made the point that the way to train this long horizon RL, it's not, I mean, earlier talking about like, well, they can think for five minutes, but not for longer. Yeah. but it's not because they can't physically output an hours worth of tokens. It's just really, at least from what I understand, what they say.

Leopold Aschenbrenner: Right, like even like Gemini has like a million contacts and the million of contacts is actually great for consumption. And it solves one important hobbling, which is the sort of onboarding problem, right? Which is, you know, a new coworker in your first five minutes, like a new smart high school intern, first five minutes, not useful at all. A month in, much more useful, right? Because they've looked at the monorepo and understand how the code works and they've read your internal docs. And so being able to put that in context, great, solves this onboarding problem. Yeah, but they're not good at sort of the production of a million tokens yet.

Dwarkesh Patel: Right. But on the production of a million tokens, there's no public evidence that there's some easy loss function where you can...

Leopold Aschenbrenner: GPT-4 has gotten a lot better since... Actually, so the GPT-4 gains since launch, I think, are a huge indicator that there's, like, You know, so you talked about this with John Sherman on the podcast. John said this was mostly post-training gains. You know, if you look at the sort of LMSIS scores, you know, it's like 100 ELO or something. It's like a bigger gap than between Clod 3 Opus and Clod 3 Haiku. And the price difference between those is 60x.

Dwarkesh Patel: But it's not more agentic. It's better in the same chatbot way.

Leopold Aschenbrenner: It's much better at math, right? It went from 40% to 70% math.

Dwarkesh Patel: The crux is whether you could be able to... No,

Leopold Aschenbrenner: but I think it indicates that clearly there's stuff to be done on hobbling. I think the interesting question is this time a year from now, is there a model that is able to think for a few thousand tokens? coherently, cohesively, agentically. And I think probably there's, you know, again, this is, I'd feel better if we had an oom or two more data, because it's like the scaling just gives you this sort of like tailwind, right? Where like, for example, tools, right? Tools, I think, you know, talking to people who try to make things work with tools, you know, actually sort of GP4 is really when tools start to work. And it's like, you can kind of make them work with GP3.5, but it's just really tough. And so it's just like having GP4, you can kind of help it learn tools in a much easier way. And so just a bit more tailwind from scaling. And then, yeah, and does, I don't know if it'll work, but it's a key question.

Dwarkesh Patel: Okay, I think it's a good place to sort of close that part where we know what the crux is and what the progress, what evidence that would look like.

Leopold Aschenbrenner: On the AGI to superintelligence,

Dwarkesh Patel: Maybe it's the case that the games are really easy right now and you can just sort of let loose an Alec Ratfer, give him a compute budget and he comes out the other end with something that is an additive, like change this part of the code, this is a compute multiplier, change this other part. What other parts of the world Okay, maybe here's an interesting way to ask this. How many other domains in the world are like this where you think you could get the equivalent of in one year you just throw enough intelligence across multiple instances and you just come out the other end with... something that is remarkably decades, centuries ahead. Like, you start off with no flight, and then you have the Wright brothers, a million instances of GPT-6, and you come out the other end with Starlink. Is that your model of how things work?

Leopold Aschenbrenner: I think you're exaggerating the timelines a little bit, but I think a decade's worth of progress in a year or something, I think that's a reasonable prompt. So I think this is where basically the automated AI researcher comes in, because it gives you this enormous headwind on all the other stuff. So it's like you automate AI research with your automated Alec Radfords. You come out the other end. You've done another five OOMs. You have a thing that is vastly smarter. Not only is it vastly smarter, you've been able to make it good at everything else, right? You're solving robotics. The robots are important, right? Because for a lot of other things, you do actually need to try things in the physical world. I mean, I don't know, maybe you can do a lot in simulation. Those are the really quick worlds. I don't know if you saw the last NVIDIA GTC, and it was all about the digital twins and just having all your manufacturing processes in simulation. I don't know. Again, if you have these super intelligent cognitive workers, can they just make simulations of everything, kind of off-the-float style, and then make a lot of progress in simulation possible? But I also just think you're going to get the robots. Again, I agree about there are a lot of real-world bottlenecks. It's quite possible that we're going to have crazy drone swarms, but also lawyers and doctors still need to be humans because of regulation. But I think you start narrowly, you broaden, and then the worlds in which you let them loose, which again, because of these competitive pressures, we will have to let them loose in some degree on various national security applications. I think quite rapid progress is possible. The other thing though, is it's sort of, you know, basically in the sort of an explosion after, there's kind of two components. There's the A, right, in the production function, like growth of technology. And that's massively accelerated by, you know, you have a billion super intelligent scientists and engineers and technicians, you know, superbly competent at everything. You also just automated labor. Right. And so it's like even without the whole technological explosion thing, you have this industrial explosion, at least if you let them let them loose, which is like now you can just build, you know, you can cover Nevada and like, you know, you start with one robot factory is producing more robots. And basically it's like just the cumulative process because you've taken labor out of the equation. Yeah, that's super interesting. Although, when you increase the K or the L without increasing the A,

Dwarkesh Patel: you can look at the Soviet Union or China, where they rapidly increased inputs. And that does have the effect of being geopolitically game-changing, where it is remarkable. You go to Shanghai over a course of decades.

Leopold Aschenbrenner: I mean, they throw up these crazy cities in a decade.

Dwarkesh Patel: Right, right. I mean, the closest thing to, like, People talk about 30% growth rates or whatever. Yeah, Asian tigers, 10%.

Leopold Aschenbrenner: It's totally possible, right?

Dwarkesh Patel: But without productivity gains, it's not like the industrial revolution where from the perspective of you're looking at a system from the outside, your goods have gotten cheaper, they can manufacture more things. But it's not like the next century is coming at you.

Leopold Aschenbrenner: Yeah, it's both, it's both. So it's, you know, both that are important. The other thing I'll say is like, all of this stuff, I think the magnitudes are really, really important, right? So, you know, we talked about a 10X of research effort or maybe 10, 30X over a decade, you know, even without any kind of like self-improvement type loop, Even in the GP4.AGI story, we're talking about an order of magnitude of effective compute increase a year. Half an order of magnitude of compute, half an order of magnitude of algorithmic progress. That sort of translates into effective compute. And so you're doing a 10x a year, basically on your labor force. It's a radically different world if you're doing a 10x or a 30x in a century versus a 10x a year on your labor force. The magnitudes really matter. They also really matter on the intelligence explosion. Just the automated AI research part. One story you could tell there is, well, ideas get harder to find. Algorithmic progress is going to get harder. Yeah, right now you have the easy wins, but in four or five years, there'll be fewer easy wins. The automated AI researchers are just going to be what's necessary to just keep it going because it's gotten harder. But that's sort of, it's like a really weird knife-edge assumption in economics, where you assume it's just enough. But isn't that the equilibrium story you were just telling with why the economy as a whole has 2% economic growth? Because you just proceeded on the equilibrium. I guess you're saying by the time you get to the equilibrium... But the result of the equilibrium here is it's like way faster, at least, you know, and it's at least, and it depends on the sort of exponents, but it's basically, it's the increase, like suppose you need to like 10x effective research effort in AI research in the last, you know, four or five years to keep the pace of progress. we're not just getting a 10X, you're getting a million X or a hundred thousand X. There's just the magnitudes really matter. And the magnitudes just basically, one way to think about this is that you have kind of two exponentials. You have your sort of like normal economy that's growing at 2% a year and you have your like AI economy and that's going at like 10X a year. And it's starting out really small, but sort of eventually it's gonna, it's just, it's way faster and eventually it's gonna overtake, right? And even if you have, you can almost sort of just do the simple revenue extrapolation, right? If you think your AI economy, that has some growth rate, I mean, it's a very simplistic way and so on, but there's, there's this sort of 10 X a year process. And that will eventually kind of like, you're going to transition the sort of whole economy from, as it broadens from the sort of, you know, 2% a year to the sort of much faster growing process. Um, and I don't know, I think that's very like, uh, um, um, consistent with historical change, you know, stories of, right. There's this sort of like, uh, you know, there's this sort of long run hyperbolic trend, you know, it manifested in the sort of like, sort of change in growth mode in the Australian revolution, but there's just this long run hyperbolic trend. And, you know, now you have this sort of, now you have that another sort of change in growth mode.

Dwarkesh Patel: Yeah, yeah. I mean, that was one of the questions I asked Tyler when I had him on the podcast is that you do go from... The fact that after 1776, you go from a regime of negligible economic growth, 2%, is really interesting. It shows that, I mean,

Leopold Aschenbrenner: from the perspective of somebody in the Middle Ages or before, 2% is equivalent to the sort of 10%. Yeah. I guess you're projecting even higher for the AI economy, but... Yeah, I mean, it depends. I think, again, and it's all this stuff, you know, I have a lot of uncertainty, right? So a lot of the time I'm trying to kind of tell the modal story because I think it's important to be kind of concrete and visceral about it. And, you know, I have a lot of uncertainty basically over how the 2030s play out. And basically the thing I know is it's going to be fucking crazy. But, you know, exactly what, you know, where the bottlenecks are and so on, I think that'll be kind of like... So let's talk through the numbers here. Hundreds of millions of AI researchers.

Dwarkesh Patel: So right now, GPD 4.0 Turbo is like 15 bucks for a million tokens outputted. And a human thinks 150 tokens a minute or something. And if you do the math on that, I think it's for an hour's worth of human output. It's like 10 cents or something.

Leopold Aschenbrenner: Mm-hmm. now cheaper than a human worker cheaper than a human worker but it can't do the job

Dwarkesh Patel: that's right that's right but by the time you're talking about models that are trained on the 10 gigawatt cluster yeah then you have something that is four orders of magnitude more expensive yeah inference three orders of magnitude something like that so that's like a hundred dollars an hour of labor and now you're having hundreds of millions of such laborers

Leopold Aschenbrenner: Is there enough compute to do with the model that is a thousand times bigger, this kind of labor? Great. Okay. Great question. So I actually don't think inference costs for sort of frontier models are necessarily going to go up that much. So, I mean, one historical data point. But isn't the test time sort of thing that it will go up even higher? I mean, we're just doing per token, right? And then I'm just saying, you know, if suppose each model token was the same as sort of a human token thing at a hundred tokens a minute. So it's like, yeah, it'll use more, but the token calculations is already pricing that in. The question is per token pricing, right? And so GBD3, when it launched, was actually more expensive than GBD4 now. And so over just fast increases in capability gains, inference costs has remained constant. That's sort of wild, and I think it's worth appreciating, and I think it gestures at sort of an underlying pace of algorithmic progress. I think there's a sort of more theoretically grounded way to why inference costs would stay constant, and it's the following story. So on Cicelli scaling laws, half of the additional compute you allocate to bigger models and half of it you allocate to more data, right? But also if we go with the sort of basic story of half an order of year more compute and half an order of magnitude a year of algorithmic progress, you're also kind of like, you're saving half an order of magnitude a year. And so that kind of would exactly compensate for making the model bigger. The caveat on that is, you know, obviously not all training efficiencies are also inference efficiencies. You know, a bunch of the time they are. Separately, you can find inference efficiencies. So, I don't know, given this historical trend, given this sort of like, you know, baseline sort of theoretical reason, you know, I don't know. I think it's not a crazy baseline assumption that actually these models, the frontier models, are not necessarily going to get more expensive per token. Oh, really? Yeah. Like, okay, that's wild. We'll see, we'll see. I mean, the other thing, maybe they get, you know, even if they get like 10x more expensive, then, you know, you have 10 million instead of 100 million, you know, so it's like, it's not really, you know,

Dwarkesh Patel: like... But, okay, so part of the intelligent explosion is that each of them has to run experiments that are GPT-4 sized, and the result, so that takes up a bunch of compute. Yes. They need to consolidate the results of the experiments, and what is the synthesized

Leopold Aschenbrenner: I mean, you have a much bigger influence street anyway than your training. But I think the experiment compute is a constraint.

Dwarkesh Patel: Going back to maybe a sort of bigger fundamental thing we're talking about here. We're projecting in a series you say we should denominate the probability of getting to AGI in terms of orders of magnitude of effective compute. Effective here, accounting for the fact that there's a quote unquote compute multiplier if you have better algorithms. Yes. And I'm not sure that it makes sense to be confident that this is a sensible way to project progress. It might be, but I'm just like, I have a lot of uncertainty about it. It seems similar to somebody trying to project when we're going to get to the moon and they're like looking at the Apollo program in the four fifties or something. And they're like, we have some amount of effective jet fuel. And if we get more efficient engines, then we have more effective jet fuel. And so we're going to like probability of getting to the moon based on the amount of effective jet fuel we have. And I don't deny that jet fuel is important to launch rockets, but that seems like an odd way to denominate when you're going to get to the moon.

Leopold Aschenbrenner: Yeah. So, I mean, I think these cases are pretty different. I don't know. I don't think there is a sort of clear, I don't know how rocket science works, but I didn't get the impression that there's some clear scaling behavior with like, you know, the amount of jet fuel. I think in AI, you know, I mean, first of all, the scaling laws, you know, they've just helped, right? And so if you, a friend of mine pointed this out, and I think it's a great point, if you kind of concatenate both these sort of original Kaplan scaling laws paper that I think went from 10 to the negative nine to 10 petaflop days, and then, you know, concatenate additional compute from from there to kind of GPT-4, you assume some algorithmic progress, you know, it's like the scaling laws have held, you know, like probably over 15 OOMs, you know, I don't know, it's rough, probably maybe even more, held for a lot of OOMs.

Dwarkesh Patel: They held for the specific loss function, which they're trained on, which is a training mix token. Whereas the progress you are forecasting will be required for further progress in capabilities. Specifically, we know that scaling can't work because of the data wall. And so there's some new thing that has to happen. And I'm not sure whether the, you can extrapolate that same scaling curve to tell us whether these hobblings will also, like, is this not on the same graph?

Leopold Aschenbrenner: The hobblings are just a separate thing. Yeah, exactly. So this is sort of like, you know, it's, yeah, so, I mean, a few things here, right? Okay, so, on the effect of compute scaling, the, um, In some sense, I think it's like people center the scaling laws because they're easy to explain and the sort of like, why is scaling matter? The scaling laws came way after people, or at least like Dario, Ilya realized that scaling mattered. And I think that almost more important than the sort of loss curve is just like, just in general, there's this great quote from Dario on your podcast. It's just like, Ilya was like, the models, they just want to learn. You make them bigger, they learn more. And that just applied just across domains generally, all the capabilities. And you can look at this in benchmarks. Again, like you say, headwind, data wall. And I'm sort of bracketing that and talking about that separately. The other thing is on hobblings, right? If you just put them on the effective compute graph, these on hobblings would be kind of huge, right? So I think- But what does it even mean?

Dwarkesh Patel: What is on the y-axis here?

Leopold Aschenbrenner: like say MLPR on this benchmark or whatever, right? And so, you know, like, you know, we mentioned the sort of the LMSIS differences, you know, RLHF, you know, again, as good as 100X more, chain of thought, right? Chain of just going from this prompting change, a simple algorithmic change can be like 10X effective compute increases on like math benchmarks. I think this is useful to illustrate that on-hobblings are large, but I think of them as slightly separate things. The way I think about it is that at a per-token level, I think GP4 is not that far away from a token of my internal monologue. Even 3.5 to 4 took us from the bottom of the human range to the top of the human range on a lot of high school tests. And so it's like a few more 3.5 to 4 jumps per token basis, like per token intelligence. And then you've got to unlock the test time. You've got to solve the onboarding problem, make it use a computer. And then you're getting real close. I'm reminded of... And again, the story might be wrong, but I think it is strikingly plausible. I agree. And in fact, I think actually, I mean, the other thing I'll say is like, you know, I say this 2027 timeline. I think it's unlikely, but I do think there's worlds that are like AGI next year. And that's basically if the test and compute overhang is really easy to crack. If it's really easy to crack, then you do like four rooms of test and compute, you know, from a few hundred tokens to a few million tokens, you know, quickly. And then, you know, again, maybe it's maybe only takes one or two, 3.5 to four jumps per token, like one or two of those jumps per token, plus uses test and compute. And you basically have the proto automated engineer. So I'm reminded of that. Uh, Steven Pinker releases his book on, um, what is it? The Better Angels of Our Nature.

Dwarkesh Patel: And it's like a couple years ago or something. And he says the secular decline in violence and war and everything. And you can just like plot the line from the end of World War II. In fact, before World War II, then these are just aberrations or whatever. And basically, as soon as it happens, Ukraine, Gaza, the everything is like...

Leopold Aschenbrenner: Impending ASI and crazy global conflict. Impending ASI and crazy new WNDs.

Dwarkesh Patel: I think this is the sort of thing that happens in history where you see a straight line and you're like, oh my gosh. And then just like as soon as you make that prediction, who was that famous author?

Leopold Aschenbrenner: So yeah, again, people are predicting deep learning will hit a wall every year. Maybe one year they're right, but it's like gone a long way and it hasn't hit a wall. We don't have that much more to go. And so yeah.

Dwarkesh Patel: Okay, so I think this is a sort of plausible story, and let's just run with it and see what it implies.

Leopold Aschenbrenner: Yeah.

Dwarkesh Patel: So in your series, you talk about alignment from the perspective of This is not about some doomer scheme to get the 0.01% probability distribution where things don't go off the rails. It's more about just controlling the systems, making sure they do what we intend them to do. If that's the case, and we're going to be in this sort of geopolitical conflict with China, and part of that will involve, and what we're worried about is them making these CCP bots that go out and take the red flag of Mao across the galaxies or something. then shouldn't we be worried about alignment as something that, if in the wrong hands,

Leopold Aschenbrenner: this is the thing that enables brainwashing, sort of dictatorial control? This seems like a worrying thing. This should be part of the sort of algorithmic secrets we keep hidden, right? How to align these models, because that's also something the CCP can use to control their models. I mean, I think in the world where you get the democratic coalition. Yeah. I mean, also just alignment is often dual use, right? Like RLHF, you know, it's like alignment team developed. It was great. You know, it was a, it was a big win for alignment, but it's also, you know, obviously makes these models useful. Right. Um, the, um, um, but yeah, so yeah, alignment enables the CCP bots. Uh, alignment also is what you need to get the, you know, get the sort of, you know, whatever USA is like follow the constitution and like disobey law, you know, unlawful orders and, you know, like respect separation of powers and checks and balances. And, um, So yeah, you need alignment for whatever you want to do. It's the sort of underlying technique.

Dwarkesh Patel: Tell me what you make of this take. I've been struggling with this a little bit. So fundamentally, there's many different ways the future could go. There's one path in which the Eliezer-type crazy AIs with the nanobots take the future and turn everything into gray goo or paperclips. And the more you solve alignment, the more that path of the decision tree is circumscribed. And then, so the more you solve alignment, the more it is just different humans and the visions they have. And of course, we know from history that things don't turn out the way you expect. So it's not like you can decide the future, but it will appear.

Leopold Aschenbrenner: It's part of the beauty of it, right? You want these mechanisms, error correction, pluralism.

Dwarkesh Patel: But from the perspective of anybody who's looking at the system, it'll be like, I can control where this thing is going to end up.

Leopold Aschenbrenner: And so the more you solve alignment and the more you circumscribe the different futures that are the result of AI will,

Dwarkesh Patel: the more that accentuates the conflict between humans and their visions of the future. And so in the world where alignment is solved, and the world in which alignment is solved is the world in which you have the most sort of human conflict over where to take AI.

Leopold Aschenbrenner: Yeah, I mean, by removing the worlds in which the AIs take over, then like, you know, the remaining worlds are the ones where it's like the humans decide what happens. And then as we talked about, there's a whole lot of, yeah, a whole lot of worlds and how that could go.

Dwarkesh Patel: And I worry, so when you think about alignment and it's just controlling these things.

Leopold Aschenbrenner: Yeah.

Dwarkesh Patel: Just think a little forward and there's worlds in which hopefully, you know, human descendants or some version of things in the future merge with super intelligences and they have the rules of their own, but they're in some sort of law and market based order. Uh huh.

Leopold Aschenbrenner: I worry about if you have things that are conscious and should be treated with rights. If you read about what alignment schemes actually are, and then you read these books about what actually happened during the Cultural Revolution, what happened when Stalin took over Russia, and you have... very strong monitoring from different instances where one, everybody's tasked with watching each other. You have brainwashing,

Dwarkesh Patel: you have red teaming, where you have the spy stuff you were talking about, where you try to convince somebody you're on like a defector and you see if they defect with you. And if they do, then you realize they're an enemy and then you... And listen, maybe I'm stretching the analogy too far, but the ease of these alignment techniques actually map onto something you could

Leopold Aschenbrenner: have read about during Mao's Cultural Revolution is a little bit troubling. Yeah, I mean, look, I think sentient AI is a whole other topic. I don't know if we want to talk about it. I agree that, like, it's going to be very important how we treat them. You know, in terms of, like, what you're actually programming these systems to do. Again, it's like alignment is just, it's a technical, it's a technical problem, a technical solution. It enables the CCP bots. I mean, in some sense, I think the, you know, I almost feel like the sort of model and also about talking about checks and balances is sort of, you know, like the Federal Reserve or Supreme Court justices. And there's a funny way in which they're kind of this like very dedicated order,

Speaker 2: you know,

Leopold Aschenbrenner: Supreme Court justices. And it's amazing. They're actually quite high quality, right? And they're like really smart people. They really believe in the Constitution. They love the Constitution. They believe in their principles. They have, you know, these wonderful, you know, and yeah, they have different persuasions, but they have sort of, I think, very sincere kind of debates about what is the meaning of the Constitution? You know, what is the best actuation of these principles? you know, I guess, you know, by the way, recommendation sort of SCOTUS oral arguments is like the best podcast, you know, when I run out of high quality content on the internet. I mean, I think there's going to be a process of like figuring out what the constitution should be. I think, you know, this constitution has like worked for a long time. You start with that. Maybe eventually things change enough that you want added to that. But anyway, you want them to like, you know, for example, for the checks and balances, they like, they really love the constitution and they believe in it and they take it really seriously. And like, look, at some point, yeah, you are going to have like AI police and AI military. But I think sort of like, you know, being able to ensure that they like, you know, believe in it in the way that like a Supreme Court justice does or like in the way that like a Federal Reserve, you know, official takes their job really seriously. Yeah. Yeah.

Dwarkesh Patel: And I guess the big open question is whether if you do the project or something like the project.

Leopold Aschenbrenner: The other important thing is a bunch of different factions need their own AIs. It's really important that each political party gets to have their own. You might totally disagree with their values, but it's really important that they get to have their own super intelligence. Again, I think it's that these classical liberal processes play out, including different people of different persuasions and so on. The AI advisors might not make them wise. They might not follow the advice or whatever, but I think it's important.

Dwarkesh Patel: Okay, so speaking of alignment, you seem pretty optimistic. So let's run through the source of the optimism. Yeah. I think you laid out different worlds in which we could get AI. Yeah. There's one that you think is low probability of next year where GPT-4 plus scaffolding plus unhopplings gets you to AGI.

Leopold Aschenbrenner: Not GPT-4, you know, like... Oh, sorry, sorry.

Dwarkesh Patel: So GPT-4, yeah.

Leopold Aschenbrenner: Yeah, yeah, yeah. And there's ones where it takes much longer. There's ones where it's something that's a couple years... 27 in a modal world, yeah. So...

Dwarkesh Patel: GPT-4 seems pretty aligned in the sense that I don't expect it to go off the rails. Yeah. And maybe with scaffolding, things might change. Yeah, I like how it works pretty good. Yeah, exactly. So, and maybe you will keep turning, there's cranks, you keep going up, and one of the cranks gets to the ASI. Yeah. Is there any point at which the sharp left turn happens? Is it when you start... Is it the case that you think plausibly when they act more like agents, this is a thing to worry about? Yeah. Is there anything qualitatively that you expect to change with regards to the enlightenment perspective? Yeah, yeah, yeah. So...

Leopold Aschenbrenner: I don't know if I believe in this concept of sharp left turn, but I do think there's basically, I think there's important qualitative changes that happen between now and kind of like somewhat superhuman systems, kind of like early on the intelligence explosion, and then important qualitative changes that happen from like early in intelligence explosion to kind of like true superintelligence and all its power and might. And let's talk about both of those. And so, okay, so the first part of the problem is, one, you know, we're going to have to solve ourselves, right? We're going to have to align the, like, initial AIs and the intelligent explosion, you know, the sort of automated Alec Bradford. I think there's kind of, like, I mean, two important things that change from GPT-4, right? So one of them is, you know, if you believe the story on, like, you know, synthetic data or L or self-play to get past the data wall, and if you believe this on a hobbling story, you know, at the end, you're going to have things, you know, they're agents, right? Including... they do long-term plans, right? Somehow they're able to act over long horizons, right? But you need that, right? That's the sort of prerequisite to be able to do the sort of automated AI research. And so I think there's basically, I basically think sort of pre-training is sort of alignment neutral in the sense of like, it has all these representations, it has good representations, it has representations of doing bad things, but it's not like scheming against you or whatever. I think this sort of misalignment can arise once you're doing more kind of long horizon training, right? And so you're training, again, too simplified example, but to kind of illustrate, you're training an AI to make money. And if you're just doing that with reinforcement learning, it might learn to commit fraud or lie or deceive or seek power simply because those are successful strategies in the real world, right? So maybe, you know, RL is basically it explores, maybe it figures out like, oh, it tries to like hack and then it gets some money and that made more money, you know, and then if that's successful, if that gets reward, that's just reinforced. So basically I think there's sort of more serious misalignments, kind of like misaligned long-term goals that could arise between now and, or that sort of necessarily have to be able to arise if you're able to get long horizon system. That's one. what you want to do in that situation is you want to add side constraints, right? So you want to add, you know, don't lie, don't deceive, don't commit fraud. And so how do you add those side constraints, right? The sort of basic idea you might have is like RLHF, right? You're kind of like, yeah, it has this goal of like, you know, make money or whatever, but you're watching what it's doing. It starts trying to like, you know, lie or deceive or fraud or whatever, break the law. You're just kind of like, thumbs down, don't do that. You anti-reinforce that, right? The critical issue that comes in is that these AI systems are getting superhuman. And they're going to be able to do things that are too complex for humans to evaluate. So again, even early on in the intelligence explosion, the automated AI researchers and engineers, they might write millions, billions, trillions of lines of complicated code. They might be doing all sorts of stuff you just don't understand anymore. And so in the million lines of code, is it somewhere hacking or exfiltrating itself or trying to go for the nukes or whatever? You don't know anymore. And so this sort of like thumbs up, thumbs down, pure RLHF doesn't fully work anymore. Second part of the picture, and maybe we can talk more about this. First part of the picture, I think it's gonna be like, there's a hard technical problem of what do you do sort of post RLHF, but I think it's a solvable problem. And it's like, there's various things I'm bullish on. I think there's like ways in which deep learning has shaped out favorably. The second part of the problem is you're going from your, like, initial systems and intelligence explosion to, like, super intelligence. And, you know, it's, like, many ooms. It ends up being, like, by the end of it, you have a thing that's vastly smarter than humans. I think the intelligence explosion is really scary from an alignment point of view, because basically if you have this rapid intelligence explosion, you know, less than a year or two years or whatever, you're going say in the period of a year from systems where like, you know, failure would be bad, but it's not catastrophic to like, you know, saying a bad word. It's like, you know, it's something goes awry to like, you know, Failure is like, you know, it exfiltrated itself. It starts hacking the military. It can do really bad things. You're going less than a year from sort of a world in which like, you know, it's some descendant of current systems and you kind of understand it. And it's like, you know, has good properties. There's something that potentially has a very sort of alien and different architecture, right? After having gone through another decade of a maladvances. I think one example there that's very salient to me is legible and faithful chain of thought, right? So a lot of the time when we're talking about these things, we're talking about, you know, it has tokens of thinking, and then it uses many tokens of thinking. And, you know, maybe we bootstrap ourselves by, you know, it's pre-trained, it learns to think in English, and we do something else on top, so it can do the sort of longer chains of thought. And so, you know, It's very plausible to me that for the initial automated alignment researchers, we don't need to do any complicated mechanistic interpretability. You can just literally read what they're thinking, which is great. It's a huge advantage. However, it's very likely not the most efficient way to do it. There's probably some way to have a recurrent architecture. It's all internal states. There's a much more efficient way to do it. That's what you get by the end of the year. You know, you're going this year from like RLHF plus plus some extension works to like, it's vastly superhuman. It's like, you know, it's to us like, you know, an expert in the field might be to like an elementary school or middle schooler. And so, you know, I think it's this sort of incredibly sort of like hairy period for alignment. Thing you do have is you have the automated AI researchers, right? And so you can use the automated AI researchers to also do alignment. And so in this world- Yeah. Why are we optimistic that the project is being run by people who are thinking?

Dwarkesh Patel: I think, so here's something to think about.

Speaker 2: Okay.

Dwarkesh Patel: The OpenAI starts off with people who are very explicitly thinking about exactly these kinds of things.

Leopold Aschenbrenner: Yes. Right? But are they still there? No, no. But so here's the thing. No, no. Even the people who are there, even like the current leadership, it's like exactly these things.

Dwarkesh Patel: You can find them in interviews in their blog posts talking about. And what happens is when, as you were talking about, when some sort of trivial... And Jan talked about it. This is not just you. Jan talked about it in his... tweet thread when there is some trade-off that has to be made with, we need to do this flashy release this week and not next week because whatever Google IO is the next week. So we're going to get it. And then the trade-off is made in favor of the less, the more careless decision.

Leopold Aschenbrenner: when we have the government or the national security advisor, the military, whatever, which is much less familiar with this kind of discourse, is it naturally thinking in this way about, ah, I'm worried the chain of thought isn't faithful and how do we think about the features that are represented here? Why shouldn't we be optimistic that a project run by people like that will be

Dwarkesh Patel: thoughtful about these kinds of considerations?

Leopold Aschenbrenner: I mean, they might not be. You know, I agree. I think... All right, a few thoughts, right? First of all, I think the private world, even if they sort of nominally care, is extremely tough for alignment. A couple of reasons. One, you just have the race between the sort of commercial labs, right? And it's like, you don't have any headroom there to be like, ah, actually, we're going to hold back for three months to get this right. And we're going to dedicate 90% of our compute to automated alignment research instead of just pushing the next zoom. The other thing, though, is in the private world, China has stolen your rights. China has your secrets. They're right on your tails. You're in this fever struggle. No room at all for maneuver. The way it's absolutely essential to get alignment right and to get it during this intelligence explosion to get it right is you need to have that room to maneuver and you need to have that clear lead. And again, maybe you've made the deal or whatever, but I think you're in an incredibly tough spot if you don't have this clear lead. Um, so I think the sort of private world is kind of rough there on like whether people will take it seriously. You know, I don't know. I have some faith in sort of, sort of normal mechanisms of a liberal society that sort of, if, if, if alignment is an issue, which, you know, we don't fully know yet, but sort of the science will develop, we're going to get better measurements of alignment, you know, and, and the case will be clear and obvious. Um, I worry about worlds where evidence is ambiguous. And I think a lot of the most scary intelligence explosion scenarios are worlds in which evidence is ambiguous. But again, if evidence is ambiguous, then that's the worlds in which you really want the safety margins. And that's also the worlds in which running the intelligence explosion is sort of like running a war. It's like, ah, the evidence is ambiguous. We have to make these really tough trade-offs. And you better have a really good chain of command for that. And it's not just yoloing at, ah, let's go. It's cool.

Dwarkesh Patel: Let's talk a little bit about Germany. We're making the analogy to World War II, and you made a really interesting point many hours ago at this point.

Speaker 4: We should saw it after, you know, after the marathon.

Leopold Aschenbrenner: The fact that throughout history, World War II is not unique, at least when you think in proportion to the size of the population. But these other sorts of catastrophes where a significant portion of the population has been killed off.

Speaker 2: Yeah.

Leopold Aschenbrenner: After that, the nation recovers and they get back to their heights. Uh-huh. So what's interesting after World War II is that Germany especially, and maybe Europe as a whole,

Dwarkesh Patel: obviously they experienced fast economic growth in the direct aftermath because of catch-up growth. But subsequently, we just don't think of Germany as...

Leopold Aschenbrenner: We're not talking about Germany potentially launching an intelligence explosion and they're going to get in and see it at the AI table.

Dwarkesh Patel: We were talking about Iran and North Korea and Russia. We didn't talk about Germany, right?

Leopold Aschenbrenner: Well, because they're our allies. Yeah, yeah, yeah.

Dwarkesh Patel: But...

Leopold Aschenbrenner: So what happened? I mean, World War Two and now it didn't like come back to the seven years war or something, right? Yeah, yeah, yeah. I mean, look, I'm generally very bearish on Germany. I think in this context, I'm kind of like, you know, it's a little bit, you know, I think you're underrating a little bit. I think it's probably still one of the, you know, top five most important countries in the world. You know, I mean, Europe overall, you know, it still has, I mean, it's GDP that's like close to the United States, the size of GDP, you know, and there's things actually that Germany is kind of good at, right? Like state capacity, right? Like, you know, the, you know, the roads are good and they're clean and they're well maintained. And, and, um, you know, in some sense, the sort of, um, a lot of this is the sort of flip side of things that I think are bad about Germany. Right. So in the U S it's a little bit like, there's a bit more of a sort of wild West feeling to the United States. Right. And it includes the kind of like crazy bursts of creativity includes like, you know, um, uh, Political candidates that are sort of you know, there's much broader spectrum and you know, you know Like both in Obama and Trump is somebody you just wouldn't see in the sort of much more confined kind of German political debate You know, I wrote this blog post at some point your political stupor about this But anyway, and so there's this sort of punctilious sort of rule following that is like good in terms of like, you know Keeping your kind of state capacity functioning but that is also I think there's a very constrained view of the world in some sense. That includes, I think after World War II, there's a real backlash against anything elite. Again, no elite high schools or elite colleges and meritocracy. Why is that the logic? Excellence isn't cherished. Yeah. Why is that the logical, intellectual thing to rebel against if you're trying to overcorrect from the Nazis? Was it because the Nazis were very much into elitism? I don't understand why that's a logical sort of counter-reaction. I don't know. Maybe it was sort of a counter-reaction against the whole Aryan race and that sort of thing. I mean, I also just think there was a certain amount in what certain... I mean, look at sort of World War I, end of World War I versus end of World War II for Germany, right? And sort of, you know, a common narrative is that the peace of Versailles, you know, was too strict on Germany. But, you know, the peace imposed after World War II was, like, much more strict, right? It was a complete, you know, I mean, the whole country was destroyed. You know, it was, you know, in most of the major cities, you know, over half of the housing stock had been destroyed, right? Like... In some birth cohorts, 40% of the men had died. Half the population displaced. Oh, yeah. Almost 20 million people displaced. Huge. Crazy, right? And the borders are way smaller than the Versailles borders. Yeah, exactly. And sort of complete imposition of a new political system on both sides. And... um yeah so it was um but in some sense that worked out better than the post-world war one piece um where then there was this kind of resurgence of german nationalism and um you know in some sense the thing that has been a pattern so it's sort of like it's unclear if you want to wake the sleeping beast i do think that at this point you know it's gotten a bit too sleepy yeah um um i do think it's an interesting point about we underrate the american political system yeah and i've been making the same correction myself yeah uh there's There was this book written by a Chinese economist called China's Worldview. And overall, I wasn't a big fan, but they made a really interesting point in there, which was the way in which candidates rise up through the Chinese hierarchy for politics, for administration. In some sense, it selects for, you're not going to get some Marjorie Taylor Greene or somebody running some... Don't get that in Germany either. Right. But he explicitly made the point in the book that that also means we're never going to get a Henry Kissinger or Barack Obama in China.

Dwarkesh Patel: We're going to get like,

Leopold Aschenbrenner: by the time they end up in charge of the Politburo, there'll be like some 60-year-old bureaucrat who's never like ruffled any feathers. Yeah, yeah, yeah. I think there's something really important about this very raucous political debate. In general, there's the sense in which in America, lots of people live in their own world. We live in this bizarre little bubble in San Francisco. I think that's important for the evolution of ideas, error correction, that sort of thing. you know, there's other ways in which the German system is more functional. Yeah. But it also leads to major mistakes, right? Like the sort of defense spending, right? And, you know, then, you know, Russia invades Ukraine and you're like, wow, what did we do? Right. No, that's a really good point, right? The main issue is there's, everybody agrees, but... Yeah. It's a consensus blob kind of thing. Right. On the China point, you know, just having this experience of like reading German newspapers and I think how much, you know, how much more poorly I would understand the sort of German debate and sort of the sort of state of mind from just kind of afar. I worry a lot about, you know, where I think it is interesting just how kind of impenetrable China is to me. It's a billion people, right? And like, you know, almost everything else is really globalized. You have a globalized internet. And I kind of have a sense of what's happening in the UK. You know, I probably, even if I didn't read German newspapers, I sort of would have a sense of what's happening in Germany. But I really don't feel like I have a sense of what, like, you know, what is the state of mind or the state of political debate, you know, of a sort of average Chinese person or like an average Chinese leader. And yeah, I think that that I find that distance kind of worrying. And I, you know, you know, and there's you know, there's some people who do this and they do really great work where they kind of go through the like party documents and the party speeches. And it seems to require a kind of a lot of interpretive ability where there's like very specific words in Mandarin that like mean will have one connotation, not the other connotation. Um, but yeah, I think it's sort of interesting given how globalized everything is. And like, I mean, now we have basically perfect translation machines and it's still so, so impenetrable.

Dwarkesh Patel: That's really interesting. I've been, I should, I'm sort of ashamed almost that I haven't done this yet. Yeah. I think many months ago, I, when Alexi interviewed me on his YouTube channel, I said, I'm meaning to go to China to actually see for myself what's going on. And actually, I should... So, by the way, if anybody listening has a lot of context on China, if I went to China, who could introduce me to people, please email me.

Leopold Aschenbrenner: You got to do some pods. And you got to find some of the Chinese AI researchers, man.

Dwarkesh Patel: I know. I think that would be so good.

Leopold Aschenbrenner: I was thinking at some point... Again, this is... Can I speak freely? I don't know if they can speak freely. I was thinking of... So, they had these papers. And on the paper, they'll say who's a co-author. It's funny because... Well, I was thinking of just cold emailing everybody, like, here's my calendar. Let's just talk. I just want to see what is the vibe. Even if they don't tell me anything, I'm just like, what kind of person is this? How westernized are they? But as I was saying this, I just remembered that, in fact, ByteDance, according to mutual friends we have at Google,

Dwarkesh Patel: they cold emailed every single person on the Gemini paper and said, if you come work for ByteDance, we'll make you an elite engineer.

Leopold Aschenbrenner: You'll report directly to the CTO. And In fact, this actually, I'm going to go. That's how the secrets go over, right?

Dwarkesh Patel: Right. No,

Leopold Aschenbrenner: I meant to ask this earlier, but suppose they hired one, if there's only a hundred or so people or maybe less, we're working on the key algorithmic secrets. Yeah, yeah, yeah. If they hired one such person, is all the alpha gone that these labs have? if this person was intentional about it, they could get a lot. I mean, they couldn't get the sort of like, I mean, actually it could probably just also exfiltrate the code. They could get a lot of the key ideas. Again, like, you know, up until recently stuff was published, but you know, they could get a lot of the key ideas if they tried, if they like, you know, I think there's a lot of people who don't actually kind of like look around to see what the other teams are doing, but you know, I think you kind of can. But yeah, I mean, they could, it's scary. Right. I think the project makes more sense there where you can't just recruit a Manhattan project engineer and then just get it. And it's like, these are secrets that can be used for like probably every training on the future. That'll be like maybe are the key to the data wall that they can't go on or they can go on that are like, you know, they're going to be worth, you know, given sort of like the multipliers on compute, you know, hundreds of billions, trillions of dollars, you know, and all it takes is, you know, China to offer a hundred million dollars to somebody and be like,

Speaker 2: ah,

Leopold Aschenbrenner: it can work for us. Right. And then, and then. Yep. I mean, yeah, I'm really uncertain on how sort of seriously China is taking AGI right now. One anecdote that was related to me on the topic of anecdotes by another sort of like, you know, kind of researcher in the field was at some point they were at a conference with somebody, a Chinese AI researcher, and he was talking to him and he was like, I think it's really good that you're here. And like, you know, we got to have the international coordination and stuff. And apparently this guy said that I'm the kind of most senior most person that they're going to let leave the country to come to things like this.

Dwarkesh Patel: Mm hmm. Wait, what's the takeaway?

Leopold Aschenbrenner: As in they're not letting really senior AI researchers leave the country. Interesting. Kind of classic, you know, Eastern Bloc move. Yeah. I don't know if this is true, but it's what I heard. That's interesting. So I thought the point you made earlier about being exposed to German newspapers and also to,

Dwarkesh Patel: because earlier you were interested in economics and law and national security. The variety and intellectual diet there has exposed you to thinking about the

Leopold Aschenbrenner: geopolitical question here in ways others talking about. I mean, this is the first episode I've done about this where we've talked about things like this,

Dwarkesh Patel: which is, now that I think about it, weird given that this is an obvious thing in retrospect. I should have been thinking about. Anyways, so that's one thing we've been missing. What are you missing? And national security, you're thinking about, so you can't say national security. What, like, key perspective are you probably underexposed to as a result? And China, I guess you mentioned.

Leopold Aschenbrenner: Yeah, so I think the China one is an important one. I mean, I think another one would be a sort of very Tyler Cowen-esque take, which is, like, you're not exposed to how, like, how will a normal person in America,

Speaker 2: like,

Leopold Aschenbrenner: you know, both, like, use AI, you know, probably not,

Speaker 2: you know,

Leopold Aschenbrenner: And that being kind of like bottlenecks to the fusion of these things. I'm overrating the revenue because I'm kind of like, ah, you know, everyone at SF is adopting it. But, you know, kind of like, you know, Joe Schmo engineer at a company, you know, like, ah, will they be able to integrate it? And also the reaction to it, right? You know, I mean, I think this was a question again hours ago where it was about like, you know, won't people kind of rebel against this? And they won't want to do the project. I don't know, maybe they will. Yeah. Yeah.

Dwarkesh Patel: Here's a political reaction that I didn't anticipate.

Leopold Aschenbrenner: So, Tucker Carlson was recently on the Joe Rogan episode.

Dwarkesh Patel: I already told you about this, but I'm just gonna tell the story again. So, Tucker Carlson is on Joe Rogan. And they start talking about World War II. And Tucker says, well, listen, I'm gonna say something that my fellow conservatives won't like, but I think nuclear weapons are immoral. I think it was obviously immoral that we used them on Nagasaki and Hiroshima. And then he says... In fact, nuclear weapons are always immoral. Except when we would use them on data centers. In fact, it would be immoral not to use them on data centers. Because look, these people in Silicon Valley, these fucking nerds are making superintelligence, and they say that it could enslave humanity. We made machines to serve humanity, not to enslave humanity. And they're just going on and making these machines. And so we should of course be nuking the data centers.

Leopold Aschenbrenner: And that is definitely not a political reaction in 2024, I was expecting. I mean, who knows? It's going to be crazy. The thing we learned with COVID is that also the left-right reactions that you would anticipate just based on hunches It completely flipped multiple times. Initially, the right was like COVID. It's so contingent. And then the left was like, this is racist. And then it flipped. The left was really into the COVID. And the whole thing also is just so blunt and crude. And so I think probably in general, I think people are really under... People like to make sort of complicated technocratic AI policy proposals. And I think especially if things go kind of fairly rapidly on the path to AGI. Yeah. you know, there might not actually be that much space for kind of like complicated, kind of like, you know, clever proposals. It might just be kind of a much cruder reaction. Yeah. Look, and then also when you mentioned the spies and the national security getting involved and everything and You can talk about that in the abstract, but now that we're living in San Francisco and we know many of the people who are doing the top AI research, it's also a little scary to think about people I personally know and friends with. It's not unfeasible if they have secrets in their head that are worth $100 billion or something. Kidnappings, assassinations, sabotage. Oh, their family. Yeah, it's really bad. I mean, this is to the point on security. Right now, it's just really foreign, but... you know at some point as it becomes like really serious it's things you know you're gonna want the security cards yeah yeah yeah so presumably you have thought about the fact that people in china will be listening to this and reading your

Dwarkesh Patel: series yeah and somehow you made the trade-off that it's better to let the whole world yeah no yeah and also including China, and make them up to AGI, which is part of the thing you're worried about, is China making up to AGI, than to stay silent. I'm just curious, walk me through how you've thought about that trade-off.

Leopold Aschenbrenner: Yeah, I actually, look, I think this is a tough trade-off. I thought about this a bunch. I think people in the PRC will read this. I think, you know, I think there's some extent to which sort of cat is out of the bag. You know, this is like not, you know, AGI being a thing people are thinking about very seriously is not new anymore. There's sort of, you know, a lot of these takes are kind of old or, you know, I've had, I had, you know, similar views a year ago, might not have written it up a year ago in part because I think this cat wasn't out of the bag enough. You know, I think the other thing is, I think to be able to manage this challenge, I think much broader swaths of society will need to wake up. And if we're going to get the project, we actually need a broad bipartisan understanding of the challenges facing us. And so I think it's a tough trade-off, but I think the need to wake up people in the United States and the Western world and the Democratic coalition is ultimately imperative. And I think my hope is more people here will read it than the PRC. You know, and I think people sometimes underrate the importance of just kind of like writing it up, laying out the strategic picture. And, you know, I think you've done actually a great service to sort of mankind in some sense by, you know, with your podcast. And... I think it's overall been good. By the way, on the topic of Germany, we were talking at some point about immigration stories. I feel like you have an interesting story you haven't told and I think you should tell. A couple years ago, I was in college and I was 20. Yeah. I was about to turn 21. Yeah. I think it was... Yeah, you came from India when you were really young.

Dwarkesh Patel: Right, yeah, yeah.

Leopold Aschenbrenner: So until I was eight or nine, I lived in India. And then we moved around all over the place. But because of the backlog for Indians... Yeah, the green card backlog.

Dwarkesh Patel: Yeah, it's... we were, we've been in the queue for like decades.

Leopold Aschenbrenner: Even though you came at eight, you're still on the, you know, H1B.

Dwarkesh Patel: Yeah. And when you're 21, you get kicked off the queue and you had to restart the process. I'm on my dad's, my dad's a doctor and I'm on his H1B as a dependent. But when you're 21, you get kicked off. Yeah. And so I'm 20 and I just like kind of dawns on me that this is my situation. Yeah. And you're completely screwed. Right. And so I also had the experience that my dad, we've moved all around the country. They have to prove that him as a doctor is like, you can't get native talent. And you can't start a startup. You're just completely changed.

Leopold Aschenbrenner: And even getting the H-1B for you would have been like a 20% lottery. So it's like, if you're lucky, you're in this.

Dwarkesh Patel: And they had to prove that they can't get native talent, which means for him, we lived in North Dakota for three years, West Virginia for three years, Maryland, West Texas. And so it kind of dawned on me, this is my situation. As I turn 21, I'll be like on this lottery.

Leopold Aschenbrenner: Even if I get the lottery, I'll be a fucking code monkey for the rest of my life because this thing isn't going to let up. Can't do a startup.

Dwarkesh Patel: Exactly. And so at the same time, I had been reading for the last year, I've been super obsessed with Paul Graham essays. My plan at the time was to make a startup or something. I was super excited about that. And it just occurred to me that I couldn't do this. That like, this is just not in the cards for me. And so I was kind of depressed about it. I remember I kind of just... I was in a daze through finals because it just occurred to me and I was really anxious about it. And I remember thinking to myself at the time that if somehow I end up getting my green card before I turn 21, there's no fucking way I'm becoming a code monkey because the thing that I've... This feeling of dread that I have is this realization that... I'm just going to have to be a code monkey. And I realized that's my default path. If I hadn't sort of made a proactive effort not to do that, I would have graduated college as a computer science student and I would have just done that. And that's the thing I was super scared about. So that was an important sort of realization for me. Anyway, so COVID happened because of that. Since there weren't foreigners coming, the backlog cleared fast. And by the skin of my teeth, like a few months before I turned 21. Extremely contingent reasons.

Speaker 4: So crazy.

Dwarkesh Patel: I ended up getting a green card.

Leopold Aschenbrenner: Because I got a green card, I could... The whole podcast, right?

Dwarkesh Patel: Exactly.

Leopold Aschenbrenner: I graduated college and I was bumming around.

Dwarkesh Patel: And I graduated a semester early. I'm going to do this podcast. See what happens. And I didn't have a green card.

Leopold Aschenbrenner: It's such a magnificent cultural artifact. And it only existed. I think it's hard. It's probably... you know, what is the impact of like immigration reform? Like what is the impact of clearing, you know, like whatever, 50,000 green cards in the backlog. And you're such like an amazing example of like, you know, all of this is only possible. And it's, yeah, it's, I mean, it's just incredibly tragic that this is so dysfunctional. Yeah.

Dwarkesh Patel: Yeah. No, it's insane. I'm glad you did it.

Leopold Aschenbrenner: I'm glad you kind of like, you know, tried the, you know, the, the, the unusual path.

Dwarkesh Patel: Well, yeah, but I could only do it, obviously I was extremely fortunate that I got the green card. I was like, I had a little bit of saved up money and I got a small grant out of college thanks to the future fund to like do this for basically the equivalent of six months. And so it turned out really well. And then at each time when I was like, oh, okay, podcast, come on. Yeah, yeah, yeah. I wasted a few months on this. Let's now go do something real. Something big would happen. I would... And you kept with it. Huh?

Leopold Aschenbrenner: You kept with it.

Dwarkesh Patel: Yeah, yeah. But there would always be, just like the moment I'm about to quit the podcast, I'm being like, Jeff Bezos will say there's something nice about me on Twitter. The Ilya episode gets like half a million views. Nice, yeah. And then now, this is my career, but it was sort of very... looking back on it, incredibly contingent that things worked out the right way.

Leopold Aschenbrenner: Yeah. I mean, look, if the AGI stuff goes down, you know, it'll be the most important kind of, like, you know, source of... It'll be how maybe most of the people who kind of end up feeling AGI are assured about it. Yeah, yeah, yeah. Also, very much, you're very linked with this story in many ways. First, the...

Dwarkesh Patel: I got, like, a $20,000 grant from a future fund right out of College, and that sustained me for... six months or however long it was.

Leopold Aschenbrenner: And without that, I wouldn't- A tiny grant is kind of crazy.

Dwarkesh Patel: Yeah, 10 grand or what was it?

Leopold Aschenbrenner: No, it's just, it's tiny, but it goes to show kind of how far small grants can go.

Dwarkesh Patel: Sort of the immersion ventures too. Exactly, the immersion ventures.

Leopold Aschenbrenner: And the, well, the last year I've been in San Francisco, we've just been... in close contact the entire time and just bouncing ideas back and forth. And just basically the alpha I have, I think people would be surprised by how much I got from you, Sholto,

Dwarkesh Patel: Trent,

Leopold Aschenbrenner: and a couple others. It's been an absolute pleasure.

Dwarkesh Patel: Yeah, likewise. It's been super fun.

Leopold Aschenbrenner: Okay, so some random questions for you.

Dwarkesh Patel: Yeah.

Leopold Aschenbrenner: If you could convert to Mormonism, and you could really believe it, would you do it? Would you push the button? Well, okay. Before I answer that question, one sort of observation about the Mormons. Actually, there's an article that actually made a big impact on me. I think it was by McKick Hoppins at some point, you know, in The Atlantic or whatever, about the Mormons. And I think the thing he kind of... And I think he even interviewed Mitt Romney in it and so on. And I think the thing I thought was really interesting in this article was he kind of talked about how the experience of kind of growing up different, growing up very unusual, especially if you grow up Mormon outside of Utah, you're like the only person who doesn't drink caffeine. You don't drink alcohol. You're kind of weird. how that kind of got people prepared for being willing to be kind of outside of the norm later on. And like, you know, Mitt Romney, you know, was willing to kind of take stands alone, you know, in his party because he believed, you know, what he believed was true. And I don't know, I mean, probably not to the same way, but I feel a little bit like this from kind of having grown up in Germany and having, you know, and really not having like this German system and having been kind of an outsider or something. I think there's a certain amount in which kind of Growing up on an outsider gives you unusual strength later on to be willing to say what you think. That is one thing I really appreciate about the Mormons, at least the ones that grow up outside of Utah. The fertility rates, they're good, they're important. They're going down as well. This is the thing that really clinched the fertility decline story for me. Even the Mormons? Yeah, even the Mormons, right? You're like, oh, this is sort of a good start. The Mormons will replace everybody. Well, no, I don't know if it's good, but it's like, at least, you know, at least come on. At least some people will maintain height. But it's no, no, even the Mormons. And sort of basically, these religious subgroups have high fertility rates. Once they kind of grow big enough, they're too close in contact with sort of normal society and become normalized. Mormon fertility rates drop from... I remember the exact numbers, maybe like four to two in the course of 10, 20 years. Um, anyway, so it's like, you know, now people point to the, you know, Amish or whatever, but I'm just like, it's probably just not scalable. And if you grow big enough, then there's just like, you know, the sort of like, you know, the sort of like overwhelming force of modernity kind of gets you. Um, no, if I could convert to Mormonism, look, I think there's something. I don't believe it. Right. If I believed it, I obviously would convert to Mormonism. Right. Cause it's, you gotta, you gotta, you can choose the world in which you do believe it. Um, yeah. I think there's something really valuable in kind of believing in something greater than yourself and believing and having a certain amount of faith. Um, you do, right. And, and, and, and, you know, um, you know, there's a, you know, feeling some sort of duty to the thing greater than yourself. Um, you know, maybe my version of this is somewhat different. You know, I think I feel some sort of duty to like, I feel like there's some sort of historical weight on like how this might play out. And I feel some sort of duty to like make that go well. I feel some sort of duty to, you know, our country, to the national security of the United States. And, um, um, I think that can be a force for a lot of good.

Dwarkesh Patel: Going back to the opening, I think, the thing that's especially impressive about that is Look, there's people at the company who have, through years and decades of building up savings from working in tech, have probably tens of millions liquid, more than that in terms of their equity. And the person, very many people were concerned about the clusters in the Middle East and the secrets leaking to China and all these things.

Leopold Aschenbrenner: But the person who actually made a hassle about it, and I think hassling people is so underrated,

Dwarkesh Patel: I think that one person who made a hassle about it is the 22-year-old who has less than a year at the company, who doesn't have savings built up, who isn't a solidified member of the... I think that's a sort of like...

Leopold Aschenbrenner: Maybe it's me being naive and not knowing how big companies work. I think sometimes I'm a bit of a speech deontologist. I kind of believe in saying what you think. Sometimes friends tell me I should be more of a speech consequentialist. No, I think I really think the amount of people who, when they have the opportunity to talk to the person, will just bring up the thing. I've been with you in multiple contexts, and I guess I shouldn't reveal who the person is or what the context was,

Dwarkesh Patel: but I've just been very impressed that the dinner begins and by the end, somebody who has a major voice in how things go is seriously thinking about a worldview they would have found incredibly alien before the dinner or something. Um, and I've been impressed that like, just like, just give them the spiel and hassle them.

Leopold Aschenbrenner: Um, um, I mean, look, I just, I think, I think I feel this stuff pretty viscerally now.

Dwarkesh Patel: You know,

Leopold Aschenbrenner: I think there's a time, you know, there's a time when I thought about this stuff a lot, but it was kind of like econ models and like, you know, kind of like these sort of theoretical abstractions and, you know, you talk about human brain size or whatever. And I think, you know, since, um, I think since at least last year, you know, I feel like, you know, I feel like I can see it, you know, and I just, I feel it. And I think I can like, you know, I can sort of see the cluster that AGI is going to be trained on. And I can see the kind of rough combination of algorithms and the people that be involved and how this is going to play out. And, you know, I think, look, we'll see how it plays out. There's many ways this could be wrong. There's many ways it could go. But I think this could get very real.

Dwarkesh Patel: Should we talk about what you're up to next?

Leopold Aschenbrenner: Sure. Yeah.

Dwarkesh Patel: Okay. So you're starting an investment firm. Yep. Anchor Investments from Nat Friedman, Daniel Gross, Patrick Collison, John Collison. First of all, why is this the thing to do? You believe the AGI is coming in a few years.

Leopold Aschenbrenner: Why the investment firm? Good question. Fair question. So, I mean, a couple of things. One is just, you know, I think we talked about this earlier, but it's like the screen doesn't go blank, you know, when sort of AGI or I think people really underrate basically the decade after. You have the intelligence explosion, that's maybe the most wild period, but I think the decade after is also going to be wild. This combination of human institutions but superintelligence, you have crazy geopolitical things going on, you have the broadening of this explosive growth. Basically, yeah. I think it's going to be a really important period. I think capital will really matter. Eventually, going to go to the stars, going to go to the galaxies. So anyway, part of the answer is just like, look, I think done right, there's a lot of money to be made. I think if AGI were priced in tomorrow, you could maybe make 100x. Probably you can make even way more than that because of the sequencing. And capital matters. I think the other reason is just some amount of freedom and independence. And I think... you know, I think there's some people who are very smart about this AI stuff and who are kind of like see it coming. But I think almost all of them, you know, are kind of, you know, constrained in various ways, right? They're in the labs, you know, they're in some, you know, some other position where they can't really talk about this stuff. And, you know, in some sense, I've really admired sort of the thing you've done, which is I think it's really important that there's sort of voices of reason on this stuff publicly or people who are in positions to kind of advise important actors and so on. And so I think there's a, you know, Basically, the thing this investment firm will be will be kind of like a brain trust on AI. It's going to be all about situational awareness. We're going to have the best situational awareness in the business. We're going to have way more situational business than any of the people who manage money in New York. We're going to do great on investing. But it's the same sort of situational awareness that I think is going to be important for understanding what's happening, being a voice of reason publicly, and sort of being able to be in a position to advise. The book about Peter Thiel, they had an interesting quote about his hedge fund. I think it got terrible returns, so this isn't the example.

Dwarkesh Patel: That's the sort of bear case, right?

Leopold Aschenbrenner: It's too theoretical.

Dwarkesh Patel: Sure, yeah. But they had an interesting quote that it's basically a think tank inside of a hedge fund. Yeah, that's what everyone tried to build. Right.

Leopold Aschenbrenner: Presumably you've thought about the ways in which these kinds of things can blow.

Dwarkesh Patel: There's a lot of interesting business history books about people who got the thesis right, but timed it wrong, where they buy that internet's gonna be a big deal. They sell at the wrong time and buy the wrong time during the dot-com boom.

Leopold Aschenbrenner: And so they miss out on the gains, even though they're right about the... Anyways, yeah. What is that trick to preventing that kind of thing? Yeah, I mean, look, obviously you can't, you know, not blowing up is sort of like, you know, task number one and two or whatever. I mean, you know, I think this investment firm is going to just be betting on AGI, you know, betting on AGI and super intelligence before the decade is out, taking that seriously, making the bets you would make,

Speaker 2: you know,

Leopold Aschenbrenner: if you took that seriously. So, you know, I think if that's wrong, you know, firm is not going to do that well. The thing you have to be resistant to is you have to be able to resist and get one or a couple or a few kind of individual calls, right? It's like AI stagnates for a year because of the data wall. Or you got the call wrong on when revenue would go up. And so that's pretty critical. You have to get timing right. I do think, in general, that the sort of sequence of bets on the way to AGI is actually pretty critical. And I think a thing people underrate. All right. I mean, yeah. So where does the story start? So obviously, the only bet over the last year was NVIDIA. And it's obvious now, very few people did it. This is also a classic debate I and a friend had with another colleague of ours, where this colleague was really into TSM and TSMC. And he was just kind of like, well, these fabs are going to be so valuable. And also, NVIDIA, there's just a lot of idiosyncratic risk. It's like maybe somebody else makes better GPUs. That was basically right, but only NVIDIA had the AI beta, because only NVIDIA was a large fraction AI. The next few doublings would just meaningfully explode their revenue, whereas TSMC was a couple percent AI. So even though there's going to be a few doublings of AI, not going to make that big of an impact. So the only place to find the AI beta basically was NVIDIA for a while. You know, now it's broadening, right? So now TSM is like, you know, 20% AI by like 27 or something is what they're saying. When we're doubling, it'll be kind of like a large fraction of what they're doing. And, you know, there's a whole, you know, whole stack, you know, there's like, you know, there's people making memory and co-ops and, you know, power, you know, utilities companies are starting to get excited about AI and they're like, oh, it'll, you know, power production in the United States will grow, you know, not 2.5%, but 5% over the next five years. And I'm like, no, it'll grow more. At some point, like a Google or something becomes interesting. People are excited about them with AI because it's like, oh, AI revenue will be 10 billion or tens of billions. I'm kind of like, ah, I don't really care about them before then. I care about it once you get the AI beta. And so at some point, Google will get $100 billion of revenue from AI. Probably their stock will explode. They're going to become $5 trillion, $10 trillion company. Anyway, so the timing there is very important. You have to get the timing right. You have to get the sequence right. At some point, actually, I think there's going to be a real tailwind to equities from real interest rates. So basically, in these sort of explosive growth worlds, you would expect real interest rates to go up a lot, basically both sides of the equation, on the demand for money side, because people are going to be making these crazy investments, you know, initially in clusters and then in the robo-factories or whatever, right? And so they're going to be borrowing like crazy. They want all this capital, higher ROI. And then on the sort of like consumer saving side, right, to like, you know, to give up all this capital, you know, sort of like Euler equation, standard sort of intertemporal transfer, you know, trade-off of consumption. Very standard. Some of our friends have a paper on this. Basically, if consumers expect real growth rates to be higher, interest rates are going to be higher because they're less willing to give up consumption in the future. Anyway, so at some point, real interest rates will go up. If sort of eta is greater than one, that actually means equities, you know, higher growth rate expectations mean equities go down because the sort of interest rate effect outweighs the growth rate effect. And so, you know, at some point, there's like the big bond short. You got to get that right. You got to get it right, that, you know, nationalization, you know, like you got, you know, anyway. So there's this whole sequence of things. You got to get that right. And the unknown unknowns. Unknown unknowns, yeah. And so you've, look, you've got to be really, really careful about your like overall risk positioning, right? Because, you know, you know, if you expect these kind of crazy events to play out, there's going to be crazy things you didn't see. You know, you do also want to make the sort of kind of bets that are tailored to your scenarios in the sense of like, you know, you want to find bets that are bets on the tails, right? You know, I don't think anyone is expecting, you know, interest rates to go above, you know, 10%, like real interest rates. But, you know, I think there's at least a serious chance of that, you know, before the decade is out. And so, you know, maybe there's some like cheap insurance you can buy on that, you know, that pays off.

Dwarkesh Patel: Very silly question. Yeah.

Leopold Aschenbrenner: In these worlds. Yeah. Are financial markets where you make these kinds of bets going to be respected? And like, you know, like, is my Fidelity account going to mean anything when we have the 50% economic growth? Like, who's like, we got to respect his property rights? I mean, that's pretty deep into it. The bond short, the sort of 50% economic growth, that's pretty deep into it. I mean, again, there's this whole sequence of things. But yeah, no, I think property rates will be destructed. Again, in the sort of modal world, the project, yeah. At some point, there's going to be figuring out the property rights for the galaxies. That'll be interesting.

Dwarkesh Patel: That will be interesting. So there's an interesting question about going back to your strategy about, well, the 30s will really matter a lot about how the rest of the future goes. And you want to be in a position of influence by that point because of capital. It's worth considering, as far as I know, there's probably a whole bunch of literature on this. I'm just riffing. But the landed gentry before the beginning of the Industrial Revolution,

Leopold Aschenbrenner: I'm not sure if they were able to leverage their position in a sort of Georgist or pickety type sense in order to accrue the returns that were...

Dwarkesh Patel: realized through the Industrial Revolution. And I don't know what happened. At some point, they just weren't the landed gentry.

Leopold Aschenbrenner: But... I'd be concerned that even if you make great investment calls, you'll be like the guy who owned a lot of land, farmland, before the Industrial Revolution. The guy who's actually going to make a bunch of money is one of the steam engines.

Dwarkesh Patel: Even if he doesn't make that much money, most of the benefits are widely diffused and so forth.

Leopold Aschenbrenner: I think the analog is you sell your land, you put it all in the people who are building the new industry. I think the real depreciating asset you know, for me is human capital, right? Yeah, no, look, I'm serious, right? It's like, you know, there's something about like, you know, I don't know, it was like Valedictorian of Columbia, you know, the thing that made you special is you're smart. But actually, like, you know, that might not matter in like four years, you know, because it's actually automatable. And so anyway, a friend joked that the sort of investment firm was perfectly hedged for me. It's like, you know, either like AGI this decade. And yeah, your human capital is depreciated, but you've turned that into financial capital. Or, you know, like no AGI this decade, in which case maybe the firm doesn't do that well. But, you know, you're still in your 20s and you're still smart.

Dwarkesh Patel: Excellent. And what's your story for why AGI hasn't been priced in?

Leopold Aschenbrenner: The story... Financial markets are supposed to be very efficient. It's very, very hard to get an edge. Here... Naively, you just say, well, I've looked at the scaling curves, and they imply that we're going to be buying much more computing energy than the analysts realize. Shouldn't those analysts be broke by now? What's going on? Yeah. I mean, I used to be a true EMH guy. I was an economist. I am, you know, I think the thing I, you know, changed my mind on is that I think there can be kind of groups of people, smart people, you know, who are, you know, say they're in San Francisco, who do just have off over the rest of society and kind of seeing the future. And so like COVID, right? Like, I think there's just honestly kind of similar group of people who just saw that and called it completely correctly, right? Um, and you know, they shorted the market, they did really well. Um, you know, a bunch of other sort of things like that. Um, so, you know, why is AGI not priced in? You know, it's sort of, um, you know, why, why hasn't the government nationalized the labs yet? Right. It's like, you know, this, you know, society hasn't priced it in yet and sort of, it hasn't completely diffused. And, you know, again, it might be wrong. Right. But, um, um, I just think sort of you know, not that many people take these ideas seriously yet.

Speaker 2: Yeah.

Dwarkesh Patel: Yeah. A couple of other sort of ideas that I was playing around with with regards to, we didn't get a chance to talk about, but the systems competition. There's a very interesting, one of my favorite books about World War II is the Victor Davis Hanson summary of everything.

Leopold Aschenbrenner: And he explains why the Allies made better decisions than the Axis. Why did they? And so obviously there were some decisions that the Axis made that were pretty, like Blitzkrieg, whatever. That was sort of by accident, though. In what sense? That they just had the infrastructure left over?

Dwarkesh Patel: Well,

Leopold Aschenbrenner: no, I mean, the sort of, I think, I mean, I don't, I mean, I think sort of my read of it is Blitzkrieg wasn't kind of some like a genius strategy. It was just kind of, it was like more like their hand was forced. I mean, this is sort of the very Adam Tuzian story of World War II, right? But it was, you know, there's sort of this long war versus short war. I think it's actually kind of an important concept. I think sort of Germany realized that if they were in a long war, including the United States, you know, they would not be able to compete industrially. So their only path to victory was, like, make it a short war, right? And that... that sort of worked much more spectacularly than they thought, right? And sort of take over France and take over much of Europe. And so then, you know, the decision to invade the Soviet Union, it was, you know, it was, look, it was about the Western Front in some sense because it was like, we've got to get the resources. You know, we don't actually have a bunch of the stuff we need, like, you know, oil and so on. You know, Auschwitz was actually just this giant chemical plant to make kind of like synthetic oil and a bunch of these things. It was the largest industrial project in Nazi Germany. Yeah. And so, you know, and sort of they thought, well, you know, we completely crushed them in World War I, you know, it'll be easy, we'll invade them, we'll get the resources, and then we can fight on the Western Front. And even during the sort of whole invasion of the Soviet Union, even though kind of like a large amount of the sort of, you know, the sort of deaths happened there, you know, like a large fraction of German industrial production was actually, you know, like planes and naval, you know, and so on that was directed, you know, towards the Western Front and towards, you know, the Western Allies. Well, and then so the point that Hansen was making was... By the way, I think this concept of like long war and short war is kind of interesting with respect to thinking about the China competition, which is like, you know, I worry a lot about kind of, you know, decline of sort of American, like latent American industrial capacity. You know, like I think China builds like 200 times more ships than we do right now. you know, some crazy way. And so it's like, maybe we have this superiority, say in the non-AI worlds, we have the superiority in military material, kind of like win a short war, at least, you know, kind of defend Taiwan in some sense. But like, if it actually goes on, you know, it's like maybe China is much better able to mobilize, mobilize industrial resources in a way that like, we just don't have that same ability anymore. I think this is also relevant to the AI thing in the sense of if it comes down to a game about building, including maybe AGI takes the trillion-dollar cluster, not the $100 billion cluster, or even maybe AGI is on the $100 billion cluster, but it really matters if you can run 10x, you can do one more order of magnitude of compute for your superintelligence or whatever, that maybe right now they're behind, but they just have this raw latent industrial capacity to outbuild us. And that matters both in the run-up to AGI and after, right? Where it's like, you have the super intelligence on your cluster. Now it's time to kind of like expand the explosive growth. And, you know, like, will we let the robo factories run wild? Like maybe not, but like maybe China will. We're like, you know, will we, yeah, will we produce the, how many of the drones will we produce? And I think, yeah, so there's some sort of like outbuilding in the industrial explosion that I worry about. You've got to be one of the few people in the world who is both concerned about alignment, but also wants to make sure that we'll let the ROA factories proceed once we get the ASI to beat out China. Which is like very... It's all part of the picture. Yeah.

Dwarkesh Patel: And by the way, speaking of the ASIs and the robot factories,

Leopold Aschenbrenner: one of the interesting things...

Dwarkesh Patel: Robo armies, too. Yeah, one of the interesting things... There's this question of what you do with industrial-scale intelligence, and obviously it's not chatbots.

Leopold Aschenbrenner: But I think it's very hard to predict in a net... But the history of oil is very interesting where in the, I think it's in the 1860s that we figure out how to refine oil, some geologist.

Dwarkesh Patel: And so then Standard Oil gets started. There was this huge boom. It changes American politics. Entire legislators are getting bought out by oil interests. And presidents are getting elected based on the divisions about oil and breaking

Leopold Aschenbrenner: the AM up and everything. And all of this has happened. The world has never illusionized before the car has been invented.

Dwarkesh Patel: And... So when the light bulb was invented, I think it was like 50 years after oil refining had been discovered,

Leopold Aschenbrenner: the majority of Standard Oil's history is before the car was invented. The kerosene lamps.

Dwarkesh Patel: Exactly.

Leopold Aschenbrenner: So it's just used for lighting. Oh, and then they thought oil would just no longer be relevant? Yeah, yeah.

Dwarkesh Patel: So there was a concern that Standard Oil would go brain corrupt when the light bulb was invented. But then there's sort of...

Leopold Aschenbrenner: you realize that there's immense amount of compressed energy here. You're going to have billions of gallons of this stuff a year.

Dwarkesh Patel: And it's hard to sort of predict in advance what you can do with that. And then later on, it turns out, oh, transportation, cars, that's what it'd be used for.

Leopold Aschenbrenner: Anyways, with intelligence, maybe one answer is the intelligence explosion. But even after that, so you have all these ASIs and you have enough compute, especially the compute they'll build. to run- Hundreds of millions of GPUs will hum.

Dwarkesh Patel: Yeah, but what are we doing with that? And it's very hard to predict in advance, and I think it'll be very interesting to figure out what the Jupiter brains will be doing. So look, there's situational awareness of where things stand now, and we've gotten a good dose of that. Obviously, a lot of the things we're talking about now, you couldn't have prejudged many years back in the past. And part of your worldview implies that things will accelerate because of AI getting into the process. Many other things that are unpredictable fundamentally, basically how people will react, how the political system will react, how foreign adversaries will react. Those things will become evident over time. So... The sexual awareness is not just knowing where the picture stands now, but being in a position to react appropriately to new information, to change your worldview as a result, to change your recommendations as a result. What is the appropriate way to think about situational awareness as a continuous process rather than as a one-time thing you realized?

Leopold Aschenbrenner: Yep. No, I think this is great. Look, I think there's a sort of mental flexibility and willing to change your mind. That's really important. I actually think this is sort of like how a lot of brains have been broken in the AGI debate. The doomers who actually, I think we're really prescient on AGI and thinking about the stuff like a decade ago, But they haven't actually updated on the empirical realities of deep learning. They're sort of like, their proposals are really kind of naive and unworkable. It doesn't really make sense. There's people who come in with sort of a predefined ideology. They're just kind of like the EACs a little bit. They like to shitpost about technology, but they're not actually thinking through. I mean, either they're sort of stagnationists who think this stuff is only going to be a chatbot, and so of course it isn't risky, or they're just not thinking through the kind of actually immense national security implications and how that's going to go. Yeah. And I actually think there's kind of a risk in having written this stuff down and put it online. I think this sometimes happens to people as a sort of calcification of the worldview, because now they've publicly articulated this position. And maybe there's some evidence against it, but they're clinging to it. And so actually, I want to give the big disclaimer. I think it's really valuable to paint a very concrete and visceral picture. I think this is currently my best guess on how this decade will go. I think if it goes anywhere like this, it will be wild. But, you know, given the map and pace of progress, we're going to keep getting a lot more information. And, you know, I think it's important to sort of keep your head on straight about that. You know, I feel like the most important thing here is that, you know, and this relates to some of the stuff we've talked about and, you know, sort of the world being surprisingly small and so on, you know, I feel like I used to have this worldview of like, look, there's important things happening in the world, but there's like people who are taking care of it, you know? And there's like the people in government and there's again, even like AI labs have idealized and, um, people are on it, you know, surely they must be on it. Right. And I think just some of this personal experience, even seeing how kind of COVID went, you know, people aren't necessarily, there's not some, there's not somebody else who's just kind of on it and making sure this goes well, however it goes. Um, you know, thing that I think will really matter is that there are sort of good people who take this stuff as seriously as it deserves and who are willing to kind of take the implications seriously, who are willing to, you know, who have situational awareness, are willing to change their minds, are willing to sort of stare the picture in the face. And, you know, I'm counting on those good people.

Dwarkesh Patel: All right, that's a great place to close, Leopold.

Leopold Aschenbrenner: Thanks so much, Tarkash.

Dwarkesh Patel: This is excellent. Hey, everybody. I hope you enjoyed that episode with Leopold. There's actually one more riff about German history that he had after a break, and it was pretty interesting, so I didn't want to cut it out. So I've just included it after this outro. You can advertise on the show now. So if you're interested, you can reach out at the form in the description below.

Leopold Aschenbrenner: Other than that, the most helpful thing you can do is just share the episode if you enjoyed it. Send it to group chats, Twitter,

Dwarkesh Patel: wherever else you think people who might like this episode might congregate. And other than that, I guess here's this riff on Frederick the Great. See you on the next one.

Leopold Aschenbrenner: I mean, I think the actual funny thing is, you know, a lot of this sort of German history stuff we've talked about is sort of like not actually stuff I learned in Germany. It's sort of like stuff that I learned after. And there's actually, you know, a funny thing where I kind of would go back to Germany over Christmas or whatever and suddenly understand the street names.

Dwarkesh Patel: You know, it's like,

Leopold Aschenbrenner: you know, Gneisenau and Scharnhorst and they're all these like Prussian military reformers. And you're like, finally understood, you know, Sansa C. And you're like, it was for Frederick. You know, Frederick the Great is this really interesting figure where he's this sort of, in some sense, kind of like, gay lover of arts, right? Where he, um, he, um, he, uh, you know, he hates speaking German. He only wants to speak French. You know, he, like, plays the flute. He composes. He has all the sort of great, you know, artists of his day, you know, over at Sanssouci. Um... And he actually had this really tough upbringing where his father was this really stern Prussian military man. And he had had Frederick the Great as a 17-year-old or whatever. He basically had a male lover. And what his father did was imprison his son and then, I think, hang his male lover in front of him. And again, his father was this very stern Prussian guy. He was this gay lover of arts. But then later on, Frederick the Great turns out to be this like, you know, one of the most kind of like, you know, successful kind of Prussian conquerors, right? Like, he gets Silesia, he wins the Seven Years' War. You know, also, you know, amazing military strategists, you know, amazing military strategy at the time consisted of, like, he was able to, like, flank the army, and that was crazy, you know, and that was brilliant. And then they, like, almost lose the Seven Years' War. And at the very end, you know, the sort of, the Russian Tsar changes, and he's like, ah, I'm actually kind of a Prussian stan. You know, I think I'm, like, I'm into this stuff. And then he lets, you know, lets Frederick the Great loose, and he kind of lets their army be okay. And, um, Anyway, kind of bizarre, interesting figure in German history.