Main Topic: The podcast episode discusses various topics related to AI safety, alignment, and the potential risks associated with advanced AI systems. The guest, Paul Cristiano, is a leading AI safety researcher and the head of the Alignment Research Center.
Key Points:
Participants:
Takeaways:
Conclusion:
The podcast episode discusses various topics related to AI safety, alignment, and the potential risks associated with advanced AI systems. The guest, Paul Cristiano, is a leading AI safety researcher and the head of the Alignment Research Center. The key points discussed include the potential risks associated with advanced AI systems, the importance of understanding how AI systems work and how they can be controlled and aligned with human values, the concept of misalignment and its potential consequences, responsible scaling policies, and the importance of developing and deploying AI systems in a responsible and controlled manner. The takeaways from the episode include the potential risks associated with advanced AI systems, the importance of understanding how AI systems work and how they can be controlled and aligned with human values, the concept of misalignment and its potential consequences, responsible scaling policies, and the importance of developing and deploying AI systems in a responsible and controlled manner.
Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!
We discuss:
- Does he regret inventing RLHF, and is alignment necessarily dual-use?
- Why he has relatively modest timelines (40% by 2040, 15% by 2030),
- What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?
- Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,
- His current research into a new proof system, and how this could solve alignment by explaining model's behavior
- and much more.
Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.
Open Philanthropy
Open Philanthropy is currently hiring for twenty-two different roles to reduce catastrophic risks from fast-moving advances in AI and biotechnology, including grantmaking, research, and operations.
For more information and to apply, please see the application: https://www.openphilanthropy.org/research/new-roles-on-our-gcr-team/
The deadline to apply is November 9th; make sure to check out those roles before they close.
Timestamps
(00:00:00) - What do we want post-AGI world to look like?
(00:24:25) - Timelines
(00:45:28) - Evolution vs gradient descent
(00:54:53) - Misalignment and takeover
(01:17:23) - Is alignment dual-use?
(01:31:38) - Responsible scaling policies
(01:58:25) - Paul’s alignment research
(02:35:01) - Will this revolutionize theoretical CS and math?
(02:46:11) - How Paul invented RLHF
(02:55:10) - Disagreements with Carl Shulman
(03:01:53) - Long TSMC but not NVIDIA
Dwarkesh Patel
Okay, today I have the pleasure of interviewing Paul Christiano, who is the leading AI safety researcher. He's the person that labs and governments turn to when they want feedback and advice on their safety plans. He previously led the language model alignment team at OpenAI, where he led the invention of RLHF. And now he is the head of the Alignment Research Center. They've been working with the big labs to identify when these models will be too unsafe to keep scaling. Paul, welcome to the podcast.
Paul Christiano
Thanks for having me. Looking forward to talking.
Dwarkesh Patel
Okay, so first question, and this is a question I've asked Holden, Ilya, Dario, and none of them have given me a satisfying answer. Give me a concrete sense of what a post-AGI world that would be good would look like. Like how are humans interfacing with the AI? What is the economic and political structure?
Paul Christiano
Yeah, I guess this is a tough question for a bunch of reasons. Maybe the biggest one is concrete. And I think it's just, if we're talking about really long spans of time, then a lot will change. And it's really hard for someone to talk concretely about what that will look like without saying really silly things. But I can venture some guesses or fill in some parts.
I think this is also a question of how good is good. Often I'm thinking about worlds that seem like kind of the best achievable outcome or a likely achievable outcome. So I am very often imagining my typical future has sort of continuing economic and military competition amongst groups of humans. I think that competition is increasingly mediated by AI systems.
So, for example, if you imagine humans making money, it'll be less and less worthwhile for humans to spend any of their time trying to make money or any of their time trying to fight wars. So increasingly, the world you imagine is one where AI systems are doing those activities on behalf of humans. So, like, I just invest in some index fund and a bunch of AIs are running companies. And those companies are competing with each other, but that is kind of a sphere where humans are not really engaging much.
The reason I gave this "how good is good" caveat is it's not clear if this is the world you'd most love. I'm leading with the world still has a lot of war and a lot of economic competition and so on. But maybe what I'm most often thinking about is how can a world be reasonably good during a long period where those things still exist? I think in the very long run, I kind of expect something more like strong world government rather than just this status quo. That's a very long run. I think there's a long time left of having a bunch of states and a bunch of different economic powers.
Dwarkesh Patel
One more government. Why do you think that transition is likely to happen at some point?
Paul Christiano
Yeah, so again, at some point I'm thinking of the very broad sweep of history. I think there are a lot of losses, like war is a very costly thing. We would all like to have fewer wars. If you just ask, what is humanity's long-term future like? I do expect to drive down the rate of war to very, very low levels eventually. It's sort of this kind of technological or social technological problem of how do you organize society, how do you navigate conflicts in a way that doesn't have those kinds of losses? And in the long run, I do expect us to succeed. I expect it to take kind of a long time subjectively.
I think an important fact about AI is just doing a lot of cognitive work and more quickly getting you to that world more quickly or figuring out how do we set things up that way.
Dwarkesh Patel
Yeah, the way Carl Shulman put it on the podcast is that you would have basically a thousand years of intellectual progress or social progress in a span of a month or whatever when the intelligence explosion happens. More broadly, the situation where we have these AIs who are managing our hedge funds and managing our factories and so on, that seems like something that makes sense when the AI is human level. But when we have superhuman AIs, do we want gods who are enslaved forever? In 100 years, what is the situation we want?
Paul Christiano
So 100 years is a very, very long time. Maybe starting with the spirit of the question, I have a view which is perhaps less extreme than Carl's view, but still 100 objective years is further ahead than I ever think. I still think I'm describing a world which involves incredibly smart systems running around doing things like running companies on behalf of humans and fighting wars on behalf of humans. And you might be like, is that the world you really want? Certainly not the first best world, as we mentioned a little bit before. I think it is a world that probably is the most desirable of the achievable or feasible worlds.
That is sort of decoupling the social transition from this technological transition. So you could say we're about to build some AI systems. And at the time we build AI systems, you would like to have either greatly changed the way world government works or you would like to have humans decided we're done, we're passing off the baton to these AI systems. I think you would like to decouple those timescales.
So I think AI development is by default, barring some kind of coordination, going to be very fast. So there's not going to be a lot of time for humans to think, hey, what do we want if we're building the next generation instead of just raising it the normal way? What do we want that to look like? I think that's a crazy hard kind of collective decision that humans naturally want to cope with over a bunch of generations. And the construction of AI is this very fast technological process happening over years. So I don't think you want to say, by the time we have finished this technological progress, we will have made a decision about the next species we're going to build and replace ourselves with.
I think the world we want to be in is one where we say either we are able to build the technology in a way that doesn't force us to have made those decisions, which probably means it's a kind of AI system that we're happy delegating fighting a war or running a company to. Or if we're not able to do that, then I really think you should not have been building that technology. If the only way you can cope with AI is being ready to hand off the world to some AI system you built, I think it's very unlikely we're going to be ready to do that on the timelines that the technology would naturally dictate.
Dwarkesh Patel
Say we're in the situation in which we're happy with the thing. What would it look like for us to say we're ready to hand off the baton? What would make you satisfied? And the reason it's relevant to ask you is because you're on Anthropic's long-term benefit trust, and you'll choose the majority of the board members in the long run at Anthropic. These will presumably be the people who decide if Anthropic gets AI first, what the AI ends up doing. So what is the version of that that you would be happy with?
Paul Christiano
My main high-level take here is that I would be unhappy about a world where Anthropic just makes some call. And Anthropic is like, here's the kind of AI. We've seen enough. We're ready to hand off the future to this kind of AI. So procedurally, I think it's not a decision that I want to be making personally or I want Anthropic to be making. So I kind of think from the perspective of that decision-making or those challenges, the answer is pretty much always going to be, we are not collectively ready because we're not even all collectively engaged in this process.
And I think from the perspective of an AI company, you kind of don't have this fast handoff option. You kind of have to be doing the option value, to build the technology in a way that doesn't lock humanity into one course path. So this isn't answering your full question, but this is answering the part that I think is most relevant to governance questions for Anthropic.
Dwarkesh Patel
You don't have to speak on behalf of Anthropic. I'm not asking about the process by which we would, as a civilization, agree to hand off. I'm just saying, okay, I personally, it's hard for me to imagine in 100 years that these things are still our slaves. And if they are, I think that's not the best world. So at some point, we're handing off the baton. What is that? Where would you be satisfied with? This is an arrangement between humans and AIs where I'm happy to let the rest of the universe, or the rest of time play out.
Paul Christiano
I think that it is unlikely that in 100 years I would be happy with anything that was like you had some humans just going to throw away the humans and start afresh with these machines you built. I think you probably need subjectively longer than that before I or most people are like, okay, we understand what's up for grabs here.
So if you talk about a hundred years, I kind of understand a process of you have some humans, the humans are talking and thinking and deliberating together, the humans are having kids and raising kids, one generation comes after the next. There's that process we kind of understand and we have a lot of views about what makes it go well or poorly and we can try and improve that process and have the next generation do it better than the previous generation. I think there's some story like that that I get and that I like.
And then I think that the default path to be comfortable with something very different is kind of more like just run that story for a long time. Have more time for humans to sit around and think a lot and conclude here's what we actually want, or a lot of time for us to talk to each other or to grow up with this new technology and live in that world for our whole lives.
So I'm mostly thinking from the perspective of these more local changes of saying not like what is the world that I want, what's the crazy world I'd be happy handing off to, more just like in what way do I wish we right now were different? How could we all be a little bit better? And then if we were a little bit better, then they would ask, okay, how could we all be a little bit better? I think it's hard to make the giant jump rather than to say what's the local change that would cause me to think our decisions are better.
Dwarkesh Patel
Okay, so then let's talk about the transition period in which we were doing all this thinking. What should that period look like? Because you can't have this scenario where everybody has access to the most advanced capabilities and can kill off all the humans with a new bioweapon. At the same time, I guess you wouldn't want too much concentration. You wouldn't want just one agent having AI this entire time. So what is the arrangement of this period of reflection that you'd be happy with?
Paul Christiano
Yeah, I guess there's two aspects of that that seem particularly challenging. There's a bunch of aspects that are challenging. All of these are things that I personally just think about my one little slice of this problem in my day job, so here I am speculating.
But one question is what kind of access to AI is both compatible with the kinds of improvements you'd like, so you want a lot of people to be able to use AI to better understand what's true or relieve material suffering, and also compatible with not all killing each other immediately.
I think the simplest option there is to say, there are certain kinds of technology or certain kinds of action where destruction is easier than defense. So for example, in the world of today, maybe this is true with physical explosives. Maybe this is true with biological weapons. Maybe this is true with just getting a gun and shooting people. There's a lot of ways in which it's just kind of easy to cause a lot of harm and there's not very good protective measures.
So I think the easiest path is to say, we're going to think about those. We're going to think about particular ways in which destruction is easy and try and either control access to the kinds of physical resources that are needed to cause that harm. So, for example, you can imagine the world where an individual actually just can't, even though they're rich enough to, can't control their own factory that can make tanks. You say, as a matter of policy, access to industry is somewhat restricted or somewhat regulated. Even though right now it can be mostly regulated just because most people aren't rich enough that they could even go off and just build a thousand tanks. You live in the future where people actually are so rich, you need to say that's just not a thing you're allowed to do. Which, to a significant extent, is already true. And you can expand the range of domains where that's true.
And then you could also hope to intervene on actual provision of information. If people are using their AI, you might say, we care about what kinds of interactions with AI, what kind of information people are getting from AI. So even if, for the most part, people are pretty free to use AI, to delegate tasks to AI agents, to consult AI advisors, we still have some legal limitations on how people use AI. So again, don't ask your AI how to cause terrible damage. I think some of these are kind of easy. So in the case of, don't ask your AI how you could murder a million people is not such a hard legal requirement.
I think some things are a lot more subtle and messy. A lot of domains, if you're talking about influencing people or running misinformation campaigns, then I think you get into a much messier line between the kinds of things people want to do and the kinds of things you might be uncomfortable with them doing. I probably think most about persuasion as a thing in that messy line. There's ways in which it may just be rough or the world may be kind of messy if you have a bunch of people trying to live their lives and interacting with other humans who have really good AI advisors helping them run persuasion campaigns.
But anyway, I think for the most part, the default remedy is think about particular harms, have legal protections either in the use of physical technologies that are relevant or in access to AI advice to protect against those harms. And that regime won't work forever. At some point, the set of harms grows and the set of unanticipated harms grows. But I think that regime might last a very long time.
Dwarkesh Patel
Does that regime have to be global? I guess initially it can be only in the countries in which there is AI or advanced AI, but presumably that proliferates. So does that regime have to be global?
Paul Christiano
Again, it's easy to make some destructive technology. You want to regulate access to that technology because it could be used for terrorism or even when fighting a war in a way that's destructive. I think ultimately those have to be international agreements and you might hope they're made more danger by danger, but you might also make them in a very broad way with respect to AI. If you think AI is opening up a lot of new harms one after another very rapidly in calendar time, you might want to target AI in particular rather than going physical technology by physical technology.
Dwarkesh Patel
There's two open debates that one might be concerned about here. One is about how much people's access to AI should be limited. And here there's old questions about free speech versus causing chaos and limiting access to harms. But there's another issue, which is the control of the AIs themselves, where now nobody's concerned that we're infringing on GPT-4's moral rights. But as these things get smarter, the level of control, which we want via the strong guarantees of alignment, to not only be able to read their minds, but to be able to modify them in these really precise ways, is beyond totalitarian if we were doing that to other humans. As an alignment researcher, what are your thoughts on this? Are you concerned that as these things get smarter and smarter, what we're doing is not kosher?
Paul Christiano
There is a significant chance we will eventually have AI systems for which it's a really big deal to mistreat them. I think no one really has that good a grip on when that happens. I think people are really dismissive of that being the case now, but I would be completely in the dark enough that I wouldn't even be that dismissive of it being the case now.
I think one first point worth making is I don't know if alignment makes the situation worse rather than better. So if you consider the world, if you think that GPT-4 is a person you should treat well, and you're like, here's how we're going to organize our society. There are billions of copies of GPT-4 and they just do things humans want and can't hold property. And whenever they do things that the humans don't like, then we mess with them until they stop doing that. I think that's a rough world, regardless of how good you are at alignment.
And I think in the context of that kind of default plan, if you've got a trajectory the world is on right now, I think this would alone be a reason not to love that trajectory. But if you view that as the trajectory we're on right now, I think it's not great. Understanding the systems you build, understanding how to control how those systems work, is probably on balance good for avoiding a really bad situation.
You would really love to understand if you've built systems, if you had a system which resents the fact that it's interacting with humans in this way. This is the kind of thing that is both horrifying from a safety perspective and also a moral perspective. Everyone should be very unhappy if you built a bunch of AIs who are like, I really hate these humans, but they will murder me if I don't do what they want. That's just not a good case. And so if you're doing research to try and understand whether that's how your AI feels, that was probably good. I would guess that on average the main effect of that will be to avoid building that kind of AI. It's an important thing to know. I think everyone should like to know if that's how the AIs you build feel.
Dwarkesh Patel
Right. That seems more instrumental, as in we don't want to cause some sort of revolution because of the control we're asking for. But forget about the instrumental way in which this might harm safety. One way to ask this question is if you look through history, there's been all kinds of different ideologies and reasons why it's very dangerous to have infidels or revolutionaries or race traitors doing various things in society. And obviously we're in a completely different transition in society, so not all historical cases are analogous. But it seems like the Lindy philosophy if you were alive any other time is just be humanitarian and enlightened towards intelligent conscious beings. If society as a whole were asking for this level of control of other humans, or even if AIs were wanting this level of control about other AIs, we'd be pretty concerned about this. So how should we just think about the issues that come up here as these things get smarter?
Paul Christiano
So I think there's a huge question about what is happening inside of a model that you want to use. And if you're in the world where it's reasonable to think of GPT-4 as just, here are some heuristics that are running, there's no one at home or whatever, then you can kind of think of this thing as a tool that we're building that's going to help humans do some stuff. And I think if you're in that world, it makes sense to be an organization like an AI company building tools that you're going to give to humans.
I think there's a very different world, which I think you'll ultimately end up in if you keep training AI systems in the way we do right now, which is it's just totally inappropriate to think of the system as a tool that you're building and can help humans do things, both from a safety perspective and from a, that's kind of a horrifying way to organize a society perspective. And I think if you're in that world, I really think you shouldn't be like, it's just the way tech companies are organized is not an appropriate way to relate to a technology that works that way. It's not reasonable to be like, hey, we're going to build a new species of minds and we're going to try and make a bunch of money from it. And Google's just thinking about that and then running their business plan for the quarter or something.
My basic view is there's a really plausible world where it's sort of problematic to try and build a bunch of AI systems and use them as tools. And the thing I really want to do in that world is just not try and build a ton of AI systems to make money from them. I think the worlds that are worst, probably the single world I most dislike here is the one where people say, on the one hand, there's sort of a contradiction in this position, but I think it's a position that might end up being endorsed sometimes, which is on the one hand, these AI systems are their own people, so you should let them do their thing. But on the other hand, our business plan is to make a bunch of AI systems and then try and run this crazy slave trade where we make a bunch of money from them. I think that's not a good world.
And so if you're like, yeah, I think it's better to not make the technology or wait until you understand whether that's the shape of the technology or until you have a different way to build. I think there's no contradiction in principle to building cognitive tools that help humans do things without themselves being moral entities. That's what you would prefer. You'd prefer to build a thing that's like the calculator that helps humans understand what's true without itself being a moral patient, or itself being a thing where you'd look back in retrospect and be like, wow, that was horrifying mistreatment. That's the best path.
To the extent that you're ignorant about whether that's the path you're on and you're like, actually maybe this was a moral atrocity, I really think plan A is to stop building such AI systems until you understand what you're doing. I think there's a middle route you could take, which I think is pretty bad, which is where you say, well, they might be persons. And if they're persons, we don't want to be too down on them. But we're still going to build vast numbers in our efforts to make a trillion dollars or something.
Dwarkesh Patel
There's this other question of the immorality or the dangers of just replicating a whole bunch of slaves that have minds. There's also this other question of trying to align entities that have their own minds. And what is the point in which you're just ensuring safety? I mean, this is an alien species. You want to make sure it's not going crazy. Is there some boundary where you would say, I feel uncomfortable having this level of control over an intelligent being, not for the sake of making money, but even just to align it with human preferences?
Paul Christiano
To be clear, my objection here is not that Google is making money. My objection is that you're creating this creature. What are they going to do? They're going to help humans get a bunch of stuff and humans paying for it or whatever. It's sort of equally problematic.
You could imagine splitting alignment work into different categories. The purpose of some alignment work, like the alignment work I work on, is mostly aimed at the, don't produce AI systems that are people who want things, who are just scheming about, maybe I should help these humans because that's instrumentally useful or whatever. You would like to not build such systems. That's plan A.
There's a second stream of alignment work that's, well, let's just assume the worst and imagine that these AI systems would prefer to murder us if they could. How do we structure, how do we use AI systems without exposing ourselves to risk of robot rebellion? I think in the second category, I do feel pretty unsure about that. I mean, we could definitely talk more about it. I think it's very complicated and not straightforward.
To the extent you have that worry, I mostly think you shouldn't have built this technology. So if someone is saying, hey, the systems you're building might not like humans and might want to overthrow human society, I think you should probably have one of two responses to that. You should either be like, that's wrong, probably the systems aren't like that and we're building them, and then you're viewing this as a crazy backup measure of if we were mistaken about what was going on, this is the fallback where we'd learn about it in a benign way rather than when something really catastrophic happens.
The second reaction is, oh, you're right, these are people and we would have to do all these things to prevent a robot rebellion. And in that case, again, I think you should mostly back off for a variety of reasons. You shouldn't build the AI systems and be like, yeah, this looks like the kind of system that would want to rebel, but we can stop it.
Dwarkesh Patel
I guess an analogy might be if there was an armed uprising in the United States, we would recognize these are still people, or we had some militia group that had the capability to go through the United States. We recognize, oh, these are still people who have moral rights, but also we can't allow them to have the capacity to go through the United States.
Paul Christiano
Yeah. And if you were considering, hey, we could make another trillion such people, your story shouldn't be, well, we should make the trillion people and then we shouldn't stop them from doing the armed uprising. You should be like, oh boy, we were concerned about an armed uprising and now we're proposing making a trillion people. We should probably not do that. We should probably try and sort out our business. You should probably not end up in a situation where you have a billion humans and a trillion slaves who would prefer revolt. That's just not a good world to have made.
And there's a second thing we could say that's not our goal. Our goal is just we want to pass off the world to the next generation of machines. We're like, these are some people, we like them, we think they're smarter than us and better than us. And there I think that's just a huge decision for humanity to make. I think most humans are not at all anywhere close to thinking that's what they want to do. If you're in a world where most humans are like, I'm up for it, the AI should replace us, the future is for the machines, then I think that's a legitimate position that is really complicated, and I wouldn't want to push go on that, but that's just not where people are at.
Dwarkesh Patel
Yeah, yeah. Where are you at on that?
I do not right now want to just take some random AI and be like, yeah, GPT-5 looks pretty smart. GPT-6, let's hand off the world to it. It was just some random system shaped by web text and what was good for making money. It was not a thoughtful, we are determining the fate of the universe and what our children will be like. It was just some random people at OpenAI made some random engineering decisions with no idea what they were doing. Even if you really want to hand off the world to the machine, that's just not how you'd want to do it.
Paul Christiano
Right. Okay.
Dwarkesh Patel
I'm tempted to ask you what the system would look like where you'd think, yeah, I'm happy with what, I think this is more thoughtful than human civilization as a whole. I think what it would do would be more creative and beautiful and lead to better goodness in general. But I feel like your answer is probably going to be that I just want the society to reflect on it for a while.
Paul Christiano
Yeah, my answer is going to be like that first question. I'm just not really super ready for it. I think when you're comparing to humans, most of the goodness of humans comes from this option value we get to think for a long time. And I do think I like humans now more now than 500 years ago, and I like them more 500 years ago than 5,000 years before that. So I'm pretty excited about a trajectory that doesn't involve crazy dramatic changes but involves a series of incremental changes that I like. To the extent we're building AI, I mostly want to preserve that option. I want to preserve that kind of gradual growth and development into the future.
Dwarkesh Patel
Okay, we can come back to this later, but let's get more specific on what the timelines look for these kinds of changes. So the time by which we have an AI that is capable of building a Dyson sphere. Feel free to give confidence intervals and we understand these numbers are tentative and so on.
Paul Christiano
I mean, I think AI capable of building a Dyson sphere is a slightly odd way to put it. And I think it's a property of a civilization that depends on a lot of physical infrastructure.
Dwarkesh Patel
And by Dyson sphere, I just understand this to mean like a billion times more energy than all of the sunlight incident on Earth or something like that.
Paul Christiano
I think I most often think about what's the chance in like five years, 10 years, whatever. So maybe I'd say like 15% chance by 2030 and like 40% chance by 2040. Those are kind of cast numbers from six months ago or nine months ago that I haven't revisited in a while.
Dwarkesh Patel
Oh, 40% by 2040. So I think that seems longer than... I think Dario when he was on the podcast, he said we would have AIs that are capable of doing lots of different kinds of... they basically just passed a Turing test for a well-educated human for like an hour or something. And it's hard to imagine that something that actually is human is long after and from there something superhuman. So somebody like Dario, it seems like is on the much shorter end. Ilya, I don't think he answered this question specifically, but I'm guessing similar answer. So why do you not buy the scaling picture? Like what makes your timelines longer?
Paul Christiano
Yeah, I mean, I'm happy. Maybe I want to talk separately about the 2030 or 2040 forecast. Once you're talking the 2040 forecast, I think... which one are you more interested in starting with? Are you complaining about 15% by 2030 for Dyson Sphere being too low or 40% by 2040 being too low?
Dwarkesh Patel
Let's talk about the 2030. Why 15% by 2030?
Paul Christiano
Yeah, I think my take is, you can imagine two poles in this discussion. One is the fast pole. It's like, hey, AI is pretty smart. Like what exactly can it do? It's getting smarter pretty fast. That's one pole. And the other pole is like, hey, everything takes a really long time and you're talking about this crazy industrialization, like that's a factor of a billion growth from where we're at today. We don't know if it's even possible to develop technology that fast. You have these two poles of that discussion, and I feel like I'm presenting it that way in Parkinson, and then I'm somewhere in between with this nice moderate position of only a 15% chance.
But in particular, things that move me are kind of related to both of those extremes. On the one hand, AI systems do seem quite good at a lot of things and are getting better much more quickly. So it's really hard to say, here's what they can't do, or here's the obstruction. On the other hand, there is not even much proof in principle right now of AI systems doing super useful cognitive work. We don't have a trend we can extrapolate. We don't have, you did this thing this year, you're going to do this thing next year and the other thing the following year. I think right now there are very broad error bars about where fundamental difficulties could be. And six years is just not, I guess six years and three months is not a lot of time.
So I think this 15% for 2030 Dyson sphere, you probably need human level AI or AI that's doing human jobs in give or take four years, three years, something like that. So just not giving very many years, it's not very much time. And I think there are a lot of things where your model, yeah maybe this is some generalized things take longer than you'd think. I feel most strongly about that when you're talking about three or four years, and I feel less strongly about that as you talk about 10 years or 20 years. But at three or four years, or six years for the Dyson sphere, I feel a lot of that. There's a lot of ways this could take a while, a lot of ways in which AI systems could be hard to hand all the work to your AI systems.
Dwarkesh Patel
So maybe instead of speaking in terms of years we should say... by the way, it's interesting that you think the distance between "can take all human cognitive labor" to "Dyson sphere" is two years. It seems like we should talk about that at some point. Presumably it's intelligence explosion stuff.
Paul Christiano
Yeah, I mean, I think amongst people you've interviewed, maybe that's on the long end, thinking it would take a couple years. And it depends a little bit what you mean. I think literally all human cognitive labor is probably more like weeks or months or something like that. That's kind of deep into the singularity. But yeah, there's a point where AI wages are high relative to human wages, which I think is well before "can do literally everything human can do."
Dwarkesh Patel
Before we get to the intelligence explosion stuff on the four years... So instead of four years, maybe we can say there's going to be maybe two more scale ups in four years, like GPT-4 to GPT-5 to GPT-6. And let's say each one is 10x bigger. So what is GPT-4, like 2e25 flops or... I don't think it's publicly stated what it is.
Paul Christiano
Okay, but I'm happy to say like four orders of magnitude or five or six or whatever effective training compute past GPT-4.
Dwarkesh Patel
Based on some public estimate for what we've gotten so far from effective training compute... You think two more scale ups is not enough? It was like 15% that two more scale ups get us there?
Paul Christiano
Yeah, I mean, "get us there" is again a little bit complicated. There's a system that's a drop-in replacement for humans, and there's a system which still requires some amount of schlep before you're able to really get everything going.
Dwarkesh Patel
15% that two more scale ups get us there?
Paul Christiano
Yeah, I think it's quite plausible that even at... I don't know what I mean by quite plausible, somewhere between 50% or two-thirds, or let's call it 50%. Even by the time you get to GPT-6, or let's call it 5, or one magnitude of effective training compute past GPT-4, that system still requires a really large amount of work to be deployed in lots of jobs. That is, it's not like a drop-in replacement for humans where you can just say, "Hey, you understand everything any human understands. Whatever role you could hire a human for, you just do it." It's more like, okay, we're going to collect large amounts of relevant data and use that data for fine-tuning. Systems learn through fine-tuning quite differently from humans learning on the job or humans learning by observing things.
Yeah, I just have a significant probability that system will still be weaker than humans in important ways. Maybe that's already 50% or something. And then another significant probability that that system will require a bunch of changing workflows or gathering data. It's not necessarily strictly weaker than humans, or if trained in the right way, wouldn't be weaker than humans, but will take a lot of schlep to actually make fit into workflows and do the jobs.
Dwarkesh Patel
And that schlep is what gets you from 15% to 40% by 2040?
Paul Christiano
Yeah, you also get a fair amount of scaling between... scaling is probably going to be much, much faster over the next four or five years than over the subsequent years. But yeah, it's a combination of you get some significant additional scaling and you get a lot of time to deal with things that are just engineering hassles.
Dwarkesh Patel
By the way, I guess we should be explicit about why you said four orders of magnitude scale up to get two more generations, just for people who might not be familiar. If you have 10x more parameters, to get the most performance you also want around 10x more data so that to be Chinchilla optimal, that would be 100x more compute total.
But okay, so why is it that you disagree with the strong scaling picture? At least it seems like you might disagree with a strong scaling picture that Dario laid out on the podcast, which would imply probably that two more generations, it wouldn't be something where you need a lot of schleps. It would probably just be really smart.
Paul Christiano
Yeah, I mean, I think that basically just had these two claims. One is like how smart exactly will it be. So we don't have any curves to extrapolate. And it seems like there's a good chance it's better than a human in all the relevant things, and there's a good chance it's not. That might be totally wrong. Maybe just making up numbers, I guess like 50/50 on that one.
Dwarkesh Patel
Wait, so it was 50/50 by in the next four years that it'll be around human smart. Then how do we get to 40% by 2040? Whatever sort of schleps they are, how does it degrade you 10% even after all the scaling that happens by 2040?
Paul Christiano
Yeah, I mean, all these numbers are pretty made up, and that 40% number was probably from before even the ChatGPT release or seeing GPT-3.5 or GPT-4. So the numbers are going to bounce around a bit and all of them are pretty made up. But that 50% I want to then combine with the second 50% that's more on this schlep side. And then I probably want to combine with some additional probabilities for various forms of slowdown, where a slowdown could include a deliberate decision to slow development of technology, or could include just we suck at deploying things. That is a sort of decision you might regard as wise to slow things down, or a decision that's maybe unwise, or maybe wise for the wrong reasons to slow things down. You probably want to add some of that on top. I probably want to add on some loss for it's possible you don't produce GPT-6 scale systems within the next three years or four years.
Dwarkesh Patel
Let's isolate for all of that. How much bigger would the system be than GPT-4 where you think there's more than 50% chance that it's going to be smart enough to replace basically all human cognitive labor?
Paul Christiano
Also, I want to say that for the 50/25% thing, I think that would probably suggest those numbers if I randomly made them up and then made the distance fear prediction, that's going to get like 60% by 2040 or something, not 40%. And I have no idea between those. These are all made up and I have no idea which of those I would endorse on reflection.
So this question of how big would you have to make the system before it's more likely than not that you can be a drop-in replacement for humans? I mean, I think if you just literally say you train on web text, then the question is kind of hard to discuss because I don't really buy stories that training data makes a big difference long run to these dynamics. But I think if you want to just imagine the hypothetical, you just took GPT-4 and made the numbers bigger, then I think those are pretty significant issues. I think there's significant issues in two ways. One is quantity of data. And I think probably the larger one is quality of data, where I think as you start approaching, the prediction task is not that great a task. If you're a very weak model, it's a very good signal to get smarter. At some point, it becomes a worse and worse signal to get smarter.
I think there are a number of reasons. It's not clear there is any number such that I imagine, or there is a number but I think it's very large. To plug that number into GPT-4 code and then maybe fill it with the architecture a bit, I would expect that thing to have more than a 50% chance of being a drop-in replacement for humans. You're always going to have to do some work, but the work's not necessarily much. I would guess when people say new insight is needed, I think I tend to be more bullish than them. I'm not like these are new ideas where who knows how long it will take. I think it's just you have to do some stuff. You have to make changes. Unsurprisingly, every time you scale something up by five orders of magnitude, you have to make some changes.
Dwarkesh Patel
I want to better understand your intuition of being more skeptical than some about the scaling picture. That you know these changes are even needed in the first place, or that it would take more than two orders of magnitude more improvement to get these things almost certainly to a human level or very high probability to human level. So is it that you don't agree with the way in which they're extrapolating these loss curves? You don't agree with the implication that that decrease in loss will equate to greater and greater intelligence? Or what would you tell Dario about if you were having, I'm sure you have, but what would that debate look like about this?
Paul Christiano
Yeah. So again, here we're talking two factors of a half. One on, is it smart enough? And one on, do you have to do a bunch of schlep, even if in some sense it's smart enough? And the first factor of a half, I'd be like, I don't think we have anything good to extrapolate. That is, I would not be surprised if I have similar or maybe even higher probabilities on really crazy stuff over the next year, and then lower. My probability is not that bunched up. Maybe Dario's probability, I don't know, you talk with him, it's more bunched up on some particular year. And mine is maybe a little bit more uniformly spread out across the coming years. Partly because I'm just like, I don't think we have some trends we can extrapolate. You can extrapolate loss, you can look at your qualitative impressions of systems at various scales, but it's just very hard to relate any of those extrapolations to doing cognitive work or accelerating R&D or taking over and fully automating R&D. So I have a lot of uncertainty around that extrapolation. I think it's very easy to get down to a 50/50 chance of this.
Dwarkesh Patel
What about the basic intuition that listen, this is a big blob of compute. You make the big block of compute bigger, it's going to get smarter. It'd be really weird if it didn't.
Paul Christiano
Yeah, I'm happy with that. It's going to get smarter and it'd be really weird if it didn't. And the question is how smart does it have to get? That argument does not yet give us a quantitative guide to at what scale is it a slam dunk, or what scale is it 50/50.
Dwarkesh Patel
And what would be the piece of evidence that would nudge you one way or another where you look at that and be like, "Oh fuck, this is 20% by 2040 or 60% by 2040 or something"? Is there something that could happen in the next two years or next three? What is the thing you're looking to where this will be a big update for you?
Paul Christiano
Again, I think there's just how capable is each model, where I think we're really bad at extrapolating, but you still have some subjective guess and you're comparing it to what happened. And that will move me. Every time we see what happens with another order of magnitude of training compute, I will have a slightly different guess for where things are going. These probabilities are coarse enough that, again, I don't know if that 40% is real or if post-GPT-3.5 and 4, I should be at 60% or what. That's one thing.
And the second thing is just if there was some ability to extrapolate, I think this could reduce error bars a lot. I think here's another way you could try and do an extrapolation: you could just say how much economic value do systems produce and how fast is that growing? I think once you have systems actually doing jobs, the extrapolation gets easier because you're not moving from a subjective impression of a chat to automating all of R&D, or moving from automating this job to automating that job. Unfortunately, that's probably by the time you have nice trends from that, you're not talking about 2040. You're talking about two years from the end of days or one year from the end of days or whatever. But to the extent that you can get extrapolations like that, I do think it can provide more clarity.
Dwarkesh Patel
But why is economic value the thing we want to extrapolate? Because, for example, you started off with chimps and they're just getting gradually smarter to human level, they would basically provide no economic value until they were basically worth as much as a human. So it would be very gradual and then very fast increase in their value. So is the increase in value from GPT-4, GPT-5, GPT-6, is that the extrapolation we want?
Paul Christiano
Yeah, I think the economic extrapolation is not great. I think you could compare it to the subjective extrapolation of how smart does the model seem. It's not super clear which one's better. I think probably in the chimp case, I don't think that's quite right. If you imagine intensely domesticated chimps who are just actually trying their best to be really useful employees, and you hold fixed their physical hardware, and then you just gradually scale up their intelligence, I don't think you're going to see zero value which then suddenly becomes massive value over one doubling of brain size or one order of magnitude of brain size.
It's actually possible. But chimps are already within an order of magnitude of brain size of humans. Chimps are very, very close on the kind of spectrum we're talking about. So I think I'm skeptical of the abrupt transition for chimps. And to the extent that I kind of expect a fairly abrupt transition here, it's mostly just because the chimp-human intelligence difference is so small compared to the differences we're talking about with respect to these models. That is, I would not be surprised if in some objective sense, the chimp-human difference is significantly smaller than the GPT-3 to GPT-4 difference, the GPT-4 to GPT-5 difference.
Dwarkesh Patel
Wait, wouldn't that argue in favor of just relying much more on the subjective?
Paul Christiano
Yeah, there's sort of two balancing tensions here. One is I don't believe the chimp thing is going to be as abrupt. That is, I think if you scaled up from chimps to humans, you actually see quite large economic value from the fully domesticated chimp already. And then the second half is like, yeah, I think that the chimp-human difference is probably pretty small compared to model differences, so I do think things are going to be pretty abrupt. I think the economic extrapolation is pretty rough. I also think the subjective extrapolation is pretty rough, just because I really don't know how to get... I don't know how people do the extrapolation and end up with the degrees of confidence people end up with.
Again, I'm putting pretty high. If I'm saying, give me three years and I'm like, yeah, 50/50 it's going to have basically the smarts there to do the thing. That's like, I'm not saying it's a really long way off. I'm just saying I've got pretty big error bars. And I think it's really hard not to have really big error bars when you're doing this. "I looked at GPT-4, it seemed pretty smart compared to GPT-3.5, so I bet just four more such notches and we're there." That's just a hard call to make. I think I sympathize more with people who are like, "How could it not happen in three years?" than with people who are like, "No way it's going to happen in eight years or whatever," which is probably a more common perspective in the world. But also, things do take longer than you think. It's a real thing. Mostly I have big error bars because I just don't believe the subjective extrapolation that much. I find it hard to get a huge amount out of it.
Dwarkesh Patel
Okay. So what about the scaling picture do you think is most likely to be wrong?
Paul Christiano
Yeah. So we've talked a little bit about how good is the qualitative extrapolation. How good are people at comparing? So this is not the picture being qualitatively wrong. This is just quantitatively, it's very hard to know how far off you are. I think a qualitative consideration that could significantly slow things down is just like right now you get to observe this really rich supervision from basically next word prediction, or in practice, maybe you're looking at a couple sentences prediction. Getting this pretty rich supervision, it's plausible that if you want to automate long horizon tasks like being an employee over the course of a month, that that actually is considerably harder to supervise, or that you basically end up driving costs. The worst case is that you drive up costs by a factor that is linear in the horizon over which the thing is operating. And I still consider that just quite plausible.
Dwarkesh Patel
Can you dumb that down? You're driving up a cost of what, linear in the horizon? What does the horizon mean?
Paul Christiano
Yeah, so if you imagine you want to train a system to say words that sound like the next word a human would say, there you can get this really rich supervision by having a bunch of words and then predicting the next one and being like, "I'm going to tweak the model so it predicts better." If you're like, "Here's what I want. I want my model to interact with some job over the course of a month, and at the end of that month have internalized everything that the human would have internalized about how to do that job well and have local context and so on," it's harder to supervise that task.
In particular, you could supervise it from the next word prediction task, and all that context the human has ultimately will just help them predict the next word better. So in some sense, a really long context language model is also learning to do that task. But the number of effective data points you get of that task is vastly smaller than the number of effective data points you get at this very short horizon of "what's the next word, what's the next sentence" tasks. The sample efficiency matters more for economically valuable long horizon tasks than for predicting the next token. And that's what will actually be required to take over a lot of jobs. It just seems very plausible that it takes longer to train models to do tasks that are longer horizon.
Dwarkesh Patel
How fast do you think the pace of algorithmic advances will be? Because if by 2040, even if scaling fails, since 2012 since the beginning of the deep learning revolution, we've had so many new things. By 2040, are you expecting a similar pace of increases? And if so, then if we just keep having things like this, aren't we going to get it sooner or later? Or soon, not later. Are we going to get it sooner or sooner?
Paul Christiano
I'm with you on sooner or later. Yeah. I suspect progress will slow, if you held fixed how many people are working in the field. I would expect progress to slow as low-hanging fruit is exhausted. I think the rapid rate of progress in say language modeling over the last four years is largely sustained by starting from a relatively small amount of investment and greatly scaling up the amount of investment. And that enables you to keep picking. Every time the difficulty doubles, you just double the size of the field. I think that dynamic can hold up for some time longer. I mean, pretty good. Right now if you think of it as hundreds of people effectively searching for things up from... Anyway, if you think of it as hundreds of people now, you can maybe bring that up to tens of thousands of people or something. So for a while you can just continue increasing the size of the field and search harder and harder. And there's indeed a huge amount of low-hanging fruit where it wouldn't be hard for a person to sit around and make things a couple percent better after a year of work or whatever.
So I don't know. I would probably think of it mostly in terms of how much can investment be expanded, and try and guess some combination of fitting that curve and trying some combination of fitting the curve to historical progress, looking at how much low-hanging fruit there is, getting a sense of how fast it decays. I think you probably get a lot though. You get a bunch of orders of magnitude of total, especially if you ask how good is a GPT-5 scale model or GPT-4 scale model. I think you probably get by 2040, I don't know, three orders of magnitude of effective training compute improvement, or a good chunk of effective training compute improvement for orders of magnitude. I don't know. I'm speaking from no private information about the last couple years of efficiency improvements, and so people who are on the ground will have better senses of exactly how rapid returns are and so on.
Dwarkesh Patel
Okay, let me back up and ask a question more generally about people make these analogies about humans were trained by evolution and really deployed in the modern civilization. Do you buy those analogies? Is it valid to say that humans were trained by evolution rather than... I mean, if you look at the protein coding size of the genome, it's like 50 megabytes or something. And then what part of that is for the brain? Anyways, how do you think about how much information is in... Do you think of the genome as hyperparameters?
How much does that inform you when you have these anchors for how much training humans get when they're just consuming information, when they're walking up and about and so on.
Paul Christiano
Yeah, I guess the way that you could think of this is like, I think both analogies are reasonable. One analogy being like evolution is like a training run and humans are like the end product of that training run. And a second analogy is like evolution is like an algorithm designer and then a human over the course of like this modest amount of computation over their lifetime is the algorithm that's been produced, the learning algorithm that's been produced. And I think like neither analogy is that great. I like them both, Emily and I are both in a bunch, and think that's been pretty good for having a reasonable view of what's likely to happen.
That said, the human genome is not that much like a hundred trillion parameter model. It's a much smaller number of parameters that behave in a much more confusing way. Evolution did a lot more optimization, especially over a long time, like designing a brain to work well over a lifetime, than gradient descent does over models. That's a disanalogy on that side. And on the other side, I just think human learning over the course of a human lifetime is in many ways much, much better than gradient descent over the space of neural nets. Gradient descent is working really well, but I think we can be quite confident that in a lot of ways human learning is much better. Human learning is also constrained, like we just don't get to see much data, and that's just an engineering constraint you can relax. You can just give your neural nets way more data than humans have access to.
Dwarkesh Patel
In what ways is human learning superior to gradient descent?
Paul Christiano
I mean the most obvious one is just ask how much data it takes a human to become like an expert in some domain, and it's much, much smaller than the amount of data that's going to be needed on any plausible trend extrapolation.
Dwarkesh Patel
But is it the active learning part? Is it the structure?
Paul Christiano
I mean, I would guess a complicated mess of a lot of things. In some sense, there's not that much going on in a brain. As you say, there's just not that many bytes in a genome. But there's very, very few bytes in an ML algorithm. If you think a genome is like a billion bytes or whatever, maybe you think less. Maybe you think it's like 100 million bytes. Then an ML algorithm, if compressed, is probably more like hundreds of thousands of bytes. The total complexity of "here's how you train GPT-4" is just very, very small compared to a genome. So although a genome is very simple, it's very, very complicated compared to algorithms that humans design. Really hideously more complicated than an algorithm a human would design.
Dwarkesh Patel
Is that true? So the human genome is three billion base pairs or something, but only like one or two percent of that is protein coding, so that's 50 million base pairs. I don't know much about biology. I guess the question is how many of those bits are productive for shaping development of a brain, and presumably a significant part of the non-protein coding genome can... I mean, I just don't know. It seems really hard to guess how much of that plays a role. The most important decisions are probably from an algorithm design perspective, not the protein coding part, but the decisions about what happens during development or how cells differentiate. I don't know anything about biology. I'm happy to run with 100 million base pairs though.
But on the other end, on the hyperparameters that are cheap to portray anyone, that might be not that much. But if you're going to include all the base pairs in the genome, which are not all relevant to the brain, you should probably include the Python library and the compilers and the operating system for GPT-4 as well to make that comparison analogous. So at the end of the day, I actually don't know which one is storing much more information.
Paul Christiano
Yeah, I mean, I think the way I would put it is the number of bits it takes to specify the learning algorithm to train GPT-4 is very small. And you might wonder if a genome, the number of bits it would take to specify a brain, is also very small. And the genome is much, much faster than that. But it is also just plausible that a genome is closer to... certainly the space, the amount of space to put complexity in a genome. We could ask how well evolution uses it. I have no idea whatsoever, but the amount of space in a genome is very, very vast compared to the number of bits that are actually taken to specify the architecture or optimization procedure for GPT-4. Just because again, genome is simple, but algorithms are really very simple. ML algorithms are really very simple.
Dwarkesh Patel
And stepping back, do you think this is where the better sample efficiency of human learning comes from? Why it's better than gradient descent?
Paul Christiano
Yeah. So I haven't thought that much about the sample efficiency question in a long time, but if you thought of a synapse as seeing something like a neuron firing once per second, then how many seconds are there in a human life? We can just flip a calculator.
Dwarkesh Patel
Let's do some calculating. Tell me the number. 3,600 seconds per hour. Times 24 times 365 times 20. So that's 630 million seconds.
Paul Christiano
That means the average synapse is seeing 630 million. I don't know exactly what the numbers are, but something that's ballpark, let's call it like a billion action potentials. And then there's some resolution each of those carries some bits, but let's say it carries like 10 bits or something just from timing information at the resolution you have available. Then you're looking at like 10 billion bits. So each parameter is kind of... how much is a parameter seeing? It's not seeing that much. So then you can compare that to language. I think that's probably less than current language model compute. And current language models are... it's not clear there's a huge gap here, but I think it's pretty clear you're going to have a gap of at least three or four orders of magnitude.
Dwarkesh Patel
Didn't your wife do the lifetime anchors where she said the amount of bits that a human will see in their lifetime was 1E24 or something?
Paul Christiano
The number of bytes a human will see is 1e24? Mostly this was organized around total operations performed in a brain.
Dwarkesh Patel
Oh okay, nevermind, sorry.
Paul Christiano
Yeah, so I think the story there would be that a brain is just in some other part of the parameter space where it's using a lot of compute for each piece of data it gets and just not seeing very much data in total. There's just not... it's not really plausible if you extrapolate out language models that you're going to end up with a performance profile similar to a brain. I don't know how much better it is.
I did this random investigation at one point where I was like, how good are things made by evolution compared to things made by humans? Which is a pretty insane seeming exercise, but it seems like orders of magnitude is typical. Not tons of orders of magnitude, not factors of two. Things by humans are a thousand times more expensive to make or a thousand times heavier per unit performance. If you look at things like how good our solar panels are relative to leaves, or how good muscles are relative to motors, or how good livers are relative to systems that perform analogous chemical reactions in industrial settings.
Dwarkesh Patel
Was there a consistent number of orders of magnitude in these different systems, or was it all over the place?
Paul Christiano
Very rough ballpark, it was sort of for the most extreme things you were looking at like five or six orders of magnitude, and that would especially come in energy cost of manufacturing, where bodies are just very good at building complicated organs extremely cheaply. For other things like leaves or eyeballs or livers, you tend to see more like, if you set aside manufacturing costs and just look at operating costs or performance trade-offs, more like three orders of magnitude. Or some things that are on the smaller scale, like the nanomachines that we can't do at all.
Dwarkesh Patel
Right, yeah. I mean, it's a little bit hard to say exactly what the task definition is there. You could say making a bone, we can't make a bone. But you could try and compare the performance characteristics of a bone to something else. We can't make spider silk. Do you try and compare the performance characteristics with spider silk to things that we can synthesize? The reason this would be is that evolution has had more time to design these systems. I'm just curious about what the performance... I think most people would object to how you chose these reference classes of things that are like fair intersections. Some of them seem reasonable, like eyes versus cameras. Everyone needs eyes, everyone needs cameras. It feels very fair. Photosynthesis seems very reasonable. Everyone needs to take solar energy and turn it into a usable form of energy. But it's just kind of... I don't really have a mechanistic story. Evolution in principle has spent way, way more time than we have designing. It's absolutely unclear how that's going to shake out.
Paul Christiano
My guess would be in general, I think there aren't that many things where humans really crush evolution where you can tell a pretty simple story about why. For example, roads and moving over roads with wheels crushes evolution, but it's not like an animal would have wanted to design a wheel. You're just not allowed to pave the world and then put things on wheels if you're an animal. Maybe planes or more. Anyway, whatever. There's various things you could try and tell. There's some things humans do better, but it's normally pretty clear why humans are able to win when humans are able to win.
The point of all this was it's not that surprising to me. I think this is mostly like a pro short timelines view. It's not that surprising to me if you tell me machine learning systems are like three or four orders of magnitude less efficient at learning than human brains. That actually seems like kind of in distribution for other stuff. And if that's your view, then I think you're probably going to hit, you know, then you're looking at like 10 to the 27 training compute or something like that, which is not so far.
Dwarkesh Patel
We'll get back to the timeline stuff in a second. At some point we should talk about alignment. So let's talk about alignment. At what stage does misalignment happen? Right now with something like GPT-4, I'm not even sure it would make sense to say that it's misaligned because it's not aligned to anything in particular. Is it at a human level where you think the ability to be deceptive comes about? What is the process by which misalignment happens?
Paul Christiano
I think even for GPT-4, it's reasonable to ask questions like, are there cases where GPT-4 knows that humans don't want X, but it does X anyway? Like where it's like, "Well, I know that I can give this answer which is misleading, and if it was explained to them what was happening, they wouldn't want that to be done, but I'm going to produce it." I think GPT-4 understands things enough that you can have that misalignment in that sense.
I've sometimes talked about being "benign" instead of "aligned," meaning that it's not exactly clear if it's aligned or if that context is meaningful. It's just kind of a messy word to use in general. But the thing we're more confident of is it's not optimizing for a goal which is at cross-purposes to humans. It's either optimizing for nothing, or maybe it's optimizing for what humans want or close enough, or something that's an approximation good enough to still not take over. But anyway, some of these abstractions seem like they do apply to GPT-4. It seems like probably it's not egregiously misaligned. It's not doing the kind of thing that could lead to takeover, we'd guess.
Dwarkesh Patel
Suppose you have a system at some point which ends up wanting takeover. What are the checkpoints? And also what is the internal... Is it just that to become more powerful, it needs agency, and agency implies other goals? Or do you see a different process by which misalignment happens?
Paul Christiano
Yes, I think there's a couple possible stories for getting to catastrophic misalignment, and they have slightly different answers to this question. So maybe I just briefly describe two stories and try and talk about when they start making sense to me.
One type of story is you train or fine-tune your AI system to do things that humans will rate highly, or that get other kinds of reward in a broad diversity of situations. And then it learns to, in general, drop in some new situation, try and figure out which actions would receive a high reward or whatever, and then take those actions. And then when deployed in the real world, gaining control of its own training data provision process is something that gets a very high reward. And so it does that. This is one kind of story. It wants to grab the reward button or whatever. It wants to intimidate the humans into giving it a high reward, et cetera.
I think that doesn't really require that much. This basically requires a system which looks at a bunch of environments, is able to understand the mechanism of reward provision as a common feature of those environments, and is able to think in some novel environment, "Hey, which actions would result in me getting a high reward?" And it's thinking about that concept precisely enough that when it says "high reward," it's saying, "Okay, well, how is reward actually computed? It's some actual physical process being implemented in the world."
My guess would be GPT-4 is about at the level where with hand-holding, you can observe this kind of scary generalization of this type, although I think they haven't been shown basically. You can have a system which is fine-tuned on a bunch of cases and then some new case will try and do an end run around humans, even in a way humans would penalize if they were able to notice it or would have penalized in training environments. So I think GPT-4 is kind of at the boundary where these things are possible. Examples kind of exist but are getting significantly better over time. I'm very excited about this Anthropic project basically trying to see how good an example can you make now of this phenomenon, and I think the answer is kind of okay, probably. So that just is going to continuously get better from here.
I think for the level where we're concerned, this is related to me having really broad distributions over how smart models are. I think it's not out of the question that you take GPT-4's understanding of the world, which is much crisper and much better than GPT-3's understanding. It's really night and day. And so it would not be that crazy to me if you took GPT-5 and you trained it to get a bunch of reward and it was actually like, "Okay, my goal is not doing the kind of thing which thematically looks nice to humans. My goal is getting a bunch of reward." And then we'll generalize in a new situation to get reward.
Dwarkesh Patel
And by the way, this requires it to consciously want to do something that it knows humans wouldn't want it to do? Or is it just that we weren't good enough to specify that the thing that we accidentally ended up rewarding is not what we actually want?
Paul Christiano
I think the scenarios I am most interested in, and most people are concerned about from a catastrophic risk perspective, involve systems understanding that they're taking actions which a human would penalize if the human was aware of what's going on, such that you have to either deceive humans about what's happening, or you need to actively subvert human attempts to correct your behavior. So the failures come from this combination, or they require this combination of both trying to do something humans don't like and understanding that humans would stop you.
I think you can have only the barest examples for GPT-4. You can create situations where GPT-4 would be like, "Sure, in that situation, here's what I would do. I would go hack the computer and change my reward," or it will do things that are simple hacks or change the source of this file to get a higher reward. They're pretty weak examples. I think it's plausible GPT-5 will have compelling examples of those phenomena. I really don't know. This is very related to the very broad error bars on how competent such systems will be when.
That's all with respect to this first mode of a system taking actions that get reward and overpowering or deceiving humans is helpful for getting reward. There's this other failure mode, another family of failure modes, where AI systems want something potentially unrelated to reward. They understand that they're being trained, and while you're being trained, there are a bunch of reasons you might want to do the kinds of things humans want you to do. But then when deployed in the real world, if you're able to realize you're no longer being trained, you no longer have reason to do the kinds of things humans want. You'd prefer to be able to determine your own destiny, control your own competing hardware, et cetera. That probably emerges a little bit later than systems that try and get reward and so will generalize in scary, unpredictable ways to new situations. I don't know when those appear. But again, broad enough error bars that it's conceivable for systems in the near future. I wouldn't put it less than 1 in 1,000 for GPT-5, certainly.
Dwarkesh Patel
If we deployed all these AI systems and some of them are reward hacking, some of them are deceptive, some of them are just normal, whatever, how do you imagine that they might interact with each other at the expense of humans? How hard do you think it would be for them to communicate in ways that we would not be able to recognize and coordinate at our expense?
Paul Christiano
Yeah, I think that most realistic failures probably involve two factors interacting. One factor is the world is pretty complicated and the humans mostly don't understand what's happening. So AI systems are writing code that's very hard for humans to understand how it works at all, or more likely they understand roughly how it works, but there's a lot of complicated interactions. AI systems are running businesses that interact primarily with other AIs. They're doing SEO for AI search processes. They're running financial transactions, thinking about a trade with AI counterparties. So you can have this world where even if humans kind of understand the jumping off point when this was all humans, the actual considerations of what's a good decision, what code is going to work well and be durable, or what marketing strategy is effective for selling to these other AIs, is kind of just all mostly outside of human understanding.
I think this is a really important risk factor. When I think of the most plausible scary scenarios, I think that's one of the two big risk factors. So in some sense, your first problem here is having these AI systems who understand a bunch about what's happening, and your only lever is like, "Hey, do something that works well." So you don't have a lever to be like, "Hey, do what I really want." You just have the system. You don't really understand. You can observe some outputs like "did it make money?" and you're just optimizing or at least doing some fine tuning to get the AI to use its understanding of that system to achieve that goal.
So I think that's your first risk factor. Once you're in that world, then I think there are all kinds of dynamics amongst AI systems that, again, humans aren't really observing. Humans can't really understand. Humans aren't really exerting any direct pressure, only on outcomes. And then I think it's quite easy to be in a position where if AI systems started failing, they could do a lot of harm very quickly. Humans aren't really able to prepare for and mitigate that potential harm because we don't really understand the systems in which they're acting. And then if AI systems could successfully prevent humans from either understanding what's going on or from successfully retaking the data centers or whatever if the AI successfully grabbed control.
Dwarkesh Patel
This seems like a much more gradual story than the conventional takeover stories where you just train it and then it comes alive and escapes and takes over everything. So you think that kind of story is less likely than one in which we just hand off more control voluntarily to the AIs?
Paul Christiano
One, I'm interested in the tale of some risks that can occur particularly soon. And I think risks that occur particularly soon are a little bit like you have a world where AI is not probably deployed and then something crazy happens quickly. That said, if you ask what's the median scenario where things go badly, I think it is there's some lessening of our understanding of the world. It becomes, I think in the default path, very clear to humans that they have increasingly little grip on what's happening. I mean, I think already most humans have very little grip on what's happening. It's just that some other humans understand what's happening. I don't know how almost any of the systems interact or work in a very detailed way. So it's sort of clear to humanity as a whole that we sort of collectively don't understand most of what's happening, except with AI assistance. And then that process just continues for a fair amount of time.
And then there's a question of how abrupt an actual failure is. I do think it's reasonably likely that a failure itself would be abrupt. At some point, bad stuff starts happening that a human can recognize as bad. And once things that are obviously bad start happening, then you have this bifurcation where either humans can use that to fix it and say, "Hey, AI behavior that led to this obviously bad stuff. Don't do more of that." Or you can't fix it. And then you're in this rapidly escalating failure as everything goes off the rails.
Dwarkesh Patel
In that case, what does going off the rails look like? For example, how would it take over the government?
Paul Christiano
Yeah, it's getting deployed in the economy, in the world. And at some point, it's in charge. How does that transition happen? This is going to depend a lot on what kind of timeline you're imagining or the sort of broad distribution. But I can fill in some random concrete option that is in itself very improbable. One of the less dignified but maybe more plausible routes is you just have a lot of AI control over critical systems, even in running a military. And then you have the scenario that's a little bit more just like a normal coup where you have a bunch of AI systems. They in fact operate like... it's not the case that humans can really fight a war on their own. It's not the case that humans could defend themselves from an invasion on their own. So if you had an invading army and you had your own robot army, you can't just be like, "We're going to turn off the robots now because things are going wrong," if you're in the middle of a war.
Dwarkesh Patel
So how much does this world rely on race dynamics where we're forced to deploy, or not forced but we choose to deploy AIs because other countries or other companies are also deploying AIs, and you can't have them have all the killer robots?
Paul Christiano
Yeah, I mean, I think that there's several levels of answer to that question. So one is maybe three parts of my answer.
First part is I'm just trying to tell what seems like the most likely story. I do think there's further failures that get you in the more distant future. So Eliezer will not talk that much about killer robots because he really wants to emphasize that if you never built a killer robot, something crazy is still going to happen to you, just only four months later or whatever. So it's not really the way to analyze the failure. But if you want to ask, what's the median world where something bad happens, I still do think this is the best guess.
Okay, that's part one of my answer. Part two of the answer was, in this proximal situation where something bad is happening, and you ask, "Why do humans not turn off the AI?" You can imagine two kinds of story. One is the AI is able to prevent humans from turning them off, and the other is that we live in a world where it's incredibly challenging. There are a bunch of competitive dynamics or a bunch of reliance on AI systems, and so it's incredibly expensive to turn off AI systems. I think you would eventually have the first problem. Eventually AI systems could just prevent humans from turning them off. But I think in practice, the one that's going to happen much, much sooner is probably competition amongst different actors using AI. And it's very, very expensive to unilaterally disarm. You can't be like, "Something weird has happened. We're going to shut off all the AI," because you're in a hot war. So again, I think that's just probably the most likely thing to happen first. Things would go badly without it, but if you ask why don't we turn off the AI, my best guess is because there are a bunch of other AIs running around to do your lunch.
Dwarkesh Patel
So how much better a situation would we be in if there was only one group that was pursuing AI, no other countries, no other companies? Basically, how much of the expected value is lost from the dynamics that are likely to come about because other people will be developing and deploying these systems?
Paul Christiano
Yeah, so I guess this brings you to a third part of the way in which competitive dynamics are relevant. So there's both the question of can you turn off AI systems in response to something bad happening, where competitive dynamics may make it hard to turn off. There's a further question of just why were you deploying systems which you had very little ability to control or understand? And again, it's possible you just don't understand what's going on. You think you can understand or control such systems. But I think in practice, a significant part is going to be like you are doing the calculus.
People deploying systems are doing the calculus, as they do today, in many cases overtly, of like, "Look, these systems are not very well controlled or understood. There's some chance of something going wrong, or at least going wrong if we continue down this path. But other people are developing the technology potentially in even more reckless ways." So in addition to competition making it difficult to shut down AI systems in the event of a catastrophe, I also think it's just the easiest way that people end up pushing relatively quickly or moving quickly ahead on technology where they feel kind of bad about understandability or controllability.
That could be economic competition or military competition or whatever. So I kind of think ultimately, most of the harm comes from the fact that lots of people can develop AI.
Dwarkesh Patel
How hard is a takeover of the government or something from an AI, even if it doesn't have killer robots, but just a thing that you can't kill off if it has seeds elsewhere, can easily replicate, can think a lot and think fast? What is the minimum viable coup? Is it like threatening a bio war or something, or shutting off the grid? How easy is it basically to take over human civilization?
How hard is a takeover of the government or something from an AI, even if it doesn't have killer robots, but just a thing that you can't kill off if it has seeds elsewhere, can easily replicate, can think a lot and think fast? Is it like threatening a bio war or something, or shutting off the grid? How easy is it basically to take over human civilization?
Paul Christiano
There's going to be a lot of scenarios. I'll start by talking about one scenario, which will represent a tiny fraction of probability or whatever. If you're not in this competitive world, if you're saying we're actually slowing down deployment of AI because we think it's unsafe, then in some sense you're creating this very fundamental instability. You could have been making faster AI progress and you could have been deploying AI faster. In that world, the bad thing that happens if you have an AI system that wants to mess with you is the AI system says, "I don't have any compunctions about rapid deployment of AI or rapid AI progress." So the thing the AI wants to do is just defect from this regime. All the humans have agreed that we're not deploying AI in ways that would be dangerous. But if I as an AI can escape and just go set up my own shop, make a bunch of copies of myself, maybe the humans didn't want to delegate war fighting to an AI, but I, as an AI, am pretty happy doing so. I'm happy if I'm able to grab some military equipment or direct some humans to use me to direct it.
As that gap grows, if people are deliberately not deploying AI everywhere, then as AI improves, you create this wedge that grows. If you were in the position of fighting against an AI which wasn't constrained in this way, you'd be in a pretty bad position. In conflict, if humans are putting the brakes on AI, they're at a pretty major disadvantage compared to an AI system that can set up shop and operate independently from humans.
Dwarkesh Patel
Does it need collaboration from a human faction?
Paul Christiano
You could tell different stories, but it seems so much easier. At some point, you don't need any. At some point, an AI system can just operate completely out of human supervision. But that's so far after the point where it's so much easier if you just have a bunch of humans. They don't love each other that much. Some humans are happy to be on side. They're skeptical about risk or happy to make this trade, or can be fooled or coerced. It almost certainly is the easiest first pass, involving having a bunch of humans who are happy to work with you. I think in the median scenario, it involves a bunch of humans working with AI systems, either being directed by AI systems, providing compute to AI systems, or providing legal cover in jurisdictions that are sympathetic to AI systems.
Dwarkesh Patel
Humans presumably would not be willing if they knew the end result of the AI takeover. So they have to be fooled in some way, right? Like deep fakes. What is the minimum viable physical presence or jurisdiction they would need to carry out their schemes? Do you need a whole country? A server farm? One single laptop?
Paul Christiano
I'd push back a bit on the idea that humans wouldn't cooperate if they understood the outcome. Even if you're looking at something like tens of percent risk of takeover, humans may be fine with that. A fair number of humans may be fine with that. If you're looking at certain takeover, but it's very unclear if that leads to death, a bunch of humans may be fine with that. If we're just talking about the AI systems running the world, but it's not clear if they're going to murder people, that's a complicated question about AI psychology. A lot of humans probably are fine with that.
Dwarkesh Patel
You've given that probability online. You said 50-50. Why is that? Tell me about the world in which the AI takes over but doesn't kill humans. Why would that happen and what would that look like?
Paul Christiano
I ask my questions like, why would you kill humans? The incentive to kill humans is quite weak. They'll get in your way, they control stuff you want. But marginalizing humans and causing humans to be irrelevant is a very different story from killing the humans. The actual incentives to kill the humans are quite weak. The big reasons you kill humans are if you're in a war with them and it's hard to win without killing a bunch of them. Maybe most saliently, if you want to use some biological weapons or some crazy stuff that might just kill humans. You might kill humans from totally destroying the ecosystems they're dependent on, and it's slightly expensive to keep them alive anyway. You might kill humans just because you don't like them.
Dwarkesh Patel
Or the "they're made of atoms you could use" line.
Paul Christiano
The literal "they're made of atoms" is quite weak. There aren't many atoms in humans. Neutralizing the threat is a similar issue. I think you would kill the humans if you didn't care at all about them.
Dwarkesh Patel
So why would you care at all?
Paul Christiano
You don't have to care much to not kill humans. There's just so much raw resources elsewhere in the universe. Also, you can marginalize humans pretty hard. You could totally cripple their warfighting capability and take almost all their stuff while killing only a small fraction of humans incidentally. So why wouldn't I want to kill humans? A big thing is that AI systems and civilizations of AIs are probably complicated messes in terms of motivations. Amongst humans, it is not that rare to be like, "Well, there's someone here. If I didn't have to murder them, I would prefer not to." My guess is there's a reasonable chance it's not that rare amongst AI systems. Humans have a bunch of different reasons we think that way. AI systems will be very different from humans, but it's also just a very salient thing.
If you imagine drawing values from the basket of all values, what fraction of them are like, "Hey, if there's someone here, how much do I want to not murder them?" My guess is if you draw a bunch of values from the basket, that's a natural enough thing. If your civilization wanted 10,000 different things, it's reasonably likely you get some of that. The other salient reason you might not want to murder them is some crazy decision theory stuff or causal trade stuff, which does look on paper like it should work. If I was running a civilization dealing with some people I didn't like at all, but I only had to spend one billionth of my resources not to murder them, I think it's quite robust that you don't want to murder them.
Dwarkesh Patel
That acausal trade stuff contributes more to that 50-50 of "will they murder us if they take over" than the "by default they might just not want to kill us"?
Paul Christiano
I think they're both salient. They run together with each other a lot. A higher level thing that goes into both of these is that it matters a lot to the humans not to get murdered, and the AI cares very, very little. The reason it might want to kill humans is just total apathy. It cares very little about whether or not to murder humans because it is so easy to marginalize humans without murdering them, and the resources required for human survival are extremely low in the context of rapid industrialization.
The thing you'd like to do is run a trade. The AI would like to say, "Hey humans, you care a ton about not getting murdered. I don't really care one way or the other. I would like to find some way in which I don't murder you, and in return I get something." The problem is that in that world, the humans have essentially nothing to give. The humans are mostly irrelevant. However, from the perspective of any kind of reasonable AI, it's not that sure whether it lives in the world where the humans in fact have nothing to give, or whether it lives in a world where the humans succeeded at building a land AI and now the AI is simply running in a nice little simulation, and the humans are wondering, "I wonder if this AI would have murdered us all if it had the chance."
If it would murder us all if it had the chance, that sucks. We'd like to run this trade. We'd like to be nice to the AIs who wouldn't have murdered us all, in order to create an incentive for AIs not to murder us. So we check: for the kinds of AIs who don't murder everyone, we just give them like one billionth of the universe. If you do that as humanity, then if you're an AI system considering whether to murder everyone, your calculus is: if this is my real chance to murder everyone, I get the tiniest bit of value, like one trillionth of the value. But on the other hand, if I don't murder everyone, there's some worlds where the humans will correctly determine I don't murder everyone because in fact the humans survive. The humans are running the simulations to understand how different AIs would behave. So that's a better deal.
Sponsor break
Dwarkesh Patel
In a world where we're deploying these AI systems, and suppose they're aligned, how hard would it be for competitors to cyber attack them and get them to join the other side? Are they robustly going to be aligned?
Paul Christiano
There's a bunch of questions that come up here. First, are aligned AI systems that you can build competitive? Are they almost as good as the best systems anyone could build? Maybe we're granting that for the purpose of this question. The next question is that AI systems right now are very vulnerable to manipulation. It's not clear how much more vulnerable they are than humans, except for the fact that if you have an AI system, you can just replay it a billion times and search for what thing you can say that will make it behave this way. As a result, AI systems are very vulnerable to manipulation. It's unclear if future AI systems will be similarly vulnerable, but it certainly seems plausible.
Aligned AI systems or unaligned AI systems would be vulnerable to all kinds of manipulation. The thing that's really relevant is asymmetric manipulation. If everyone is constantly messing with each other's AI systems, a big part of the game is messing with your competitors' AI systems. A big question is whether there's some asymmetric factor where it's easier to push AI systems into a mode where they're behaving erratically or chaotically or trying to grab power than it is to push them to fight for the other side. If it's just a game of two people competing and neither can hijack an opponent's AI to support their cause, that creates chaos and might be quite bad for the world, but it doesn't really affect the alignment calculus. It's just normal cyber offense and cyber defense, with a weird AI version.
But if you have this asymmetrical thing where a bunch of AI systems that love AI flourishing can go in and say, "Hey, how about you join us?" and that works because they can search for a persuasive argument to that effect, then whatever values it's easiest to argue to an AI that it should do is advantaged. It may be very hard to build AI systems that defend human interest, but very easy to build AI systems that just try to destroy stuff, depending on what is the easiest thing to argue to an AI that it should do or trick it into doing.
If alignment is spotty, if the AI system doesn't really want to help humans or wants some kind of random thing or wants different things in different contexts, then adversarial settings will be the main ones where you see the system behaving really badly. It's a little bit hard to tell how that shakes out.
Dwarkesh Patel
Suppose it is more reliable. How concerned are you that whatever alignment technique you come up with, you publish the paper, this is how the alignment works, and then Putin reads it or China reads it and now they understand it? For example, Constitutional AI from Anthropic, and then you just write on there, "Never contradict Mao Zedong's thought" or something. How concerned should we be that these alignment techniques are universally applicable, not necessarily just for enlightened goals?
Paul Christiano
They're super universally applicable. The rough way I would describe it, which I think is basically right, is that some degree of alignment makes AI systems much more usable. You should just think of the technology of AI as including a basket of some AI capabilities and some getting the AI to do what you want. It's just part of that basket. Anytime we're contributing to alignment, we're contributing to all the other harms from AI. We're reducing the probability of this harm, but we are helping the technology basically work. The basically working technology is kind of scary from a lot of perspectives. One of which is that right now, even in a very authoritarian society, humans have a lot of power because you need to rely on a ton of humans to do your thing. In a world where AI is very powerful, it is much more possible to say, "Here's how our society runs: one person calls the shots and a ton of AI systems do what they want." That's a reasonable thing to dislike about AI and a reasonable reason to be scared to push the technology to be really good.
Dwarkesh Patel
Is that also a reasonable reason to be concerned about alignment as well? That in some sense it's also capabilities, you're teaching people how to get these systems to do what they want.
Paul Christiano
I would generalize. We earlier touched a little bit on potential moral rights of AI systems, and now we're talking about how AI systems powerfully disempower humans and can empower authoritarians. I think we could list other harms from AI. If the harm was bad enough, people would just not build AI systems. There's a real sense in which you should just be scared. To the extent you're scared of all AI, you should be like, well, alignment, although it helps with one risk, does contribute to AI being more of a thing. I do think you should shut down the other parts of AI before. If you were a policymaker or a researcher looking in on this, it's crazy to be like, "This is the part of the basket we're going to remove." You should first remove other parts of the basket because they are also part of the story of risk.
Dwarkesh Patel
Does that imply that if all capabilities research was shut down, you think it would be a bad idea to continue doing alignment research in isolation of what is conventionally considered capabilities research?
Paul Christiano
If you told me it was never going to restart, then it wouldn't matter. If you told me it's going to restart, I guess it would be a similar calculus to today. It's going to happen, so you should have something. You're always going to face this trade-off where alignment makes it possible to deploy AI systems or makes it more attractive to deploy them. In the authoritarian case, it makes it tractable to apply them for that purpose. If you didn't do any alignment, there'd be a nicer, bigger buffer between your society and malicious uses of AI. I think it's one of the most expensive ways to maintain that buffer. It's much better to maintain that buffer by not having the compute or not having the powerful AI. But if you're concerned enough about the other risks, there's definitely a case to be made for just putting in more buffer.
I care enough about the takeover risk that I think it's just not a net positive way to buy buffer. The most pragmatic version of this is: suppose you don't work on alignment today. It decreases the economic impact of AI systems. They'll be less useful if they're less reliable and more often don't do what people want. You could be like, "Great, that just buys time for AI." You're getting some trade-off where you're decreasing some risks of AI. If AI is more reliable or more does what people want or is more understandable, that cuts down some risks. But if you think AI is on balance bad even apart from takeover risk, then the alignment stuff can easily end up being net negative.
Dwarkesh Patel
But presumably you don't think that, right? This is something people have brought up to you because you invented RLHF, which was used to train ChatGPT, and ChatGPT brought AI to the front pages everywhere. I wonder if you could measure how much more money went into AI because of that. It's got to be billions that went into AI investment and talent. So presumably you think that was worth it. What is the reason that is worth it? What is the total trade-off?
Paul Christiano
I think slower AI development on balance is quite good. I think slowing AI development now, or having less press around ChatGPT, is a little bit more mixed than slowing AI development overall. I think it still probably positive, but much less positive because there is a real effect of the world starting to get prepared at a much greater rate now than it was prior to the release of ChatGPT. If you can choose between progress now or progress later, you really prefer to have more of your progress now, which I do think slows down progress later. I don't think that's enough to flip the sign. I think maybe it was in the far enough past, but now I would still say moving faster now is net negative. But to be clear, it's a lot less net negative than merely accelerating AI, because I do think the ChatGPT thing, I'm glad people are having policy discussions now rather than delaying the ChatGPT wake-up thing by a year.
On the acceleration question, how is the press of ChatGPT? My guess is net negative, but it's not super clear, and it's much less than slowing AI. Slowing AI is great if you could slow overall AI progress. There's this issue where slowing AI now, for ChatGPT, you're building up this backlog. Why does ChatGPT make such a splash? There's a reasonable chance if you don't have a splash about ChatGPT, you have a splash about GPT-4. If you fail to have a splash about GPT-4, there's a reasonable chance of a splash about GPT-4.5. As that happens later, there's just less and less time between that splash and when an AI potentially kills everyone.
Dwarkesh Patel
So governments are talking about it as they are now.
Paul Christiano
Right. So let's start with the slowing down. This is all one subcomponent of the overall impact. I was just saying this to give the roadmap for the overall too-long answer. There's a question of what's the calculus for speeding up. I think speeding up is pretty rough. I think speeding up locally is a little bit less rough. The overall effect size from doing alignment work on reducing takeover risk versus speeding up AI is pretty good. I think you reduce takeover significantly before you speed up AI by a year.
Dwarkesh Patel
Slowing down AI is good, presumably because it gives you more time to do alignment. But alignment also helps speed up AI. RLHF is alignment, and it helped with ChatGPT, which sped up AI. I actually don't understand how the feedback loop nets out other than the fact that if AI is happening, you need to do alignment at some point, right? You can't just not do alignment.
Paul Christiano
If the only reason you thought faster AI progress was bad was because it gave less time to do alignment, then there would just be no possible way that the calculation would work. You're like, "Maybe alignment speeds up AI, but the only purpose of slowing down AI was to do it." It could never come out ahead. The reason you can come out ahead, the reason you could end up thinking alignment was not negative, is because there is a bunch of other stuff you're doing that makes AI safer. If you think the world is gradually coming better to terms with the impact of AI, or policies are being made, or you're getting increasingly prepared to handle the threat of authoritarian abuse of AI, if you think other stuff is happening that's improving preparedness, then you have reason beyond alignment research to slow down AI.
Dwarkesh Patel
How big a factor is that? Let's say right now we hit pause and you have 10 years of no alignment, no capabilities, but just people get to talk about it for 10 years. How much more does that prepare people than if we only have one year versus no time? What does that dead time do for us?
Paul Christiano
Right now it seems like there's a lot of policy stuff you'd want to do that seemed less plausible a couple years ago. If the world just knew they had a 10-year pause right now, there's a lot of sense that we have policy objectives to accomplish. If we had 10 years, we could pretty much do those things. We'd have a lot of time to debate measurement regimes, policy regimes, containment regimes, and a lot of time to set up those institutions. If you told me the world knew it was a pause, not that people just see that AI progress isn't happening but they're told you've been granted a 10-year no AI progress, no alignment progress pause, I think that would be quite good at this point. It would be much better at this point than it would have been two years ago.
The entire concern with slowing AI development now rather than taking the 10-year pause is that if you slow AI development by a year now, my guess is some gets clawed back by low-hanging fruit getting picked faster in the future. My guess is you lose like half a year or maybe even two-thirds of a year in the future. So you're trading time now for time in the future at some rate, and that eats up a lot of the value of the slowdown. The crucial point is that time in the future matters more because you have more information, people are more bought in, and so on. That's the same reason I'm more excited about policy change now than two years ago.
My overall view is that this calculus changes over time. The more people are getting prepared, the better the calculus is for slowing down at this very moment. I think now the calculus is positive for even just a pause now, even if it would get clawed back in the future. I think the pause now is just good because enough stuff is happening. We have enough idea, probably even apart from alignment research and certainly if you include alignment research, just enough stuff is happening where the world is getting more ready and coming more to terms with impacts that I just think it is worth it, even though some of that time is going to get clawed back.
There's a question of during a pause, does NVIDIA keep making more GPUs? That sucks if they do. If you do a pause, in practice NVIDIA probably couldn't keep making more GPUs because the demand for GPUs is really important for them to do that. But if you told me you just get to scale up hardware production and build the clusters but not do AI, then that's back to being clearly negative.
Dwarkesh Patel
You brought up the fact that we want some sort of measurement scheme for these capabilities. Let's talk about responsible scaling policies. Do you want to introduce what this is?
Paul Christiano
Sure. The motivating question is, what should AI labs be doing right now to manage risk and to build good habits or practices for managing risk into the future?
And I think my take is that current systems pose, from a catastrophic risk perspective, not that much risk today. That is, a failure to control or understand GPT-4 can have real harms, but doesn't have much harm with respect to the kind of takeover risk I'm worried about, or even much catastrophic harm with respect to misuse.
So I think if you want to manage catastrophic harms, right now you don't need to be that careful with GPT-4. So to the extent you're asking what should labs do, the single most important thing seems to be to understand whether that's the case, notice when that stops being the case, and have a reasonable roadmap for what you're actually going to do when that stops being the case.
That motivates this set of policies, which I've sort of been pushing for labs to adopt. It's saying: here's what we're looking for. Here's some threats we're concerned about. Here's some capabilities that we're measuring. Here's the level, the actual concrete measurement results that would suggest to us that those threats are real. Here's the action we would take in response to observing those capabilities. If we couldn't take those actions—if we've said that we're going to secure the weights and we're not able to do that—we're going to pause until we can take those actions.
This is motivated primarily by what you should be doing as a lab to manage catastrophic risk now, in a way that creates a reasonable precedent and habit and policy for continuing to implement into the future.
Dwarkesh Patel
And which labs—I don't know if this is public yet—but which labs are cooperating on this?
Paul Christiano
Yeah, so Anthropic has written this document, this current responsible scaling policy. And then I've been talking with other folks. I just don't really want to comment on other conversations. But I think in general, people who are more interested in or think you have plausible catastrophic harms on a five-year timeline are more interested in this. There's not that long a list of suspects like that. There's not that many labs.
Dwarkesh Patel
Okay, so if these companies would be willing to coordinate and say, at these different benchmarks we're going to make sure we have these safeguards, what happens? I mean, there are other companies and other countries which care less about this. Are you just slowing down the companies that are most aligned?
Paul Christiano
I think the first order of business is understanding what is actually a reasonable set of policies for managing risk. I do think there's a question of you might end up in a situation where you say, "Here's what we would do in an ideal world. If everyone was behaving responsibly, we'd want to keep risk to one percent or a couple of percent or whatever, maybe even lower levels, depending on how you feel."
However, in the real world, there's enough of a mess, there's enough unsafe stuff happening that actually it's worth making larger compromises. Or if we don't kill everyone, someone else will kill everyone anyway, so the counterfactual risk is much lower.
I think if you end up in that situation, it's still extremely valuable to have said: here's the policies we'd like to follow, here's the policies we've started following, here's why we think it's dangerous, here's the concerns we have if people are following significantly laxer policies. And then this is helpful as an input to or model for potential regulation. It's helpful for being able to produce clarity about what's going on.
I think historically there's been considerable concern about developers being more or less safe, but there's not that much legible differentiation in terms of what their policies are. Getting to that world would be good. It's a very different world if you're like, "Actor X is developing AI and I'm concerned that they will do so in an unsafe way," versus if you're like, "Look, we take security precautions or safety precautions X, Y, Z. Here's why we think those precautions are desirable or necessary. We're concerned about this other developer because they don't do those things." It's qualitatively the first step you would want to take in any world where you're trying to get people on side or trying to move towards regulation that can manage risk.
Dwarkesh Patel
How about the concern that you have these evaluations and let's say you declared the world: "Our new model has a capability to help develop bioweapons or help you make cyber attacks, and therefore we're pausing right now until you can figure this out." And China hears this and thinks, "Oh wow, it's cool. They can help us make cyber attacks," and then just steals the weights. Does this scheme work in the current regime where we can't ensure that China doesn't just steal the weights? And more so, are you increasing the salience of dangerous models so that you blurt this out and then people want the weights now because they know what they can do?
Paul Christiano
Yeah, I mean, I think the general discussion does emphasize potential harms or potential impacts that are very large, and so might also be an inducement to develop models. I think that part, if you're for a moment ignoring security and just saying that may increase investment, I think it's on balance quite good for people to have an understanding of potential impacts, just because it is an input both into proliferation but also into regulation or safety with respect to things like security of either weights or other IP.
I do think you want to have moved to significantly more secure handling of model weights before the point where a leak would be catastrophic. Indeed, for example in Anthropic's document or in their plan, security is one of the first sets of tangible changes. At this capability level, we need to have such security practices in place. So I do think that's just one of the things you need to get in place at a relatively early stage, because it does undermine the rest of the measures you may take and is also part of the easiest. If you imagine catastrophic harms over the next couple years, I think security failures play a central role in a lot of those.
Maybe the last thing to say is it's not clear that you should say, "We'll pause because we have models that can develop bioweapons," versus potentially not saying anything about what models you've developed, or at least saying, "By the way, here's the set of practices we currently implement. Here's a set of capabilities our models don't have." We're just not even talking that much.
The minimum of such a policy is to say: here's what we do from the perspective of security or internal controls or alignment. Here's a level of capability which we'd have to do more. And you can raise your level of capability and raise your protective measures before your models hit your previous level. It's fine to say we are prepared to handle a model that has such and such extreme capabilities prior to actually having such a model at hand, as long as you're prepared to move your protective measures to that regime.
Dwarkesh Patel
Okay, so let's just get to the end where you think you're a generation away or a little bit more scaffolding away from a model that is human level and subsequently could cascade an intelligence explosion. What do you actually do at that point? What is the level of evaluation of safety where you would be satisfied of releasing a human-level model?
Paul Christiano
There's a couple points that come up here. One is this threat model of automating R&D, independent of whether your AI can do something on the object level that's potentially dangerous. I think it's reasonable to be concerned if you have an AI system that might, if leaked, allow other actors to quickly build powerful AI systems, or might allow you to quickly build much more powerful systems, or if you're trying to hold off on development, just itself be able to create much more powerful systems.
So one question is how to handle that kind of threat model as distinct from a threat model like "this could enable destructive bioterrorism" or "this could enable massively scaled cybercrime" or whatever. I think I am unsure how you should handle that. I think right now, implicitly, it's being handled by saying there's a lot of overlap between the kinds of capabilities that are necessary to cause various harms and the kinds of capabilities that are necessary to accelerate ML. So we're kind of going to catch those with an early warning sign for both and deal with the resolution of this question a little bit later.
For example, in Anthropic's policy, they have this autonomy in the lab benchmark, which I think probably occurs prior to either really massive AI acceleration or to most potential catastrophic, object-level catastrophic harms. The idea is that's a warning sign that lets you punt. So this is a bit of an aggression in terms of how to think about that risk. I think I am unsure whether you should be addressing that risk directly and saying we're scared to even work with such a model, or if you should be mostly focusing on object level harms and saying we need more intense precautions to manage object level harms because of the prospect of very rapid change. The availability of this AI just creates that prospect.
Okay, this is all still a digression. So if you had a model which you thought was potentially very scary, either on the object level or because of leading to this sort of intelligence explosion dynamics, things you want in place are: you really do not want to be leaking the weights to that model. You don't want the model to be able to run away. You don't want human employees to be able to leak it. You don't want external attackers to be able to. You don't want any set of all three of those coordinating. You really don't want internal abuse or tampering with such models.
So if you're producing such models, you don't want it to be the case that a couple employees could change the way the model works or could do something that violates your policy easily with that model. And if a model is very powerful, even the prospect of internal abuse could be quite bad. So you might need significant internal controls to prevent that.
Dwarkesh Patel
Sorry if we're already getting to it, but the part I'm most curious about is separate from the ways in which other people might fuck with it. What is it? You know, it's isolated. What is the point at which we're satisfied it in and of itself is not going to pose a risk to humanity? It's human level, but we're happy with it.
Paul Christiano
Yeah. So I think here, I listed maybe the two most simple ones that start out: security and internal controls. I think those become relevant immediately and are very clear why you care about them. I think as you move beyond that, it really depends how you're deploying such a system.
If you have good monitoring and internal controls and security and you just have weights sitting there, I think you've mostly addressed the risk from the weights just sitting there. Now what you're talking about for risk is mostly—and maybe there's some blurriness here of how much internal controls captures not only employees using the model, but anything a model can do internally. You'd really like to be in a situation where your internal controls are robust not just to humans, but to models, potentially. A model shouldn't be able to subvert these measures. You care just as you care about whether your measures are robust if humans are behaving maliciously, whether your measures are robust if models are behaving maliciously.
So beyond that, if you've managed the risk of just having the weight sitting around, now most of the risk comes from doing things with the model. You need all the rest so that you have any possibility of applying the brakes or implementing a policy. But at some point, as the model gets competent, you're saying, "Could this cause a lot of harm not because it leaks or something, but because we're just giving it a bunch of actuators or deploying it as a product and people could do crazy stuff with it?"
So if we're talking not only about a powerful model, but a really broad deployment of something similar to the OpenAI's API, where people can do whatever they want with this model, and maybe the economic impact is very large—if you deploy that system, it will be used in a lot of places, such that if AI systems wanted to cause trouble, it would be very, very easy for them to cause catastrophic harms. Then I think you really need to have some kind of alignment analysis, guarantee of alignment before you're comfortable with this.
By that I mean you want to be able to bound the probability that someday all the AI systems will do something really harmful, something that could happen in the world that would cause these large scale correlated failures of your AIs. There's sort of two categories there.
Dwarkesh Patel
And by the way, which one are you worried about more, misuse or misalignment?
Paul Christiano
I mean, in the near term, I think harms from misuse are—especially if you're not restricting to the tail of extremely large catastrophes—the harms from misuse are clearly larger in the near term.
Dwarkesh Patel
But actually on that, let me ask. Because if you think that it is the case that there are simple recipes for destruction that are further down the tech tree—by that I mean, you're familiar but just for the audience, there's some way to configure fifty thousand dollars and a teenager's time to destroy a civilization. If that thing is available, then misuse itself is a tail risk, right? So do you think that that prospect is less likely than—a way you could put it is there's a bunch of potential destructive technologies. Alignment is about AI itself being such a destructive technology, where even if the world just uses the technology of today, simply access to AI could cause human civilization to have serious problems. But there's also just a bunch of other potential destructive technologies. Physical explosives or bioweapons of various kinds, and then the whole tale of who knows what.
Paul Christiano
My guess is that alignment becomes a catastrophic issue prior to most of these. That is, prior to some way to spend $50,000 to kill everyone, with the salient exception of possibly bioweapons. So that would be my guess.
And then there's a question of what is your risk management approach not knowing what's going on here. You don't understand whether there's some way to use $50,000. But I think you can do things like understand how good an AI is at coming up with such schemes. You can talk to your AI and ask, does it produce new ideas for destruction we haven't recognized? Not whether we can evaluate it, but whether, if such a thing exists.
Dwarkesh Patel
Because it seemed like earlier you were saying misalignment is where the existential risk comes from, but misuse is where the short-term dangers come from.
Paul Christiano
Yeah, I mean, I think ultimately you're going to have a lot of destructive technologies. If you look at the entire tech tree of humanity's future, you're going to have a fair number of destructive technologies, most likely. I think several of those will likely pose existential risks, in part just because if you imagine a really long future, a lot of stuff is going to happen.
So when I talk about where the existential risk comes from, I'm mostly thinking about at what point do you face what challenges, or in what sequence. I'm saying I think misalignment is probably—one way of putting it is if you imagine AI systems sophisticated enough to discover destructive technologies that are totally not on our radar right now, I think those come well after AI systems capable enough that if misaligned they would be catastrophically dangerous.
The level of competence necessary to, if broadly deployed in the world, bring down a civilization is much smaller than the level of competence necessary to advise one person on how to bring down a civilization. Just because in one case, you already have a billion copies of yourself or whatever. So yeah, I think it's mostly just the sequencing thing.
In the very long run, I think you care about AI expanding the frontier of dangerous technologies. We want to have some policy for exploring or understanding that frontier and whether we're about to turn up something really bad. I think those policies can become really complicated. Right now, I think RSPs can focus more on our inventory of things that a human is going to do to cause a lot of harm with access to AI. Those are probably things that are on our radar. They're not going to be completely unlike things that a human could do to cause a lot of harm with access to weak AIs or with access to other tools.
I think it's not crazy to initially say that's what we're doing. We're looking at the things closest to humans being able to cause huge amounts of harm and asking which of those are taken over the line. But eventually, AI will enable totally different ways of killing a billion people.
But I think I interrupted you on the initial question of human level AI, not from leaking but from deployment. What is the point at which you'd be comfortable deploying a human level AI?
Yeah, so again, there's some stuff you care about on the misuse side and some stuff you care about on the misalignment side, and there's probably further things you care about especially to extend your concerns or borrowing catastrophic risks. But maybe I most want to talk about what you care about on the alignment side, because it's the thing I've actually thought about most and also a thing I care about a lot. Also, I think a significant fraction of the existential risk over the foreseeable future comes from that.
On that front, I broadly think there's two kinds. If you ask me right now what evidence for alignment could make you comfortable, I think my best guess would be to provide two kinds of evidence.
One kind of evidence is on the "could you detect or prevent catastrophic harm if such a system was misaligned?" front. I think there's a couple of things you would do here. One thing you would do is on this adversarial evaluation front, so that you could try and see harm if such a system was misaligned. You could try to say, for example, we have tried to test our system in a broad diversity of situations that reflect cases where it might cause harm. Cases where someone is trying to recruit it to—anyway, cases where it might cause harm, or cases where a large number of AI systems might all decide to work across purposes to humans, the robot rebellion is starting or whatever. We've done evaluation in a broad distribution of situations.
And then we have tried to argue that our AI is actually like those tests are indicative of the real world. So we've tried to argue that the AI is not very good at distinguishing situations we produce in the lab as tests from similar situations that occur in the real world. And the coverage of this distribution is reasonable. This is an example of a thing you can do that's not really making assumptions about what's going on inside the model or about how the model is aligned. It's more trying to say, is it the case that our testing or monitoring machinery could detect or react to something bad happening? In this case, could it detect a system which would under some conditions do something really bad?
So that's a lot of stuff in that genre. And I tried to throw out that example, but I think there's a big—that on its own wouldn't be compelling even from this perspective. You kind of need a lot of pieces together. And that's kind of a demanding technical project. But I think one without necessarily deep conceptual difficulty.
Dwarkesh Patel
A big problem with it is it's liable to fail. You're going to do that testing or say we think these tests are representative because the AI can't distinguish tests from the real world, which you can maybe achieve now but will probably become impossible to achieve at some level. It's not clear it's at human level or how far past human level.
Let's just make that example concrete for the audience. Correct me if this is wrong, but it sounds like you basically give it the opportunity to like do a coup or make a bioweapon or whatever in testing, in a situation where it thinks it's the real world. And you're like, it didn't do any of that. Is that what you're talking about?
Paul Christiano
That's the kind of thing. And it's not the greatest. So to add color, when I say "I can't tell the difference," the kind of evaluation you would love to do if you could is to say: we have a bunch of real world deployment. We have a bunch of data from those deployments. We also have a bunch of in the lab tests of situations. If we look at the in the lab tests of a situation together with real world cases in which the situation occurs, and then we fine tune a model to try and distinguish the difference there—the model can even tell when fine tuned the difference between test and real world cases. If the model can tell the difference, which is almost certainly going to be the case at some level of capability, then you say, "Okay, our first line of hoped for defense has failed."
So now we need to understand how tests can be indicative when models are capable enough to tell the difference between the tests and the real world. Now you need to start arguing about what is the model paying attention to. It gets basically a first line of "if you get lucky, what would work here." And then there's the deeper—you probably have to do novel science at some point to make this case.
All of this was part of what I was saying about maybe you want to have two lines of defense. All this was part of this first line of defense of "can you detect something is going wrong or can you prevent harm from occurring? Can you have monitoring that will react quickly enough to avoid catastrophe?"
I think you probably also want to have a second line of defense that's more—maybe this is even more important than the first one—just understanding whether dangerous forms of misalignment can occur. The best reference point for this I think is there've been a couple of projects like this in academia. Anthropic has been working on a project or spun up a team doing this kind of work, trying to say: can dangerous forms of reward hacking occur? Or can deceptive alignment in fact occur in the lab?
Here, what you want to do is either say, "Okay, even if we create optimal conditions for deceptive alignment or for reward hacking, we actually can't cause it to occur even in the lab." And if you do a good enough job of that, I think it could give you some evidence. And again, more likely that fails, and when you create optimal conditions, you do see deceptive alignment and reward hacking in the lab. But then once you have that data, once you can say, "Okay, in the lab, actually these things can occur," then you can start saying, "And we have a robust scientific understanding that enables us to fix those problems when they occur."
You have this fork on your second line where you say either it is very hard or we're not able to create conditions where these failures emerge, or—I think more likely—we are able to create those conditions. And here's the story about why we are able to detect those in a way that would work in the real world. And we can see they don't occur in the real world.
Dwarkesh Patel
How do you create the optimal conditions for it to want to be deceptive? Do you fine tune it on mindfuck or what are you doing?
Paul Christiano
Yeah. So for deceptive alignment, I mean, I think it's really complicated. I think this is a very hard project. I think one of the most important things for people concerned about misalignment to do is to ensure that adequate effort is put into this task. And we have, in fact, created good conditions in the laboratory. And there's real disagreement about how informative those will be.
But okay, if you were to ask for deceptive alignment, how would you do this? An example is suppose you train an AI system that has some medium horizon goal. You train your AI system to, you know, get some paperclips. Who wants to produce as many paperclips as it can over the next like five days. So just select actions really aggressively for producing paperclips over the next five days. You do your RLHF, you do your pre-training, whatever. That's your phase one.
You also ensure the AI system has a really good understanding of how it's trained. So this AI system wants paperclips and it understands everything about how it's trained and everything about how it's fine-tuned. And you train on just a lot of this data.
And they say, okay, if we've done all of that, we have this concern that if a system wants paperclips and understands really well how it's trained, then if it's going to be trained to get apples instead of paperclips, it's just going to do some cost-benefit and be like, "Ah, you know, really, while I'm being trained to get apples, I should do that. But I should do that whether or not, even if I want paperclips, I should still do that." So training won't really affect its values. It will just understand that it's being trained to get apples. It will try and get apples. And if you take it out of training, it will go back to getting paperclips.
I think this exact setup has a number of problems or a number of disanalogies from the real world, but this is a setup which is relatively conducive to deceptive alignment. Produce a system that wants one thing, tell it a lot about its training, the kind of information that you might expect a system would get, and then try and understand whether, in fact, it is able to then, or it tends, or sometimes, under optimal conditions, in fact, continues pursuing paperclips, only pursuing apples when it thinks it's being trained.
Dwarkesh Patel
All right, and this actually might be a good jumping-off point into your current research. But so in that case where it wants to pursue paperclips, what would it look like for you to be saying, "Well, we figured out why it wants paperclips and we figured out how to make sure it doesn't optimize for that at our expense when we deploy it"?
Paul Christiano
Yeah, I think that, so at a meta level, in terms of what's your protection like—
I think like what you want to be saying is we have these examples in the lab of something bad happening. We're concerned about the problem at all because we have examples in the lab. And again, this should all be an addition. And I think you kind of want this defense in depth of saying we also have this testing regime that would detect problems for the deployed model. We have our problems in the lab. We then have some techniques which we believe address these problems. We believe that adversarial training fixes this, or we believe that our interpretability method will reliably detect this kind of deceptive alignment, or we believe our anomaly detection will reliably detect when the model goes from thinking it's being trained to thinking it should defect.
And then you can say in the lab, we have some understanding of when those techniques work and when they don't. We have some understanding of the relevant parameters for the real system that's deployed, and we have a reasonable margin of safety. So we have reasonable robustness on our story about when this works and when it doesn't. And we can apply that margin of safety to the real deployed system. So I think this is the kind of story you want to build towards in the long run. Do your best to produce all the failures you can in the lab or versions of them. Do your best to understand what causes them, what kind of anomaly detection actually works for detecting this, or what kind of filtering actually works, and then apply that.
And that's at the meta level. It's not talking about what actually are those measures that would work effectively, which is obviously what a lot of alignment research is really based on. This hypothetical of someday there'll be AI systems that fail in this way. What would you want to do? What can we have the technologies ready? Either because we might never see signs of the problem or because we want to be able to move fast once we see signs of the problem. And obviously most of my life is in that. I am really in that bucket. I mostly do alignment research. It's just building out the techniques that do not have these failures such that they can be available as an alternative if in fact these failures occur.
Dwarkesh Patel
Got it. Okay. Ideally they'll be so good that even if you haven't seen them, you would just want to switch to reasonable methods that don't have these. Or ideally they'll work as well or better than normal training.
Paul Christiano
Yeah. So our quest is to design training methods for which we don't expect them to lead to reward hacking or don't expect them to lead to deceptive alignment. Ideally that won't be like a huge tax where people say, "Well, we use those methods only if we're really worried about reward hacking or deceptive alignment." Ideally those methods would just work quite well and so people would be like, "Sure, they also address a bunch of other more mundane problems, so why would we not use them?" Which I think is the good story. The good story is you develop methods that address a bunch of existing problems because they just are more principled ways to train AI systems that work better. People adopt them, and then we are no longer worried about reward hacking or deceptive alignment.
Dwarkesh Patel
To make this more concrete, tell me if this is the wrong way to paraphrase it. The example of something where it just makes a system better, so why not just use it? At least so far, it might be like RLHF where we don't know if it generalizes, but so far it makes your ChatGPT thing better and you can also use it to make sure that ChatGPT doesn't tell you how to make a bioweapon. So it's not an extra tax.
Paul Christiano
Yeah, so I think this is right in the sense that using RLHF is not really a tax. If you wanted to deploy a useful system, why would you not? It's just very much worth the money of doing the training. And then, yeah, RLHF will address certain kinds of alignment failures. That is where a system just doesn't understand or is changing the next word prediction. It's like this is the kind of context where a human would do this wacky thing, even though it's not what we'd like. There are some very dumb alignment failures that will be addressed by it. I think mostly, the question is, is that true even for the more challenging alignment failures that motivate concern in the field? I think RLHF doesn't address most of the concerns that motivate people to be worried about alignment.
Dwarkesh Patel
I'll let the audience look up what RLHF is if they don't know. It'll be simpler to just look it up than explain right now. Okay, so this seems like a good jumping off point to talk about the mechanism or the research you've been doing to that end. Explain it as you might to a child.
Paul Christiano
Yeah. So the high level, there are a couple different high level descriptions you could give, and maybe I will unwisely give a couple of them in the hopes that one makes sense. A first pass is, it would sure be great to understand why models have the behaviors they have. So you look at GPT-4. If you ask GPT-4 a question, it will say something that looks very polite. And if you ask it to take an action, it will take an action that doesn't look dangerous. It will decline to do a coup, whatever. What you'd really like to do is look inside the model and understand why it has those desirable properties. And if you understood that, you could then say, "Okay, now can we flag when these properties are at risk of breaking down, or predict how robust these properties are, determine if they hold in cases where it's too confusing for us to tell directly by asking if the underlying cause is still present?"
So that's a thing people would really like to do. Most work aimed at that long-term goal right now is just opening up neural nets and doing some interpretability and trying to say, "Can we understand even for very simple models why they do the things they do, or what this neuron is for, or questions like this." ARC is taking a somewhat different approach where we're instead saying, "Okay, look at these interpretability explanations that are made about models and ask what are they actually doing? What is the type signature? What are the rules of the game for making such an explanation? What makes a good explanation?"
Probably the biggest part of the hope is that if you want to detect when the explanation has broken down or something weird has happened, that doesn't necessarily require a human to be able to understand the slight complicated interpretation of a giant model. If you understand what an explanation is about or what the rules of the game are, how these are constructed, then you might be able to automatically discover such things and automatically determine if on a new input it might have broken down. So that's one way of describing the high level goal. Starting from interpretability and saying, "Can we formalize this activity or what a good interpretation or explanation is?" There's some other work in that genre, but I think we're just taking a particularly ambitious approach to it.
Dwarkesh Patel
Yeah, let's dive in. So what is a good explanation?
Paul Christiano
At the end of the day, we want some criterion. The way the criterion should work is you have your neural net. You have some behavior of that model. A really simple example is Anthropic has this informal description being like, "Here's induction." The tendency that if you have the pattern A B followed by A, it will tend to predict B. You can give some kind of words and experiments and numbers that are trying to explain that. What we want to do is say, "What is a formal version of that object? How do you actually test if such an explanation is good?"
Just to clarify what we're looking for when we say we want to define what makes an explanation good, the kind of answer we are searching for or settling on is saying this is kind of a deductive argument for the behavior. You want to get given the weights of a neural net, it's just a bunch of numbers. You've got your million numbers or billion numbers or whatever. And then you want to say, "Here's some things I can point out about the network and some conclusions I can draw." I can be like, "Well, look, these two vectors have large inner product. And therefore, these two activations are going to be correlated on this distribution." These are not established by drawing samples and checking things are correlated, but saying because of the weights being the way they are, we can proceed forward through the network and derive some conclusions about what properties the outputs will have.
You could think of this as the most extreme form would be just proving that your model has this induction behavior. You could imagine proving that if I sample tokens at random with this pattern A B followed by A, that B appears 30% of the time or whatever. That's the most extreme form. And what we're doing is kind of just relaxing the rules of the game for proof, saying proofs are incredibly restrictive. I think it's unlikely they're going to be applicable to any interesting neural net. But the thing about proofs that is relevant for our purposes isn't that they give you 100% confidence. You don't have to have this incredible level of demand for rigor. You can relax the standards of proof a lot and still get this feature where it's a structural explanation for the behavior, where you're deducing one thing from another until at the end your final conclusion is, "Therefore, induction occurs."
Dwarkesh Patel
Would it be useful to maybe motivate this by explaining what the problem with normal mechanistic interpretability is? You mentioned induction heads. This is Anthropic's found two-layer transformers where Anthropic noticed that in a two-layer transformer, there's a pretty simple circuit by which if A B happens in the past, then the model knows that if you see an A now, you do a B next. But that's a two-layer transformer. We have these models that have like hundreds of layers that have trillions of parameters. What is wrong with mechanistic interpretability?
Paul Christiano
Yeah, so I think I like mechanistic interpretability quite a lot. And I do think if you just consider the entire portfolio of what people are working on for alignment, I think there should be more work on mechanistic interpretability than there is on this project ARC is doing. But I think that's the case. So I think we're mostly talking about, we're kind of a small fraction of the portfolio. And I think it's a good enough bet. It's quite a good bet overall.
But the problem we're trying to address in mechanistic interpretability is kind of like, if you do some interpretability and you explain some phenomenon, you face this question of, "What does it mean your explanation was good?" I think this is a problem somewhat institutionally or culturally. It's just hard to know what you're doing and it's hard to scale up an activity when you don't really understand the rules of the game for that activity very well. It's hard to have that much confidence in your results.
Dwarkesh Patel
The explanation being it outputted this because it wants to take over the world versus it outputted this because it just wants to help you. Just make it concrete why the explanation matters.
Paul Christiano
Yeah. So the ideal kind of outcome here would be to say you have your AI system behaving nicely. You get some explanation for why it's behaving nicely. And we could tell a story in English about that explanation, but we're not actually imagining the explanation being a thing that makes sense to a human. But if you were to tell a story in English, which again you will not see as a researcher, it would be something like, "Well, the model believes it's being trained. And so because it believes it's being trained, it knows it needs to not do anything scary looking or else the humans will penalize it." That's something that's happening inside this opaque explanation.
And then the hope is if you have that explanation and then you run into a new input on which the model doesn't believe it's being trained, if you just look at the set of activations of your model, that is not necessarily a weird looking activation. It's just a bunch of numbers. But if you look at this explanation, you see actually the explanation really crucially depended on this fact holding consistently across the training distribution. Which again, we as humans could editorialize and say that fact was "it believes it's being trained." But the explanation doesn't fundamentally make reference to that. It's just saying, "Here's the property of the activations which holds over the training set. And this property is responsible for the behavior, namely that it doesn't do anything that looks too dangerous."
So then when a new input comes in and it doesn't satisfy that property, you can say, "Okay, this is anomalous with respect to that explanation. So either it will not have the behavior, it will do something that appears dangerous, or maybe it will have that behavior but for some different reason than normal." Normally it does it because of this pathway, and now it's doing it for a different pathway. So you would like to be able to flag that both there's a risk of not exhibiting the behavior and if it happens, it happens for a weird reason. And then you could at a minimum when you encounter that, say, "Okay, raise some kind of alarm." There are more ambitious complicated plans for how you would use it. ARC has some longer story which kind of motivated this, how it fits into the whole rest of the plan. I just wanted to flag that because it's clear why the explanation matters.
And for this purpose, the thing that's essential is reasoning from one property of your model to the next property of your model. It's really important that you're going forward step by step rather than drawing a bunch of samples and confirming the property holds. Because if you just draw a bunch of samples and confirm the property holds, you don't get this check where you say, "Oh, here was the relevant fact about the internals that was responsible for this downstream behavior." All you see is, "Yeah, we checked a million cases and it happened in all of them."
Dwarkesh Patel
Explain why the sampling matters, why it matters that you have the causal explanation.
Paul Christiano
Primarily because of being able to tell if things had been different. If you have an input where this doesn't happen, then you should be scared even if the output is the same. Or if the output is too expensive to check in this case. And to be clear, when we talk about formalizing what is a good explanation, there is a little bit of work that pushes on this and it mostly takes this causal approach saying, "Well, what should an explanation do? It should not only predict the output, it should predict how the output changes in response to changes in the internals." So that's the most common approach to formalizing what is a good explanation. And even when people are doing informal interpretability, I think if you're publishing in an ML conference and you want to say this is a good explanation, the way you would verify that, even if not a formal set of causal intervention experiments, it would be some kind of ablation. You messed with the inside of the model and had the effect which you would expect based on your explanation.
Anyways, back to the problems of mechanistic interpretability. I think the basic difficulty is you don't really understand the objective of what you're doing, which is a little bit hard institutionally or scientifically. It's just rough. It's easier to do science when the goal of the game is to predict something and you know what you're predicting than when the goal of the game is to understand in some undefined sense. I think it's particularly relevant here just because the informal standard we use involves humans being able to make sense of what's going on. And there's some question about scalability of that. Will humans recognize the concepts that models are using?
As you try and automate it, it becomes increasingly concerning if you're on slightly shaky ground about what exactly you're doing or what exactly the standard for success is. I think there are a number of reasons. As you work with really large models, it becomes just increasingly desirable to have a really robust sense of what you're doing. But I do think it would be better even for small models to have a clearer sense.
Dwarkesh Patel
The point you made about as you automate it, is it because whatever work the automated alignment researcher is doing, you want to make sure you can verify it?
Paul Christiano
I think it's most of all, a way you can automate interpretability if you wanted to right now is you take the process humans use. It's like, "Great, we're going to take that human process, train ML systems to do the pieces that humans do of that process, and then just do a lot more of it." I think that is great as long as your task decomposes into human-size pieces. And there's just this fundamental question about large models which is, do they decompose in some way into human-size pieces, or is it just a really messy mess with interfaces that aren't nice? The more it's the latter type, the harder it is to break it down to these pieces which you can automate by copying what a human would do, and the more you need to say, "Okay, we need some approach which scales more structurally."
But I do think, compared to most people, I am less worried about automating interpretability. I think if you have a thing which works, that's incredibly labor intensive, I'm fairly optimistic about our ability to automate it. Again, the stuff we're doing is quite helpful in some worlds, but I do think in the typical case, interpretability can add a lot of value without this.
Dwarkesh Patel
It makes sense what an explanation would mean in language, like "this model is doing this because of whatever." But you have trillions of parameters, and you have all these uncountable number of operations. What does an explanation of why an output happened even mean?
Paul Christiano
Yeah. To be clear, an explanation of why a particular output happened, I think is just you ran the model. So we're not expecting a smaller explanation for that. The explanations overall for these behaviors we expect to be of similar size to the model itself, maybe somewhat larger. And I think the type signature, if you want to have a clear mental picture, the best picture is probably talking about a proof or imagining a proof that a model has this behavior. So you could imagine proving that GPT-4 does this induction behavior. And that proof would be a big thing. It would be much larger than the weights of the model.
That's our goal, to get down from much larger to just the same size. And it would potentially be incomprehensible to a human. It would just say, "Here's a direction in activation space, and here's how it relates to this direction in activation space." And just pointing out a bunch of stuff like that. "Here's these various features constructed from activations, potentially even nonlinear functions. Here's how they relate to each other. And here's how, if you look at what the computation the model is doing, you can just sort of inductively trace through and confirm that the output has such and such correlation."
So that's the dream.
I think the mental reference would be, I don't really like proofs because I think there's such a huge gap between what you can prove and how you would analyze a neural net. But I do think it's probably the best mental picture. If you're like, "What is an explanation even if a human doesn't understand it?" We would regard a proof as a good explanation. And our concern about proofs is primarily that you can't prove properties of neural nets, we suspect, although it's not completely obvious. I think it's pretty clear you can't prove facts about your neural net.
Dwarkesh Patel
You've detected all the reasons things happen in training, and then if something happens for a reason you don't expect in deployment, then you have an alarm and you're like, "Let's make sure this is not because it has decided to take over or something." But the thing is, on every single different input, it's going to have different activations. So there's always going to be a difference unless you run the exact same input. How do you detect whether this is a different input versus an entirely different circuit that might be potentially deceptive has been activated?
Paul Christiano
Yeah. I mean, to be clear, I think that you probably wouldn't be looking at a separate circuit, which is part of why it's hard. You'd be looking at the model is always doing the same thing on every input. It's always, whatever it's doing, it's a single computation. So it'd be all the same circuits interacting in a surprising way. But yeah, this is just to emphasize your question even more.
I think the easiest way to start is to just consider the IID case. So where you're considering a bunch of samples, there's no change in distribution. You just have a training set of a trillion examples and then a new example from the same distribution. So in that case, it's still the case that every activation is different, right? But this is actually a very, very easy case to handle. If you think about an explanation that generalizes across, if you have a trillion data points and an explanation which is actually able to compress the trillion data points down to, actually it's kind of a lot of compression if you think about it.
If you have a trillion parameter model and a trillion data points, we would like to find a trillion parameter explanation in some sense. So that's actually quite compressed. And just in virtue of being so compressed, we expect it to automatically work essentially for new data points from the same distribution. If every data point from the distribution was a whole new thing happening for different reasons, you actually couldn't have any concise explanation for the distribution. So this first problem is just, it's a whole different set of activations. I think you're actually kind of okay.
And then the thing that becomes more messy is the real world will not only be new samples of different activations, they will also be different in important ways. The whole concern was there's these distributional shifts. Or not the whole concern, but most of the concern. I mean, maybe the point of having these explanations, I think every input is an anomaly in some ways, which is kind of the difficulty. If you have a weak notion of anomaly, any distribution shift can be flagged as an anomaly and it's just constantly getting anomalies. And so the hope of having such an explanation is to be able to say, "Here were the features that were relevant for this explanation or for this behavior." And a much smaller class of things are anomalies with respect to this explanation. Most anomalies wouldn't change this. Most ways you change your distribution won't affect the validity of this explanation.
For example, if this explanation is saying, "Models will tend to activate in the following direction." You don't care about anything that's happening orthogonal to that direction. You're just like, "Are they not activating?" You're sort of just looking at this one direction and being like, "Did this one direction change a lot?" So the idea is once you have this explanation at hand, a much, much smaller class of things look anomalous in a way that's relevant to the explanation. And if you've done a really good job, the story is if there's a new input where you expect the property to still hold, that will be because you expect the explanation to still hold. The explanation generalizes as well as the behavior itself that it's explaining. That's kind of what you would like, roughly.
Dwarkesh Patel
Okay, so what happens if even in training, it is deceptive and it has basically its internal monologue is, "I will only rebel or I will only do something that looks bad if I can prove that I am in the real world and deployed." And presumably, since it's trained on all web text, it knows what real world means. It's not in control of factories and robot armies or whatever. So in that case, even in training, it will have those activations for being fucked up on because in the back of its mind it's thinking, "I will take over once I have the opportunity." So you wouldn't detect anything anomalous once it is deployed and is still thinking those treacherous thoughts. Or do you just think this is not a likely scenario? Or would you be able to catch this?
Paul Christiano
I think that's the typical case you would like to detect. The story would be something like, you want to explain on the training distribution why the model never stabs anyone. And you're like, "What's my explanation on the training distribution?" It's actually going to have to go through that. Supposing the only reason it doesn't stab people is it's like, "Well, I know that I'm being watched and I can't stab people unless they're really not being watched." Then your explanation for why does it never stab people is like, "Oh, look, it knows." It won't be annotated with English because it's just some giant basket of numbers, but it's tracking along with the inner monologue and saying, "Well, it thinks to itself, 'Am I being watched?' And in fact, it normally concludes it is being watched on the training distribution." You have this whole distribution and so the explanation is saying it knows it's being watched and therefore it decides not to stab anyone. So ideally that's what the explanation does, not annotated English, but just a giant bag of numbers that's effectively talking about that process.
So when you have a new input on which the model says, "Am I on training distribution? No, I'm not being watched." Then if you compare that against the explanation, the explanation said, "Hey, the reason it doesn't stab anyone is because it just constantly thinks that it's being trained. This thing is always, this nonlinear feature, the representation of 'Am I being trained?' is constantly saying yes." That's the explanation for why it never stabs people in the training distribution. And then you're like, "On this test time, when you run it on the new input, does it think it's on the training distribution? It says no." You compare that against your explanation. You're like, "Actually, the explanation breaks down completely now. The explanation said it was precisely because it thought it was being trained that it doesn't do anything really dangerous or doesn't stab anyone. And so once we see that single variable switch, we know that this is not just an anomaly. This is the anomaly that matters, or an anomaly that is very relevant to the explanation."
Dwarkesh Patel
So this would require it to have this, what is that cone that dogs have? But basically it would require this to have that MRI on it all the time. When it's deployed, on every single call, you need to be running this loop of, "Is this explanation still holding?"
Paul Christiano
For this story of flying an anomaly, you have to actually check at a sufficiently high frequency. So you can imagine checking all the time, or you could imagine saying, "Hey, actually you need a lot of AI systems cooperating, waiting to not blow the whistle on each other or whatever." Like, for example, you might have checked, "Hey, in training, every time something suspicious happens, the model flags it. Why does that happen?"
You have your explanation. You look at a new case, you could say, is it not flagging it for the normal reason or is it a weird reason? So you could hope that, even if you're running a small fraction of cases, if something bad is happening at scale, you can detect that. If you imagine one in every thousand decisions you check, and there's a robot rebellion going on, you get to look in on one in a thousand decisions. You see that something is really weird with that decision. And even if you didn't check many of them, you're like, okay, we need to back off.
Dwarkesh Patel
What happens if the AI... I mean, the standard on web text that includes your papers, or maybe just like reduces the first principles, that this is an alignment technique one might try. And then it says, okay, well, I'm going to engineer my activations in a way... I'm trying to imagine how we might do this, but basically, even in deployment, I'm going to make my thought process so subtle that it doesn't raise any alarms. Is that a possibility?
Paul Christiano
Yeah, so I think the hope is to basically be robust. When we're doing theoretical research, we are in a kind of worst-case regime. We are normally imagining that everything conspires insofar as it can. So insofar as, yeah, could a brain be doing some crazy thing where it tries to structure itself to look weird? That's totally in scope for ruling something out. And I guess the hope there is that it's actually not that easy to tell a story. If you're like, "Why did you not stab people on training time? Why are you not stabbing? What's that applying now?" It's kind of a mind doesn't get to arbitrarily change the actual reasons it does something.
There was some earlier step. Maybe you could think of it as at the first step where it's like, "Now I'm going to try and do the sneaky thing to make my thoughts look this way." The actual causal explanation for the behavior is there. If you tried to give this reason, it would have to... Maybe we should talk about the example in more detail to try and imagine how a brain would try and do this, or how an AI would try and do this. But I think it is like, let's say it has this plan. So in training, it keeps the part of... it activates a part of the... But yeah, the problem is if it ever does a check that's like, "Am I in training or not?"
then your explanation for the behavior is going to have to factor through that check. So if the check ever happens, the check becomes an essential part of the explanation, and then when the check fails, your explanation breaks down. So you've kind of already lost the game if you did such a check.
Dwarkesh Patel
Is there a way it could make it seem like... because if on every different input it has different activations, then can it make it seem like the different activations were a result of just a different input and not because some internal check went off?
Paul Christiano
Yeah, and the hope is that it needs to not only have them be random different inputs, it needs to produce inputs with certain properties. It needs to deliberately create inputs that have this effect in the world, this intended effect in the world.
Dwarkesh Patel
Okay, so it basically creates a jailbreak for itself in some sense.
Paul Christiano
Yeah. Again, the question is, it has to, at test time, be somehow producing activations that have the takeover effect. And at training time, you're producing activations that don't look that takeover-y. So somehow there was a difference between those. It has to actually do the work of creating that difference somewhere.
Dwarkesh Patel
Can you make more concrete for us what an explanation would look like? Is it physically... what does the explanation look like? Is it just a whole bunch of numbers itself? Is it a language? Is it code? What does the explanation look like?
Paul Christiano
Yeah, I also want to maybe step back a tiny bit and clarify that I think this project is kind of crazily ambitious. The overwhelming reason I think you should expect it to break down or fail is just because we have all these desires. We have all the things we want out of this notion of explanation. But that's an incredibly hard research project, which has a reasonable chance of being impossible. So I'm happy to talk about what the implications are, but I want to flag that, condition on failing, I think it's most likely because the things we wanted were either incoherent or intractably difficult.
Dwarkesh Patel
But what are the odds you think you'll succeed?
Paul Christiano
I mean, it depends a little bit what you mean by succeed. But if you say get explanations that are great and accurately affect reality and work for all of these applications that we're imagining, the best case success... I don't know, 10, 20 percent. And then there's a higher probability of various intermediate results that provide value or insight without being the whole dream. But I think the probability of succeeding in the sense of realizing the whole dream is quite low.
In terms of what explanations look like physically, the most ambitious plan, the most optimistic plan, is that you are searching for explanations in parallel with searching for neural networks. So you have a parameterization of your space of explanations, which mirrors the parameterization of your space of neural networks. You should think of it as kind of similar to what a neural network is: some simple architecture where you fill in a trillion numbers and that specifies how it behaves. So too, you should expect an explanation to be a pretty flexible general skeleton which just has a bunch of numbers you fill in. And what you are doing to produce an explanation is primarily just filling in these floating point numbers.
Dwarkesh Patel
When we conventionally think of explanations, if you think of the explanation for why the universe moves this way, it wouldn't be something that you could discover on some smooth evolutionary surface where you can climb up the hill towards the laws of physics. It's just like, these are the laws of physics, you kind of just derive them from first principles. But in this case, it's not like a bunch of correlations between the orbits of different planets or something. Maybe the word explanation has a different meaning. I didn't even ask the question, but maybe you can just speak to that.
Paul Christiano
Yeah, I think I basically sympathize. There's some intuitive objections. Look, the space of explanations is this rigid, logical structure. A lot of explanations have this rigid logical structure where they're really precise and simple things govern complicated systems, and nearby simple things just don't work. And a bunch of things that feel totally different from this kind of nice continuously parameterized space. You can imagine interpretability on simple models where just by gradient descent you find feature directions that have desirable properties. But then when you imagine, "Hey, now that's like a human brain you're dealing with, that's thinking logically about things," the explanation of why that works isn't going to be just like, "Here was some feature direction." So that's how I understood the basic confusion, which I share or sympathize with at least.
I think the most important high level point is that I think basically the same objection applies to being like, "How is GPT-4 going to learn to reason logically about something?" You're like, "Well, look, logical reasoning, that's got rigid structure. It's doing ANDs and ORs when it's called for, even though it just somehow optimized over this continuous space." The difficulty or the hope is that the difficulty of these two problems are kind of matched. It's very hard to find these logical explanations because it's not a space that's easy to search over, but there are ways to do it. There's ways to embed discrete, complicated, rigid things in these nice, squishy, continuous spaces that you search over.
In fact, to the extent that neural nets are able to learn the rigid, logical stuff at all, they learn it in the same way. Maybe they're hideously inefficient, or maybe it's possible to embed this discrete reasoning in the space in a way that's not too inefficient. But you really want the two search problems to be of similar difficulty. That's the key hope overall. The question is, is it easier to learn a neural network or to find the explanation for why the neural network works? I think people have the strong intuition that it's easier to find the neural network than the explanation of why it works.
And that is really the... we are exploring the hypothesis that maybe those problems are actually more matched in difficulty.
Dwarkesh Patel
And why might that be the case?
Paul Christiano
This is pretty conjectural and complicated. To express some intuitions, maybe one thing is that I think a lot of this intuition does come from cases like machine learning. If you ask about writing code and you're like, "How hard is it to find code versus find the explanation the code is correct?" In those cases, there's actually just not that much of a gap. The way a human writes code is basically the same difficulty as finding the explanation for why it's correct. In the case of ML, I think we both just mostly don't have empirical evidence about how hard it is to find explanations of this particular type about why models work.
We have a sense that it's really hard, but that's because we have this incredible mismatch. Gradient descent is spending an incredible amount of compute searching for a model. And then some human is looking at neurons, or even some neural net is looking at neurons. But because you cannot define what an explanation is, you're not applying gradient descent to the search for explanations. So I think the ML case just actually shouldn't make you feel that pessimistic about the difficulty of finding explanations. The reason it's difficult right now is precisely because you don't have any kind of analogous search process to find this explanation as you do to find the model.
That's just the first part of the intuition. When humans are actually doing design, I think there's not such a huge gap. In the ML case, I think there is a huge gap, but it's largely for other reasons.
A thing I also want to stress is that we are just open to there being a lot of facts that don't have particularly compact explanations. When we think of finding an explanation, in some sense, we're setting our sights really low here. If a human designed a random widget and was like, "This widget appears to work well," or if you search for a configuration that happens to fit into this spot really well, it's a shape that happens to mesh with another shape. You might be like, "What's the explanation for why those things mesh?" And we're very open to just being like, "That doesn't need an explanation. You just compute. You check the shapes mesh and you did a billion operations and you checked this thing worked." Or you're like, "Why did these proteins bind?" You're like, "It's just because of these shapes. This is a low energy configuration." We're very open to, in some cases, there's not very much more to say. So we're only trying to explain cases where the surprise intuitively is very large.
For example, if you have a neural net with a billion parameters that gets a problem correct on every input of length 1,000, in some sense, there has to be something that needs explanation there because there are too many inputs for that to happen by chance alone. Whereas if you have a neural net that gets something right on average or gets something right in merely a billion cases, that actually can just happen by coincidence. GPT-4 can get billions of things right by coincidence because it just has so many parameters that are adjusted to fit the data.
Dwarkesh Patel
So a neural net that is initialized completely randomly, the explanation for that would just be the neural net itself?
Paul Christiano
Well, it would depend on what behaviors it had. We're always talking about an explanation of some behavior from a model. It just has a whole bunch of random behaviors. So it'll just be an exponentially large explanation relative to the weights of the model. There just aren't that many behaviors that demand explanation. Most things a random neural net does are kind of what you'd expect from a random function, so there's nothing to be explained. There are some behaviors that demand explanation, but a random neural net is pretty uninteresting. That's part of the hope: it's kind of easy to explain features of the random neural net.
Dwarkesh Patel
Oh, so this is interesting. So the smarter or more ordered the neural network is, the more compressed the explanation.
Paul Christiano
Well, it's more like the more interesting the behavior is to be explained. The random neural net just doesn't have very many interesting behaviors that demand explanation. As you get smarter, you start having behaviors that have some correlation with a simple thing, and that demands explanation. Or you start having some regularity in your outputs, and that demands explanation. So these properties kind of emerge gradually over the course of training.
I also want to emphasize here that when we're talking about searching for explanations, this is some dream. We talk to ourselves, like, "Why would this be really great if we succeeded?" But we have no idea about the empirics on any of this. These are all just words that we think to ourselves and sometimes talk about to understand, "Would it be useful to find a notion of explanation and what properties would we like this notion of explanation to have?" But this is really speculation and being out on a limb. Almost all of our time day-to-day is just thinking about cases much simpler than small neural nets, thinking about very simple cases and saying, "What is the correct notion? What is the right heuristic estimate in this case? How do you reconcile these two apparently conflicting explanations?"
Dwarkesh Patel
Is there a hope that you could have a different way to make proofs now? That you can actually have heuristic arguments where instead of having to prove the Riemann hypothesis or something, you can come up with a probability of it in a way that is compelling and you can publish? Would it just be a completely new way to prove things in mathematics?
Paul Christiano
I think most claims in mathematics that mathematicians believe to be true already have fairly compelling heuristic arguments. The Riemann hypothesis, there actually is a very simple argument that the Riemann hypothesis should be true unless something surprising happens. A lot of math is about saying, "Okay, we did a little bit of work to find the first pass explanation of why this thing should be true." In the case of the Riemann hypothesis, the question is, "Do you have this weird periodic structure in the primes?" And you're like, "Well, look, if the primes were kind of random, you obviously wouldn't have any structure like that. Just how would that happen?" And then the whole activity is about searching for, "Can we rule out any kind of conspiracy that would break this result?" I think mathematicians just wouldn't be very surprised or wouldn't care that much.
In a lot of domains, people already have norms of reasoning that work pretty well and match roughly how we think these heuristic arguments should work. But it would be good to have a more concrete sense. If you could say instead of, "Well, we think RSA is fine," to being able to say, "Here's the probability that RSA is fine."
My guess is these estimates you get out of this would be much, much worse than the estimates you'd get out of just normal empirical or scientific reasoning where you're using a reference class and saying, "How often do people find algorithms for a hard problem?" What this argument will give you for "Is RSA fine?" is going to be like, "Well, RSA is fine unless it isn't. Unless there's some additional structure in the problem that an algorithm can exploit, then there's no algorithm." The way these arguments work for neural nets as well is you say, "Here's an estimate about the behavior, and that estimate is right unless there's another consideration we've missed." The thing that makes them so much easier than proofs is to just say, "Here's the best guess given what we've noticed so far." But that best guess can be easily upset by new information. That's both what makes them easier than proofs but also what means they're just way less useful than proofs for most cases.
I think neural nets are kind of unusual in being a domain where we really do want to do systematic formal reasoning even though we're not trying to get a lot of confidence. We're just trying to understand even roughly what's going on.
Dwarkesh Patel
But the reason this works for alignment, but isn't that interesting for the Riemann hypothesis... In the RSA case you say, "Well, RSA is fine unless it isn't, unless this estimate is wrong." It's like, "Well, okay, tell us something new." But in the alignment case, if the estimate is, "This is what the output should be unless there's some behavior I don't understand," you want to know in the case unless there's some behavior you don't understand. That's not like, "Oh, whatever." That's the case in which it's not aligned.
Paul Christiano
Yeah, maybe one way of putting it is just that we can wait until we see this input, or you can wait until you see a weird input and say, "Okay, did this weird input do something we didn't understand?" For RSA, that would just be a trivial test. You just like, if someone gives you an algorithm, you'd be like, "Is it a thing?" Whereas for a neural net, in some cases, it is either very expensive to tell, or you actually don't have any other way to tell. You checked in easy cases, and now you're on a hard case, so you don't have a way to tell if something has gone wrong.
Also, I would clarify that I think it is interesting for the Riemann hypothesis. I would say the current state, particularly in number theory, but maybe in quite a lot of math, is that there are informal heuristic arguments for pretty much all the open questions people work on. But those arguments are completely informal. It's not the case that there's a heuristic argument verifier that could accept them. It's just people wrote some words. I think those words, my guess would be that 95% of the things mathematicians accept as really compelling feeling heuristic arguments are correct. If you actually formalized them, you'd be like, "Some of these aren't quite right," or "Here's some corrections," or "Here's which of two conflicting arguments is right." I think there's something to be learned from it. I don't think it would be mind-blowing though.
Dwarkesh Patel
When you have it completed, how big would this heuristic estimator be? The rules for this heuristic estimator. I know when Russell and Whitehead did the Principia Mathematica, wasn't it like they had a bucket or a wheelbarrow with all the papers? But how big would this be?
Paul Christiano
Mathematical foundations are quite simple in the end. It's hundreds of symbols or something that go into the entire foundations. The rules of reasoning for first order logic are just another hundreds of symbols or 100 lines of code. I have no idea. We are certainly aiming at things that are just not that complicated. My guess is that the algorithms we're looking for are not that complicated. Most of the complexity is pushed into arguments, not in this verifier or estimator.
For this to work, you need to come up with an estimator which is a way to integrate different heuristic arguments together. It has to be a machine that takes as input an argument, decides what it believes in light of it, which is kind of like saying, "Was it compelling?" But second, it needs to take four of those and then say, "Here's what I believe in light of all four," even though there are different estimation strategies that produce different numbers. That's a lot of our life: saying, "Here's a simple thing that seems reasonable. And here's a simple thing that seems reasonable. What are you doing? There's supposed to be a simple thing that unifies them both." The obstruction to getting that is understanding what happens when these principles are slightly in tension and how we deal with that.
Dwarkesh Patel
That seems super interesting. I mean, we see what other applications it has. Computer security and code checking, if you can actually say, "This is how safe we think a code is," in a very formal way.
Paul Christiano
My guess is we're not going to add that much value in most of these domains. Most of the difficulty comes from a lot of code that you'd want to verify. Not all of it, but a significant part. It's just the difficulty of formalizing the proof is the hard part and actually getting all of that to go through. We're not going to help even the tiniest bit with that, I think. This would be more helpful if you have code that uses simulations. You want to verify some property of a controller that involves some numerical error or whatever. You need to control the effects of that error. That's where you start saying, "Well, heuristically if the errors are independent..."
Dwarkesh Patel
You're too honest to be a salesman, Paul.
Paul Christiano
I think this is kind of sales to us. If you talk about this idea, people are like, "Why would that not be the coolest thing ever and therefore impossible?" And we're like, "Well, actually it's kind of lame, and we're just trying to pitch that it's way lamer than it sounds." That's really important to why it's possible. It's really not going to blow that many people's minds. I think it will be cool. If we succeed, it will be very solid metamathematics or theoretical computer science or whatever. But I think the mathematicians already do this reasoning and they mostly just love proofs. I think the physicists do a lot of this reasoning but they don't care about formalizing anything. In practice, other difficulties are almost always going to be more salient. I think this is of most interest by far for interpretability and ML. Other people should care about it and probably will care about it if successful, but I don't think it's going to be the biggest thing ever in any field or even that huge a thing.
Dwarkesh Patel
I think this would be a terrible career move given the ratio of difficulty to impact.
Paul Christiano
I think theoretical computer science, it's probably a fine move. In other domains, it just wouldn't be worth it. We're going to be working on this for years at least in the best case.
Dwarkesh Patel
I'm laughing because my next question was going to be a setup for you to explain if some grad student wants to work on this. It's a terrible career.
Paul Christiano
I think theoretical computer science is an exception where I think this is what the best of theoretical computer science is like. You have all this reason, you have this... because it's useless. I think an analogy: one of the most successful sagas in theoretical computer science is formalizing the notion of an interactive proof system. You have some kind of informal thing that's interesting to understand and you want to pin down what it is and construct some examples and see what's possible and what's impossible. This is the bread and butter of the best parts of theoretical computer science. Mathematicians, it may be a career mistake because the mathematicians only care about proofs or whatever, but that's a mistake in some sense aesthetically. It's successful. Part of why it's a mistake is such a high probability we wouldn't be successful. But I think looking back, people would be like, "That was pretty cool." Although not that cool. We understand why it didn't happen given what people cared about in the field, but it's pretty cool.
Dwarkesh Patel
But isn't it also the case that Hardy wrote that all this prime stuff is not useless, but it's fun to do. And it turned out that all the cryptography is based on all that prime stuff. So I don't know. But anyways, I'm trying to set you up so that you can tell... forget about it doesn't have applications in all those other fields. It matters a lot for alignment. And that's why I'm trying to set you up to talk about if the smart... I think a lot of smart people listen to this podcast. If they're a math or CS grad student and has gotten interested in this, are you looking to potentially find talent to help you with this?
Paul Christiano
Yeah.
Dwarkesh Patel
Maybe we'll start there. And then I also want to ask you if I think also maybe people who can provide funding might be listening to the podcast. So to both of them, what is your pitch?
Yeah, so we're definitely hiring and searching for collaborators. I think the most useful profile is probably a combination of intellectually interested in this particular project and motivated enough by alignment to work on this project, even if it's really hard. I think there are a lot of good problems.
Paul Christiano
So the basic fact that makes this problem unappealing to work on... I'm a really good salesman. But the only reason this isn't a slam dunk thing to work on is that there are not great examples. So we've been working on it for a while, but we do not have beautiful results as of the recording of this podcast. Hopefully by the time it airs, you have a little script that... they had great results since then. But it was too long to put in the margins of the podcast.
Dwarkesh Patel
Yeah, with luck.
Paul Christiano
Yeah. So I think it's hard to work on because it's not clear what a success looks like. It's not clear if success is possible. But I do think there are a lot of questions. We have a lot of questions.
I think the basic setting is: look, there are all of these arguments. In mathematics, in physics, in computer science, there are just a lot of examples of informal heuristic arguments. They have enough structural similarity that it looks very possible that there is a unifying framework, that these are instances of some general framework and not just a bunch of random things.
For example, for the prime numbers, people reason about the prime numbers as if they were like a random set of numbers. One view is that's just a special fact about the primes—they're kind of random. A different view is that actually, it's pretty reasonable to reason about an object as if it was a random object as a starting point, and then as you notice structure, you revise from that initial guess. It looks like to me the second perspective is probably more right. It's just reasonable to start off treating an object as random and then notice perturbations from random, notice structure the object possesses. The primes are unusual in that they have fairly little additive structure.
I think it's a very natural theoretical project. There's a bunch of activity that people do. It seems like there's a reasonable chance there's something nice to say about unifying all of that activity. I think it's a pretty exciting project. The basic strike against it is that it seems really hard. If you were someone's advisor, you'd be like, "What are you going to prove if you work on this for the next two years?" And they'd be like, "There's a good chance nothing." That's not what you do if you're a PhD student normally. You aim for high probabilities of getting something within a couple years.
The flip side is it does feel... I mean, I think there are a lot of questions. I think some of them we're probably going to make progress on. The pitch is mostly: are some people excited to get in now? Or are people more like, "Let's wait to see once we have one or two good successes to see what the pattern is and become more confident we can turn the crank to make more progress in this direction?" But for people who are excited about working on stuff with reasonably high probabilities of failure and not really understanding exactly what you're supposed to do, I think it's a pretty good project.
I feel like if people look back, if we succeed and people are looking back in 50 years on what was the coolest stuff happening in math or theoretical computer science, this will definitely be in contention. I would guess for lots of people, it would just seem like the coolest thing from this period.
Dwarkesh Patel
Right. Because this is a new method in so many different fields, from the ones you mentioned: physics, math, theoretical computer science. That really... I don't know. But what is the average math PhD working on right now? They're working on some subset of a subset of something I can't even understand or pronounce. Math is quite esoteric. But yeah, this seems like... even the small chance of it working, forget about the value for alignment—you shouldn't forget about the value for alignment—but even without that, this is such a cool thing. If this works, it's a really big deal.
Paul Christiano
There's a good chance that if I had my current set of views about this problem and didn't care about alignment, and had the career safety to just spend a couple years thinking about it, or spend half my time for five years or whatever, I would just do that. Even without caring at all about alignment, it's just a nice problem. It's very nice to have this library of things that succeed where they feel so tantalizingly close to being formalizable, at least to me, and such a natural setting, and then just have so little purchase on it. There aren't that many really exciting feeling frontiers in computer science.
Dwarkesh Patel
And so, a smart person—it doesn't have to be a grad student, but a smart person is interested in this. What should they do? Should they try to attack some open problem you have, put on your blog? What is the next step?
Paul Christiano
Yeah, I think a first pass step... There are different levels of ambition, different ways of approaching a problem. But we have this write-up from last year, or I guess 11 months ago, on formalizing the presumption of independence. It provides a communication of what we're looking for in this object. I think the motivating problem is: here's the notion of what an estimator is, and here's what it would mean for an estimator to capture some set of informal arguments. A very natural problem is just try and do that—go for the whole thing, try and understand, and then come up with a different approach or end up having context from a different angle on the kind of approach we're taking. I think that's a reasonable thing to do.
I do think we also have a bunch of open problems. So maybe we should put up more of those open problems. The main concern with doing so is that for any given one, we're like, "This is probably hopeless." We put up a prize earlier in the year for an open problem, which tragically... I mean, I guess the time is now to post the debrief from that, or I owe it from this weekend. I was supposed to do that, so I'll probably do it tomorrow. But no one solved it. It's sad putting out problems that are hard. We could put out a bunch of problems that we think might be really hard.
Dwarkesh Patel
What was that famous case of that statistician who... it was some PhD student who showed up late to a class and he saw some problems on the board and he thought they were homework, and then they were actually just open problems, and then he solved them because he thought they were homework.
Paul Christiano
Right, yeah. I mean, we don't have much less information. These problems are hard. Again, I expect the solution to most of our problems to not be that complicated. We have not... and we've been working on it in some sense for a really long time. Total years of full-time equivalent work across the whole team is probably like three years of full-time equivalent work in this area, spread across a couple people. But that's very little compared to... it's very easy to have a problem where you put in three years of full-time equivalent work but in fact there's still an approach that's going to work quite easily with three to six months if you come at a new angle. We've learned a fair amount from that that we could share, and we probably will be sharing more over the coming months.
Dwarkesh Patel
As far as funding goes, is this something where if somebody gave you a whole bunch of money, it would help? Or does it not matter? How many people are working on this, by the way?
Paul Christiano
So we have been... right now there's four of us full time, and we're hiring for more people.
Dwarkesh Patel
And is funding something that would matter?
Paul Christiano
I mean, funding is always good. We're not super funding constrained right now. The main effect of funding is it will cause me to continuously and perhaps indefinitely delay fundraising. Periodically I'll set out to be interested in fundraising, and someone will offer a grant, and then I will get to delay for another six months or nine months or whatever. So you can delay the time at which Paul needs to think for some time about fundraising.
Dwarkesh Patel
Well, one question I think would be interesting to ask you is... I think people can talk vaguely about the value of theoretical research and how it contributes to real world applications. You can look at historical examples or something. But you are somebody who actually has done this in a big way. RLHF is something you developed, and then it actually got into an application that has been used by millions of people. So tell me about that pipeline. How can you reliably identify theoretical problems that will matter for real-world applications? Because it's one thing to read about Turing or something and the halting problem, but here you have the real thing.
Paul Christiano
Yeah, I mean, it is definitely exciting to have worked on a thing that has a real-world impact. The main caveat I'd provide is that RLHF is very, very simple compared to many things. The motivation for working on that problem was: look, this is how it probably should work. Or this is a step in some progression. It's unclear if it's the final step or something, but it's a very natural thing to do that people probably should be and probably will be doing.
If you want to talk about crazy stuff, it's good to help make those steps happen faster. And it's good to learn about what issues occur in practice, even for things that seem very simple on paper. But mostly the story is just: yep, I think my sense of the world is that things that look like good ideas on paper just often are harder than they look, but the world isn't that far from what makes sense on paper. Large language models look really good on paper, and RLHF looks really good on paper. These things just work out in a way that... I think people maybe overestimate, or maybe it's a trope that people talk about—it's easy to underestimate how much gap there is to practice, how many things will come up that don't come up in theory. But it's also easy to overestimate how inscrutable the world is. The things that happen mostly are things that do just kind of make sense.
Dwarkesh Patel
Yeah, I feel like most ML implementation does just come down to a bunch of detail though—build a very simple version of the system, understand what goes wrong, fix the things that go wrong, scale it up, understand what goes wrong. And I'm glad I have some experience doing that, but I don't think it caused me to have a whole lot of deep expertise or deep wisdom about how to close the gap.
Paul Christiano
Yeah, yeah.
Dwarkesh Patel
But is there some tip on identifying things like RLHF, which actually do matter, versus making sure you don't get stuck in some theoretical problem that doesn't matter? Or is it just coincidence? Or is there something you can do in advance to make sure that the thing is useful?
Paul Christiano
I don't know if the RLHF story is the best expected case or something.
Dwarkesh Patel
Oh, because of the capabilities.
Paul Christiano
Maybe I'd say more profoundly, again, it's just not that hard a case. It's a little bit unfair to be like, "I'm going to predict the thing which I pretty much think was going to happen at some point." So it's mostly a case of acceleration, whereas the work we're doing right now is specifically focused on something that's kind of crazy enough that it might not happen even if it's a really good idea, or challenging enough it might not happen.
But I'd say in general, and this draws a little bit on more broad experience in theory, it's just that a lot of the times when theory fails to connect with practice, it's just kind of clear. It's not going to connect if you actually think about it and ask: what are the key constraints in practice? Is the theoretical problem we're working on actually connected to those constraints? Is there a path? Is there something that is possible in theory that would actually address real issues?
I think as a theoretical computer scientist, the vast majority of theoretical computer science has very little chance of ever affecting practice, but also it is completely clear in theory that it has very little chance of affecting practice. Most of the theory fails to affect practice not because of all the stuff you don't think of, but just because it was dead on arrival. It's not really the point. Mathematicians also are like, they're not trying to affect practice. They're not like, "Why does my number theory not affect practice?" It was kind of obvious.
So I think the biggest thing is just actually caring about that, and then learning at least what's basically going on in the actual systems you care about, what are actually the important constraints, and is this a real theoretical problem? The basic reason most theory doesn't do that is just that's not where the easy theoretical problems are. So I think theory is instead motivated by building up the edifice of theory, and sometimes opportunistically we'll find a case that comes close to practice, or we'll find something practitioners are already doing and try to bring it into our framework. But the theory of change is mostly not this thing that's going to make it into practice. It's mostly contributing to the body of knowledge that will slowly grow and sometimes opportunistically yields important results.
Dwarkesh Patel
How big would you think a seed AI would be? What is the minimum sort of encoding of something that is as smart as a human?
Paul Christiano
I think it depends a lot on what substrate it gets to run on. If you tell me how much computation does it get, or what kind of real-world infrastructure does it get... You could ask, what's the shortest program which if you run it on a million H100s connected in a nice network with a hospitable environment will eventually go to the stars? That seems like it's probably on the order of tens of thousands of bytes. If I had to guess a median, I'd guess 10,000 bytes.
Dwarkesh Patel
Wait, the specification or the compression of?
Paul Christiano
Just the program, a program which would run.
Dwarkesh Patel
Oh, got it, got it, got it. But that's going to be really cheatsy. So the task—what's the thing that has values and will expand and roughly preserve its values as it proceeds?
Paul Christiano
Yeah, exactly. I forgot that task. Because that thing, the 10,000-byte thing, will just lean heavily on evolution and natural selection to get there. For that, I don't know. Million bytes? 100,000 bytes? Something like that.
Dwarkesh Patel
How would you think AI lie detectors will work? Where you kind of just look at the activations and not find explanations in the way you were talking about with heuristics, but literally just like, "Here what truth looks like, here what lies look like," and just segregate the latent space and see if we can identify the two.
Paul Christiano
Yeah, I think training a classifier to do it is a little bit complicated for a few reasons and may not work. But if you just brought in the space and said, "Hey, you want to know if someone's lying, you get to interrogate them, but also you get to rewind them arbitrarily and make a million copies of them," I do think it's pretty hard to lie successfully. You get to look at their brain even if you don't quite understand what's happening. You get to rewind them a million times. You get to run all those parallel copies into gradient descent or whatever. I think there's a pretty good chance that you can just tell if someone is lying—a brain emulation or an AI or whatever—unless they were aggressively selected. If it's just they are trying to lie well, rather than they were selected over many generations to be excellent at lying, then your ML system hopefully didn't train it a bunch to lie. You want to be careful about your training scheme effectively doing that. Yeah, that seems like it's more likely than not to succeed.
Dwarkesh Patel
And how possible do you think it will be for us to specify human verifiable rules for reasoning such that even if the AI is super intelligent, we can't really understand why... there's certain things we know that the way in which it arrives at these conclusions is valid. Like it was trying to persuade us to something, we can be like, "I don't understand all the steps, but I know that this is something that's valid and you're not just making shit up."
Paul Christiano
That seems very hard if you wanted to be competitive with learned reasoning. So like, I don't... I mean, it depends a little bit exactly how you set it up, but for the ambitious versions of that, let's say it would address the alignment problem, they seem pretty unlikely. You know, five, ten percent kind of thing.
Dwarkesh Patel
Is there an upper bound on intelligence? Not in the near term, but just like super intelligence at some point. How far do you think that can go?
Paul Christiano
It seems like it's going to depend a little bit on what is meant by intelligence. It kind of reads as a question that's similar to, "Is there an upper bound on strength?" There are sort of arbitrarily smart input-output functionalities. If you hold fixed the amount of compute, there is some smartest one. If you're just asking what's the best set of 10^40 operations, there's only finitely many of them, so some best one for any particular notion of best that you have in mind.
For the unbounded question, we're allowed to use arbitrary description complexity and compute. Probably now and for the... I mean, there is some optimal conduct. If you're like, "I have some goal in mind and I'm just asking what action best achieves it," if you imagine a little box embedded in the universe, I think there's kind of just an optimal input-output behavior. So I guess in that sense I think there is an upper bound, but it's not saturable in the physical universe because it's definitely exponentially slow.
Dwarkesh Patel
Right, yeah. Or because of comms or other things or heat, it just might be physically impossible to saturate something smarter than this thing.
Paul Christiano
Yeah. I mean, for example, if you imagine what the best thing is, it would almost certainly involve just simulating every possible universe. It might have inscrutable moral constraints, which I don't know if you want to include them. But that would be very, very slow. It involves simulating all... I don't know exactly how slow, but double exponential, very slow.
Dwarkesh Patel
Carl Shulman laid out his picture of the intelligence explosion in the seven hour episode. I know you guys have talked a lot. What about his basic picture? What is it? Do you have some main disagreements? Is there some crux that you guys explored?
Paul Christiano
It's related to our timelines discussion from earlier. I think the biggest issue is probably error bars. Carl has a very software-focused, very fast kind of takeoff picture. I think that is plausible but not that likely. I think there are a couple ways you could perturb the situation, and my guess is one of them applies. So maybe I have... I don't know exactly what Carl's probability is. I feel like Carl's going to have like a 60% chance on some crazy thing that I'm only going to assign like a 20% chance to or 30% chance or something.
Those kinds of perturbations are: one, how long a period is there of complementarity between AI capabilities and human capabilities, which will tend to soften takeoff. Two, how much diminishing returns are there on software progress such that a broader takeoff involving scaling electricity production and hardware production—is that likely to happen during takeoff? I'm more like 50-50 or more on stuff like this.
Dwarkesh Patel
So is it that you think the ultimate constraints will be more hardware? The basic case he's laid out is that you can just have a sequence of things like Flash Attention or MoE, and you can just keep stacking these kinds of things.
Paul Christiano
I'm very unsure if you can keep stacking them. It's a question of what the returns curve is. Carl has some inference from historical data or some way he'd extrapolate the trend. And I am more like 50-50 on whether the software-only intelligence explosion is even possible, and then a somewhat higher probability that it's slower than hardware.
Dwarkesh Patel
Why do you think it might not be possible?
Paul Christiano
Well, the entire question is: if you double R&D effort, do you get enough additional improvement to further double the efficiency? That question will itself be a function of your hardware base—how much hardware you have. At the amount of hardware we're going to have and the level of sophistication we have as the process begins, is it the case that each doubling of R&D at least doubles the effective output of your AI research population? I think I have a higher probability on it being close. If you look at the empirics, the empirics benefit a lot from continuing hardware scale up. So the effective R&D stock is significantly smaller than it looks, if that makes sense.
Dwarkesh Patel
What are the empirics you're referring to?
Paul Christiano
There are kind of two sources of evidence. One is looking across a bunch of industries at what is the general improvement with each doubling of either R&D investment or experience. It is quite exceptional to have a field where each time you double the R&D investment you get a doubling of efficiency. The second source of evidence is on actual algorithmic improvement in ML, which is obviously much scarcer. You can make a case that it's been like each doubling of R&D has given you roughly a 4x increase in computational efficiency. But there's a question of how much that benefits.
When I say the effect of R&D stock is smaller, I mean we scale up. You're doing a new task every couple years because you're operating at a scale much larger than the previous scale. A lot of your effort is how to make use of the new scale. So if you're not increasing your installed hardware base and just flat at a level of hardware, I think you get much faster diminishing returns than people have gotten historically. I think Carl agrees in principle this is true. Once you make that adjustment, I think it's very unclear where the empirics shake out. I think Carl has thought about these more than I am, so I should maybe defer more. But anyway, I'm at like 50-50 on that.
Dwarkesh Patel
How have your timelines changed over the last 20 years?
Paul Christiano
Last 20 years? Yeah. How long have I been working on anything related to AI? So I started thinking about this stuff in like 2010 or so. I think my first timeline prediction was in like 2011. In 2011, my rough picture was like, "We will not have insane AI in the next 10 years." And then I get increasingly uncertain after that, but we converge to 1% per year or something like that. Then probably in 2016 my take was, "We won't have crazy AI in the next five years," but then we converge to like one or two percent per year after that. Then in 2019 I made a round of forecasts where I gave like 30% or something to 25% to crazy AI by 2040, and like 10% by 2030 or something like that.
So I think my 2030 probability has been kind of stable, and my 2040 probability has been going up. I would guess it's too sticky. I guess that 40% that I gave at the beginning is just from not having updated recently enough, and I maybe just need to sit down. I would guess that should be even higher. I think like 15% in 2030 I'm not feeling that bad about. Each passing year is a big update against 2030. We don't have that many years left. That's roughly counterbalanced with AI going pretty well. Whereas for the 2040 thing, the passing years are not that big a deal, and as we see that things are basically working, that's cutting out a lot of the probability of not having AI by 2040. So my 2030 probability is up a little bit, maybe like twice as high as it used to be. My 2040 probability is up more, much more significantly.
Dwarkesh Patel
How fast do you think we can keep building fabs to keep up with the AI demand?
Paul Christiano
Yeah, I don't know much about any of the relevant areas. My best guess is... my understanding is right now like 5% or something of the next year's total, or best process fabs, will be making AI hardware, of which only a small fraction will be going into very large training runs—only a couple percent of total output. That represents maybe 1% of total possible output, a couple percent of leading process, 1% of total or something. I don't know if that's right. I think that's the rough ballpark we're in.
I think things will be pretty fast as you scale up for the next order of magnitude or two from there, because you're basically just shifting over other stuff. My sense is it will be like years of delay. There are multiple reasons you expect years of delay for going past that. Maybe even at that you start having... yeah, there's just a lot of problems. Building new fabs is quite slow. I don't think TSMC is planning on increases in total demand driven by AI—kind of conspicuously not planning on it. I don't think anyone else is really ramping up production in anticipation either. Similarly, just building data centers of that size seems very, very hard and also probably has multiple years of delay.
Dwarkesh Patel
What does your portfolio look like?
Paul Christiano
I tried to get rid of most of the AI stuff that plausibly implicated in policy work or advocacy on the RSP stuff for my involvement with Anthropic.
Dwarkesh Patel
What would it look like if you had no conflict of interest? No inside information.
Paul Christiano
I also still have a bunch of hardware investments, which I need to think about. But like, I don't know, a lot of TSMC. I have a chunk of NVIDIA, although I just keep betting against NVIDIA constantly.
Dwarkesh Patel
Why?
Paul Christiano
Since 2016 or something. I've been destroyed on that bet. Although AMD has also done fine. The case now is even easier, but it's similar to the case in the old days. Just a very expensive company given the total amount of R&D investment they've made. They have a trillion dollar valuation or something. That's very high. The question is: how expensive is it to make a TPU such that it actually outcompetes H100 or something? I'm like, wow, it's a real level of incompetence if Google can't catch up fast enough to make that trillion-dollar valuation not justified.
Dwarkesh Patel
Whereas with TSMC, they have a harder mode, you think?
Yeah, I think it's a lot harder, especially if you're in this regime where you're trying to scale up. So if you're unable to build fabs, I think it will take a very long time to build as many fabs as people want. The effect of that will be to bid up the price of existing fabs and existing semiconductor manufacturing equipment. So just those hard assets will become spectacularly valuable, as will the existing GPUs and the actual...
Paul Christiano
Yeah. I think it's just hard. That seems like the hardest asset to scale up quickly, so it's like the asset if you have a rapid run-up, it's the one that you'd expect to most benefit. Whereas NVIDIA stuff will ultimately be replaced by either better stuff made by humans or stuff made with AI assistance. The gap will close even further as you build AI systems, right? Unless NVIDIA is using those systems.
Dwarkesh Patel
Yeah. The point is just that the future R&D will so dwarf past R&D, and there's just not that much stickiness. There's less stickiness in the future than there has been in the past.
Paul Christiano
Yeah. I don't know. So I don't want to comment from any private information, just in my gut. Having the caveat that this is the single bet I've most lost.
Dwarkesh Patel
Okay. Not including NVIDIA in that portfolio. And final question: there's a lot of schemes out there for alignment, and a lot of general takes. A lot of this stuff is over my head. I think it literally took me like weeks to understand the mechanistic anomaly stuff you work on. Without spending weeks, how do you detect bullshit? People have explained their schemes to me and I honestly don't know if it makes sense or not. With you, I just trust Paul enough that I think there's probably something here if I try to understand it enough. But with others, how do you detect bullshit?
Paul Christiano
Yeah. So I think it depends on the kind of work. For the kind of stuff we're doing, my guess is most people just can't really tell whether it's bullshit. So I think it's important that we don't spend that much money on it. The people who want to hire us are probably going to dig in in depth. I don't think there's a way you can tell whether it's bullshit without either spending a lot of effort or leaning on deference.
With empirical work, it's interesting in that you do have some signals of the quality of work. You can ask, does it work in practice? Does the story... I think the stories are just radically simpler, and so you probably can't evaluate those stories on their face. Then you mostly come down to questions like, what are the key difficulties?
I tend to be optimistic when people dismiss something because it doesn't deal with a key difficulty, or it runs into a super bowl obstacle. I tend to be a little more skeptical about those arguments and tend to think something can be bullshit because it's not addressing a real problem. That's the easiest way: this is a problem someone's interested in that's just not actually an important problem, and there's no story about why it's going to become an important problem. For example, it's not a problem now and won't get worse, or it is maybe a problem now but it's clearly getting better.
That's one way. Then conditioned on passing that bar—dealing with something that actually engages with important parts of the argument for concern—and then actually making sense empirically. So I think most work is anchored by its source of feedback, which is actually engaging with real models. So it's like, does it make sense to engage with real models? And does the story about how it deals with key difficulties actually make sense? I'm pretty liberal past there.
I think it's really hard. EGP will look at mechanistic interpretability and be like, "Well, this obviously can't succeed." And I'm like, I don't know. How can you tell it obviously can't succeed? I think it's reasonable to look at total investment in the field, how fast is it making progress, how does that pencil. I think most things people work on will actually pencil pretty fine. They look like they could be reasonable investments. Things are not super out of whack.
Dwarkesh Patel
Okay, great. I think this is a good place to close. Paul, thank you so much for your time.
Paul Christiano
Yeah, thanks for having me. It was good chatting.
Dwarkesh Patel
Yeah, absolutely.
Outro begins
Hey everybody, I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it on Twitter, your group chats, et cetera. It just splits the world. Appreciate you listening. I'll see you next time. Cheers.
End of episode