Holden Karnofsky — History's most important century

Dwarkesh Patel: If we had AI systems that could do everything humans do to advance science and technology, that would be insane. We live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. You just imagine a universe where there actually are some people who live in an especially important time. And then there's a bunch of other people who like tell stories to themselves about how, what, you know, whether they do, how would you want all those people to behave? And it's like, to me, the worst possible rule is all those people should just be like, nah, this is crazy and forget about it. All right. Today, I have the pleasure of speaking with Holden Karnofsky, who is the co-CEO of Open Philanthropy. In my opinion, Holden is one of the most interesting intellectuals alive, well, given your role. So Holden, welcome to the Lunar Society.

Holden Karnofsky: Thanks for having me.

Dwarkesh Patel: Okay. So let's start off by talking about the most important century thesis. Do you want to explain what this is for the audience?

Holden Karnofsky: You know, my story is I originally co-founded an organization called GiveWell that helps people decide where to give as effectively as possible. I'm no longer there, but I'm on the board. And it's a website called GiveWell.org that I think makes good recommendations where to give to charity to help a lot of people. And, you know, as we were working at GiveWell, we met Cari Tuna and Dustin Moskovitz. Dustin is the co-founder of Facebook and Asana and started a project that became Open Philanthropy to try to help them give away their large fortune, again, to help as many people as possible. And so I've kind of spent my career looking for ways to do as much good as possible with a dollar or with an hour, with whatever resources you have, and especially with money. And so I've kind of developed this professional specialization in looking for ideas that are underappreciated, underrated, tremendously important because a lot of the time that's where I think you can find just kind of outsized, what you might call outsized return on investment opportunities to spend some money and just get an enormous impact because you're doing something very important that is being ignored by others.

And so it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention. And then I encountered the effective altruist community, which is a community of people basically built around the idea of doing as much good as you can. And so it's through that community that I encountered the idea of the most important century. It's not my idea at all. I got to it from a lot of people. And the basic idea is that if we developed the right kind of AI systems this century, and that looks reasonably likely, that could make this century the most important of all time for humanity.

So the basic mechanics of why that might be or how you might think about that. So one thing is that if you look back at all of economic history, just the rate at which the world economy has grown, you see acceleration. You see that it's growing a lot faster today than it ever was. And one theory of why that might be, or one way of thinking about it through the lens of basic economic growth theory, is that in normal circumstances, you can imagine a kind of feedback loop where you have people have ideas, and the ideas lead to greater productivity and more resources. And then when you have more resources, you can also have more people, and then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources. And starting a couple hundred years ago, if you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster.

And basically, if you take the story of our economy to date and you just kind of plot it on a chart and do the kind of simplest thing you can to project it forward, you project that it will go, that our economy will reach like an infinite growth rate this century. And the reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple of hundred years ago. So it goes more people, more ideas, more resources, more people, more ideas, more resources. A couple of hundred years ago, people stopped having more children when they had more resources. They got just more, they got richer instead of more populous. And this is all discussed in the most important century page on my blog, Cold Takes.

And so what happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop. And if you had AI systems that could do all the things humans do to advance science and technology, meaning the AI systems could fill in that more ideas part of the loop, then you could get that feedback loop back. And then you could get sort of this unbounded, heavily accelerating, explosive growth in science and technology. So that's the basic dynamic at the heart of it.

So that's kind of a way of putting it that's trying to use familiar concepts from economic growth theory. Another way of putting it might just be, gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane. You know, what if we were to take the things that humans do to create new technologies that have transformed the planet so radically and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology? So either way you think about it, you could imagine the world changing incredibly quickly and incredibly dramatically.

And so I argue in the most important century series that it looks reasonably likely, in my opinion, more than 50-50, that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology. That if that happens, we'll see explosive progress in science and technology. The world will quickly become extremely different from how it is today. You might think of it as if there was thousands of years of changes packed into a much shorter time period. And then if that happens, I argue that you could end up in a deeply unfamiliar future. I give one example of what that might look like using this hypothetical technology idea called digital people. That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us.

And when you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. You could basically find most of what there is to find and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, and is very stable for very long periods of time and basically looks sort of post-human in a lot of relevant ways. And if you think that, then this is basically our last chance to shape how this happens. So that's the most important century hypothesis in a nutshell is that if we develop AI that can do all the things humans do to advance science and technology, we could very quickly reach a very futuristic world, very different from today's, could be a very stable, very large world. This is our last chance to shape it.

Dwarkesh Patel: Gotcha. Okay, so I and many other people are going to find that very wild. So maybe you can walk us through the process by which you went from doing global development stuff to thinking this way. So in 2014, for example, you had an interview or a conversation. And this is a quote from there. Maybe you can walk me through how you got from there to where you are today. "I have looked at the situation in Africa, have understanding of the situation in Africa, and see a path of doing a lot of good in Africa. I don't know how to look into the far future situation, don't understand the far future situation, and don't see a path to doing good on that front I feel good about."

Holden Karnofsky: Yeah, first, I think I just, I went on for a while, but I want to come back and connect this back up to how this relates to the work I was even doing at GiveWell. Why this is all kind of one theme. If we are kind of on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's like very good or very bad. And I think there are decent arguments that if we move forward without caution and we develop kind of sloppily designed AI systems, they can end up with goals of their own. and basically we'd end up with a universe that contains very little that humans value or a galaxy that does. We could also imaginably end up with a world in which very powerful technologies are used by just not very well-meaning governments to create a world that isn't very good or a world where we kind of eliminate a lot of forms of material scarcity and have a world that's much better than today's.

And so a lot of what I ask, a lot of what I ask the give-all is how can we help the most people possible per dollar spent? And if you ask how can we help the most people possible per dollar spent, then if you think that funding some work to help shape that transition, to help make sure that we don't move forward too incautiously, to help make sure that we do increase the odds that we do get like a good future world instead of a bad future one, that's helping a huge number of people per dollar spent. So that's the motivation.

And now you're quoting me, you know, a discussion, an argument I was having where we posted a transcript back in 2014. And at that time was was that was part of my journey of getting here. So I was talking to people who are saying, Holden, you want to help a lot of people with your resources. You should be focused on this massive event that could be coming this century that that very few people are paying attention to that there might be a chance to make this go well or poorly for humanity. And I was saying, gosh, like that sure is interesting. And I did think it was interesting. That's why I was spending the time and doing the conversation. But I said, you know, when I look at global poverty and global health, I see what I can do. I see the evidence. I see the actions I can take. And I'm not seeing that with this stuff. So what changed?

I would say a good chunk of what changed is maybe like the most boring answer possible is I just kept at it. So I think I was sitting there in 2014 saying, gosh, this is really interesting, but it's all a bit overwhelming. It's all a bit crazy. I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today. and now I've just been thinking about this for a much longer time period. And I do believe, you know, most things you could say about the far future are very unreliable and not worth taking action on. But I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some things I'm willing to say. I'm willing to say it would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad and I do, and I am more familiar today than I was then with the research and the work people can do to make that less likely and the actions people can take to make that less likely. So that's probably more than half the answer.

But another thing that would be close to half the answer is I think the world has changed. And I think that basically there's been big changes in the world of AI since then. So I think basically in 2014, that was the beginning of what's sometimes called the deep learning revolution. And since then, we've basically seen these very computation intensive, but fundamentally simple AI systems achieve a lot of progress on a lot of different unrelated tasks. And it's looking to me like not totally crazy to imagine that the current way people are developing AI systems, cutting edge AI systems, could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science and technology. it's not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there. And if you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them. So I don't think it's necessarily the leading possibility, but it's enough to sort of start thinking concretely about the problem.

Dwarkesh Patel: Another quote from the interview that today I find appealing because I haven't done the work you have yet. "Does even the upper crust of humanity have a track record of being able to figure out the kinds of things MIRI claims to have figured out?" And by the way, for context for the viewers, MIRI is the organization Eliezer was leading, which is who we were talking to at the time.

Holden Karnofsky: Yeah, well, I don't remember exactly what kinds of things MIRI was trying to figure out, and I'm not sure that I even understood what they were that well. So I definitely think it is true that it is hard to predict the future, no matter who you are, no matter how hard you think, no matter how much you've studied. I think that is true. I think parts of our kind of world or memeplex or whatever you want to call it, overblow this at least a little bit. And I think I was I was kind of buying into that a little bit more than I could. So I think, you know, probably in 2014, I would have said something like, gosh, you know, really no one's ever done something like making smart statements about what the, you know, several decades out future could look like, or making smart statements about what we would be doing today to prepare for it.

Since then, I think a bunch of people have looked into this and looked for like historical examples of people kind of making long term predictions and long term interventions. And I don't think it's amazing, but I think I think I wrote a recent blog post entitled "The Track Record of Futurists Seems Dot Dot Dot Fine." Fine is how I'd put it, where I don't I don't think there's anyone who has demonstrated a real ability to predict the future with precision and know exactly what we should do. But I also don't think human's track record of this is so bad and so devastating that we shouldn't think we are capable of at least giving it a shot. And I think if you enter into this endeavor with self-awareness about the fact that everything is less reliable than it appears and feels at first glance, and you look for the few things that you would really bet on, I think it's worth doing. I think it's worth the bet.

My job is to find things that we might do 10 things and have nine of them fail embarrassingly and one of them be such a big hit that it makes up for everything else. And a lot of my job is to find stuff like that. So I don't think it's totally crazy to think we could make meaningful statements about how things we do today could make these future events go better, especially if the future events are crazily far away, especially if they're within the next few decades. So that's that is something I've changed my mind on at least to some degree.

Dwarkesh Patel: Gotcha. OK, so we'll get to the forecasting stuff in a second. But let's let's continue on the object level conversation about the most important century. So I want to make sure I have the thesis right. So is the argument that because we're living in a weird time, we shouldn't be surprised if something like transformative AI happens this century? Or is the argument that since transformative AI could happen this century, it's a weird time?

Holden Karnofsky: Yeah. So something we haven't covered yet but I think is worth throwing in now is that a significant part of the Most Important Century series is kind of just making the case that even if you ignore AI, there's a lot of things that are very strange about the time that our generation lives in. The reason I spent so much effort on this is because my number one, back in 2014 and before that, my number one objection to these stories about transformative AI is, you know, it's not anything about whether the specific claims about AI or economic models or, you know, alignment research makes sense. It's just this whole thing sounds crazy. And it's just suspicious. It's just suspicious if someone says to you, you know, this could be the most important century of all time for humanity. I titled the series that way because I wanted people to know that I was saying something kind of crazy and that I should have to defend it. I didn't want to be backpedaling or soft pedaling or hiding what a big claim I was making.

And so I think my biggest source of skepticism has been just like, I don't have any specific objection. It just sounds kind of crazy and suspicious to say that we might live in one of the most significant times or the most significant time for humanity ever. And so a lot of my series is just kind of saying it is weird. It is weird to think that, but we already have a lot of evidence that we live in an extraordinarily weird time that would be on the short list of contenders for most important time ever before you get into anything about AI. Just using like completely commonly accepted facts about the world.

For example, if you chart the history of economic growth, you see that the last couple hundred years have seen faster growth by a lot than like anything else in the history of humanity or the world. And if you chart anything about like scientific and technological developments, you can see that everything significant is packed together in the recent past. And there's almost no way to cut it. You know, I've looked at many different cuts of this. There's almost no way to cut it that won't give you that conclusion. One way to put it is that, you know, the universe is something like 11 or 12 billion years old. Humanity or sorry, life on Earth is like three billion years old. And then humanity is just a blink of an eye compared to that. You could call it 300,000, three million years old. human civilization is a blink of eye compared to that. And we're in this really, really tiny sliver of time, couple hundred years when we've seen basically, you know, all or like just a huge amount of the technological advancement, economic growth. So that's weird.

You know, I also talk about the fact that the current rate of economic growth seems high enough that we can't keep going for that much longer. If it went for another 10,000 years, that's another blink of an eye on galactic timescales. It looks to me and we can get to this like we would run out of atoms in the galaxy wouldn't have anywhere to go. And so I think there are a lot of signs that we just live in a really strange time.

I think one more I'll just throw in there, then we can move on, is that, you know, I think a lot of people who disagree with my take would say, look, I do believe eventually we will develop space colonization abilities, go to the stars, fill up the galaxy with life, you know, maybe have artificial general intelligence. I just, to say it's a century is crazy. I think it might be 500 years. I think it might be a thousand years. I think it might be 5,000 years. And a big point I make in the series is I say, well, even if it's a hundred thousand years, that's still an extremely crazy time to be in, in the scheme of things. If you just make, you know, if you make a graphic timeline and you kind of show my view versus yours, they look exactly the same down to the pixel. And so there's already a lot of reasons to think we live in a very weird time. We're on this planet where there's no other sign of life anywhere in the galaxy. We believe that we could fill up the galaxy with life. That alone would make us among the earliest life that has ever existed or would make us the earliest life that has ever existed in the galaxy, a tiny fraction of it. So that is a lot of what the series is about.

And I have sort of answered your question, but I'll do it explicitly. You ask, you know, is this series about transformative AI could come and therefore this century could be weird or is it about this century could be weird, therefore transformative AI could come. The central claim is that transformative AI could be developed in this century. And the stuff about how weird a time we live in is just a response to an objection. It's a response to a point of skepticism. It's a way of saying there's already a lot of reasons to think we live in a very weird time. And so actually this thing about AI is only a moderate quantitative update, not a complete revolution in the way you're thinking about things.

Dwarkesh Patel: There's a famous comedian who has a bit where he's imagining what it must have been like to live in 10 B.C. Yeah. People are just numbering. It's 10 years before zero. It's nine years before zero. It's going to happen. Let's say somebody comes with the proof that current deep learning techniques are not scalable for some reason and that transformative AI is very unlikely this century. I don't know if this is the hypothetical where that would happen, but let's just say that's what it is. Even if this is a weird time in terms of economic growth, does that have any implications other than transformative AI of what you should be doing?

Holden Karnofsky: Yeah, I think in general, having kind of learned about just how strange the time we live in is when you look at it in context. And here I just encourage people to go to my series because I have a bunch of charts illustrating this and it can be a little bit hard to do concisely. But I think the biggest thing I take away from that is we should just, you know, we should really look for the next big thing. I think if you'd been living 300 years ago and you'd been talking about the best way to help people, a lot of people might have been talking about various forms of helping low-income people. Maybe they probably would have been talking about spreading various religious beliefs. And I think it would have seemed crazy to think that what you should be thinking about, for example, was the steam engine and how that might change the world. But I think the Industrial Revolution was actually just an enormous deal and probably was the right thing to be thinking about if there's any way to be thinking about it, if there was any way to be thinking about how that would change the world and what one might do to make that a world that could be better.

So that's basically where I'm at on this stuff, is I just think, you know, as a world, as a global civilization, we should have a really high priority on just saying, we live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. We should make a list of all the things that might radically transform the world. We should make sure we've done everything we can to think about them, identify the ways we might be able to do something today that would actually help. And maybe after we're done doing all that, and we have a lot of the world's brightest mind doing their best to think of stuff, and they can't think of any more, then we can go back to to all the other things that we worry about. But I think right now the world invests so little in that kind of speculative, hey, what's the next big thing, even if it's not super productive to do so even if there's not that much to learn, I feel the world should be investing more in that than there is because the stakes than it is because the stakes are extremely high. And I think it is just like a reasonable guess that we're living in a world that's recently been incredibly transformed by the industrial revolution and the future could be incredibly transformed by the next thing. I just don't think this gets a lot of discussion in basically any circles. And if I got some, I would feel a lot more comfortable with the whole thing. I don't think the whole world should just obsess over what the next transformative event is. But I think right now there's so little attention to it.

Dwarkesh Patel: I'm glad you brought up the industrial revolution, because I feel like there's two implicit claims within the most important century thesis that don't seem perfectly compatible. One is that, you know, we live in an extremely wild time, that the transition here is potentially wilder than any other transition there has been before. And the second is we have some sense of what we can be doing to make sure this transition goes well. I'm curious if you think that somebody at the beginning of the Industrial Revolution, if knowing what they knew then could have done something significant to, you know, make sure that it went as favorably as possible, or do you think that that's a bad analogy for some reason?

Holden Karnofsky: It's a pretty good analogy for being thought-provoking and for thinking, gosh, if you had seen the industrial revolution coming in advance, and this is when economic growth really reached a new level back in the 1700s and 1800s, what could you have done? And I think part of the answer is it's not that clear, and I think that is a bit of an argument that we should maybe not get too carried away today by thinking that we know exactly what we can do. But I don't think the answer is quite nothing. So I have a I have kind of a goofy Cold Takes post. I never that I never published and may never publish because I don't know. I kind of lost track of it. But it's it's kind of it's kind of saying, well, what if what if you'd been sitting in that time and you had known the Industrial Revolution was coming or you had thought it might be? You'd ask yourself what you could be doing.

One answer you might have given is you might have said, well, gosh, if this happens, whatever country it happens in might be just like disproportionately influential. And you know what would be great is if I could help transform the thinking and the culture in that country to have a better handle on human rights and more value on human rights and individual liberties and a lot of other stuff. And gosh, it kind of looks like people were doing that, and it kind of looks like it worked out. So this is the Enlightenment, and I think – I even give this kind of goofy example. I could look it up, and it's all kind of a trollish post. But the example is it's like someone's kind of thinking through, hey, you know, I'm thinking about this sort of esoteric question about what a government owes to its citizens or, you know, when a citizen has a right to overthrow a government or when it's acceptable to kind of enforce certain beliefs and not. And it's like the other person in the dialogue is just like, this is the weirdest, most esoteric question. Why does this matter? Why aren't you helping poor people?

Holden Karnofsky: But, you know, these are the questions that the Enlightenment thinkers were thinking about. And I think there is a good case that they came up with a lot of stuff that really shaped the whole world since then because of the fact that the UK became so influential. It really laid the groundwork for a lot of stuff about the rights of the governed, free speech, individual rights, and human rights.

Holden Karnofsky: And then I go to the next analogy. I'm like, okay, now we're sitting here today and someone is saying, "Well, instead of working on global poverty, I'm studying this kind of esoteric question about how you get an AI system to do what you wanted it to do instead of doing its own thing." It's not completely crazy to see them as analogous. Now, I don't think this is what the Enlightenment thinkers were actually doing. I don't think they were saying, "This could be the most important millennium, so let's do this." But it is interesting that it doesn't look like there was nothing to be had there. It doesn't look like there's nothing you could have come up with. In many ways, it looks like what the Enlightenment thinkers were up to had the same kind of esoteric, strange, overly cerebral feel at the time and ended up mattering a huge amount. So it doesn't feel like there's zero precedent either.

Dwarkesh Patel: Yeah. Maybe I'm a bit more pessimistic about that because people like the people who are working on individual rights frameworks like Locke, I don't think they were like anticipating an industrial revolution.

Holden Karnofsky: Yeah. I mean, I feel like the person actually did anticipate the industrial revolution. And probably his political philosophy was actually probably a negative given, you know, Karl Marx. OK. Yeah. So it's not obvious to me that even if you saw something like this happening, that it's totally not obvious. I mean, I think my basic position here is I'm not sitting here highly confident. I'm not saying there's tons of precedent and we know exactly what to do. That's not what I believe. I believe we should be giving it a shot. I think we should be trying and I don't think we should be totally defeatist and saying, "Well, it's so obvious that there's never anything you could have come up with throughout history and humans have been helpless to predict the future." I don't think that is true. So, yeah, I think that's enough of an example to illustrate that.

Holden Karnofsky: And I mean, gosh, you could make the same statement today. You could say, "Look, doing research on how to get AI systems to behave as intended is a perfectly fine thing to do at any period in time. It's not like a bad thing to do." And I think John Locke was doing his stuff because he felt it was a good thing to do at any period in time. But the thing is that if we are at this crucial period of time, it becomes an even better thing to do. And it becomes magnified to the point where it could be more important than other things.

Dwarkesh Patel: Now, the one reason I might be skeptical of this theory is that I could say, "Oh, gosh, if you look throughout history, people were often convinced they were living in the most important time, or at least an especially important time." And if you go back, I mean, everybody can be right about living in the most important time. So maybe I should just have a very low prior that anybody is right about this kind of thing. How do you respond to that kind of logic?

Holden Karnofsky: Well, I mean, so first off, I don't know that it's really true that it's that common for people who said they're living in the most important time in history. I mean, this would be an interesting thing to look at. But just from stuff I've read about whatever past works on political philosophy and stuff, I don't exactly see this claim all over the place. It definitely happens. It's definitely happened. And I think a way of thinking about it is there's kind of two reasons that you might think you are especially important. One is that you actually are and you made reasonable observations about it. And another is that you want to be or you want to think you are, you're self-deceiving. And so, over the long sweep of history, a lot of people will come to this conclusion for the second reason. And most of the people who think they're the most important will be wrong. So that's all true. And that certainly could apply to me and it certainly could apply to others.

Holden Karnofsky: So I think that's just completely fine and completely true. And I think we should have some skepticism when we find ourselves making these kinds of observations. And at the same time, I think it would be a really bad rule or a really bad norm that every time you find yourself thinking the stakes are really high or that you're in a really important position that you just decide to ignore the thought. I think that would be very bad. If you just imagine a universe where there actually are some people who live in an especially important time and then there's a bunch of other people who tell stories to themselves about how... how would you want all those people to behave? To me, the worst possible rule is all those people should just be like, "Now this is crazy," and forget about it. That's the worst possible rule because the people who are living at the important time will then do the wrong thing.

Holden Karnofsky: I think another bad rule would be that everyone should take themselves completely seriously and literally and just promote their own interests ahead of everyone else's. And a rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much and justify the means reasoning. Whatever people are trying to do to help the most important century go well, it's totally good and fine to do research on AI alignment, but people shouldn't be going and telling lies or breaking the law in order to further their ends. That would be my proposed rule: when we have these high stakes, crazy thoughts, we should do what we can about them and not go so crazy about them that we break all the rules of society. And that seems like a better rule. That's the rule I'm trying to follow.

Dwarkesh Patel: Okay. Can you talk more about that? So if for some reason you become convinced that the expected value calculation was immense, that you have to break some law in order to increase the odds that the AI goes well... I don't know what hypothetical this would be true. And is it just that you're not sure that you would be right, so you just want to err on the side of caution?

Holden Karnofsky: Yeah, I'm really not a fan of ends justify the means reasoning. I just think historically, that's the thing that looks really, really bad. It's people saying, "It's worth doing horrible things and coercing each other and using force to accomplish these things because the ends we're trying to get to are more important than everything else." So I'm against that stuff. I think that stuff looks a lot worse historically than people trying to predict the future and do helpful things. So I see my main role in the world as trying to predict the future and do helpful things. I can do that without doing a bunch of harmful or common sense unethical stuff. Maybe someday there will be one of these intense trade-offs. I haven't really felt like I've run into them yet. And if I ever ran into one of those intense trade-offs, I'd have to ask myself how confident I really am. The current level of information and confidence I have is, in my opinion, not enough to do really unjustified, ends-justify-the-means stuff.

Dwarkesh Patel: Okay. So let's talk about the potential implausibility of continued high growth. One thing somebody might think is, okay, maybe 2% growth can't keep going on forever. But maybe growth slows down to 0.5% a year or something. And as you know, small differences in growth rates have big effects on the end result. So by the point that we've exhausted all the possible growth in the galaxy, we'll probably be able to expand to other galaxies. What is wrong with that kind of train of logic where, if there's like 0.5% growth, that still doesn't imply a lock-in or it'd be weird if that implied lock-in.

Holden Karnofsky: I think we might want to give a little bit more context here. One of the key arguments of the most important century is just one of the arguments that we live in a strange time. But I'm arguing the current level of economic growth just looks too high to go on for another 10,000 years or so. One of the points I make, which is a point I got from Robin Hanson, it's not my own originally, is that if you just take the current level of economic growth and extrapolate it out 10,000 years, you end up having to conclude that we would need multiple stuff that is worth as much as the whole world economy is today, multiple times that per atom in the galaxy. And if you believe we can't break the speed of light, then we can't get further than that. We can't get outside the galaxy. So in some sense, we run out of material.

Holden Karnofsky: So you're saying, well, okay, but what if the growth rate falls to 0.5%? The growth rate now I ballparked in the post is around 2%. That's the growth rate generally in the most developed countries. Let's say it falls to 0.5%. And then I'm like, okay, so the growth rate falls to 0.5% just like for how long? Did you calculate how long it would take to get to the same place?

Dwarkesh Patel: Yeah, I think it was like 25,000 years. 0.5% gets you like one world-size economy. It's 10,000 versus 25,000, but 25,000 is the amount of light years between us and the next galaxy. That doesn't sound right. I don't think this galaxy calculation was very close. There's also going to be a bunch of dead space as you get to the outer reach of the galaxy. There's not going to be as much there. That doesn't sound super right, but let's just roll with it.

Holden Karnofsky: Sure, let's just say that. Let's just say that you had 2% today and then growth went down to 0.5% and just stayed there forever. Yeah, I'm pretty sure that's still too big. I'm pretty sure you're still going to hit limits in some reasonable period of time. But also that would still be just weird on its own. It would just be like, well, we lived in the 200 year period when we had 2% growth and then we had 0.5% growth forever. That would still make this an interesting time. It would be like the most dynamic, fastest changing time in all of human history, not by a ton, but it's also like you pick the number that's the closest and the most perfectly optimized here. So if it went down to 0.1% or even down to 0.01%, then it would take longer to run out of stuff, but it would be even stranger, the 2% versus the 0.01%. So I don't really think there's any way out of this. It looks like it's probably going to end up looking like a very special time or a very weird time.

Dwarkesh Patel: This is probably not getting hung up on, but from that perspective, then the century in which we had 8% growth because of the Industrial Revolution, you would say that maybe that's the most important century.

Holden Karnofsky: Oh, sure. Yeah, yeah, yeah. No, totally. The thing about the rapid growth is not supposed to be on its own an argument that this century is the special one. By growth standards, this century looks less special than the last one or two. It's just saying this century is one of a handful. Or I think when I say one of 80 of the most significant centuries or something by economic growth standards. And that's only one argument. And then I look at a lot of other ways in which this century looks unusual. But just to be clear, to say that something is the most important century of all time sounds totally nuts because there's so many centuries. There's just so many centuries in the history of humanity even, and especially if you want to think about it on galactic timescales. And even once you narrow it down to 80, it's just way less weird. It's just like, well, if I've already convinced you using non-controversial reasoning that we're in one of the 80 most important centuries, it shouldn't take me nearly as much further evidence to say actually this one might be number one out of 80 because your starting odds are more than 1%. So to get you up to 10% or 20% or 30%, it doesn't necessarily require a massive update the way that it would if we're just starting from nowhere.

Dwarkesh Patel: But I guess I'm still not convinced that just because this is a weird century, that has any implications for why or whether we should see transformative AI this century. So if we have a model about when transformative AI happens, and if one of the variables that goes into that is what is the growth rate in 0 AD and what is the growth rate in 2000 AD, it just feels weird to have as a parameter for when the specific technological development is going to happen.

Holden Karnofsky: It's just one argument in the series. I think the way that I would come at it is I would just say, hey, look at AI systems. Look at what they're doing. Look at how fast the rate of progress is. Look at these five different angles on imagining when AI might be able to do all the things humans do to advance science and technology. And just imagine that we get there this century. Wouldn't it be crazy to have AI that could do all the things humans do to advance science and technology? Wouldn't that lead to just a lot of crazy stuff happening? There's only ever been one species in the history of the universe that we know of that can do the kinds of things humans do. Wouldn't it be weird if there were two? That would be crazy. And one of them was a new one we built that could be copied at will, run at different speeds, run on any hardware you have. That would be crazy.

Holden Karnofsky: And then you might come back and say, "Yeah, that would be crazy. This is too crazy. I'm ruling this out because this is too crazy." And then I would say, "Okay, well we have a bunch of evidence that we live in an unusual, crazy time. And you actually should think that this century is not just a random century picked from a sample of millions of centuries." So that's the basic structure of the argument. As far as the growth rate in zero AD, I think it matters. I think you're asking the question of why do the dynamics of growth in zero AD matter at all for this argument? And I think it's because it's a question of how does economic growth work generally and how has it worked and what is the trend that we're on and what happens if that trend continues. So if around zero AD growth was very low, but accelerating, and if that was also true at 180 and a thousand AD and negative a thousand, or a thousand BC, if all those things are true, then it starts to point to a general pattern that growth is accelerating and maybe accelerating for a particular reason. And therefore you might expect more acceleration.

Dwarkesh Patel: Gotcha. Gotcha. Okay. So let's just talk about transformative AI then. Yeah. Can you describe what success looks like concretely? Are humans part of the post-transformative AI world? Are we hoping that these AIs become enslaved gods that help us create a utopia? What is the concrete success scenario look like?

Holden Karnofsky: I mean, I think we've talked a lot about the difficulty of predicting the future. And I think I do want to emphasize that I really do believe in that. So my attitude to the most important century is not at all, "Hey, I know exactly what's going to happen and I'm making a plan to get us through it." It's much more like there's a general fuzzy outline of a big thing that might be approaching us. There's maybe like two or three things we can come up with that seem good to do. Everything else we think about, we're not going to know if it's good to do or bad to do. And so I'm just trying to find the things that are good to do so that I can make things go a little bit better. That is my general attitude. It's like if you were on a ship in a storm and you saw some very large fuzzy object obscured by the clouds, you might want to steer away from it. You might not want to say, "Well, I think that is, it's an island and I think there's probably a tiger on it. And if we go and train the tiger in the right way, blah, blah, blah." You don't want to get into that. So that is the general attitude I'm taking.

Holden Karnofsky: So what does success look like to me? I mean, success could look like a lot of things, but one thing success would look like to me would frankly just be that we get something not too different from the trajectory we're already on. So in other words, if we could have AI systems that behaved as intended, acted as tools and amplifiers of humans, did the things they're supposed to do, and if we could avoid a world where those AI systems got all controlled by one government or one person, avoid a world where that caused a huge concentration of power. If we could just have a world where AI systems are just another technology, they help us do a lot of stuff, we invent lots of other technologies, and everything is relatively broadly distributed and everything works roughly as it's supposed to work, then you might be in a world where we continue the trend we've seen over the last couple hundred years, which is that we're all getting richer. We're all getting more tools. We all hopefully get increasing ability to understand ourselves, study ourselves, understand what makes us happy, what makes us thrive. And hopefully the world just gets better over time. And we have more and more new ideas. The ideas make us hopefully wiser.

Holden Karnofsky: I do think that in most respects the world of today is just a heck of a lot better than the world of 200 years ago. I don't think the only reason for that is wealth and technology, but I think they played a role. And I think that if you'd gone back to 200 years ago and said, "Holden, how would you like the world to develop? A bunch of new technologies as long as they're evenly distributed and they behave roughly as intended and people mostly just get richer and discover new stuff," I'd be like, "That sounds great." I don't know exactly where we're going to land. I can't predict in advance whether we're going to decide that we want to treat our technologies as having their own rights. That's stuff that the world will figure out. But I'd like to avoid massive disasters that are identifiable, because I think if we can, we might end up in a world where the future is wiser than we are and is able to do better things.

Dwarkesh Patel: Okay. The way you put it with AI enabling humans, that doesn't sound like something that could last for thousands of years. It almost sounds as weird as like chimps saying, "What we would like as humans to be our tools." At best, maybe they could hope we would give them nice zoos. But like, what is the role for humans in this future?

Holden Karnofsky: I mean, a world I could easily imagine, although that doesn't mean it's realistic at all, is a world where we do build these AI systems. They do what they're supposed to do. And we kind of use them to gain more intelligence and wisdom. I've talked a little bit about this hypothetical idea of digital people. Maybe we develop something like that. And then after 100 years of this, we've been around and people have been having discussions in the public sphere and people kind of start to talk about whether the AIs themselves do have rights of their own and should be sharing the world with us. And then maybe they do get rights and maybe some AI systems end up voting or maybe we decide they shouldn't and they don't. And either way, you have this kind of world where there's a bunch of different beings that all have rights and interests that matter and they vote on how to set up the world so that we can all hopefully thrive and have a good time. We have less and less material scarcity, so fewer and fewer trade-offs need to be made. That would be great. I don't know exactly where it ends or what it looks like.

Dwarkesh Patel: Does anything strike you as unimaginable about that? The fact that you can have beings that can be copied at will, but also there's some method of voting that... yeah.

Holden Karnofsky: So yeah, that's a problem that would have to be solved. I mean, we have a lot of attention paid today to how voting systems work, who gets to vote, and how we avoid things being unfair. And yeah, it's definitely true that if we decided there was some kind of digital entity that should have the right to vote and that digital entity was able to copy itself, well, you could get some havoc right there. So you'd want to come up with some system that maybe restricts how many copies you can make of yourself or restricts how many of those copies can vote. These are problems that I am hoping can be handled in a way that, while not perfect, could be non-catastrophic by a society that hasn't been derailed by some huge concentration of power or misaligned AI systems.

Dwarkesh Patel: So that sounds like that might take time, but let's say you didn't have time. So let's say you get a call and somebody says, "Holden, next month my company is deploying a model that might plausibly lead to AGI." What does Open Philanthropy do? What do you do?

Holden Karnofsky: Well, I need to distinguish. You may not have time to avoid some of these catastrophes like a huge concentration of power or AI systems that don't behave as intended and have their own goals. If you can prevent those catastrophes from happening, you might then get more time after you build the AIs to have these tools that help us invent new technologies and help us perhaps figure things out better and ask better questions. And then you could have a lot of time or you could figure out a lot in a little time if you had those things.

Dwarkesh Patel: But if someone said, "How long did you give me?" A month. A month. Let's say three months. So it's a little bit more.

Holden Karnofsky: Yeah, I would find that extremely scary. And I think that would be one of the worlds in which I might not even be able to offer an enormous amount. My job is in philanthropy and a lot of what philanthropists do historically or have done well historically is we help fields grow. We help do things that operate on very long timescales. So an example of something Open Philanthropy does a lot of right now is we fund people who do research on AI alignment and we fund people who are thinking about what it would look like to get through the most important century successfully. And a lot of these people right now are very early in their career and just figuring stuff out. And so a lot of the world I picture is like it's 10 years from now, it's 20 years from now, it's 50 years from now. And there's this whole field of expertise that got support when traditional institutions wouldn't support it. And that was because of us.

Holden Karnofsky: And then you come to me and you say, "We've got one week left. What do we do?" And I'm like, "I don't know. We did what we could do. Go back in time and try to prepare for this better." So that would be a lot of my answer. I mean, I could say more specific things about what I'd say in the one to three month time frame, but a lot of it would be flailing around and freaking out, frankly.

Dwarkesh Patel: Gotcha. Okay. So maybe we can reverse the question. And let's say you found out that it actually is going to take much longer than you thought. And you have like more than five decades. What changes? What are you able to do that you might not otherwise be able to do?

Holden Karnofsky: Yeah. I mean, I think the further out things are, the more I think it's valid to say that humans have trouble making predictions on long time frames. And the more I'm interested in focusing more on other causes or on very broad things we do, such as trying to grow the set of people who thinks about issues like this, rather than trying to specifically study, for example, how to get AI systems like today's to behave as intended. So I think that's a general shift, but I would say that I tend to feel a bit more optimistic on longer timeframes because I do think that the world just isn't ready for this and isn't thinking seriously about this. And a lot of what we're trying to do at Open Philanthropy is create support that doesn't exist in traditional institutions for people to think about these topics. And that includes doing the AI alignment research. That also includes thinking through how we want politically, what regulations we might want to prevent disaster. I think those are a lot of the things.

Holden Karnofsky: So it's kind of a spectrum. I would say, if it's in three months, I would probably be trying to hammer out a reasonable test of whether we can demonstrate that the AI system is either safe or dangerous. And if we can demonstrate it's dangerous, use that demonstration to really advocate for a broad slowing of AI research to buy more time to figure out how to make it less dangerous. But I don't know that I feel that much optimism. If AI is like 500 years off, then I'm kind of inclined to just ignore it and just try and make the world better and more robust and wiser. But I think if we've got 10 years, 20 years, 50 years, 80 years, something in that range, I think that is the place where supporting early careers and supporting people who are going to spend their lives thinking about this, then we flash forward to this crucial time and there's a lot more people who spent their lives thinking about it. And I think that would be a big deal.

Dwarkesh Patel: Let's talk about the question of whether we can expect AI to be smart enough to disempower humanity, but dumb enough to have that kind of goal. When I look out at smart people in the world, it seems like a lot of them have very complex, nuanced goals that they've thought a lot about what is good and how to do good. A lot of them don't. Does that overall make you more optimistic about AIs?

Holden Karnofsky: I am not that comforted by that. This is a very, very old debate in the world of AI alignment. Eliezer Yudkowsky has something called the orthogonality thesis. I don't remember exactly what it says, but it's something like you could be very intelligent about any goal. You could have the stupidest goal and be very intelligent about how to get it. And in many ways, a lot of human goals are pretty silly. Like a lot of the things that make me happy are not things that are profound or wonderful. They're just things that happen to make me happy. And you could very intelligently try to get those things. So it doesn't give me a lot of comfort.

Holden Karnofsky: I think basically my picture of how modern AI works is that you're basically training these systems by trial and error. So you're taking an AI system, you're encouraging some behaviors, discouraging other behaviors, and you might end up with a system that's been encouraged to pursue something that you didn't mean to encourage and it does it very intelligently. And I don't see any contradiction there. If you were to design an AI system and you were giving it encouragement every time it was getting more money into your bank account, if you were encouraging it to do that, you might get something that's very, very good at getting money into your bank account to the point where it's going to disrupt the whole world to do that. And you will not automatically get something that thinks, "Gosh, is this a good thing to do?" And I think a lot of human goals are just like, there's not really a right answer about whether our goals actually make sense. They're just the goals we have.

Dwarkesh Patel: You've written elsewhere about how moral progress is something that's real, that's historically happened, and it corresponds to what actually counts as moral progress. I wonder, do you think there's reason to think the same thing might happen with AI?

Dwarkesh Patel: Like whatever the process, whatever the processes that creates is moral progress. I kind of don't in particular. So I've kind of used the term moral progress as just a term to refer to changes in morality that are good. I think there has been moral progress, but I don't think that means moral progress is something inevitable or something that happens every time you are intelligent or something like that.

So I just, an example I use a lot is just attitude towards homosexuality. It's a lot more accepted today than it used to be. I call that moral progress because I think it's good. And some people will say, "Well, I don't believe that morality is objectively good or bad. I don't believe there is any such thing as moral progress. I just think things change randomly." And that will often be an example I'll pull out. And I say, "But do you think that was a neutral change?" I just think it was good. I think it was good, but that's not because I believe there's some underlying objective reality. It's just my way of tagging or using language to talk about moral changes that seem like they were positive to me.

I don't particularly expect that an AI system would have the same sort of evolution that I've had in reflecting on morality, or would come to the same conclusions I've come to, or would come up with moralities that seem good to me. I don't have any reason to think any of that. I do think historically there have been some cases of moral progress.

Dwarkesh Patel: All right. What do you think is the explanation for that historical progress?

Holden Karnofsky: Well, one thing that I would say is that humans have a lot in common with each other. So I think some of history contains cases of humans just learning more about the world, learning more about themselves, debating each other. Maybe I think a lot of moral progress has just come from humans getting to know other humans who they previously were stereotyping and judging negatively and afraid of. So I think there's some way in which humans learning about the world and learning about themselves leads them to have conclusions that are more reflective and more intelligent for their own goals. But if you brought in something into the picture that was not a human at all, it might be very intelligent and reflective about its goals. And those goals might have zero value from our point of view.

Dwarkesh Patel: Recent developments in AI have made many people think that AI could happen much sooner than they otherwise thought. Have the release of these new models impacted your timelines?

Holden Karnofsky: Yeah, I definitely think that recent developments in AI have made me a bit more freaked out. And even since I wrote the Most Important Century series and before that, there were years when Open Philanthropy was very interested in AI risk, but it's become more so as we've seen the progress in AI.

So I think what we're seeing is these very simple systems. It's kind of the same type of system over and over again, very general, and able to do a lot of different tasks. One of the most interesting... I think people are interested in this. You should maybe Google GPT-3. There's a lot of compilations of what this very simple language model—by the way, my wife and brother-in-law both worked on it, so that's a disclosure—but this very simple language model that just predicts the next word it's going to see in a stream of text. People have gotten it to tell stories. People got similar, though not identical, models to analyze and explain jokes. People have gotten it to play role-playing games, write poetry, write lyrics, answer multiple choice questions, answer trivia questions.

One of the results that I found most ridiculous and strange and weird was this thing called Minerva, where people took one of these language models and with very little intervention, very little special intervention, they got it to do difficult math problems and explain its reasoning and get them right about half the time. And it wasn't really trained in a way that was very specialized for these math problems. So I think we just see AI systems having all these unpredictable human-like abilities just from having this very simple training procedure. And that is something I find kind of wild and kind of scary. I don't know exactly where it's going or how fast.

Dwarkesh Patel: So if you think transformative AI might happen this century, what implications does that have for the traditional global health and well-being stuff that Open Philanthropy does? Will that have persistent effects if AI gets aligned and creates a utopia for us anyway?

Holden Karnofsky: I mean, I don't know about utopia. My general take is that anything could happen. And the reason the most important century stuff is so important is because it's easy to imagine a world that is really awesome and free from scarcity, where we see more of the progress we've seen over the last 200 years and end up in a really great place. It's also easy to imagine a horrible dystopia.

But my take is that the more likely you think all of this is—the more likely you think transformative AI is—the more you should think that should be the top priority, that we should be trying to make that go well instead of trying to solve more direct problems that are more short term. I'm not an extremist on this. So Open Philanthropy does both. Open Philanthropy works on speculative far-off future risks and also does a bunch of more direct work. We direct and recommend a lot of money to GiveWell's top charities, which do things like distributing bed nets in Africa to help prevent malaria, treating children for intestinal parasites. Open Philanthropy does a lot of advocacy for more money going to foreign aid or for better land use policies to have a stronger economy. We do a bunch of scientific research work that is more aimed at direct medical applications, especially in poor countries. So I support all that stuff. I'm glad we're doing it. And it's just a matter of how real and how imminent you think this transformative stuff is. The more real and more imminent, the more of our resources should go into it.

Dwarkesh Patel: Yeah, that makes sense to me. But I'm curious, whatever work you do elsewhere, does that still have persistent effects after transformative outcomes? Or do you think that does wash out in comparison to the really big stuff?

Holden Karnofsky: I think in some sense, the effects are permanent in that if you cause someone to live a healthier, better life, that's a thing that happened. And nothing will ever erase that life or make that life unimportant. But in terms of the effects on the future, I do expect it mostly to wash out. I expect mostly whatever we do to make the world better in that way will not persist in any kind of systematic, predictable way past these kind of crazy changes. And I think that's probably how things look pre- and post-industrial revolution. There's probably some exceptions, but that's my guess.

Dwarkesh Patel: You've expressed skepticism towards the competition frame around AI, or you try to make capabilities go faster for the countries or companies you favor most. But elsewhere, you've used the "innovation as mining" metaphor—maybe you can explain that when you're giving the answer. But it seems like this frame should imply that actually the second most powerful AI company is probably right on the heels of the first most powerful. And so if you think the first most powerful is going to take safety more seriously, you should try to boost them. How do you think about how these two different frames interact?

Holden Karnofsky: I think it's common for people who become convinced that AI could be really important to just jump straight to, "Well, I want to make sure that people I trust build it first." And that could mean my country, that could mean my friends, people I'm investing in. I have generally called that the competition frame: wanting to win a competition to develop AI. And I've contrasted it with a frame that I also think is important, which is the caution frame: that we need to all work together to be careful to not build something that spins out of control and has all these properties and behaves in all these ways we didn't intend.

I do think we're likely, if we do develop these very powerful AI systems, to end up in a world where there's multiple players trying to develop it, and they're all hot on each other's heels. And I am very interested in ways to find ways for us all to work together to avoid disaster as we're doing that. And I am maybe less excited than the average person who first learns about this is about picking the one I like best and helping them race ahead. Although I am somewhat interested in both.

Dwarkesh Patel: But if you take the "innovation is mining" metaphor seriously, doesn't that imply that the competition is really a big factor here?

Holden Karnofsky: Oh, so the innovation mining metaphor is from another bit of Cold Takes. It's an argument I make that you should think of ideas as being somewhat like natural resources in the sense that once someone discovers a scientific hypothesis, or even once someone writes a certain great symphony, that's something that can only be done once. That's an innovation that can only be done once. And so it gets harder and harder over time to have revolutionary ideas because the most revolutionary, easiest to find ideas have already been found. So there's an analogy to mining.

I don't think it applies super importantly to the AI thing because all I'm saying is that success by person one makes success by person two harder. I'm not saying that it has no impact or that it doesn't speed things up. So just to use a literal mining metaphor, let's say there's a bunch of gold in the ground. It is true that if you rush and go get all that gold, it'll be harder for me to now come in and find a bunch of gold. That is true. What's not true is that it doesn't matter if you do it. I mean, you might do it a lot faster than me. You might do it a lot ahead of me. That totally matters.

Dwarkesh Patel: Fair enough. So maybe one piece of skepticism that somebody could have about transformative AI is that all this is going to be bottlenecked by the non-automatable steps in the innovation sequence, so there won't be these feedback loops that speed up. What is your reaction?

Holden Karnofsky: I think the single best criticism and my biggest point of skepticism on this most important century stuff is the idea that you could build an AI system that's very impressive, that could do pretty much everything humans can do, but there might be one step that you still have to have humans do, and that could bottleneck everything. And then you could have the world not speed up that much and science and technology not advance that fast because the AIs are doing almost everything, but humans are still slowing down this one step, or the real world is slowing down one step. So you have to do, let's say, real world experiments to invent new technologies, and they just take how long they take.

I think this is the best objection to this whole thing and the one that I'd most like to look into more. But I ultimately think that if you had AI systems that had human-like reasoning and analysis capabilities, you shouldn't count on this kind of bottleneck causing everything to go really slow. I write about that in a piece called "Weak Point in the Most Important Century: Full Automation." Part of this is just that you don't need to automate the entire economy to get this crazy growth loop. You can automate just a part of it—specifically the parts that have to do with very important tech like energy and AI itself. And those actually seem in many ways less bottlenecked than a lot of other parts of the economy.

So you could be developing better AI algorithms and AI chips, manufacturing them mostly using robots, using those to come up with even better designs. And then you could also be designing more and more efficient solar panels, using those to collect more and more energy to power your AIs. So a lot of the crucial pieces here just actually don't seem all that likely to be bottlenecked. And also, at the point where you have something that has the ability to have creative new scientific hypotheses the way a human does—which is a debate over whether we should ever expect that and when—but once you have that, I think you should figure there's just a lot of ways to get around all your other bottlenecks because you have this potentially massive population of thinkers looking for them. An example is, you could with enough firepower, enough energy, enough AI, enough analysis, probably find a way to simulate a lot of the experiments you need to run.

Dwarkesh Patel: It seems like the specific examples you used of energy and AI innovations are probably the hardest things to automate, given the fact that those are the ones that humanity's only gotten around to advancing most recently. Can you talk through the intuition that those might be easier?

Holden Karnofsky: I think some of the stuff that might be hardest to automate would be stuff that in some sense doesn't have anything to do with software or capabilities. An example of something that might be extremely hard to automate might be trust—like making a business deal or providing care for someone who's sick. Even if an AI system has all the same intellectual capabilities as a human, can write poetry just as well, can have just as many ideas, can have just as good a conversation, it just doesn't look like a human. So people don't want that. Maybe you can create a perfect representation of a human on a screen, but it's still on the screen.

In general, I see the progress in AI as being mostly on the software front, not the hardware front. So AIs are able to do a lot of incredible things with language, things with math, things with board games. I wouldn't be surprised if they could write hit music in the next decade or two. But people really are not making the same kind of progress on robotics. So weirdly, a task that might be among the hardest to automate—especially if things go fast—might be the task of taking this bottle of water and taking off the cap. Because I have this hand that is just well designed for that. Well, it's clearly not designed for that, but these hands, I can do a lot of stuff. And we aren't seeing the same kind of progress there.

So I think there are a lot of places where AI systems might have brains that can do roughly everything human brains can, but there's some other reason they can't do some key economic tasks. And I think these are not the tasks I see likely to bottleneck the R&D as much.

This is an argument I make in one of my more obscure Cold Takes posts. I say that AI that could actually take everyone's job—like every human's job—might be a lot harder than AI that could radically transform the galaxy via new technology. Because it might be easier to take a scientist's job than a teacher's job or a doctor's job, because the teachers and the doctors are regulated and people might just say, "I want human teachers, I don't want an AI teacher." Whereas you can sit there in your lab with your AI scientists and find new theories that change the world. So some of this stuff is very counterintuitive, but I can imagine worlds where you get really wacky stuff before you get self-driving cars out on the road, just because of the way the regulation is working.

Dwarkesh Patel: Gotcha. Let's talk about another weak point, or the one you identify: lock-in. What do you think are the odds of lock-in given transformative AI?

Holden Karnofsky: So lock-in is my term—or I don't know if it's my term, but it's a term I use—to talk about the possibility that we could end up with a very stable civilization. I talk about that in another post called "Weak Point in the Most Important Century: Lock-in." I wrote posts about the weakest points in the series.

The idea is basically that throughout history so far, let's say someone becomes in charge of a government and they're very powerful and they're very bad. Usually this is generally considered to be temporary in at least some sense—the kind of thing that's not going to go on forever. There's a lot of reasons the world is dynamic, and the ways the world is tend to not stay that way completely. The world has changed a lot throughout history. It's kind of a dumb thing to say, but I'll get to why this might be important.

So if someone is running a country in a really cruel, corrupt way, for one thing, at some point they're going to get old and die. And someone else is going to take over, and that person will probably be different from them. For another thing, the world is changing all the time. There's new technologies, new things are possible, there's new ideas. And so the most powerful country today might not be the most powerful tomorrow. The people in power today might not be the ones in power tomorrow. And I think this gets us used to the idea that everything is temporary, everything changes.

A point I make in the Most Important Century series is that you can imagine a level of technological development where there just aren't new things to find and there isn't a lot of new growth to have, and people aren't dying because—for whatever reason—it seems like it should be medically possible for people not to age or die. And so you can imagine a lot of the sources of dynamism in the world actually going away if we had enough technology. You could imagine a government that was able to actually surveil everyone—which is not something you can do now—with a dictator who actually doesn't age or die, who knows everything going on, who's able to respond to everything. And then you could imagine that world just being completely stable.

I think this is a very scary thought, and it's something we have to be mindful of: that if the rate of technological progress speeds up a lot, we could quickly get to a world that doesn't have a lot more dynamism and is a lot more stable. What are the odds of this? I don't know. It's very hard to put a probability on it. I think when you imagine that we're going to get this explosion in scientific and technological advancement, you have to take pretty seriously the idea that that could end by hitting a wall, that there could not be a lot of room for more dynamism, and that we could have these kind of very stable societies. What does "seriously" mean? I don't know, a quarter, a third, a half, something like that. I'm making up numbers. I think it's serious enough to think about it and think about it as something that affects the stakes of what we're talking about.

Dwarkesh Patel: So I'm curious if you're concerned about lock-in just from the perspective of locking in a negative future, or if you think it might be intrinsically bad to lock in any kind of future. If you could right now press a button and lock in a reasonably positive future, but that won't have any dynamism, or one where dynamism is guaranteed but net expected positive is not, how would you make that determination?

Holden Karnofsky: Well, I don't think a lot about what I would do with unrealistic buttons where I have crazy amounts of power that I'll never have and shouldn't have. I think of lock-in by default as mostly a bad thing. I feel like mostly we want to at least preserve optionality and have a world where it's not just one person running the show with their values set up the way they want forever. I think of it mostly that way.

I can imagine some future world where the civilization has been around long enough, we've learned what there is to learn, and we know what a good world looks like. And most people feel pretty confident about that, and they're right to feel confident. And maybe lock-in is not so bad. But I mostly think of lock-in as a bad thing.

I also imagine that you could sort of lock in some things about the world in order to avoid locking in others. So I can imagine if you had this enormous amount of power over how the world works—and some of this is more explained in my Digital People series, how this could work—but if you had this kind of world where you completely control the environment, you might want to lock in the fact that you never have one person with all the power. That might be a thing you might want to lock in, and that prevents other kinds of lock-in.

Dwarkesh Patel: Do you worry about AI alignment as being a form of lock-in? In some sense, if the goal of the research is to prevent drift from human values, then you might just be locking in values that are suboptimal.

Holden Karnofsky: I mostly think of AI alignment as just trying to avoid a really bad thing happening. So I mostly think of it like the thing we don't want to happen is we have some AI system. We thought we were designing it to help us. We're actually designing it to do something extremely random. Again, these systems work by trial and error, by encouragement, discouragement, or positive and negative reinforcement. And so we might have not even noticed that through the pattern of reinforcement we were giving, we trained some system to want to put as much money as possible into one bank account, or gain as much power as possible, or control as much energy as possible, or something like that, or just set its own reward number, its own score, to the highest possible number.

I think that would be a form of lock-in if we had systems more powerful than humans that had these kind of random goals. That would be like locking in a future that is not related to the things that humans value and care about. That's an example of a thing I think would be really bad.

Now, if we got these systems to behave as intended, we still might have problems because we might have humans doing really stupid things and locking in really bad futures. I think that's an issue too. I feel reasonably comfortable—though not 100% confident—saying that we'd like to avoid that slip-up. We'd like to avoid having these systems that have these random goals we gave them by accident, and they're very powerful, and they're better at setting up the world than we are. And so we get this world that's just doing this random thing that we did by accident. I think that's a thing worth avoiding.

Dwarkesh Patel: What is your biggest disagreement with Will MacAskill's new book on longtermism?

Holden Karnofsky: I like Will's book. I think it's worth reading Will MacAskill's book about how the future could be very large and very important. I think he presents a picture where, in my opinion, if you want to talk about the long-run future and how to make the long-run future go well, you're starting from a place where, by default, almost nothing I can do will actually make sense. I do really believe it's hard to understand the long-run future and it's hard to make specific plans about it.

So I would say compared to Will, I am very picky about which issues I think are big enough and serious enough to actually pay attention to. And so I feel the issue of AI—AI would be transformative enough. It looks likely enough that it'll be soon. If it's soon, that means there might actually be things we can do that have predictable effects. I think this misaligned AI thing is a real threat. The way people design AI systems today would be really bad. I am ready to put some resources into preventing it. But that's crossing my threshold. Most things don't.

So if you make a list of ways to make the next million years go well, I'll look at most of them and be like, "I don't know, I don't really believe in this, I wouldn't really invest in this." So I think Will is a bit broader than I am, in a sense. He's interested in more things. And I am pickier than he is, because I think it's so hard to know what's really going to affect the long-run future that I'm just looking for a really short list of things that are worth paying special attention to.

Dwarkesh Patel: Is there a specific thing that he points out in the book that you think is hard to grapple with?

Holden Karnofsky: I don't remember super well. The book is a really broad survey of lots of stuff. An example I might give is he talks about the risk of stagnation, for example. The risk that growth might just stop or something, or growth might slow to very low levels. And then that kind of implies that what we should be trying to do is make sure we continue to innovate and continue to have growth. But then there's other parts of the book that kind of make it sound like we shouldn't move too fast and we shouldn't innovate too much because we don't want to get to our future before we've achieved some sort of civilizational maturity beyond what we have now to decide what we want that future to look like. Like we don't want to build these powers before we have a better idea of what to do with them.

So I think these are examples of things where I'm just like, "Gosh, I don't know." It could be good to have more growth, could be bad. Good to have less growth. It could be that stagnation is a big threat. It could be that building powerful technologies too fast is a big threat. I just don't really know. I'll tell you what I'm thinking about. I'm thinking about AI because I think it's a big enough deal and likely enough and that we've got enough traction on some of the major risks.

Dwarkesh Patel: When I look throughout history, it often seems like people who are predicting long-term trends are too pessimistic. So in the seventies, you might have been too pessimistic about the ability to find more oil or feed a growing population. And they couldn't have predicted the technological breakthroughs that might have made these things possible. Does this inform some sort of vague optimism about the future for you with regards to AI, or no?

Holden Karnofsky: Yeah, I think historically people have been overly pessimistic about future technologies. And by default, the picture with AI looks just really scary. It just looks like it would be really easy to get a bad future in a lot of different ways if we just didn't move cautiously. So these two considerations kind of balance each other out a little bit for me.

I know a lot of people who believe that we're in deep, enormous trouble, that this outcome where you get AI with its own goals, wiping humanity off the map, is almost surely going to happen. I don't believe that. And this is part of the reason I don't believe it. I actually think this situation looks very challenging, very scary by default. And I think we're tending to overestimate how bad and how dire things are. And so they balance out a little bit for me.

Dwarkesh Patel: But in many of these cases, it seems like it would be impossible to see the positive scenario come about. For example, if you were forecasting population in the 70s, is there some reasonable method by which you would have predicted this is not going to lead to some massive famine that kills a billion people? Or would that have been your focus in the 70s if Open Phil was a thing back then?

Dwarkesh Patel: I think it's really hard to know how knowable the future was in the past and what that means for today. I do think that when you look back—and again, this is something I've looked at a fair amount—when you look back at people trying to predict the future in the past, it just looks deeply unserious. You could say that future people will say the same about us. I'm sure they'll think we look less serious than they are, but I think there's a difference.

I really do think that attempts to rigorously make predictions about the future haven't been numerous historically. And I don't think it's obvious that people were doing the best they can and that we can't do better today. So, this population is an example. It doesn't seem necessarily true to me that you couldn't have been at that time and said, "Gosh, population has been going up for a while now, and people keep inventing new ways to come up with more resources. Maybe that'll keep happening." I'm just really not convinced you couldn't have said that. I'm definitely not convinced no one did say it. I think some people did say it. So I'm hesitant to get too defeatist just from the fact that some people were wrong about the future in the past. It's hard to know if there was really no way to know or if they just weren't trying very hard.

Holden Karnofsky: Well, one thing you just said a minute ago was that we are better at making predictions than people in the past were. So that alone should make us more optimistic about our ability to predict the future.

Dwarkesh Patel: It's just a guess.

Holden Karnofsky: But I do think there's been a lot of progress. I mean, this is what society is—we have a lot of progress on all kinds of intellectual fronts. And I think there has been a lot of progress on what it looks like to make good, reasonable predictions about the future. I think that's something that's happened. So I think we should be expecting ourselves to do a bit better than people did in the past. And future people will probably do better than we do.

Dwarkesh Patel: Right. But I often wonder when I look at a report like Biological Anchors whether Asimov just shooting the shit about screens and what we would do with them might have had fewer sources of error than this eight-step methodology where you might not even be aware of a ninth or tenth missing step that might make the whole thing invalid, and where many of the inputs have multiple orders of magnitude, wide confidence intervals. What do you think of that general skepticism?

Holden Karnofsky: I mean, Biological Anchors is a very important input into my thinking. It's not the only input. So I think my views on AI timelines are a mix of a few things. First, just looking at AI systems today, looking at what they did 50 years ago, looking at what they did 10 years ago, and just being like, "Well, gosh, it sure looks plausible that these will be able to do all the things humans can do to advance science and technology pretty soon." That's one input.

Another input is what we call the semi-informative priors analysis. It's a complex report because it looks from a lot of different angles. But I think you can summarize the highlights as saying that most of the effort that has ever gone into making AI in the history of humanity will be in this century. The field of AI is not very old, and the economy and the amount of effort invested have gone up dramatically. I think that's a data point in favor of not being too skeptical that we could be on the cusp of transformative AI. In some sense, the world has not been trying very hard for very long. So that's an input.

Another input is just expert surveys. When people ask AI researchers when they think AI will be able to do everything humans can, they tend to say it's a few decades. That can be biased and unreliable in all kinds of ways. All these things have their problems. That's a data point.

And then there's Biological Anchors. It's a very complex report with a lot of different angles and questions. But I think you can boil it down to some fairly simple observations. In some important sense, we've never built AI systems before that do as much computation per second as a human brain does. So it shouldn't be surprising that we don't have AI systems that can do everything humans do, because humans are actually doing more work in their brains than a normal AI system is doing. However, it also looks like within this century, we probably will have AI systems that are that big. And if we estimate how much it would take to train them and how much it would cost to build them, that looks like it will probably be affordable this century.

You can talk about all the different ways you could define this and quantify that and all the different assumptions you could put in. But my bottom line is that almost any way you slice it, most angles suggest it looks reasonably likely it will happen this century. That's a data point for me. That matters. So all those are data points feeding into my view.

Dwarkesh Patel: Okay. So I'm stealing this from Eliezer who asked on Twitter, "Has there literally ever in the entire history of AI been any case of anybody successfully calling the development timing of literally any novel AI capability using a bio-anchored or bio-inspired calculation?" I saw some discussion of this on his Facebook, and I think the answer might be yes, but I mostly want to attack the premise.

Holden Karnofsky: I mean, I mostly just want to say that there haven't been a lot of cases of people predicting AI milestones with great precision. And that's also not what I'm trying to do. A lot of what I'm trying to say is that this century, it looks like more likely than not that we'll get something hugely transformative. He's asking about some history of AI that's a few decades old, and there haven't even been a lot of people trying to make predictions. A lot of the predictions have been way more narrow and specific than that. So this isn't a very important or informative question.

And I think all the work he's doing—has there ever been an example of someone using the kind of reasoning Eliezer is using to predict the end of the world or something, which is what he's predicting? I mostly just want to challenge the premise and say, look, we're not working with a lot of sample size here. It's not like this is some big, well-developed field where people have tried to do the exact same thing I'm trying to do 25 times and failed each time. This is mostly people in academia trying to advance AI systems. They don't try to predict when AI systems can do much. We're not working with a lot here. We have to do the best we can, make our best guess, use our common sense, use our judgment, use the angles we've got. That's my main answer.

But another answer I would give is that Hans Moravec was kind of the original biological anchors person. And I think he predicted artificial general intelligence around 2020 or something. And on Eliezer's own views, it's going to look like he was unbelievably close. Eliezer believes we'll see it by 2030. So if that's what you believe, then I think we'll look back on that as the greatest technology prediction ever, by a lot.

Dwarkesh Patel: Yeah, I think the answer is maybe. There was also some discussion in the comments about whether Moravec called big progress on AIs doing well at vision by examining the retina. There was some debate about that. I think it's all very muddy. I don't think this is much of a knockdown argument against thinking about biological anchors.

Holden Karnofsky: It is only one input into my thinking. I do think it looks kind of good for biological anchors that we have seen this deep learning revolution and we have seen these brute force AI training methods working really well when they didn't used to work well. And it is when AI systems started to be about the size of insect or small animal brains, or within a few orders of magnitude of human brains. You can call that a wild coincidence, but it doesn't feel—probably these numbers are all off by 10X, 100X, 1000X. But we're talking about very important things and trying to get our best handle and our best guess. I think biological anchors looks fine so far. It doesn't look amazing. It looks fine.

Dwarkesh Patel: Now, as I'm sure you're aware, many people have proposed that increasing progress in science and technology and increasing academic growth are the most compelling things to be doing instead of working on transformative AI. I just want to get your broad reaction to that first.

Holden Karnofsky: Sure. I think we're talking about the progress studies crowd. I wrote a piece about this on Cold Takes called "Rowing, Steering, Anchoring, Equity and Mutiny," where I discuss different ways of thinking about what it means to make the world good. I do have some sympathy for the idea that a lot of the way the world has gotten better over the last couple hundred years is just we've gotten richer, we've had more technological capabilities. Maybe we should try and do more of that. I don't think this is a nutty thing to think. I think this is somewhat reasonable.

But I overall feel that even a couple hundred years is not that big a percentage of the history of humanity. I wrote a series called "Has Life Gotten Better" that asks what the whole graph of quality of life looks like over the course of humanity. And there is precedent for technological development seeming to make things worse. That's kind of what it looks like happened in the agricultural revolution. So I have some sympathy for saying, "Hey, this thing has been good for 200 years. Let's do more of it." But I don't think it's the tightest, most conclusive argument in the world.

And I think we do have some specific reasons to believe that developing some particular new technologies—not only AI, but also potentially bioweapons—could be so catastrophically dangerous. I think Toby Ord uses the analogy of humanity being like a child who's becoming an adolescent. It's great to become stronger up to a certain point. That feels good. And then at a certain point, you're strong enough to really hurt yourself and really get yourself in trouble, or maybe strong enough that you don't know your own strength. I think there's a pretty good case that humanity is reaching that point. We're reaching the point where we could have a nuclear war or a bioweapon or AI systems that really changed the world forever.

So it might have made sense 300 years ago when we were all struggling to feed ourselves to say, "Hey, we want more power. We want more technology. We want more abilities." Today, I think we're starting to enter the gray zone. Maybe we should slow down a little bit, be a little bit more careful. I'm not saying to literally slow down, but I'm talking about priorities. I would rather look at what I think are dramatically neglected issues that might affect all of humanity's future and at least do the best we can to have a handle on what we want to do about them, than put my effort into throwing more juice and more gas behind this ongoing technological progress, which I think is a good thing. It's just a matter of priority.

Dwarkesh Patel: Okay. Do you think that the entire vision of increasing progress is kind of doomed if ideas get harder to find?

Holden Karnofsky: I do think there's eventually limits to how much. I've said the atoms of the galaxy argument. But I think just a broader common sense take would be that the world over the last couple of hundred years has changed incredibly dramatically. We've had new, exciting technologies and capabilities every year. I think a good guess would be that that hits a wall at some point. It might be the atoms of the galaxy. It might just be something much more boring. It might just be a gradually slowing down, which is actually what we seem to observe when we look at the numbers. We are seeing a bit of stagnation, a bit of slowing down, and probably will keep slowing down by default. So I think probably a good guess is that the world is changing at an incredible pace that has not been the case for most of history, and it probably won't be the case for the whole future.

Dwarkesh Patel: Okay, gotcha. I guess there's several reactions somebody could have to the idea that ideas are getting harder to find and therefore that this makes progress studies less relevant. One is just if you look at your own blog, right? The entire thing is just you complaining about all this low-hanging fruit that people are not plucking. Nobody's thinking seriously about transformative AI. Nobody's thinking seriously about utopia. Nobody's thinking seriously about good governance. How do you square this with the general thing about ideas getting harder to find?

Holden Karnofsky: Oh, I think there's just a ton of really important stuff today that not enough people are paying attention to. That was also true 50 years ago. That was also true 100 years ago. It was probably more true 50 years ago than it is today. It was probably more true 100 years ago than 50 years ago. The supply of amazingly important ideas that are not getting any attention is gradually getting harder to find. Harder doesn't mean impossible.

I do actually think that if people want to do something that's really new and world-changing and dramatic and revolutionary, maybe the worst way to do that is to go into some well-established scientific field and try to revolutionize that. I think it's better to just use your common sense and ask an important question about the world that no one's working on because it isn't a scientific field, because it isn't a field of academia, because it doesn't have institutions, and work on that. A lot of my blog does advocate that.

For example, AI itself is a very well-established field, but AI alignment is a weird field that doesn't really have academic departments right now. A lot of the stuff I'm talking about—trying to predict what the future is going to look like—it's kind of a weird, low prestige thing that you can't easily explain to your extended family. And I do think that's probably the best place to look if you want to do something that's going to be super significant, super revolutionary. That is why I've kind of professionally been drawn to it, looking for potential big wins that philanthropy could get.

Dwarkesh Patel: But isn't another way to read what you said—that we shouldn't follow in the footsteps of the greats to be a great person, to have great achievements yourself—isn't another way to think about that that you should probably also ignore the advice that you're giving or that 80,000 Hours gives, because those specific things are not what's going to make you the next... I mean, a fair number of the dialogue I see in the skeptical of futurism world feels like it's almost just getting unnecessarily fancy or something.

I kind of just want to say, who is someone who really revolutionized the way the world thinks about stuff? I don't know, like Darwin. Now, what was Darwin doing? Was Darwin saying, "Well, I really don't want to think about this thing because I don't believe humans are capable of thinking about that thing. And I don't want to think about this topic because I think it's too hard to know the future and blah, blah, blah"? Was he doing all that stuff, or was he just asking an interesting question? Was he just saying, "Hey, this thing seems important. I'm going to use my common sense and judgment to figure out how it works and I'm going to write about it"?

I think some of this stuff gets too fancy. So I think today, if I just look at the world and I say, "What are the most important things that could matter for the world being a good or bad place in the future?" I've looked at a lot of possibilities. I think that AI is one of the leading examples, and I don't see a lot of people paying attention to it. And so that's what I want to work on. A lot of the people who have done revolutionary work that we now look back on—and a lot of people try to imitate—they weren't really doing all this stuff. They weren't trying to imitate the stuff that usually works and stay away from the stuff that wasn't. They were just asking interesting, important questions and working on them.

As far as myself and 80,000 Hours, I just don't feel that we're well known enough or influential enough that our advice—that the stuff we're interested in is obvious—is automatically therefore not neglected. I think the stuff we're talking about is very neglected. But if you find something that's even more neglected and more important, more power to you.

Dwarkesh Patel: Gotcha. Let's say the total amount of money given to EA just increased by an order of magnitude or something. What could be possible at that point that's not possible now?

Holden Karnofsky: I don't know. I think even then, the amount of money we'd be working with would be really small by the standards of any kind of government budget. In general, with philanthropy, I'm always looking for things where we can seed the creation of a field, fund people to introduce new ideas. But still, we're very small compared to the overall economy and the overall government. I think even multiplying everything by 10, that would still be true. I'm not sure exactly what we'd do with 10x as much money. I'm not even sure what we're going to do with the money that already exists.

Dwarkesh Patel: Yeah. But do you think there will be more EA billionaires in the coming future?

Holden Karnofsky: I would guess so.

Dwarkesh Patel: Does that imply that you should be spending money faster now?

Holden Karnofsky: Yeah, I mean, we're trying to model it. In theory, we have all these models that say, "Here's our guess at how much money is eventually going to be available. And here's our guess at how many giving opportunities will eventually be there to fund. And this is our guess at what's good enough to fund and what's not." That's a very tentative guess. A lot of it is really imprecise stuff, but we have to have some view on it. Anyone who's spending money does.

So I do tend to assume that Sam Bankman-Fried and Dustin Moskovitz and Cari Tuna are not the last billionaires who are interested in doing as much good as possible. But it is really hard to model this stuff. Frankly, we have various rough models we've made over the years. We also sometimes use our intuition and just say, "We fund the stuff that seems quite good and exciting, and we don't fund stuff that doesn't." That's an input into our thinking too.

Dwarkesh Patel: I gotcha. And then how do you think about the risk that some of your giving might have negative impacts? People have brought this up in the context of your million investment in OpenAI, but in all sorts of contexts, especially when you're talking about political advocacy, people might think that the thing you do has negative side effects that counteract the positive effects. How do you think about that? Is it just a straight EV calculation?

Holden Karnofsky: I mean, in theory, what we want is to make grants that have more upside than downside, or have expected net positive effects. I think we tend to be, in a common sense way, a little bit conservative with the negative effects in that the thing we don't want to do is enter some field on a theory that's just totally messed up and wrong in a way that we could have known if we just done a little bit more homework. I think there's something kind of irresponsible and uncooperative about that.

So in general, when we are making big decisions—big dollar decisions or going into a new cause—we often try really hard to do everything we can to understand the downsides. And after we've done roughly everything we can up to some reasonable diminishing returns, if we believe as far as we can tell the upsides outweigh the downsides, we're generally going to go for it. Our goal is not to avoid harm at all costs. Our goal is to operate in a cooperative, high integrity way, always doing our best, always trying to anticipate the downsides, but recognize we're going to have unintended side effects sometimes. That's life, and anything you do has unintended side effects.

I don't agree with the specific example you gave as an example of something that was net negative.

Dwarkesh Patel: Are you talking about OpenAI?

Holden Karnofsky: Yeah. Many people on Twitter reminded me about our $30 million grant to OpenAI. I think it was back in 2016. We wrote about some of the thinking behind it. Part of that grant was getting a board seat for Open Philanthropy for a few years so we could help with their governance at a crucial early time in their development.

I think some people believe that OpenAI has been net negative for the world because they have contributed a lot to AI advancing and to AI being sort of hyped, and they think that gives us less time to prepare for it. I do think that AI advancing faster gives us less time to prepare and is a bad thing. But I don't think it's the only consideration. I think OpenAI has done a number of good things too, has set some important precedents, and is probably much more interested in the risks from advanced AI than the company that I would guess would exist if they didn't, that would be doing similar things. I don't really accept that OpenAI is a negative force. I think it's highly debatable. We could talk about it all day.

And then if you look at our specific grant, that's even a completely different thing, because a lot of that was not just about boosting them, but was about getting to be part of their early decision-making. I think that was important. So my overall view is that I look back on that grant as one of the better grants we've made, not one of the worst ones.

But certainly we've done a lot of things that have not worked out. And I think there are some times when we've done things that have consequences we didn't intend. No philanthropist can be free of that. What we can try and do is be responsible, seriously do our homework to try to understand things beforehand, see the risks that we're able to see, and think about how to minimize them.

Dwarkesh Patel: So let's talk about ethics. I think you have a very interesting series of blog posts about future-proof ethics. Do you want to explain what this is first?

Holden Karnofsky: Sure. I wrote a short blog post series just trying to explain some of the philosophical views and ethical views that are common among people who call themselves effective altruists. One of the ideas I appealed to is what I think a lot of people I know are trying to do: they're trying to come up with a system of morality and ethics that would survive a lot of moral progress. If they later became a lot wiser and learned a lot more and reflected on their morality, they wouldn't look back on their earlier actions and think they were doing horrible, monstrous mistakes.

A lot of history is just people doing things they thought were fine and right at the time, and now we look back and we're horrified. So you could think of yourself as asking, "What morality can I have that would make it not so likely that, if there is a bunch more moral progress and people do learn a lot more, the future won't look back on me and be horrified at what I did?"

I wrote a bit of a series about what it might look like to try to do that, laid out a few principles of it, and tried to use this to explain the moral systems a lot of effective altruists tend to use, which tends to be some flavor of utilitarianism, often very expansive about whose rights count. So effective altruists are very interested in future generations that don't exist yet. They're interested in animals being mistreated on factory farms. They're interested in various populations that a lot of people don't care about today, but that there are large numbers of. So I tried to explain that.

A thing that's important is I laid this view out partly so I could argue against it later, and I haven't done the latter yet. So I have a lot of reservations too about the ethical systems that are common with effective altruists.

Dwarkesh Patel: Okay. So let's talk about some of the pillars you laid out in this piece. Sentientism seems pretty reasonable to me.

Holden Karnofsky: So sentientism is the idea that—well, let me back up. There are three principles that I roughly outlined that you might want for a morality that is going to stand up to scrutiny, or that you won't be so likely to change your mind about if you learn more and get better.

One principle is just systematization. It's better to have morality based on simple general principles that you apply everywhere, rather than just deciding what feels right at the moment. The latter could be subject to a lot of the biases of your time, and the former lets you stress test the core ideas.

And then two of the core ideas I propose are what I call thin utilitarianism, which is basically the greatest good for the greatest number, and sentientism, which is basically saying that someone counts or someone matters if they're able to suffer or have pleasure.

You said sentientism seems reasonable to you. I think sentientism might be the weakest part of the picture to me. If you have a morality where you insist on saying that everyone counts equally in proportion to the amount of pain or pleasure they're able to have, you run into a lot of weird dilemmas that you wouldn't have to run into if you didn't have that view. So I think it's very strange, but I think it is actually one of the more questionable parts of the view.

It's kind of saying, when I'm deciding whether I care about someone, it doesn't matter at all if they're way in the future, if they're way far away, if they're totally different from me, if they're not human, if I've never met them. All that matters is if they can have pain or pleasure. I think it sounds great. And I completely get why someone listening to it would be like, "How could you ever disagree with that?" But I do think there's various challenges with it, which I have not had the chance to write about yet. And I doubt I can be very convincing on this podcast yet because I haven't thought about it enough.

Dwarkesh Patel: All right, yeah, sounds good. Let's talk about the other two. So yeah, systematization. Doesn't the fact that you have lots of complex and sometimes contradictory moral intuitions suggest that maybe the whole goal of having some fundamental principles you extrapolate the rest of morality from is a kind of doomed project?

Dwarkesh Patel: Doesn't the fact that you have lots of complex and sometimes contradictory moral intuitions suggest that maybe the whole goal of having some fundamental principles you extrapolate the rest of morality from is a kind of doomed project?

Holden Karnofsky: I think it does somewhat suggest that. I am somewhat partial to that view. And that's something I may be writing in my rebuttal. But I also think it's possible to just be confused. I think it's possible to just have lots of stuff going on in your brain. Some of it might be based on really good intentions of treating other people fairly and being good to other people. And some of it might be based on other weird stuff, like wanting to stand up for people who look like you or help people who look like you.

So I do have some sympathy for the project of trying to say, "My intuitions contradict each other, but some of them are coming from a good place. Some of them are coming from a bad place. If I thought more about it, I would realize which ones are which." And I want to try and do that.

Let's talk about the utilitarianism. There's this question from an old Scott Alexander post where he asks, would you rather the medieval church spent all of its money helping the poor rather than supporting the arts? So maybe there were fewer poor people back in the medieval times, but you wouldn't have any cathedrals or the Sistine Chapel. I don't know how you would answer that if you were in medieval times. It doesn't sound like the strongest version of this argument to me. Like, maybe that would be fine or good. I don't know. My wife really loves these old churches, so maybe if I had more of her attitude, I would be more horrified by this idea. Low income people had a rough time in the past, and having better lives seems pretty appealing. So I don't really know if that's the best version of this argument.

Dwarkesh Patel: So how much of Future Proof Ethics is basically you're very confident that a future Holden will have a much more developed and better set of ethics? Or how much do you think people in general, or humanity in general, will get better ethics over time?

Holden Karnofsky: This has been a point of confusion in this series. I use the term "moral progress" and I just use it to refer to things getting better. I think sometimes there is such a thing as thinking more about your morality, gaining some insight, and ending up in a better place as a result. I think that is a real thing. There are some people who believe morality is an objective truth, but I'm not one of those people. But even though I believe morality is not objective, I still think there's a meaningful notion of moral progress. There's such a thing as having more reasonable moral views than I used to.

What I didn't mean to say is that moral progress has any inevitability about it. I didn't mean to say that moral progress necessarily happens just because time goes on. I just think it's a thing that can happen. So I do think a future Holden will probably be better at morality just because I'm really interested in the topic and I'm going to keep trying to improve it. I think we have some reason to think that actually just does help a bit, a really tiny bit. But I'm not confident in that at all. And I certainly don't think that society is going to have moral progress just necessarily. But I do think we've had some in the past.

Dwarkesh Patel: Okay. But then it seems weird to label the system of ethics "Future Proof Ethics," right? Maybe just be "Future Holden-Proof Ethics."

Holden Karnofsky: Yeah, possible. I talk about this a bunch in the series and I think I just didn't do a great job with this. What I was trying to do was use a term that you didn't have to be a moral realist to get behind. What I was really trying to capture is: can I think now to reduce the odds that if later I improve, I'll be horrified by my early actions? That was what I was trying to capture the concept of. I'm not sure I really did it successfully.

Dwarkesh Patel: Gotcha. Okay, so you had a recent post on the EA forum that I thought was really interesting. A quote from that is: "My view is that for the most part, people who identify as EAs tend to have unusually high integrity. But my guess is that this is more despite utilitarianism than because of it." So what do you think is the explanation for this coincidence where a group of reasonable, non-fanatical, high integrity people also happen to be a community of utilitarians?

Holden Karnofsky: You might have a set of people who think of themselves as trying really hard to be the kind of person they should be, trying really hard to bring their actions into line with their beliefs and their statements. That drives them to be honest a lot of the time and follow a lot of our common sense rules of morality. But it also drives them to really try to get that ethics right and land on ideas like utilitarianism that are very systematic and pure and give you this clear theoretical guidance. So it could drive both those things.

Whereas I believe that if you're a utilitarian, it's really unclear whether utilitarianism actually tells you to do things like avoiding lying. Some people think it does. Some people think it doesn't. I think it's very unclear.

Dwarkesh Patel: You've advocated for the Moral Parliament approach when you're trying to make decisions. What is the right level of organization at which to use that approach? Should individuals be making decisions based on having multiple different moral parties inside them? Is that the right approach for entire movements, but individuals should be specializing?

Holden Karnofsky: There's something I hope to write about in the future: moral uncertainty. The basic idea is there might be a bunch of different ways of thinking about what the right thing to do is. You might look at the world from one angle and say, "What matters is the total sum of all the pleasures. So a bigger world would be better. I should be really obsessed with getting the world to be as big as possible." There might be another perspective that says what really matters is suffering. We should minimize suffering. We should want the world to be small. There might be another perspective that says it doesn't matter what happens to the world, it matters how I act. What matters is that I act with integrity, that I tell the truth.

There are these interesting debates about what do you do when you think you have some sympathy for all these views. How do you choose an action that some perspectives would say is the best thing you've ever done and some would say is the worst thing you've ever done? The Moral Parliament idea was laid out by Nick Bostrom in an Overcoming Bias post about a decade ago. I think about it as if I'm just multiple people all living inside my head arguing about what to do. They all are friends and they all care about each other and they want to get along. They're trying to reach a deal that all of them can feel fairly good about. That is how I tend to think about dealing with different moral views.

I tend to want to do things that are really good according to one view and not too bad according to the rest, and have the different parts of myself making deals with each other. That relates to something I said at the beginning about not being into "ends justify the means." I put a lot of effort into doing things that would be really good if this Most Important Century stuff came out true, but also not too catastrophic if it didn't. There are lines I'm not willing to cross. There are behaviors I'm not willing to engage in to promote the goals of people who worry about AI safety. So it's a moderating approach.

Dwarkesh Patel: That makes a lot of sense for somebody who's the CEO of Open Philanthropy, that you would want the decisions you make to reflect uncertainties. But if it's just somebody like me where I'm not in some sort of leadership position with a large amount of resources, should I just specialize in that particular moral view I have? Or should I also be trying to allocate my time and resources according to different moral views?

Holden Karnofsky: I think no matter what position I was in, however many resources I had, I would feel that my decisions were significant in some sense and affected people. So I think it's very natural to me to think there's a lot of different perspectives on what it means to be a good person. Rather than trying to turn them into a single unifying mathematical equation and take the expected value—which is another approach I think is interesting—the approach I tend to prefer is to imagine the different perspectives as different people trying to get along and make a deal with each other.

Dwarkesh Patel: Let's talk about governance and management. In software, there's a concept of a 10x engineer. Is there something similar in the kinds of work a research analyst at Open Philanthropy does? Is it meaningful to say that two people doing the same job, one can be orders of magnitude more effective than another?

Holden Karnofsky: Yeah, some people are much better at any given thing in Open Philanthropy than others. I don't think that's very surprising. I think it's true in many jobs. I don't really know the reasons for it. It could be any combination of talent, interest, how hard someone works. So yeah, I certainly think there's a lot of variance. Hiring people who can do a great job at the work OpenPhil does has been a lifelong challenge.

Dwarkesh Patel: You've written about the Bayesian mindset. You know many billionaires, many of them are donors to Open Philanthropy. In your experience, do these startup founders who end up becoming very successful have a Bayesian mindset? Or is that the wrong way to characterize them?

Holden Karnofsky: I wrote about this idea of Bayesian mindset, which is basically being willing to put a probability on anything and use your probabilities as a way of discovering why it is you think what you think, and using expected value calculations similarly. I think this is much more common among successful tech founders than it is among the general population. But there are plenty of tech founders who don't think this way at all. I don't think it's a super well-tested, well-proven social technology that does amazing things. I think it's more like an interesting thing to be experimenting with.

Dwarkesh Patel: The general population, it's practically unheard of.

Holden Karnofsky: Yeah. Not even just the name, just the whole idea of thinking expected value and subjective probabilities all the time. Almost no one does that. I think tech founders probably do it more than the average person.

Dwarkesh Patel: Do you think that adopting more of a Bayesian mindset would help somebody at the top levels be more successful?

Holden Karnofsky: It's really TBD and unclear. I just think the Bayesian mindset is a cool thing to experiment with. I experiment with it a lot. I feel like it helps me sometimes. Like most things, it's good in moderation and with taste and not using it for every single thing. Maybe 10 years from now, as it gets more popular, we'll have a better sense of where the actual applied strengths and weaknesses are.

Dwarkesh Patel: As I'm sure you're aware, there are many prizes floating around for all kinds of intellectual work in effective altruism. Some of them have come from Open Philanthropy. Are you optimistic about their ability to resurface or surface new ideas?

Holden Karnofsky: I would say I'm medium optimistic about the impact of all these prizes. I've been part of designing some of them. I've seen other ones where people say, "We'll pay you X dollars if you can give us a good critique of our reasoning about what the best charities are." Or Open Philanthropy has a prize for showing us a cause we should be looking at that we're not. I think it will get some interest and get some people to pay attention who weren't otherwise. Some of those people might have good ideas. I don't think it's the only way to solve these problems or that it will automatically solve them. That's generally how the people designing the prizes think about them, too.

Dwarkesh Patel: You have an interesting post about stakeholder management, that over time institutions have to take into account the interest of more and more stakeholders. Do you expect that this will be a major factor in how Open Philanthropy acts in the future? And what will be the impact on how Open Philanthropy runs overall?

Holden Karnofsky: In general, the bigger your organization is, the bigger your city is, the bigger your society is, there are more people you're going to have to make happy if you want everyone to be happy. By default, as a company grows, it gets less able to make a lot of disruptive, quick changes. A lot of people in the tech world like to use very negative terms for big company properties and very positive terms for small company properties. Small companies are nimble, quick, practical, adaptive, dynamic, high productivity. Big companies are bureaucratic, slow, and non-adaptive. I think that's all fair.

But I also think that big companies often just produce more stuff than they could if they were small. If Apple were still 10 people, it might be a more exciting place to work, but they wouldn't be able to make all those iPhones. There are a lot of iPhones going out to a lot of people, serving a lot of different people's needs, abiding by a lot of regulatory requirements. There's a lot of work to be done. So I don't think it's necessarily a bad thing, but it's a trade-off for a company to grow.

Open Philanthropy is in the business of doing unconventional giving and using a lot of judgment calls. I tend to think we benefit a lot from staying as small as we can, and I have generally fought for us to stay as small as we can while doing our work. But we still have to grow from where we are.

Dwarkesh Patel: Do you mean stay small in terms of funds or in terms of people?

Holden Karnofsky: People. We're bigger than we were a year ago. We'll be bigger in a year. So it's definitely not true that I'm just trying to minimize the size of the company. We're growing, but I think we want to treat each hire as something we only do because we had a really good reason to. There are some companies that may have more to gain from being 10,000 people. I don't think we'll ever be 10,000 people.

Dwarkesh Patel: Your written career advice emphasizes building aptitudes and specializing. But when I look at your career, it's all over the place. You started off in GiveWell, then you were working at Open Philanthropy, and now you're forecasting AI. How do you think about this? Are you specializing, or what's going on here?

Holden Karnofsky: I don't know if I really forecast AI. I mostly distill and bring together analyses that others have done, and I manage people who work on that sort of thing. The career advice I often give is that it's really good to just have something you're very focused on that you're specializing in, that you're trying to be the best in the world at.

The way my career went is that the general theme is just taking questions—especially questions about how to give effectively—where no one's really gotten started on this question. So even doing a pretty crappy analysis can be better than what already exists. What I consider myself to have specialized in is I do the first cut, crappy analysis of some question that has not been analyzed much and is very important. Then I build a team to do better analysis of that question. That's been my general pattern. That's the most generalizable skill I've had.

But I have switched around because at various points in my career I've just said, "Here's something that's getting very little attention and it's very important, and it's worth the sacrifice of the specialized knowledge I built up in one area to switch into this other area that I think I ought to be working on."

Dwarkesh Patel: What does the logo on the Cold Takes blog mean?

Holden Karnofsky: There is no logo. I think you're talking about the browser icon. That is a stuffed animal named Mora. At some point, if I get enough subscribers, I will explain who all these stuffed animals are. My wife and I basically use a stuffed animal personality classification system where we compare someone to various stuffed animals to explain what their strengths and weaknesses are. Mora is a pink polar bear who's very creative but also very narcissistic and loves attention. She's kind of the mascot of the blog because it's this blog that's very crazy, very out there, and is just me writing in public. It just felt like her spirit.

Dwarkesh Patel: What is the goal of the blog? Having a second job as a blogger in addition to being the CEO of a big organization?

Holden Karnofsky: I think it fits into my job reasonably well. I didn't want it to be Open Philanthropy branded. I just wanted the freedom to write about things the way I wanted. But we make these high stakes decisions based on very unconventional views about the world, and I think it's good for us to be trying to make those views have contact with the rest of the world. There would be something not ideal about being a large foundation giving large amounts of money and just quietly going around believing these enormously important true things that no one else believes.

If we put the views out into the world, all the people seeking money from us have a better idea of where we're coming from—why it is that we're interested in funding what we're interested in funding. To the extent people find the arguments compelling, or even just understand them, that can help create more grantees for us. It can cause the world to be a place where there's more good stuff for us to fund because more people see where we're coming from, hopefully agree with it, and are trying to work on the things we consider important.

And to the extent that my stuff is actually screwed up and wrong, and I've got mistakes in there and I've thought it all through wrong, this is also the best way I know of to discover that. I don't know how else I'm going to get people to critique it except by putting it out there and getting some attention for it. So that's how I think of it: taking views that are very important to the decisions we're making and trying to express them so that we can either get more people agreeing with us who we can fund and support and work with, or learn more about what we're getting wrong.

Dwarkesh Patel: Has that happened? Has there been an important view expressed on the blog that because of feedback you changed your mind on? Or is it mostly about the communication part?

Holden Karnofsky: There's definitely been a lot of interesting stuff. For example, I put up this post on the track record of futurists. Then there was a post by Dan Liu recently that I haven't read yet, but it has its own analysis of the track record of futurists. I need to compare them and think about what I really think about how good humans have historically been about predicting the future. But he certainly has a ton of data in there that I was not aware of. That feels like a bit of a response, may or may not have been prompted by it.

There's been a lot of commentary. You wrote some critiques of some of the stuff I've written in The Most Important Century. There've been other critiques. A lot of the stuff I wrote about the biggest weak points of The Most Important Century were based on the public criticism that was coming in. I think I have become more aware of a lot of the parts of my thinking that are the least convincing or the most weak or the most in need of argument, and I have paid more attention to those things because of that.

Dwarkesh Patel: It sounds like maybe you've learned about how people react to the Most Important Century thesis, but it doesn't seem like something has surfaced which has made you change your mind on it a lot.

Holden Karnofsky: I haven't dropped my view that we could be in the most important century for humanity. That's still what I believe. I've also heard from people who think I'm underselling the whole thing like crazy—people who think I should be planning on transformative AI much sooner than what I imply in the series. I put out this Most Important Century thing, and I don't believe any of the critiques have been deep enough and strong enough to make me just drop that whole thing. But it's a big picture with a lot of moving parts, and I have deepened my understanding of many of the parts.

Dwarkesh Patel: One thing I find really interesting about your work is how much it involves the CEO having a deep understanding of all the issues involved. You are the one who has to deeply understand the Moral Parliament, or specific forecasts about AI, or biological anchors. It seems maybe in other organizations, the CEO just delegates this kind of understanding and asks for the bullet points. Is this something you think more leaders should be doing? Or is there something special about your position?

Holden Karnofsky: I know much less about any given topic than the person who specializes in it. What I try to do is know enough about the topic that I can manage them effectively. That's a pretty general corporate best practice. It varies a lot. For something like keeping our books, doing the financial audits, it's really easy to just judge the outputs without knowing much about finance at all. You can just say, "Was this compliant? Did we pass the audit? How much money do we have in the bank?" You don't need to know much about it to judge it effectively.

But at any company, you may need to know a fair amount about some topics in order to judge them very effectively. If your company is making computers or phones and design is very important, the CEO would be really bad if they just had no opinions on design and just said, "I'm going to let our design person decide." It's a central thing to the company. They should know some things about it.

For things that are really central to Open Philanthropy—what does it mean to do good? How do we handle uncertainty about how to do good? What are the most important causes? If AI might be one of the most important causes, when might we see transformative AI? What would that mean? How big is the risk of misaligned AI? I think I need to understand those issues well enough to effectively manage people who know a lot more about them than I do.

Dwarkesh Patel: I'm just curious, what do you think about this whole Most Important Century stuff? Does this just strike you as crazy? What do you think when you read the series?

Holden Karnofsky: I've been trying to nitpick small things throughout this interview. But when I really think about what the main claim is—that this could be the most important century, transformative AI could happen this century, and if it does, it's a really big deal—I don't disagree. That makes a lot of sense. Throughout preparing for this interview and trying to come up with objections, I've just been a little bit more and more convinced. Thinking about, "Is there actually something I could do over my early career that matters? Or is that something I should just hold off on thinking about?"

Dwarkesh Patel: Glad to hear it. Do you have any ideas what you might do?

Holden Karnofsky: No.

Dwarkesh Patel: Really? Literally no ideas? You haven't been like, "Can I work on AI alignment or..."

Holden Karnofsky: Oh, well, in that sense, I thought a little bit about it. Probably in two months or three months, I'll think real hard about what I actually want to do for a career.

[Outro music begins]

Dwarkesh Patel: Hey, thanks for listening. If you enjoyed that episode, I would really appreciate it if you could share it. This is still a pretty small podcast, so it is a huge help when any one of you shares an episode that you like. Post it on Twitter, send it to friends who you think might like it, put it in your group chats. Just let the word go forth. It helps out a ton.

Many thanks to my amazing editor, Graham Bessalou, for producing this podcast, and to Mia Ayana for creating the amazing transcripts that accompany each episode, which have helpful links. You can find them at the link in the description below. Remember to subscribe on YouTube and your favorite podcast platforms. Cheers. See you next time.