--:-- Dwarkesh Patel: Today, I have the pleasure of interviewing Fin Moorhouse, who is a research scholar at Oxford University's Future of Humanity Institute, and he's also an assistant to Toby Ord, and also the host of the Hear This Idea podcast. Fin, I know you've got a ton of other projects under your belt. So do you want to talk about all the different things you're working on and how you got into EA and this kind of research?
--:-- Fin Moorhouse: I think you nailed the broad strokes there. I think, yeah, I've kind of failed to specialize in a particular thing. And so I found myself just dabbling in projects that seem interesting to me, trying to help get some projects off the ground and just doing research on things which seem maybe underrated. I probably won't bore you with the list of things.
--:-- And then, yeah, how did I get into EA? Actually, also a fairly boring story, unfortunately. I really loved philosophy. I really loved kind of pestering people by asking them all these questions: "Why are you not... why are you still eating meat?" I read Peter Singer and Will MacAskill, and I realized I just wasn't actually living these things out myself. I think there's some force of consistency that pushed me into really getting involved.
--:-- I think the second piece was just the people. I was lucky enough to have this student group where I went to university. And I think there's some dynamic of realizing that this isn't just a free-floating set of ideas, but there's also a community of people I really get on with and have all these incredibly interesting personalities and interests. So those two things, I think.
--:-- Dwarkesh Patel: Yeah. And then so what was the process like? I know a lot of people who are vaguely interested in EA, but not a lot of them then very quickly transitioned to working on research with top EA researchers. So yeah, just walk me through how you ended up where you are.
--:-- Fin Moorhouse: Yeah, I think I got lucky with the timing of the pandemic, which is not something I suppose many people can say. I did my degree, I was quite unsure about what I wanted to do. There was some option of taking some kind of close to default path of maybe something like consulting or whatever. And then I was forced into this natural break where I had time to step back. I was lucky enough that I could afford to spend a few months just figuring out what I wanted to do with my life. And that space was enough to start reading more about these ideas, also to try teaching myself skills I hadn't really tried yet. So I tried to learn to code for a lot of this time and so on.
--:-- And then I just thought, well, I might as well wing it. There are some things I could apply to. I don't really rate my chances, but the cost to apply to these things is so low, it just seems worth it. And then, yeah, I guess I got very lucky and here I am.
--:-- Dwarkesh Patel: Awesome. Okay. So let's talk about one of these things you're working on, which is that you've set up and are going to be helping judge these prizes about EA writing. One is you're giving out five prizes for $100,000 each for blogs that discuss effective altruist-related ideas. Another is five prizes of $20,000 each to criticize EA ideas. So talk more about these prizes. Why is now an important time to be talking about and criticizing EA?
--:-- Fin Moorhouse: That is a good question. Honestly, I'm reluctant to frame this as like me personally. I certainly have helped set up these initiatives.
--:-- Dwarkesh Patel: Well, I heard on the inside that actually you've been pulling a lot of the weight on these projects.
--:-- Fin Moorhouse: Certainly, yeah. I've found myself with the time to get these things over the line, which I'm pretty happy with. So yeah, the criticism thing, let's start with that.
--:-- I want to say something like: in general, being receptive to criticism is obviously really important. And if as a movement you want to succeed—where succeed means not just achieve things in the world but also end up having as close to correct beliefs as you can get—then having this property of being anti-fragile with respect to being wrong, really celebrating and endorsing changing your mind in a loud and public way, that just seems really important.
--:-- So I know this is just a prima facie obvious case of wanting to incentivize criticism, but you might also ask: why now? There's a few things going on there. One is that the effective altruism movement overall has reached this place where it's actually beginning to do a lot of really incredible things. There are a lot of funders now excited to find fairly ambitious, scalable projects. And so it seems like if there's an inflection point, you want to get the criticism out the door and you want to respond to it earlier rather than later, because you want to set the path in the right direction rather than have to course-correct later on, which is more expensive.
--:-- Will MacAskill made this point a few months ago. You can also point to this dynamic in some other social movements where the really exciting beliefs that have this period of plasticity in the early days, they kind of ossify. And you end up with this set of beliefs that it's trendy or socially rewarded to hold. In some sense, you feel like you need to hold certain beliefs in order to get credits from certain people. And the cost to publicly questioning some practices or beliefs becomes too high. That is a failure mode, and it seems like one of the more salient failure modes for a movement like this.
--:-- So it just seems really important to be quite proactive about celebrating this dynamic where you notice you're doing something wrong and then you change track. And maybe that means shutting something down. You set up a project, the project seems really exciting, you get some feedback from the world, the feedback looks more negative than you expected, and so you stop doing the project. In some important sense, that is a success. You did the correct thing, and I'm going to celebrate that. So I think these are some of the things that go through my head, just framing criticism in this kind of positive way. Yeah, it seems pretty important.
--:-- Dwarkesh Patel: Right, right. I mean, analogously, it's said that losses are as important as profit in terms of motivating economic incentives. And it seems very similar here. In a Slack, we were talking and you mentioned that maybe one of the reasons it's important now is if a prize of $20,000 can help somebody figure out how to better allocate $10 million, that's a steal. It's really impressive that effective altruism is a movement that is willing to fund criticism of itself. I don't know, is there any other example of a movement in history where that's been so interested in criticizing itself and becoming anti-fragile in this way?
--:-- Fin Moorhouse: I guess one thing I want to say is the proof is in the pudding here. It's one thing to make noises to the effect that you're interested in being criticized—and I'm sure lots of movements make that—and another thing to really follow through on them. EA is a fairly young movement, so I guess time will tell whether it really does that well. I'm very hopeful. I also want to say that this particular prize is one part of a much bigger thing, hopefully.
--:-- That's a great question. I actually don't know if I have good answers, but that's not to say that there are none. I'm sure there are. Like political liberalism as a strand of thought in political philosophy comes to mind as maybe an example.
--:-- One other random thing I want to point out: you mentioned profits and just doing the maths and what's the EV of investing in just red teaming an idea, like shooting an idea down. I think thinking about the difference between the for-profit and non-profit space is quite an interesting analogy here. You have this very obvious feedback mechanism in for-profit land, which is you have an idea. No matter how excited you are about the idea, you can very quickly learn whether the world is as excited, which is to say you can just fail. And that's a tight, useful feedback loop to figure out whether what you're doing is worth doing.
--:-- Those feedback loops don't, by default, exist if you don't expect to get anything back when you're doing these projects. And so that's a reason to want to implement those things artificially. One way you can do this is with charity evaluators, which in some sense impose a kind of market-like mechanism where now you have an incentive to actually be achieving the thing that you are ostensibly setting out to achieve because there's this third party that's assessing whether you're getting it. But that framing—I mean, we can say more about it, but that's a really useful framing, I think, to me anyway.
--:-- And yeah, but one other reason this seems important to me is if you have a movement that's about 10 years old. We have strains of ideas that are thousands of years old that have significant improvements made to them that were missing before. So just on that alone, it seems to me that the reason to expect some mistakes—either at a theoretical level or in the applications—does seem like I do have a strong prior that there are such mistakes that could be identified in a reasonable amount of time.
--:-- Dwarkesh Patel: Yeah, I guess one framing that I like as well is not just thinking about "here's a set of claims we have, we want to figure out what's wrong," but some really good criticism can look like: "Look, you just missed this distinction which is a really important distinction to make," or "You missed this addition to this naive conceptual framework you're using, and it's really important to make that addition."
--:-- A lot of people are skeptical about progress in non-empirical fields like philosophy, for instance. It's like, "Oh, we've been thinking about these questions for thousands of years but we're still kind of unsure." And I think that misses a really important kind of progress, which is something you might call conceptual engineering or something—finding these really useful distinctions and then building structures on top of them. So it's not like you're making claims which are necessarily true or false, but there are other kinds of useful criticism which include just getting models more useful.
--:-- Speaking of just making progress on questions like these, one thing that's really surprising to me—and maybe this is just my ignorance of the philosophical history here—it's super surprising to me that a movement like longtermism, at least in its modern form, took thousands of years of philosophy before somebody had the idea that "Oh, the future could be really big, therefore the future matters a lot."
--:-- Yeah, and so maybe you could say there have been lots of movements in history that have emphasized that civilizational collapse is a very prominent factor that might be very bad for many centuries, so we should try to make sure society is stable or something. But do you have some sense—you have a philosophy background—so do you have some sense of what the philosophical background here is? And to the extent that these are relatively new ideas, how did it take so long?
--:-- Fin Moorhouse: Yeah, that's such a good question. I think one name that comes to mind straight away is this historian called Tom Moynihan. He wrote this book about the history of how people think about existential risk, and more recently he's been doing work on the question you asked: what took people so long to reach this what now seems like a fairly natural thought?
--:-- I think part of what's going on here is it's really hard—or easy, I should say—to underrate just how much conceptual apparatus we have going on. It's a bit like the water we swim in now, and so it's hard to notice. So one example that comes to mind is thinking about probability as this thing we can talk formally about. This is a shockingly new thought. Also, the idea that human history might end, and furthermore that that might be within our control—to decide or to prevent that happening prematurely—these are all really surprising new thoughts.
--:-- I think it just requires a lot of imagination and effort to put yourself into the shoes of people living earlier on who just didn't have the tools for thinking that make these ideas pop out much more naturally. And of course, as soon as those tools are in place, then the conclusions fall out pretty quickly. But it's not easy. And I agree that actually wasn't a very good answer, just because it's such a hard question.
--:-- Dwarkesh Patel: Yeah, so what's interesting is that more recently—maybe I'm unaware of the full context of the argument here—but I think I heard Holden Karnofsky write somewhere that he thinks that there is more value in thinking about the issues that EA has already identified rather than identifying some sort of unknown risk. For example, what AI might have been like 10, 20 years ago. AI alignment, I mean.
--:-- Given this historical experience that you can have some very fundamental tools for thinking about the world missing and consequently miss some very important moral implications, does that imply that we should expect the space that AI alignment occupies in terms of our priorities—should we expect something as big or bigger coming up? Or just generally tools of thinking like expected value thinking, for example.
--:-- Fin Moorhouse: Yeah, that's a good question. I think one thing I want to say there is it seems pretty likely that the most important, useful concepts for finding important things are also going to be the lowest hanging. And I don't know, I think it's roughly correct that we did, over the course of building out conceptual frameworks, pick the most important ideas first, and now we're kind of refining things and adding maybe somewhat more peripheral things. If that trend is roughly going to hold, that's a reason for not expecting to find some kind of earth-shattering new concept from left field. Although I think that's a very weak and vague argument, to be honest.
--:-- Also, I guess it depends on what you think your time span is. If your time span is the entire span of time that humans have been thinking about things, then maybe you would think that actually it's kind of strange it took 3,000 years—maybe even longer, I guess it depends on when you define the start point—it took 3,000 years for people to realize, "Hey, we should think in terms of probabilities and in terms of expected impact." In that sense, it took 3,000 years of thinking to get to this very basic idea, what seems to us like a very important and basic idea.
--:-- I feel like I want to say two things. If you imagined lining up every person who ever lived in a row and then you walked along that line and saw how much progress people have made across the line—so you're going across people rather than across time—then I think progress in how people think about stuff looks a lot more linear and in fact started earlier than maybe you might think by just looking at progress over time. And if it was faster early on, then if you're following the very long-run trend, then maybe you should expect not to find these kind of totally left-field ideas soon.
--:-- But I think a second thing, which is maybe more important, is I also buy this idea that in some sense, progress in thinking about what's most important is really kind of boundless. David Deutsch talks about this kind of thought a lot: when you come up with new ideas, that just generates new problems, new questions, please some more ideas. That's very well and good. I think there's some sense in which one priority now could just be framed as giving us time to make that progress. And even if you thought that we have this boundless capacity to come up with a bunch of new important ideas, it's pretty obvious that that's a prerequisite. And therefore, in some sense, that's a robust argument for thinking that trying not to throw humanity off course and preventing or mitigating some of these catastrophic risks is always just going to shake out as a pretty important thing to do, maybe one of the most important things.
--:-- Dwarkesh Patel: Yeah, I think that's reasonable. But then there's a question of even if you think that existential risk is the most important thing, to what extent have you discovered all of the X-risk arguments? And by the way, earlier what you said about trying to extrapolate what we might know from the limits of physical laws—if that can kind of constrain what we think might be possible, I think that's an interesting idea. But I wonder, partly, one argument is just we don't even know how to define those physical constraints. And before you had a theory of computation, it wouldn't even make sense to say, "Oh, like this much matter can sustain so many floating point operations per second." And even if you know that number, it still doesn't tell you what you could do with it.
--:-- I think an interesting thing that Karnofsky talks about is he has this article called "This Can't Go On," where he makes the argument that if you just have compounding economic growth, at some point you'll get to the point where you'll have many, many, many times Earth's economy per atom in the effectable universe. And so it's hard to see how you could keep having economic growth beyond that point. But that itself seems like—if that's true, then there has to be a physical law that's like the maximum GDP per atom is this, right? If there's no such constant, then you should be able to surpass it. I guess it still leaves a lot to be desired. Even if you could know such a number, you don't know how interesting or what kinds of things could be done at that point.
--:-- Fin Moorhouse: Yeah, I guess the first one is, even if you think that preventing these very large-scale risks that might curtail human potential—even if you think that's just incredibly important, you might miss some of those risks because you just aren't able to articulate them or really conceptualize them.
--:-- I feel like I just want to say at some point, we have a pretty good understanding of roughly what looks most important. For instance, if you get stranded on a camping trip and you're like, "We need to just survive long enough to make it out," and it's like, "Okay, what do we look out for? I don't really know what the wildlife is here because I haven't been here before, but probably it's going to look a bit like this. I can at least imagine the risk of dying of thirst, even though I've never died of thirst before." And then it's like, "What if we haven't even begun to think of the other..." It's like, yeah, maybe. But there's just some practical reason for focusing on the things which are most salient. And definitely spending some time thinking about the things we haven't thought of yet. But it's not like that list is completely endless. And there's a reason for that.
--:-- And then you said the second thing, which I don't actually know if I have a ton of interesting things to say about. Although maybe you could try zooming in on what you're interested in there.
--:-- Dwarkesh Patel: I come to think of it, I don't think the second thing has big implications for this argument. But the two—yeah, we have like 20 other topics that are just as interesting. We can't move on to, but yeah, just as a closing note, the analogy is very interesting to me. I can't think—if you're trying to do what needs to be done to survive. I don't know.
--:-- Okay. So to extend an analogy, it might be like somebody discovers, "Oh, that berry that we're all about to eat because we feel like that's the only way to get sustenance here while we're just almost starving—don't eat that bear because that berry is poisonous." And then maybe somebody could point out, "Okay, so given the fact that we've discovered one poisonous food in this environment, should we expect there to be other poisonous foods that we don't know about?" I don't know if there's anything more to say on that topic.
--:-- Fin Moorhouse: I mean, one angle you can put on this is you can ask this question: we have precedent for a lot of things. We know now that igniting nuclear weapons does not ignite the atmosphere, which was a worry that some people had. So we at least have some kind of balance on how bad certain things can be. And so if you ask this question, "What is worth worrying about most in terms of what kinds of risks might reach this level of potentially posing an existential risk?" Well, it's going to be the kinds of things we haven't done yet, that we haven't got some experience with.
--:-- And so you can ask this question: what things are there in this space of big-seeming but totally novel, precedent-free changes or events? And it actually does seem like you can try generating that list and getting at answers. This is why—or at least one reason why—AI sticks out, because it fulfills this criteria of being pretty potentially big and transformative and also the kind of thing we don't have any experience with yet. But again, it's not as if that list is in some sense endless. There are only so many things we can do in the space of decades, right?
--:-- Dwarkesh Patel: Okay, yeah. So moving on to another topic: we're talking about for-profit entrepreneurship as a potentially impactful thing you can do. Sorry, maybe not in this conversation, but we separately had one point.
--:-- [Discussion continues...]
--:-- Fin Moorhouse: Yeah, yeah. Um, yeah, so okay. So to clarify, this is not just for profit in order to do earning to give—so you become a billionaire and you give your wealth away. To what extent can you identify opportunities where you can just build a profitable company that solves an important problem area or makes people's lives better? One example of this is Wave. It's a company that helps with transferring money and banking services in Africa. It probably has a boost at people's well-being in all kinds of different ways. So to what extent can we expect just a bunch of for-profit opportunities for making people's lives better?
--:-- Fin Moorhouse: Yeah, that's a great question. And there is really a sense in which some of the more innovative big for-profit companies just are doing an incredibly useful thing for the world. They're providing a service that wouldn't otherwise exist, and people are obviously using it because they are a successful for-profit company.
--:-- Yeah, so I guess the question is something like: you stepping back, you asking, "How can I have a ton of impact with what I do?" The question is, "Am I underrating just starting a company?" So I feel like I want to throw a bunch of disconnected observations. We'll see if they tie together.
--:-- There is a reason why you might in general expect a non-profit route to do well. And this is obviously very naive and simple, but where there is a for-profit opportunity, you should just expect people to take it. This is why we don't see $20 bills lying on the sidewalk. But the natural incentives for taking opportunities to help people where there isn't a profit opportunity—they're going to be weaker. And so if you're thinking about the difference you make compared to whether you do something or whether you don't do it, in general you might expect that to be bigger where you're doing something non-profit. In particular, this is where there isn't a market for a good thing. It might be because the things you're helping aren't humans, maybe because they live in the future so they can't pay for something. It could also be because maybe you want to get a really impactful technology off the ground. In those cases, you get a kind of free rider dynamic. Where you can't protect the IP and patent something, there's less reason to be the first mover.
--:-- And so this is maybe not for profit, but starting to help get a technology off the ground which could eventually be a space for a bunch of for-profit companies to make a lot of money—that seems really exciting. Also, creating markets where there aren't markets seems really exciting. So for instance, setting up AMCs, advanced market commitments, or prizes, or just giving—creating incentives where there aren't any so you get the efficiency and competition gains that you get from the for-profit space. That seems great.
--:-- But that's not really answering your question, because the question is: what about actual for-profit companies? I don't know what I have to say here in terms of whether they're being underrated.
--:-- Dwarkesh Patel: Yeah, actually I'm just curious what you think. Okay, so I think I have like four different reactions. I've been remembering the number four just in case I'm at three and I'm like, "I think I had another thing to say."
--:-- Okay, so yeah, I had a draft of an essay about this. I didn't end up publishing, but that led to a lot of interesting discussions. So that's why we might have—I don't know, in case the audience feels like they're interrupting a conversation that was already preceded the beginning here.
--:-- So one is: to what extent should we expect this market to be efficient? One thing you can think is, listen, the amount of potential startup ideas are so vast and the amount of great founders is so small that you can have a situation where the most profitable ideas are—yeah, it's right that somebody like Elon Musk will come up and pluck up all the maybe hundred billion dollar ideas. But if you have a company like Wave, I'm sure they're doing really well. But it's not obvious how it becomes the next Google or something. And I guess more importantly, if it requires a lot of context—for example, you talked about neglected groups. I guess this doesn't solve for animals and future people. But if you have something in global health where a neglected group is, for example, people living in Africa, right?
--:-- So one is: The people who could be building companies don't necessarily have experience with the problems that these neglected groups have. So if you have—it's very likely, or I guess it's possible, that you could come upon an idea if you were specifically looking at how to help, for example, people suffering from poverty in the poorest parts of the world. You could identify a problem that people who are programmers in Silicon Valley just wouldn't know about.
--:-- Dwarkesh Patel: Okay, so a bunch of other ideas regarding the other things you said. One is, okay, maybe a lot of progress depends on fundamental new technologies and companies coming at the point where the technology is already available and somebody needs to really implement and put all these ideas together.
--:-- Fin Moorhouse: Yeah, two things on that. One is, we don't need to go down a rabbit hole on this. One is the argument that the invention itself, not just the innovation, is a very important aspect and potentially a bottleneck aspect of getting an invention off the ground and scaled. Another is, if you can build a hundred billion dollar company or a trillion dollar company—or maybe not even just a billion dollar company—you have the resources to actually invest in R&D. I mean, think of a company like Google. How many billions of dollars have they basically poured down the drain on harebrained schemes? You can have reactions to DeepMind with regards to AI alignment, but just other kinds of research things they've done seem to be really interesting and really useful. And all the other FAANG companies have a program like this, like Microsoft Research or Amazon's thing.
--:-- Another thing you can point out is with regards to setting up a market that would make other kinds of ideas possible and other kinds of businesses possible. In some sense, you could maybe make the argument that some of the biggest companies, that's exactly what they've done. If you think of Uber, it's not a market for companies, or maybe Amazon is a much better example here. Theoretically, you had an incentive before—like if a pandemic happens, I'm going to manufacture a lot of masks—but Amazon makes the market so much more liquid so that you can just start manufacturing masks and immediately put them up on Amazon. So it seems in these ways, actually starting a company is an effective way to deal with those kinds of problems.
--:-- Dwarkesh Patel: Yeah, man, we've gone so async here. I should have just said one thing and then... Yeah, so I'm sorry for throwing lots of things at you.
--:-- Fin Moorhouse: There's a lot there. As far as I can remember, those are all great points. Yeah, I think my high-level thought is I'm not sure how much we disagree. But I guess one thing I want to say is, again thinking about what you expect the real biggest opportunities to typically be for having a kind of impact. One thing you might think of is, if you can optimize for two things separately—that is, optimize for the first thing and then use that to optimize for the second thing—versus trying to optimize for some combination of the two at the same time, you might expect to do better if you do the first thing.
--:-- So for instance, you can do a thing which looks a bit like trying to do good in the world and also make a lot of money, like social enterprise. And often that goes very well. But you can also do a thing which is try to make a lot of money and just make a useful product that is not directly aimed at improving humanity's prospects or anything, but it's just great, and then use the success of that first thing to then think squarely about how to just do the most good, without worrying about whether there's some kind of profit mechanism. I think often that strategy is going to pan out well.
--:-- There's a thought about the tails coming apart. If you've had this thought that at the extremes of scalability in terms of opportunity to make a lot of profit, and at the extreme of doing a huge amount of good, you might expect it to be not such a strong correlation. One reason in particular that you might think that is because you might think the future really matters, like humanity's future. And sorry to be like a stuck record, but there's not really a natural market there because these people haven't been born yet.
--:-- That's a rambly way of saying that it's not always going to be true, but I basically agree. I would want to resist a framing of doing good which just leaves out also doing some successful for-profit company. There are just a ton of really excellent examples of where that's just been a huge success and should be celebrated. So yeah, I think I disagree with the spirit. Maybe we disagree somewhat on how much we should relatively emphasize these different things, but it doesn't seem like a very deep disagreement.
--:-- Dwarkesh Patel: Yeah, yeah. Maybe I've been spending too much time with Bryan Caplan or something. And by the way, the tails coming apart I think is a very interesting way to think about this. Scott Alexander had a good article on this. One thing he points out is that generally you expect different parts of different types of strength to correlate, but the guy who has the strongest grip strength in the world is probably not the guy who has the biggest squat in the world, right?
--:-- Fin Moorhouse: Yeah.
--:-- Dwarkesh Patel: Okay, so I think that's an interesting place to leave that idea. Okay, another thing I wanted to talk to you about was backtesting EA. So if you have these basic ideas of we want to look at problems that are important, neglected, and tractable, and apply them throughout history—a thousand years back, two thousand years back, a hundred years back—is there a context in which applying these ideas would maybe lead to a perverse outcome, an unexpected outcome? And are there examples where things you could have easily made much better, but maybe it made it much better than even conventional morality or present day ideas would have made them?
--:-- So we get to the first part of the question, which is something like: can we think about what some kind of effective altruism movement, or if these ideas were in the water significantly earlier, whether they might have misfired sometimes, or maybe they might have succeeded? In fact, how do we think about that at all?
--:-- Fin Moorhouse: I guess one thing I want to say is that very often the correct decision ex ante is a decision which might do really well in some possible outcomes, but you might still expect to fail. The kind of mainline outcome is this doesn't really pan out, but it's a moonshot, and if it goes well, it goes really well. This is similar to certain kinds of investing. If that's the case, then you should expect, even if you follow the exact correct strategy, you should expect to look back on the decisions you made and see a bunch of failures, where failure is you just have very little impact. And I think it's important not to kind of resist the temptation to really negatively update on whether that was the correct strategy because it didn't pan out.
--:-- So I don't know if something like EA-type thinking was in the water and was thought through very well. I think it would go wrong a bunch of times, and that shouldn't be terrible news. When I say go wrong, I mean not pan out rather than do harm. If it did harm, that's a different thing.
--:-- I think one thing this points to, by the way, is you could choose to take a strategy which looks something like minimax regret. You have a bunch of options, you can ask about the roughly worst-case outcome, or just the default outcome on each option. One strategy is just choose the option with the least bad meh case. If you take that strategy, you should expect to look back on the decisions you make and not see as many failures. That's one point in favor of it.
--:-- Another strategy is just do the best thing in expectation. If I made these decisions constantly, what in the long run just ends up making the world best? This looks a lot like just taking the highest EV option. Maybe you don't want to run the risk of causing harm, so that's okay to include. But I happen to think that the second strategy is very often going to be a lot better. And it's really important not to be misguided by this feature of the minimax regret strategy where you look back and feel a bit better about yourself in many cases, if that makes sense.
--:-- Dwarkesh Patel: Yeah, that's super interesting. I mean, if you think about backtesting in terms of models for the stock market, one thing that tends to happen is that a strategy of just trying to maximize returns from a given trade results very quickly in you going bankrupt, because sooner or later there will be a trade where you lose all your money. And so there's something called the Kelly criterion where you reserve a big portion of your money and you only bet with a certain part of it, which sounds more similar to the minimax regret thing here. Unless your expected value includes a possibility that—I mean, in this context, losing all your money is like an existential risk, right? So maybe you bake into the cake in the definition of expected value the odds of losing all your money.
--:-- Fin Moorhouse: Yeah, yeah, that's a great point. I guess in some cases you want to take something which looks a bit more like the Kelly bet. But if you're at the margins, relatively small margins compared to the pot of resources you have, then I think it often makes sense to just do the best thing and not worry too much about the size of the Kelly bet. But yeah, it's a great point. A naive version of doing this is just losing your bankroll very quickly because you've taken two enormous bets and forgotten that it might not pan out. So I appreciate that.
--:-- Dwarkesh Patel: What did you mean by "at the margins"?
--:-- Fin Moorhouse: If you think that there's a pool of resources from which you're drawing, which is something like philanthropic funding for the kind of work you're interested in doing, and you're only a relatively marginal actor, then that's unlike being an individual investor where you're more sensitive to the risk of just running out of money. When you're more like an individual investor, then you want to pay attention to what the size of the Kelly bet is. If you're acting at the margins, then maybe that is less of a big consideration, although it is obviously still a very important point.
--:-- Dwarkesh Patel: Well, and then, by the way, I don't know if you saw my recent blog post about why I think there'll be more EA billionaires.
--:-- Fin Moorhouse: Yes.
--:-- Dwarkesh Patel: Okay, yeah, yeah. I don't know what your reaction to any of the ideas there is, but my claim is that we should expect the total funds dedicated to EA to grow quite a lot.
--:-- Fin Moorhouse: Yeah, I think I really liked it, by the way. I think it was great. One thing it made me think of is that there's quite an important difference between trying to maximize returns for yourself and then trying to get the most returns just for the world. That is, just doing the most good. One consideration we've just talked about is the risk of just losing your bankroll, which is where Kelly betting becomes relevant.
--:-- Another consideration is that as an individual just trying to do the best for yourself, you have pretty steeply diminishing returns from money, or just how well your life goes with that extra money. If you have 10 million in the bank and you make another 10 million, does your life get twice as good? Obviously not. And as such, you should be kind of risk-averse when thinking about the possibility of making a load of money.
--:-- If, on the other hand, you just care about making the world go well, then the world's an extremely big place and so you basically don't run into these diminishing returns at all. For that reason, if you're making money at least in part to give it away or otherwise just have a positive effect in some impartial sense, then you're going to be less risk-averse, which means maybe you fail more often, but it also means that people who succeed succeed really hard. I don't know if that's in some sense just recycling what you said, but I think it's a neat observation.
--:-- Dwarkesh Patel: Well, and another interesting thing is that not only is that true, but you're also in a movement where everybody else has a similar idea. And not only is that true, but the movement is full of people who are young, techie, smart, and as you said, risk-neutral. So basically people who are going to be way overrepresented in the ranks of future billionaires. And they're all hanging out and they have this idea that we can become rich together and then make the world better by doing so. You would expect that this would be exactly the kind of situation that would lead to people teaming up and starting billion dollar companies.
--:-- All right. So a bunch of other topics in effective altruism that I wanted to ask you about. So one is: should it impact our decisions in any way if the many worlds interpretation of quantum mechanics is true? I know the argument that you can just translate amplitudes to probabilities, and if it's just probabilities then decision theory doesn't change. My problem with this is—I've gotten very lucky in the last few months. Now I think it changes my perception of that if I realize actually most mes—and I know there's problems with saying "mes" and to what extent they're fungible—most branches of the multiverse, I'm significantly worse off. That makes it worse than "oh, I just got lucky, but now I'm here."
--:-- Another thing is, if you think of existential risk and do think that even if existential risk is very likely, in some branch of the multiverse, humanity survives. I don't know, that seems better in the end than "oh, the probability was really low, but it just resolved to we didn't survive." Does that make sense?
--:-- Fin Moorhouse: Okay. All right. There's a lot there. I guess rather than doing a terrible job at trying to explain what this many worlds thing is about, maybe it's worth just pointing people towards just Googling it. I should also add this enormous caveat that I don't really know what I'm talking about. This is just an outsider who's taking this stuff as interesting.
--:-- So there's this question of what if the many worlds view is true? What, if anything, could that mean with respect to questions about what should we do or what's important? One thing I want to say is, without zooming into anything, it just seems like a huge deal. Every second, every day, I'm in some sense just dissolving into this cloud of mes, an unimaginably large number of mes, and each of those mes is in some sense dissolving into more cloud. This is just wild. Also seems somewhat likely to be true, as far as I can tell.
--:-- So what does this mean? You point out that you can talk about having a measure over worlds. In some sense, there's actually a problem of how you get probabilities or how you make sense of probabilities on the many worlds view. There's a kind of neat way of doing that which makes use of questions about how you should make decisions. You should just weigh future use according to how likely they are, but it's really the reverse—you're explaining what it means for them to be more likely in terms of how it's rational to weigh them.
--:-- Then I think there's a ton of very vague things I can try saying. Maybe I'll just try doing a brain dump of things you might think. Many worlds being true could push you towards being more risk-neutral in certain cases if you weren't before, because in certain cases you're translating from some chance of this thing happening or not into some fraction of worlds this thing does happen and another fraction it doesn't. I do think it's worth reading too much into that because I think a lot of the important uncertainties about the world are still subjective uncertainties about how most worlds will in fact turn out. But it's interesting and notable that you can convert between overall uncertainty about how things turn out to more certainty about the fraction of ways things turn out.
--:-- I think another interesting feature of this is that the question of how you should act is no longer the question of how should you benefit this person who is you in the future who's one person. It's more like how do you benefit this cloud of people who are all successors of you, just kind of diffusing into the future. I think you point out that you could basically salvage all decisions even if that's true, but the picture of what's going on changes. In particular, just intuitively, it feels to me like the gap between acting in a self-interested way and acting in an impartial way where you're helping other people kind of closes a little. You're already benefiting many people by doing the thing that's rational to benefit you, which isn't so far from benefiting people who aren't continuous with you in this special way. So I kind of like that as a thing.
--:-- Dwarkesh Patel: Hmm, it's interesting.
--:-- Fin Moorhouse: Yeah, and then there is also this slightly more out-there thought. Here's the thing: if many worlds is true, then there is at least a sense in which there are very, very many more people in the future compared to the past, just unimaginably many more. Even the next second from now, there are many more people. So you might think that should make us have a really steep negative discount rate on the future, which is to say we should value future times much more than present times, in a way which would just kind of explode how we should think about this.
--:-- This definitely doesn't seem right. Maybe one way to think about this is that if this thought was true or was kind of directionally true, then that might also be a reason for being extremely surprised that we're both speaking at an earlier time rather than a later time. Because if you think you're just randomly drawn from all the people who ever lived, it's absolutely mind-blowing that we get drawn from today rather than tomorrow, given that there are 10 to the something many more people tomorrow. So it's probably wrong, and wrong for reasons I don't have a very good handle on because I just don't know what I'm talking about. I can try parroting the reasons, but it's something I'm interested in trying to really crack those reasons a bit more.
--:-- Dwarkesh Patel: That's really interesting. I didn't think about that argument for the selection argument. I think one resolution I heard about this is that you can think of the proportion of Hilbert space, or the proportion of the universe's wave function, that could be the probability rather than each different branch. You know what? I just realized the selection argument you made—maybe that's an argument against Bostrom's idea of we're living in a simulation. Because basically his argument is that there will be many more simulations than there are real copies of you, therefore you're probably in a simulation. The thing about saying that all the simulations plus you, your prior should be equally distributed among them, seems similar to saying your prior of being distributed along each possible branch of the wave function should be the same. Whereas I think in the context of the wave function, you were arguing that maybe you shouldn't think about it that way. You should think about maybe a proportion of the total Hilbert space.
--:-- Does that make sense? I don't know. Wait, say it again, how it links into simulation type stuff. Instead of thinking about each possible simulation as an individual thing across which it is equally as likely—each individual instance of a simulation is equally as likely as you're living in the real world—maybe simulation as a whole is equally likely to you living in the real world. Just as you being alive today rather than tomorrow is equally likely, despite the fact that there will be many more branches of the wave function tomorrow.
--:-- Fin Moorhouse: Yeah, okay. There's a lot going on. I feel like there are people who actually know what they're talking about here, just tearing their hair out. Like, you've been doing this obvious thing.
--:-- Dwarkesh Patel: That's the nature of being a podcast. That's the point.
--:-- Fin Moorhouse: But by the way, if you are one such person, please do email me or DM me or something. I'm very interested. So yeah, you mentioned obviously there is a measure over worlds, and this lets you talk about things being sensible again. Also, maybe one minor thing to comment on is talking about probabilities is kind of hard because in many worlds, everything happens that can happen. So it's difficult to get the language exactly right. But anyway, totally get the point.
--:-- And then the question of how it maps onto simulation type thoughts... I know a maybe thought which kind of connects to this. Do you know sleeping beauty type problems?
--:-- Dwarkesh Patel: No, no.
--:-- Fin Moorhouse: Okay, only vaguely remembered example. But let's start. In the original sleeping beauty problem, you go to sleep, and then I flip a coin. If it comes up tails, they wake you up once. If it comes up heads, they wake you up once, and then you go back to sleep and your memory is wiped, and then you're woken up again as if you're being woken up in the other world. So you go to sleep, you wake up, and you're asked what is the chance that the coin came up heads or tails. And it feels like there's really intuitive reasons for both 50 and one third.
--:-- Here's a related question which is maybe a bit simpler, at least in my head. I flip a coin. It comes up heads, I make a world with one observer in it. If it comes up tails, I make a world with 100 observers in it. Maybe it could be like running simulations with 100 people. You wake up in one of these worlds. You don't know how many other people are there in the world. You just know that someone has flipped a coin and decided to make a world with either one or 100 people in it. What is the chance that you're in the world with 100 people? There's a reason for thinking it's half, and there's a reason for thinking that it's 100 over 101. Does that make sense?
--:-- Dwarkesh Patel: I understand the logic behind the half. What is the reason for thinking—I mean, regardless of where you ended up as the observer, it seems like if the odds are like the coin coming up—oh, I guess, is it because you'd expect there to be more observers in the other universe? Like, wait, yeah. So what is the logic for thinking it might be 100 over 101?
--:-- Fin Moorhouse: Well, you might think of it like this. How should I reason about where I am? Maybe it's something like this: I'm just a random observer, right?
--:-- Fin Moorhouse: Of all the possible observers that could have come out of this. And there are 101 possible observers. And you can just imagine that I've been like randomly drawn. If I'm randomly drawn from all the possible observers, then it's overwhelmingly likely that I'm in the big world.
--:-- Dwarkesh Patel: Huh, that's super interesting.
--:-- Fin Moorhouse: I should say, I should plug someone who does know what they're talking about on this, which is Joe Carl Smith, who has a series of really excellent blog posts. He's coming on the podcast next week.
--:-- Dwarkesh Patel: Yes.
--:-- Fin Moorhouse: He's going to talk to you about it. I don't want to scoop him, but one thought that comes from him which is really cool, just to kind of round this off: if you're a 100 over 101 on examples like this, and you think there's any chance that the universe is infinite in size, then you should think that the chance you're in a universe that is infinite in extent is just one, or close to one. If that makes sense.
--:-- Dwarkesh Patel: I see. Yeah, yeah. Okay, so in the end, does your awareness of many worlds provide a good explanation? Has that impacted your view of what should be done in any way?
--:-- Fin Moorhouse: I don't really know if I have a good answer. My best guess is that things just shake out to where they started, as long as you started off in a relatively risk-neutral place. I suspect that if many worlds is true, this might make it much harder to hold on to intuitive views about personal identity, because there isn't this one person who you're continuous with throughout time and no other people, which is how people tend to think about what it is to be a person. And then there's this vague thing where occasionally, every other month or so, I remember that maybe many worlds is true, and it blows my mind. I don't know what to do about it, and I just go about my day. That's about where I am.
--:-- Dwarkesh Patel: Okay. Other interesting topics to talk about: talent search. What is EA doing about identifying more people like you, basically? But maybe even people like you who are not in places where they're next to Oxford. If they're from China or India or something, what is EA doing to recruit more Fins from places where they might not otherwise work on EA?
--:-- Fin Moorhouse: It's a great question. To be clear, I just won the lottery on things going right to be lucky enough to do what I'm doing now. In some sense, the question is how do you print more winning lottery tickets and find those people who really deserve them but are currently not being identified? A lot of this comes from reading the book *Talent* by Tyler Cowen and Daniel Gross. There's something really powerful about this fact that the business of finding really smart, driven people and connecting them with opportunities to do the things they really want to do is still really inefficient. There are still so many people out there who aren't getting those opportunities.
--:-- I actually don't know if I have much more insight to add there, other than this is just a big deal. There's a sense in which it is an important consideration for this project of trying to do the most good. You really want to find people who can put these ideas into practice, and I think there's a special premium on that kind of person now, given that there's a lot of philanthropic funding ready to be deployed. There's also a sense in which this is a cause in its own right. It's kind of analogous to open borders in that sense, at least in my mind. I hadn't really appreciated it on a visceral level before I read that book.
--:-- Another thing the book talks about is you want to get them when they're young. You can really shape somebody's ideas about what's worth doing, and also their ambition about what they can do, if you catch them early. Tyler Cowan also had an interesting blog post a while back where he pointed out that a lot of people applying to his Emergent Ventures program, a lot of young people, are heavily influenced by effective altruism. That seems like it's going to be a very important factor in the long term. Eventually these people will be in positions of power.
--:-- Dwarkesh Patel: So maybe effective altruism is already succeeding to the extent that a lot of the most ambitious people in the world are identified that way. At least given the selection effect that the telecom program has. But what is it that can be done to get people when they're young?
--:-- Fin Moorhouse: It's a very good question. What you point out is right. Nick Whitaker has a blog post called "The Lamplight Model of Talent Curation." He draws this distinction between casting a very wide net that's very legibly prestigious and then filtering through thousands of applications, versus putting out the bat signal that in the first instance attracts the really promising people and maybe drives away people who would be a better fit for something else.
--:-- An example: if you were to hypothetically write a quite wonky economics blog every day for however many years and then run some program, you're automatically selecting for people who read that blog. That's a pretty good starting population to begin with. I really like that thought: not needing to be incredibly loud and prestigious sounding, but rather being quite honest about what this thing is about, so you just attract the people who really sort it out, because that's a good feature.
--:-- Another thing, and this is not a very interesting point to make, but something I've really realized the value of is having physical hubs. There's this model of running fellowships where you find really promising people and put them in the same place, surrounding them with people who are a bit more senior. You let this natural process happen where people get really excited that there is this community of people working on stuff that previously you had just been reading about in your bedroom on some blogs. As a source of motivation, it's less tangible than other things, but it's so powerful. It's probably one of the reasons I'm working here.
--:-- Dwarkesh Patel: It is one aspect of working from home that you don't get. Regarding the first point, maybe that should update in favor of not doing community outreach and community building. Maybe that's negative marginal utility. For example, there was an effective altruism group at my college that I didn't attend, and there's also an effective altruism group for the city as a whole in Austin that I don't attend. The reason is because I don't know the people. There is some sort of adverse selection here where the people who are leading organizations like this are people who couldn't just aren't directly doing the things that effective altruism says they might consider doing, and are more interested in the social aspects of altruism. I would be much less impressed with the movement if my first introduction to it was these specific groups that I personally interacted with, rather than just hearing Will MacAskill on a podcast. By the way, the fourth ladder was my first introduction to effective altruism.
--:-- Fin Moorhouse: Interesting. I feel like I really don't want to underwrite the job that community builders are doing. I think it has turned out to be incredibly valuable, especially looking at the numbers of what you can achieve as a group organizer at your university. You could change the course of more than one person's career over the course of a year of your time. That's pretty incredible.
--:-- Part of what's going on is that the difference between going to your local group or engaging with stuff online is that you get to choose the stuff you engage with. One upshot here is that there's a set of ideas that might get associated with EA that is very big, and you don't need to buy into all of it or be passionate about all of it. If this AI stuff seems interesting but other stuff is more peripheral, this could push towards wanting to have a specific group for people who think the AI stuff seems cool and other stuff isn't their cup of tea. In the future, as things get scaled up, having this differentiation and diversification of different groups seems pretty good. But just more of everything also seems good.
--:-- Dwarkesh Patel: I'm probably over-indexing on my own experience, given that I didn't actively interact with any of those communities. I'm probably not even informed on the experiences of others. But there was an interesting post on the effective altruism forum that somebody sent me where they were making the case that at their college, the EA community building stuff had a negative impact because people were kind of turned off by their peers. There's a difference between somebody like Sanjay Bakshi or William MacAskill advising you virtually to do these kinds of things, versus some sophomore at your university starting a philosophy club.
--:-- Fin Moorhouse: I think my guess is that these efforts are still overwhelmingly positive. But I think it's pretty interesting that people have the experience you describe as well, and it's interesting to think about ways to get around that.
--:-- Dwarkesh Patel: So, long reflection. It seems like a bad idea.
--:-- Fin Moorhouse: No, I'm so glad you asked. I want to say no. I think in some sense I've come around to it as an idea.
--:-- Dwarkesh Patel: Oh, really interesting.
--:-- Fin Moorhouse: Maybe it's worth trying to explain what's going on with this idea. If you zoom out really far over time and consider our place now in history, you could ask this question: suppose humanity became perfectly coordinated, what's the plan? What should we be prioritizing and in what stages?
--:-- You might say something like this: this moment in history, this century, looks wildly and unsustainably dangerous. So many things are happening at once, it's really hard to know how things will pan out, but it's possible to imagine things panning out really badly, badly enough to more or less end history. So before we can worry about longer-term considerations, let's just get our act together and make sure we don't mess things up. That seems like a pretty good first priority.
--:-- But suppose you succeed in that and we're in a significantly safer time. What then? We might notice that the scope for what we could achieve is really extraordinarily large, maybe larger than most people typically entertain. We could do a ton of really exceptional things. But also, in the not especially long-term future, we might for the first time be able to embark on these really ambitious projects that are in some important sense really hard to reverse.
--:-- That might make you think: at some point it'd be great to achieve our potential. A lower bound on this is lifting everyone out of poverty who remains in poverty, and then going even further, making everyone even wealthier, able to do more things they want to do, making more scientific discoveries. We want to do that. But maybe something should come in between these two things: figuring out what is actually good.
--:-- Why should we think this? One thought is that it's very plausible, and this links to what we were talking about earlier, that the way we think about really positive futures is really incomplete. Almost certainly we're getting a bunch of things wrong. By a pessimistic induction on the past, a bunch of smart people thought really reprehensible things 100 years ago, so we're getting things wrong. A second thought is that it seems possible to actually make progress here in thinking about what's good. Most work in moral philosophy has focused on the negatives: avoiding doing things wrong, fixing harms, avoiding bad outcomes. But this idea of studying the positive, studying what we should do if we can do many different things, is super early. We should expect to be able to make a ton of progress.
--:-- So, imagining the world is perfectly coordinated, would it be a good idea to spend some time, maybe a long period of time, deliberately holding back from embarking on these huge irreversible projects which might involve leaving Earth in certain scenarios, or otherwise doing things which are hard to undo? Should we spend some time thinking before then? Yeah, sounds good.
--:-- The very obvious response is: that's a pretty huge assumption, that we can just coordinate around that. The answer is yep, it is. But as a directional ideal, should we push towards or away from the idea of taking our time, holding our horses, getting people together who haven't really been part of this conversation, and hearing them? Definitely seems worthwhile.
--:-- Dwarkesh Patel: Okay, so I have another good abstract idea that I want to entertain by you. It seems kind of wasteful that we have these different companies that are building the same exact product. Because they're the same exact product, they don't have economies of scale and they don't have coordination. There's a whole bunch of loss that comes from that. Wouldn't it be better if we could just coordinate and figure out the best person to produce something together, and then just have them produce it? And we could also coordinate to figure out what is the right quantity and quality for them to produce. I'm not trying to say this is communism or something. I'm just saying it's ignoring what would be required. In this analogy, you're ignoring what kinds of information gets lost and what it requires to do that so-called coordination. In the communism example, whatever would be required to prevent somebody from realizing, let's say somebody has a vision for colonizing a star system or making some new technology, that's part of something that the long reflection would curtail. Maybe you're getting this wrong, but it seems like it would require almost a global panopticon totalitarian state to be able to prevent people from escaping the reflection.
--:-- Fin Moorhouse: There's a continuum here, and I basically agree that some kind of panopticon thing not only is impossible but actually sounds pretty bad. But something where you're just pushing in the direction of being more coordinated on the international level about things that matter seems desirable and possible. In particular, preventing really bad things rather than trying to get people to all do the same thing. The Biological Weapons Convention strikes me as an example which is imperfect and underfunded, but nonetheless directionally good.
--:-- An extra point here is that there's a sense in which the long reflection option, or the better framing is aiming for a bit more reflection rather than less, is the conservative option. It's doing what we've already been doing, just a bit longer, rather than some radical option. I agree it's pretty hard to imagine some super long period where everyone's perfectly agreed on doing this. But framing it as a directional ideal seems worthwhile. I'm maybe kind of naively hopeful about the possibility of coordinating better around things like that.
--:-- Dwarkesh Patel: There are two reasons why this seems like a bad idea to me. One is: who is going to be deciding when we come to a good consensus? We decide this is the way things should go, now we're ready to escape the long reflection and realize our vision for the rest of the lifespan of the universe. Who is going to be doing that? It's the people who are presumably in charge of the long reflection. Almost by definition, it'll be the people who have an incentive in preserving whatever power balances exist at the end of the long reflection.
--:-- The second thing is: there's a difference between having a consensus on not using biological weapons or something like that, where you're limiting a negative, versus when we've required society-wide consensus on what we should aim towards achieving. The outcome has not been good in history. It seems better on the positive end to just leave it open-ended and then maybe say that the very bad things we might want to restrict together.
--:-- Fin Moorhouse: I think I just agree with a lot of what you said. The best framing is the version where you're preventing something which most people can agree is negative, which is some actor unilaterally deciding to set out on this huge irreversible project.
--:-- Dwarkesh Patel: Something you said was that the outcome is going to reflect the values of whoever is in charge. And not just the values, but think about how guilds work. Whenever we let an industry decide how the industry should progress collectively by the people who are currently dominant in the industry, like guilds or industrial conspiracies, the outcome is bad. My prior would be that at the end of such a situation, our ideas about what we should do would actually be worse than going into the long reflection.
--:-- Fin Moorhouse: It really depends on how it's implemented, right?
--:-- Dwarkesh Patel: I'm not saying that, but just broadly given all possible implementations, and maybe the most likely implementation given how governments run now.
--:-- Fin Moorhouse: I should say that I am in fact pretty absurd. I just know it's more enjoyable to give this thing its hearing.
--:-- Dwarkesh Patel: No, I enjoy the parts where we have disagreements.
--:-- Fin Moorhouse: One thought here: if you're worried about the course of the future being determined by some single actor, that worry is just symmetrical with the worry of letting whoever wins some race first go and do the project where they more or less determine what happens to the rest of humanity. The option where you deliberately wait and let people have some global conversation seems less worrying, even if the worry is still there.
--:-- I should also say I can imagine the outcome is not unanimity. In fact, it'd be pretty wild if it was. But you want the outcome to be some kind of stable friendly disagreement, where we can reach some kind of Coasian solution and go and do our own things. There are a bunch of projects which kind of go off at once. That feels really great to me compared to whoever gets there first determining how things turn out. It's hard to talk about stuff because it's somewhat speculative, but I think it's a useful north star to try pointing towards.
--:-- Dwarkesh Patel: Maybe to make it more concrete, I wonder if your expectation that the consensus view would be better than the first mover view. In today's world, either we have the form of government and the industrial and logistical organization that Elon Musk has designed for Mars. If he's the first mover for Mars, would you prefer that? Or we have the UN come to a consensus between all the different countries about how we should have the first Mars colony organized? Would the Mars colony run better if after 10 or 20 years of that, they decide how the first Mars colony goes? Is the global consensus view better than the first mover view?
--:-- Fin Moorhouse: That's a good question. One obvious point is not always right. There are certainly cases where the consensus view is just somewhat worse. I think you limit the downside with a consensus view.
--:-- Fin Moorhouse: Because you give people space to express why they just don't think this one idea is bad. I don't know if this is not your question, but it's a really good one. You can imagine the kind of the UN-led thing is going to be way slower, it's going to probably be way more expensive. The International Space Station is a good example where, I don't know, I think that turned out pretty well, but a private version of that would have happened a lot more effectively.
--:-- I guess I'm not like the Elon example is kind of a good one because it's not obvious why that's super worrying. The thing I have in mind in the long reflection example is maybe a bit more wild, but it's really hard to make it concrete, so I'm somewhat floundering.
--:-- There's also another reason. To the extent that somebody has the resources—I don't know, maybe this just gets to an irreconcilable question about your priors about other kinds of political things—but to the extent that somebody has been able to build up resources privately to be able to be a first mover in a way that is going to matter for the long term, what do you think about what kind of views they're likely to have and what kind of competencies they're likely to have? Versus assuming that the way governments work and function and the quality of their governance doesn't change that much for the next 100 years, what kind of outcomes you will have?
--:-- Basically, if you think the likelihood of leaders like Donald Trump or Joe Biden is going to be similar for the next 100 years, and if you think the richest people in the world, the first movers, are going to be people that are similar to Elon Musk, I can see people having genuinely different, reasonable views about who should have the power to decide the long-run course of humanity. Should it be Elon Musk 100 years from now, or Joe Biden 100 years from now? Is that a fulcrum in this debate that you think is important, or is that maybe not as relevant as I might think?
--:-- Dwarkesh Patel: Yeah, I guess I'll try saying some things and maybe it'll respond to that. Two things are going through my head.
--:-- So one is something like: you should expect these questions about what we should do when we have the capacity to do a far larger range of things than we currently have the capacity to do. That question is going to hinge much more importantly on theories people have and worldviews and very kind of particular details, much more than it does now.
--:-- I'm going to do a bad job of trying to articulate this, but there's some kind of analogy here. If you're fitting a curve to some points, you can overfit it. In fact, you can overfit it in various ways and they all look pretty similar, but then if you extend the axis so you see what happens to the curves beyond the points, those different ways of fitting it can go all over the place. So there's some analogy here where when you expand the space of what we could possibly do, different views which look kind of similar right now, or at least come to similar conclusions, they just go all over the shop.
--:-- That is not responding to your point, but I think it's maybe worth saying. This is a reason for expecting reflecting on what the right view is to be quite important.
--:-- And then I guess I'll listen to a second thought, which is something like: I guess there's two things going on. One is the thing you mentioned, which is there are basically just a bunch of political dynamics where you can just reason about where you should expect values to head for political reasons. In some sense, is now better than the defaults? And what is that default?
--:-- Then there's a kind of different way of thinking about things, which is: separately from political dynamics, can we actually make progress and think better about what's best to do, in the same way that we can make progress in science, kind of separately from the fact that people's views about science are influenced by political dynamics? Maybe a disagreement here is a disagreement about how much scope there is to just get better at thinking about these things.
--:-- One reason I can give—I guess I kind of mentioned this earlier—is this project of thinking about what's best to do, maybe thinking better about ethics, is not the thing. It's maybe more relevant to think this is on the order of kind of 30 years old rather than on the order of 2000 years old. You might call it secular ethics. Parfit writes about this, he talks about this kind of thing. There are at least reasons for hope. We haven't ruled out that we can make a lot of progress. Because the thing we were doing before, like we were trying to think systematically about what's best to do, was just very unlike the thing that we should be interested in.
--:-- I'm sorry, that was like a huge ramble, but hopefully there's something there.
--:-- Fin Moorhouse: Yeah, I want to go back to what you were saying earlier about how you can think of a global consensus as the reduced variance version of future views. I think that to the extent that you think a downside is really bad, I think that's a good argument. And then, yeah, it's similar to my argument against monarchism, which is that actually, I think it is reasonable to expect that if you could reliably have people like Lee Kuan Yew who are in charge of your country and you have a monarchy, things might be better than a democracy. It's just that the bad outcome is just so bad that it's better just having a low variance thing like democracy. It's a fun one to talk about.
--:-- Maybe one last trailing thought on what you said. I think Popper has this thought, and also David Deutsch did a really good job of explaining it, about one underrated value of democracy. It's not just in some sense having this function to combine people's views into some kind of optimal path which is some mishmash where everyone thinks. It's also having the ability for people who are being governed to just cancel this current experiment in governance and try again. It's like, we'll give you freedom to implement this kind of governance plan that seems really exciting, and then we're just going to pull the brakes when it goes wrong. That kind of option to start again in general just feels like really important as some kind of tool you want in your toolkit when you're thinking about these pretty big futures.
--:-- Dwarkesh Patel: I guess my hesitation about this is I can't imagine a form of government where at the end of it, I would expect that a consensus view from—I mean, not just like nerdy communities like EA, but an actual global consensus—would be something that I think is a good path. Maybe it's something like I don't think it's the worst possible path. But one thing about reducing variance is if you think the far future could be really, really good, then by reducing variance, you're cutting off a lot of expected value.
--:-- And then you can think democracy works much better in cases where the problem is closer to something that the people can experience. Democracies don't have famines because if there's a famine you get voted out. Or you have major wars as well, but if you're talking about some form of consensus way of deciding what the far future should look like, it's not clear to me why the consensus view on that is likely to be correct.
--:-- Fin Moorhouse: Yeah, yeah, yeah. I think maybe what's going on here is I'd want to resist—and it's my fault for suggesting this framing—that you just spend a bunch of time thinking and having this conversation and then you just have this international vote on what we should do.
--:-- I think maybe another framing is something like: let's just give the time for the people who want to be involved in this to make the progress that could be possible on thinking about these things, and then just see where we end up. I know there's a very weak analogy to progress in other fields where we don't make progress in mathematics or science by taking enormous votes on what's true. But we can by just giving people who are interested in making progress the space and time to do that, and then at the end it's often pretty obvious what turns out.
--:-- That's very begging the question because it's way more obvious what's right and wrong if you're doing math compared to doing this kind of thing. But also, what happens if this seems similar to the question about monarchy where it's like: what happens if you pick the wrong person or the wrong politburo to pick the charter you take to the rest of the universe? It seems like a hard problem to ensure that you have the group of people who will be deciding this, either if it's a consensus or if it's a single person or anything in between. It has to be some decision maker.
--:-- Dwarkesh Patel: I think you can just imagine there being no decision maker. The thing could be: let's agree to have some time to reflect on what is best, and we might come to some... and then at the end, one version of this is just let things happen. There's no final decision. The time between doing the thing and thinking about it, just extending that time for a bit seems good.
--:-- I see. Okay, okay. Yeah, sorry I missed that earlier.
--:-- Okay, so actually one of the major things we're going to discuss—this is one of the quadrants of the conversation. Actually, before we talk about space governance, let's talk about podcasting. So you have your own podcast, I have my own. What have you... where did you start it and what have been your experiences so far? What have you learned about the joy and impact of podcasting?
--:-- Fin Moorhouse: The story is Luca, who's a close friend of mine who I do this podcast with. We're both at university together and we're both podcast nerds. I think I remember we were in our last year and we had this conversation. We're surrounded by all these people who just seem incredibly interesting, all these academics we really love to talk about or talk to. If we just emailed them saying we're doing a podcast and wanted to interview them, that could be a pretty good excuse to talk to them. So let's see how easy it is to do this.
--:-- Turns out the startup costs of doing a podcast are pretty low if you want to do a scrappy version of it. Did that. Turns out that academics especially, but just tons of people, really love being asked to talk about the things they think about all day. It's a complete win-win where you're trying to boost the ideas of someone who you think deserves more air time, and that person gets to talk about their work and spread their ideas. There's no downsides to doing this other than the time.
--:-- Also, I should say that the yes rate on our emails was considerably higher than we thought. We were two random undergrads with microphones. But this is a really nice snowball effect where if someone who is well known is gracious enough to say yes despite not really knowing what you're about, and then you do an interview and it's a pretty good interview, when you're emailing the next person you don't have to sell yourself. You can just be like, "Hey, I spoke to this other impressive person." So you get this snowball.
--:-- No, it's definitely a fancy scheme. It's great, it's the best kind of fancy scheme. Podcasts as a form of media are just incredibly special. There's something about the incentives between guest and host being aligned so much better than if this was some journalistic interview, which would be way more uncomfortable. There's something about the fact that it's still kind of hard to search transcripts, so there's less of a worry about forming all your words in the right way. So it's just more relaxed. Yeah, I recommend it.
--:-- Dwarkesh Patel: Yeah, no, it's such a natural form. You can think of writing as a sort of way of imitating conversation, and audiobooks is a way of trying to imitate a thing that's trying to imitate conversation. Audiobooks seem like they're supposed to... because you get the writing, you're visually perceiving what was originally an ability you had for understanding audible ideas. But then audiobooks, you're going through two layers of translation where you don't have the natural repetition and the ability to gauge the other person's reaction and the back and forth that an actual conversation has. That's why people potentially listen to podcasts too much. I don't know that they just have something in their ears the whole day, which you can't imagine for audiobooks.
--:-- Fin Moorhouse: Yeah, a few things this makes me think of. One is there's some experiment—you can just do it yourself—where if you force people not to use disfluencies, like "ums" and "ahs," those people just get much worse at reading words in some sense. Disfluencies help us communicate thoughts for some reason. If you take what I'm saying and word for word transcribe it, it's hot garbage. It's like I've just learned how to talk. But that pattern of speech, as you point out, is in fact easier to digest, or at least it requires less stamina or effort.
--:-- Nassim Taleb has an interesting point about this in *Antifragile*. I'm vaguely remembering this, but he makes a point that sometimes when a signal is distorted in some way, it makes you retain or absorb more of it because you have to go through extra effort to understand it. His example was if somebody is speaking but they're far away so their audio is muted, you have to apply more concentration, which means you retain more of their content. So if you overlay what someone says with a bit of noise or you turn the volume down, people have better comprehension because of the thing you just said: you're paying more attention.
--:-- Also, I think maybe I was misremembering the thing I mentioned earlier, or maybe it's a different thing. You can take perfect speech recordings and then you can insert "ums" and "ahs" and make it worse, and then you can do a comprehension test where people listen to the different versions and they can remember it. They do better with the versions which are less perfect.
--:-- Dwarkesh Patel: Is it just about having more space between words, or is it actually the "um"? If you just added space instead of "ums," would that have the same effect? Or is there something specific about... there's a limit to how much I can stretch some global psychology, some global consonant that just evokes absolute concentration.
--:-- Fin Moorhouse: Yeah, exactly.
--:-- Dwarkesh Patel: I'm curious to ask you, I want to know what you feel like you've learned from doing podcasting. What's some underappreciated difficulty of trying to ask good questions? I mean, you're obviously currently asking excellent questions, so what have you learned?
--:-- One thing I've heard is this advice: you want to do something where a thing that seems easier to you is difficult for other people. One obvious thing you can do is ask on Twitter, "Hey, I'm interviewing this person, what should I ask them?" And you'll observe that all the questions that people propose are terrible. But maybe it's just adverse selection: the people who actually could come up with good questions are not going to spend the time to reply to your tweet.
--:-- But then I've even tried to hire research partners or research assistants who can help me come up with questions more recently, and the questions they come up with also seem... it's like, "How did growing up in the Midwest change your views about blah blah?" It's just a question whose answer is not interesting. It's not a question you would organically have. I hope you wouldn't have organically wanted to ask them if you were only talking to them one-on-one.
--:-- So it does seem like the skill is harder, or rarer, than I would have expected. I don't know why. I don't know if you have a good sense of it, because you have an excellent podcast where you ask good questions. I don't know, what do you think? Have you observed this, where asking good questions is a rarer skill than you might think?
--:-- Dwarkesh Patel: Have you observed this where asking good questions is a rarer skill than you might think?
--:-- Fin Moorhouse: Certainly I've observed that it's really hard skill. I still feel like kind of... I still feel like it's really difficult. I also at least like to think that we've got a bit better.
--:-- First thing I thought was this example you gave of like "what was it like growing up in the Midwest?" We always ask those kinds of questions. So you know like "how did you get into behavioral economics and why do you think it was so important?" These are just like guaranteed to be kind of uninspiring answers. So specificity seems like a really good kind of...
--:-- Dwarkesh Patel: What, what is your book about? Yeah, exactly.
--:-- Fin Moorhouse: Exactly. "Tell us about yourself." This is why I love conversations with Tyler. It's one of the many reasons I love it. He'll just launch with the first question being about some footnote in this person's undergrad dissertation, and that just sets the tone so well.
--:-- Also, I think cutting off—which I've made very difficult for you, I guess—cutting off answers. What is the interesting thing that has been said? The elaboration or the caveats on the meat of the answer are often just like way less worth hearing.
--:-- I think I'm trying to ask questions which a person has no hope of knowing the answer to, even though it would be great if they knew the answer. Like "so what should we do about this policy?" is a pretty bad move.
--:-- Also, if you speak to people who are familiar with asking questions about their book for instance, in some sense you need to flush out the kind of pre-prepared spiel that they have in their heads. You could even just do this before the interview, right? And then it gets to the good stuff where they're actually being made to think about things.
--:-- Rob Wiblin has a really good list of interview tips which I think... I guess the reason this is kind of nice to talk about, other than the fact that this is good to have some inside baseball talk, is that skills of interviewing feel pretty transferable to just asking people good questions, which is a generally useful skill.
--:-- So yeah, I guess I've found that it's really difficult. I still get pretty frustrated with how hard it is, but it's a cool thing to realize that you are able to slowly learn.
--:-- Dwarkesh Patel: So how do you think about the value you're adding through your podcast, and what advice do you have for somebody who might want to start their own?
--:-- Fin Moorhouse: One reason you might think podcasts are really useful in general is—the way I think about this is—you can imagine there's a stock of ideas that seem really important. If you just have a conversation with someone who's researching some cool topic and they tell you all this cool stuff that isn't written up anywhere, you're like "oh my god, this needs to exist in the world."
--:-- I think in many cases, this stock of important ideas just grows faster than you're able to pay it down and put it out into the world. That's just a bad thing. So there's this overhang you want to fix. Then you can ask this question: what is the most effective way to communicate ideas relatively well and put them out into the world? Having a conversation with that person is just one of the most efficient ways of doing it.
--:-- I think it's interesting in general to consider the rate of information transfer for different kinds of media—transmitting and receiving ideas. On the best end of the spectrum, I'm sure you've had conversations where everyone you're talking with shares a lot of context, so you can just blurt out this slightly incoherent three-minute "I just had this thought in the shower" and they can fill in the gaps and basically just get the idea.
--:-- At the opposite end, maybe you want to write an article for a kind of prestigious outlet, so you're covering all your bases and making it really well written. The information per unit of effort is just so much lower. Certain kinds of academic papers are way out on the other side.
--:-- So as a way of solving this problem of there being an overhang of important ideas, podcasts just seem like a really good way to do that. When you don't successfully put ideas out into the world, you get these little clusters or fogs of contextual knowledge where everyone knows these ideas in the right circles, but they're hard to pick up from legible sources. It maps onto this idea of context being that thing which is scarce. I remember Tyler Cowen talking about that. It eventually made sense in that context.
--:-- Dwarkesh Patel: The thing you mentioned about either just head off on a podcast and explain your idea, or take the time to do it in a prestigious place—it seems very much like a barbell strategy. Whereas the middle ground of spending four or five hours writing a blog post where it's not going to be in something super prestigious, you might as well either just put it up in a podcast if it's a thing you just want to get over with, or spend a little more time getting into more prestigious places.
--:-- The argument against it, I guess, is that the idea seems more accessible if it's in the form of a blog post for posterity. If you want it to be the canonical source for something, you should just make it a more official thing. Because if it's just a YouTube clip, it's a little difficult for people to reference.
--:-- But you can get the best of both worlds. You can put your recording into transcription software. If you're lucky enough to have someone to help you with this, you can get someone—or you can do it yourself—to go through the podcast transcript to make sure there aren't any glaring mistakes. Now you have this artifact that is in text form that lives on the internet, and it's just way cheaper than writing it in the first place.
--:-- Fin Moorhouse: That's a great point. And also people should read your... "Barbells for Life" is that it? "Strategies for Life"?
--:-- Dwarkesh Patel: Yeah, that's it.
--:-- Fin Moorhouse: Cool. Maybe one last thing that seems worth saying on this topic of podcasting is that it's quite easy to start doing a podcast, and my guess is it's often worth at least trying. There are probably a few people listening to this who have entertained the idea.
--:-- One thing to say is it doesn't need to be the case that if you just stop doing it after five episodes or even fewer, it's a failure. You can frame it as: I wanted to make a small series. It's just a useful artifact to have in the world. Here's this bit of history that I think is underrated, I'm going to tell the story in four different hour-long episodes. If you set out to do that, then you have this self-contained chunk of work.
--:-- So yeah, maybe that's a useful framing. There's a bunch of resources which might be possible to link to on just how to set up a podcast. I tried writing, collecting some of those resources. The thing to emphasize, I think, is that I've talked to at least three or four people at this point who have told me "Oh, I have this idea for a podcast. It's going to be about architecture" or "It's going to be about VR" or whatever. They seem like good ideas. I'm not making the ideas themselves. But I talk to them six months later and they haven't started it yet. I just tell them: literally just email somebody right now, whoever you want to be your first guest. I cold emailed Bryan Caplan. He ended up being my first guest. Just email them and set something on the calendar.
--:-- Because I don't know what it is. Maybe just about life in general. I don't know if it's specific to podcasting, but the amount of people I've talked to who have vague plans of starting a podcast and have nothing scheduled or no immediate... I don't know what they're expecting, like some MP3 file to appear on their hard drive on some fine day. So yeah, just do it. Get it on the calendar now.
--:-- Dwarkesh Patel: That seems good. Also, there's some way of thinking about this where you could just write off in advance that your first seven episodes are just going to be embarrassing to listen to. That is more freeing because it probably is the case. You need to go through the bad episodes before you start getting good at anything. That's not even a podcast point.
--:-- Also, if you're just brief and polite, there's very little cost in being ambitious with the people you reach out to. So just go for it.
--:-- Bryan had this interesting argument about this somewhere. He was pointing out that actually the cost of cold emailing is much lower if you're an unknown quantity than if you're somebody who has somewhat of a reputation. Because if you're just nobody, they're going to forget you ever cold emailed them. They'll just ignore it in their inbox. If you ever run into them in the future, they won't have registered you the first time. If you're somebody who has somewhat of a reputation, then there's a mystery of "why are we not getting introduced to somebody who should know both of us?" if you claim to be a professor who wants to start a podcast. But anyways, just reinforcing the point that the cost is really low.
--:-- Fin Moorhouse: All right, cool.
--:-- Dwarkesh Patel: Okay, let's talk about space governance. This is an area where you've been writing about and researching recently.
--:-- Okay, one concern you might have: Toby Ord has that book, *The Precipice*, about how we're in this time of peril where we have a one in six odds of going extinct this century. Is there some reason to think that once we get to space, this will no longer be a problem? That the risk of extinction for humanity will asymptote to zero?
--:-- Fin Moorhouse: I think one point here... Actually, maybe it's worth beginning with a kind of naive case for thinking that spreading through space is just the ultimate hedge against extinction. You can imagine just duplicating civilization, or at least having civilizational backups. Things which are in different places in space. If the risk of any one of them being hit by an asteroid or otherwise encountering some existential catastrophe—if those risks are independent—then you get this exponential, power law effect where every new backup is like having multiple backups of some data in different places in the world. If those risks are independent, then going to space is just an incredibly good strategy.
--:-- I think there are pretty compelling reasons to think that a lot of the most worrying risks are really not independent at all. One example is you can imagine very dangerous pathogens. If there's any travel between these places, the pathogens are going to travel. But maybe the more pertinent example is if you think it's worth being worried about artificial general intelligence that is unaligned, that goes wrong and relentlessly pursues really terrible goals. Then just having some physical space between two different places is really not going to work as a real hedge.
--:-- So I'd say something like: space seems net useful to diversify, go to different places, but absolutely not sufficient for getting through this kind of time of perils.
--:-- Then there's this follow-up question: why expect that there is any hope of getting the risk down to sustainable levels? If you're sympathetic to the possibility of really transformative artificial general intelligence arriving, you might think that in some sense, getting that transition right—where the outcome is that now you have this thing on your side which has your interests in mind, or has good values in mind, but has this general purpose reasoning capability—in some sense this just tilts you towards being safe indefinitely long. One reason is if bad things pop up, like some unaligned thing, then you have this much better established safe and aligned thing which has this defensive advantage. That's one consideration.
--:-- If you're less sympathetic to this AI story, you'd also just tell a story about being optimistic for our capacity to catch up along some kind of wisdom or coordination dimension. If you really zoom out and look at how quickly we just invented all this insane technology that is roughly an exponential process, you might think that might eventually slow down, but our improvements in how well we're able to coordinate ourselves continues to increase. So you get this defensive advantage in the long run.
--:-- Those are two pretty weak arguments. So I think it's actually just a very good question to think about, and I acknowledge it's not a very compelling answer.
--:-- Dwarkesh Patel: I'm wondering if there are aspects that you can discern from first principles about the safety of space which suggests that either there's no good reason to think the time of peril ever ends. Because the thing about AI is that's true whether you go to space or not. If it's aligned, then I guess they can indefinitely reduce existential risk.
--:-- One thought you can have is, contra the long reflection thing we're talking about, if you think that one of the bottlenecks to a great future could be some sort of tyrannical... "Tyrannical" is a kind of coded term in terms of conventional political thought, but you know what I mean. Then the diversity of political models that being spread out would have, maybe that's a positive thing.
--:-- On the other hand, Gwern has this interesting blog post about space wars, where he points out that the logic of mutually assured destruction goes away in space. So maybe we should expect more conflict because it's hard to identify who the culprit is if an asteroid was redirected to your planet. And they can sweep it up sufficiently fast, they can basically destroy your above-ground civilization.
--:-- Is there something we can discern from first principles about how violent and how pleasant the time in space will be?
--:-- Fin Moorhouse: It's a really good question. I will say that I have not reflected on that question enough to give a really authoritative answer. Incidentally, one person who absolutely has is Anders Sandberg, who has been thinking about almost exactly these questions for a very long time. At some point in the future, I might have a book about this. So watch that space.
--:-- One consideration is that you can start at the end. You can consider what happens very far out in the future. It turns out that because the universe is expanding, for any point in space—if you just consider the next cluster over, or maybe even the next galaxy over—there'll be a time in the future where it's impossible to reach that other point in space no matter how long you have to get there. Even if you sent out a signal in the form of light, it would never reach there because there'll always be a time in the future where it starts expanding faster than the speed of light relative to that other place. So there's a small consolation there: if you last long enough to get to this kind of era of isolation, then suddenly you become independent again in the strict sense. I don't think that's especially relevant when we're considering relatively nearer term things.
--:-- Gwern's point is really nice. Gwern starts by pointing out that we have this logic with nuclear weapons on Earth, mutually assured destruction, where the emphasis is on a second strike. If I receive a first strike from someone else, I can identify who that first strike came from, and I can credibly commit to retaliating. The thought is that this disincentivizes that person from launching the first strike in the first place, which makes a ton of sense.
--:-- Gwern's point, the thing you already mentioned, is that in space there are reasons for thinking it's going to be much harder to attribute where a strike came from. That means that you don't have any kind of credible way to threaten a retaliation. So mutually assured destruction doesn't work. That's a bit of an uncomfortable thought because the alternative to mutually assured destruction is just first strike: if you're worried about some other actor being powerful enough to destroy you, then you should destroy their capacity to destroy you. So it's a slightly bleak blog post.
--:-- I think there are a ton of other considerations, some of which are a bit more hopeful. One is that you might imagine there is in general a kind of defensive advantage in space over offensive. One reason is that space is this dark canvas in 3D where there's absolutely nowhere to hide, so you can't sneak up on anyone.
--:-- But yeah, I think there's a lot of stuff to say here, and a little bit I don't quite fully understand yet. But I guess that makes it an interesting and important subject to be studying. We don't know that much about how it's going to turn out.
--:-- Dwarkesh Patel: Von Neumann has this vision that you would set up a virus-like probe that infests a planet and uses all of its usable resources to build more probes which go on to infect more planets. Is the long-run future of the universe that all the available low-hanging resources are burnt up in some sort of expanding fire of Von Neumann probes? Because it seems like as long as one person decides that this is something they want to do, the low-hanging fruit in terms of spreading out will just be burned up by somebody who built something like this.
--:-- Fin Moorhouse: That's a really good question. So, okay, maybe there's an analogy here where on Earth we have organisms which can convert raw resources plus sunlight into more of them, and they replicate. It's notable that they don't blanket the Earth. Although I do just as a tangent, I remember someone mentioned this thought: an alien arrives on Earth and asks the question "what is the most successful species?"
--:-- Fin Moorhouse: It would probably be grass. But again, the reason that particular organisms that just reproduce using sunlight don't just have this green goo dynamic is because there are competing organisms. There are things like antivirals and so on.
--:-- So I guess, as you mentioned, as soon as this thing gets seeded, it's game over. You can imagine trying to catch up with these things and stop them. I don't know what the equilibrium is where you have things that are trying to catch things and things which are also spreading. It's pretty unclear, but it's not clear that everything gets burned down, although I know it seems worth having on the table as a possible outcome.
--:-- Another thought is something you also basically mentioned. Robin Hanson has this paper called, I think, "Burning the Cosmic Commons." The things he says are a little bit subtle, but to bastardize the overall point, there's an idea that you should expect kind of selection effects on what you observe in the long run of which kinds of things have won out in this race for different parts of space. In particular, the things you should expect to win out are these things which burn resources very fast and are greedy in terms of grabbing as much space as possible. I don't know, that seems roughly correct.
--:-- He also has a more recent bit of work called "Grabby Aliens." There's a website, grabbyaliens.com, which expands on this point and asks this question about what we should expect to see from grabby civilizations.
--:-- One maybe slightly fanciful upshot here is you don't want this greedy, evil Von Neumann-type probes to win out, but also they're just dead, they have nothing of value. So if you think you have something of value to spread, maybe that is a reason to spread more quickly than you otherwise would have planned, once you've figured out what that thing is, if that makes sense.
--:-- Dwarkesh Patel: Yeah, so then does this militate towards the logic of a space race? Similar to the first strike where if you're not sure that you're going to retaliate, you want to do a first strike. Maybe there's a logic that as long as you have at least somewhat of a compelling vision of what the far future should look like, you should try to make sure it's you who's the first actor that goes out into space, even if you don't have everything sorted out, even if you have concerns about how...
--:-- Fin Moorhouse: Yeah, yeah. My guess is that the time scales on which these dynamics are relevant are extremely long time scales compared to what we're familiar with. So I don't think that any of this straightforwardly translates into wanting to speed up on the order of decades. In fact, if any delay on the order of decades or presumably also centuries gives you a marginal improvement in your long run speed, then just because of the time scales and the distances involved, you almost always want to take that trade-off.
--:-- So yeah, I'd be wary of reading too much into all this stuff in terms of what we should expect for some kind of race in the near term. It just turns out space is extremely big and there's a ton of stuff there. In anything like the near term, I think this reasoning about running out of useful resources probably won't kick in. But that's just me speculating. I don't have a clear answer to that.
--:-- Dwarkesh Patel: Okay, so if we're talking about space governance, is there any reason to think that in the far future, space will be colonized either by a fully artificial intelligence or by simulations of humans like M-s? In either case, it's not clear that these entities would feel constrained by whatever norms of space governance we detail now. What is the reason for thinking that any sort of charter or constitution that the UN might build, regardless of how sane it is, will be the basis on which the actual long run fate of space is decided upon?
--:-- Fin Moorhouse: Yeah, so the first thing I want to say is that it does feel like an extremely long shot to expect that any kind of norms you end up agreeing on now, even if they're good, flow through to the point where they really matter, if they ever do. But you can ask, what are the worlds in which this early thinking does end up being good on the endpoint? I don't know. I can imagine, for instance, the U.S. Constitution surviving in importance at least to some extent if digital people come along. It's not obvious why there's some discontinuity there.
--:-- I guess the important thing is considering what happens after anything like transformative artificial intelligence arrives. My guess is that the worlds in which this is even remotely relevant—like, super long-term, what norms should we have for settling space—the worlds in which this matters or does anything worthwhile are worlds where alignment goes well, right? And it goes well in the sense that there's a significant sense in which humans are still in the driving seat, and when they're looking for precedence, they just look to existing institutions and norms. There are so many variables here that this seems like a fairly narrow set of worlds, but it seems pretty possible.
--:-- And then it's also kind of like settling the moon or Mars, where it's just much easier to imagine how this stuff actually ends up influencing or positively influencing how things turn out. It feels worth pointing out that there are things that really plausibly matter when we're thinking about space that aren't just thinking about these crazy, very long-run sci-fi scenarios, although they are pretty fun to think about.
--:-- One is that there's a ton of pretty important infrastructure currently orbiting the Earth, and anti-satellite weapons are being built. My impression is that there is a worryingly small amount of agreement and regulation about the use of those weapons. Maybe that puts you in a kind of analogous position to not having many agreements over the use of nuclear weapons, although maybe less worrying in certain respects, but still it seems worth taking that seriously and thinking about how to make progress there.
--:-- Yeah, there are a ton of other near-term considerations. There's this great graph on Our World in Data, which I can send you the link to after this, which shows the number of objects launched into orbit, especially low Earth orbit, just over time. It's a perfect hockey stick. It's quite a nice illustration of why you might want to think about how to make sure this stuff goes well.
--:-- The story behind that graph is kind of fun as well. I was messing around on some UN website which had this incredible database with more or less every officially recorded launch log and all this data about how many objects were contained. It was the clunkiest API you've ever seen. You have to manually click through each page, it takes like five seconds to load, and you have to scrape it. I was like, "Okay, this is great that this exists, I am not remotely sophisticated enough to know how to make use of it." But I emailed the Our World in Data people saying, "FYI, this exists, if you happen to have a ton of time to burn, then have at it." And Ed from Our World in Data got back to me like a month later saying, "Hey, I had a free day, all done." And it's up on the website. It's so cool.
--:-- Okay, I think that's my space rambling. I'd quite like to ask you a couple questions if that's all right. I realize I've been kind of hogging the airwaves.
--:-- Dwarkesh Patel: Yeah, so here's one thing I've just been interested to know. You're doing this blogging and podcasting right now, but what's next? 2024? What is he doing?
--:-- Fin Moorhouse: I think I'll probably be... The idea of building a startup has been very compelling to me. Not necessarily because I think it's the most impactful thing that could possibly be done, although I think it is very impactful. But it's just something that's stuck in your head, like how people tend to have different things like "I want to be a doctor" or something. So yeah, I think that's probably what I'll be attempting to do in 2024.
--:-- I think the situation in which I remain a blogger and podcaster is if it just turns out to be really huge. At that point, it might make more sense. Currently, I think the impact the podcast has is like 0.0000001, and the 0.01 is just me getting to learn about a lot of different things. So for it to have any impact—not necessarily that it has to be thought of in terms of impact, but in terms of how useful it is—I think it's only the case if it really becomes much bigger.
--:-- Dwarkesh Patel: Nice, that sounds great. Maybe this is going to start that conversation. What about a non-profit startup? Is there the same excitement? If you have a great idea, you kind of skip the fundraising stage, more freedom because you don't need to make...
--:-- Fin Moorhouse: Well, no, you still have to raise money, right?
--:-- Dwarkesh Patel: Sure, but if it's a great idea, then I'm sure there'll be support to make it happen.
--:-- Fin Moorhouse: Yeah, if there's something where I don't see a way to profitably do it and I think it's very important that it gets done, I definitely wouldn't be opposed to it. Is that by the way where you're leading? Like, I asked you "What is Fin doing in 2024?" and you have a non-profit startup?
--:-- Dwarkesh Patel: I don't have something concrete in mind, but that kind of thing feels very exciting to me to at least try out. Gotcha. Yeah, I think my prior is that there are profitable ways to do many things if you're more creative about it. There are obvious counter examples of so many different things where I could not tell you how you could make that profitable. Like if you have something like One Day Sooner where they're trying to speed up challenge trials, how is that a startup? It's not clear. So yeah, I think there's a big branch of the decision tree where I think that's the most compelling thing I could do.
--:-- Fin Moorhouse: Nice. And maybe a connected question is, I'm curious what you think EA in general is underrating from your point of view. Also maybe another question you can answer is what you think I am personally getting wrong or got wrong, but maybe the more general question is a more interesting one for most people.
--:-- So I think when you have statements which are somewhat ephemeral or ambiguous—let's say there's some historian like Toynbee, right? He wrote "A Study of History" and one of the things he says in it is that civilizations die when the elites lose confidence in the norms that they're setting and they lose the confidence to rule. I don't think that's actually an existential risk, I'm just trying to use that as an example. It's the kind of thing that could be true, but I don't know how I would think about it using the modes of analyzing importance of topics that we've been using throughout this conversation. I don't know what that implies for you because it's not clear to me. Maybe EA shouldn't be taking things that are vague and ambiguous like that seriously to begin with.
--:-- Right. Yeah. If there is some interesting way to think about statements like that from a perspective that EAs could appreciate, including myself, I'd be really interested to see what that would be. Because there does seem to be a disconnect where when I talk to my friends who are intellectually inclined, who have a lot of interesting ideas, it requires a sort of translation layer, almost like a compiler or a transpiler that converts code from this language into assembly here. It does create a little bit of inefficiency and potentially a loss of topics that could be talked about.
--:-- Fin Moorhouse: Nice. That feels like a great answer. I just say it's something I'm kind of right about as well, especially in leaning towards a more kind of speculative, longtermist end. It seems really important to keep hold of some real truth-seeking attitudes where the obvious feedback of whether you're getting things right or wrong is much harder, and often you don't have the luxury of having that. So yeah, just keeping that attitude in mind seems very important.
--:-- I like that. What is your answer by the way? What do you think that you should improve on?
--:-- Dwarkesh Patel: Yeah, off the top of my head, maybe I have two answers which go in exact opposite directions. One answer is that a failure mode I'm a bit worried about is if the movement grows significantly, then the ideas that originally motivated it, that were quite new and exciting and important, somewhat dilute. Maybe it's related to what you said—you lose these attitudes of just taking weird ideas seriously, scrutinizing one another quite a lot, and it becomes a bit like greenwashing where the language stays but the real fire behind it of taking impact really seriously rather than just saying the right things fades away. I don't think I want to say EA is currently underrating that in any important sense, but it's something that seems worth having as a worry on the radar.
--:-- And then the roughly opposite thing that seems also worth worrying about is I think it's just really worth paying attention to best case outcomes when a lot of this stuff maybe grows quite considerably. Thinking about how this stuff could become mainstream, thinking about really scalable projects as well as just little fun interventions on margins—there's at least some chance that becomes very important. One part of that is maybe just learning to make a lot of these fields legible and attractive to people who could contribute who are learning about it for the first time. Just in general, planning for the best case, which could mean thinking in very ambitious terms, thinking about things going very well, that just also seems worth doing. So it's a very vague answer, but maybe that's my answer.
--:-- Perhaps opposite to what you were saying about not taking weird ideas too seriously in the future is maybe they are taking weird ideas too seriously now. It could be the case that just following basic common sense morality, kind of like what Tyler Cowen talks about in "Stubborn Attachments," is really the most effective way to deal with many threats, even weird threats. If you have areas that are more speculative like biorisk or AI, where it's not even clear that the things you're doing to address them are necessarily making them better—I know there's concern in the movement that the initial grant they gave to OpenAI might have sped up AI doom. Maybe the best case scenario in cases where there's a lot of ambiguity is to just do more common sense things. And maybe this is also applicable to things like global health where my layer nets are great, but the way that hundreds of millions of people have been left out of poverty is just through implementing capitalism, not through targeted interventions.
--:-- Again, I don't know what this implies for the movement in general. Even if just implementing the neoliberal agenda is the best way to decrease poverty, what does that mean somebody should do with a marginal million dollars? It's not clear to me. It's something I hope I'll know more about in five to ten years. I'd be very curious to talk to future me about what he thinks about common sense morality versus taking weird ideas seriously.
--:-- Fin Moorhouse: I think one way of thinking about weird ideas is that in some sense they are the result of taking a bunch of common sense starting points and then just really reflecting on them hard and seeing what comes out. So I think maybe the question is, how much trust should we place on those reflective processes? What should my prior be on weird ideas being true because they're weird? Is that good or bad?
--:-- And then separately, one thing that just seems obvious and important is if you take these ideas, first of all you should ask yourself whether you actually believe them or whether they are kind of fun to say, or you're just kind of saying that you believe them. Sometimes it's fun to say weird ideas, but it's like, okay, I actually don't have good grounds to believe this. And second of all, if you do in fact believe something, it's really valuable to ask if you think this thing is really important and true, why aren't you working on it if you have the opportunity to work on it? This is the Hamming question, right? "What's the most important thing in your fields?" and then "What's stopping you from working on it?" Obviously many people don't have the luxury of dropping out everything and working on the things they believe are really important, but if you do have that opportunity, that's a question which is maybe just valuable to ask.
--:-- Dwarkesh Patel: Maybe this is a meta objection to EA, which is that I'm aware of a lot of potential objections to EA like the ones we were just talking about, but there are so many other ones where people will identify an interesting point and then nobody knows what to do about it. It's like, "Oh, you know, should we take common sense morality more seriously or weird ideas more seriously?" It's like, "Oh, that is an interesting debate." But then how do you resolve that? I don't know how to resolve that. I don't know if somebody's come up with a good way to resolve that.
--:-- Fin Moorhouse: I guess it kind of hooks into the long reflection stuff a little bit because one answer here is just time. I think the story of people raising concerns about AI is maybe instructive here. Early on, you get some real radical out-there researchers or writers who are raising this as a worry. There's a lot of weird baggage attached to what they write. Then maybe you get a first book or two, and then you get more prestigious or established people expressing concerns. I think one way to accelerate that process when it's worth accelerating is just to ask that question: "Do I in fact see, can I go along with this argument? Do I see a hole in it?" And if the answer is no, if it just kind of checks out, even if you're always going to be uncertain, but if it seems kind of reasonable, then by default you might just spend a few years living like, "Oh yeah, this is a thing that I guess I think is true, but I'm not really acting on it." You can't just skip that step and be like, "Well, just act on it now."
--:-- Dwarkesh Patel: I'm not sure I agree. Maybe an analogy here is like, you're in a relationship and you think, "Oh well, I don't see what's wrong with this relationship." So instead of just waiting a few years to try to find something wrong with it, might as well just tie the knot now and get married. I think it's something similar. A failure mode—maybe not because EA would see it, but we can see generally in the world—is that people just come to conclusions about how the world works or how the world ought to work too early in life, when they don't seem to know that much about what is optimal and what is possible.
--:-- Fin Moorhouse: That's a great point. So yeah, maybe they should just wait a little longer. Maybe just integrate these weird radical ideas as things that exist in the world and wait until you're in your late 20s until you decide, "Actually, this is the thing I should do with the rest of my career or with my political rights or whatever."
--:-- Dwarkesh Patel: Yeah, I think that's actually a really good point. I think maybe I'd want to walk back what I said based on that. But I think there's some version of it which I'd still really endorse, which is maybe you've spent some time reflecting on this such that you don't expect further reflection is going to radically change what you think. You can maybe talk about this being the case for a group of people rather than a particular person. I could just really see this thing playing out where I just believe it's important for a really long time without acting on it, and that's the thing which seems worth skipping.
--:-- To be a little bit more concrete, if you really think some of these potentially catastrophic risks are real and you think there are things that we can do about it, then sure, it seems good to start working on this stuff. You really want to avoid that regret of some years down the line being like, "Oh, I really could have just started working on that earlier." There are occasions where this kind of thinking is useful, or at least asking this question: "What would I do right now if I just did what my idealized self would endorse doing?" Maybe that's useful.
--:-- Fin Moorhouse: So it seems that if you're trying to pursue a career related to EA, there are two steps. The first step is you have to get a position like the one you have right now where you're learning a lot and figuring out future steps. And then the one after that is where you actually lead or take ownership of a specific project, like a nonprofit startup or something. Do you have any advice for somebody who is before step one?
--:-- Dwarkesh Patel: Hmm, that's a really good question. I also will just do the annoying thing of saying, definitely other things you can do other than that kind of two-step trajectory.
--:-- Fin Moorhouse: As in go directly to step two? Or just never go to step two and just be a really excellent researcher or communicator or anything else?
--:-- Dwarkesh Patel: Sure, sure. I think where you have the luxury of doing it, not kind of rushing into the most salient career option and then retroactively justifying why it was the correct option, is quite a nice thing to bear in mind. I suppose often it's quite uncomfortable.
--:-- Fin Moorhouse: Do you mean something like consulting?
--:-- Dwarkesh Patel: Yeah, something like that.
--:-- Fin Moorhouse: Yeah, I mean, the obvious advice here is that there is a website designed to answer this question, which is 80,000 Hours.
--:-- Dwarkesh Patel: Oh, yeah.
--:-- Fin Moorhouse: So maybe some particular bit of advice from 80,000 Hours which I found very useful. After I left uni, I was really unsure what I wanted to do. I was choosing between a couple of options. And I was like, oh my God, this is such a big decision because in this context, not only do you have to answer the question of what might be a good fit for me and what I might enjoy, but also sometimes what is actually most important, and how am I supposed to answer that given that there's a ton of disagreement?
--:-- I just found myself bashing my head against the wall trying to get to a point where I discerned that one option was better than the other. The piece of advice I found useful was that often you should just write off the possibility of becoming fully certain about what option is best. Instead, you should reflect on the decision proactively. Talk to people, write down your thoughts, and just keep iterating on that until the dial stops moving backwards and forwards and just kind of settles on some particular uncertainty.
--:-- So it's like, look, I guess I'm 60-70% that option A is better than B, and that hasn't really changed having done a bunch of extra thinking. That's roughly the point where it might be best to make the decision rather than holding out for certainty. Does that make sense?
--:-- Dwarkesh Patel: Yeah, it's like gradient descent where if the loss function hasn't changed in the last iteration, you call it. That's super interesting though. I guess one problem maybe that somebody might face is that before they've actually done things, it's hard to know.
--:-- Like, this is actually going to be my career, but the podcast was just something I did as in I was bored during COVID. The classes went online and I just didn't have anything else to do. I don't think it's something I would have pursued if I ever thought of it as a career. But just doing things like that can potentially lead you down interesting paths.
--:-- Fin Moorhouse: Yeah, I think that's a great point. We're both involved with this blog prize, and there was a kind of mini prize last month for people writing about the idea of agency. What you just said links into that really nicely. There's this property of going from realizing you can do something to doing it, which seems both really valuable and learnable. Just having that going from the idea of "I could maybe do a little podcast series" to actually testing it and being open to the possibility that it fails but you learn something from it is really valuable.
--:-- Also, we were talking about sending cold emails in that same bit of the conversation. If there's someone you look up to and you think it's very plausible that you might end up in their line of research, and you think there's a bunch of things you could learn from them, as long as you're not demanding a huge amount of their time or attention, then you can just ask to talk to them. Finding a mentor in places like this is just so useful. Just asking people if they could fill that role in a kind of friendly way. Maybe it's a move people don't opt for a lot of the time, but it's taking the non-obvious options, being proactive about connecting to other people, seeing if you can physically meet other people who are interested in the same kind of weird things as you.
--:-- This is all extremely obvious, but I guess it's stuff I would have really benefited from learning earlier on.
--:-- Dwarkesh Patel: Yeah. And the unfortunate thing is it's not clear how you should apply that in your own circumstance when you're trying to decide what to do.
--:-- Okay, let's close out by talking about plugging the Effective Ideas blog prize you just mentioned and the Red Teaming EA contest. You already mentioned that earlier. If you just want to leave links and summarize them, I'd appreciate that.
--:-- Fin Moorhouse: Yeah, so the criticism contest, the deadline is the 1st of September. The canonical post that announces that is an EA Forum post, which I'd be very grateful if you could link to somewhere. The prize pool is at least a hundred thousand dollars, but possibly more if there are just a lot of exceptional entries. Hopefully all the relevant information is there.
--:-- And then this blog prize as well, which I've been helping run. I think you mentioned it right at the start. The overall prize is a hundred thousand dollars and up to five of those prizes, but also there are these smaller monthly prizes. Last month the theme was agency. The theme this month is to write some response or some reflection on this series of blog posts called "The Most Important Century" by Holden Karnofsky. Incidentally, people should just read that anyway. I think it's truly excellent and kind of remarkable that one of the most affecting series of blog posts I've basically ever read was written by the co-CEO of this enormous philanthropic organization in his spare time. It's just kind of insane.
--:-- The website is effectiveideas.org.
--:-- Dwarkesh Patel: And then where can people find you? Your website, Twitter handle, and where can people find your podcast?
--:-- Fin Moorhouse: Oh yeah. Website is myname.com. Twitter is my name. And the podcast is called Hear This Idea, as in "listen, this idea." So it's just that phrase calm. I'm sure if you Google it, it'll come up.
--:-- Dwarkesh Patel: By the way, what is your distribution of how impactful these criticisms end up being? Or just how good they end up being? If you had to guess, what is your median outcome? And then what is your 99th or 90th percentile outcome of how good these end up being?
--:-- Fin Moorhouse: Yeah, okay, that's a good question. I feel like I want to say that doing this stuff is really hard, so I don't want to discourage posting by saying this, but I think maybe the median submission is really robustly useful, absolutely worth writing and submitting. That said, maybe the difference between the most valuable posts of this kind and the median kind of effort is probably very large. Which is to say that the ceiling is really high. If you think you have a one percent chance of influencing 100 million dollars of philanthropic spending, then there is some sense in which an impartial philanthropic donor might be willing to spend roughly one percent of that amount to find out that information. Which is like a million dollars. So this stuff can be really, really important.
--:-- Dwarkesh Patel: Yeah, okay, excellent. The stuff you're working on seems really interesting, and the blog prizes seem like they might have a potentially very big impact. I mean, our worldviews have been shaped so much by some of these bloggers we've talked about. So if this leads to one more of those, that alone could be very valuable.
--:-- Fin, thanks so much for coming on the podcast. This was the longest, but also one of the most fun conversations I've gotten a chance to do.
--:-- Fin Moorhouse: The whole thing was so much fun. Thanks so much for having me.
--:-- Dwarkesh Patel: Thanks for watching. I hope you enjoyed that episode. If you did and you want to support the podcast, the most helpful thing you can do is share it on social media and with your friends. Other than that, please like and subscribe on YouTube and leave good reviews on podcast platforms. Cheers. I'll see you next time. Thank you.
