The main topic of this podcast episode is a conversation with Joe Carlsmith, a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford. Joe discusses various topics, including his work on existential risk from artificial intelligence, his blog on philosophy and the future, and his thoughts on utopia and long-termism.
Key points from the conversation include:
Takeaways from the conversation include:
Overall, this conversation with Joe Carlsmith offers a thought-provoking exploration of various philosophical and futurist ideas, and highlights the importance of considering the long-term implications of our actions and the potential for a better future.
Joseph Carlsmith is a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford.
We discuss utopia, artificial intelligence, computational power of the brain, infinite ethics, learning from the fact that you exist, perils of futurism, and blogging.
Watch on YouTube. Listen on Spotify, Apple Podcasts, etc.
Episode website + Transcript here.
Follow Joseph on Twitter. Follow me on Twitter.
Subscribe to find out about future episodes!
Timestamps
(0:00:06) - Introduction
(0:02:53) - How to Define a Better Future?
(0:09:19) - Utopia
(0:25:12) - Robin Hanson’s EMs
(0:27:35) - Human Computational Capacity
(0:34:15) - FLOPS to Emulate Human Cognition?
(0:40:15) - Infinite Ethics
(1:00:51) - SIA vs SSA
(1:17:53) - Futurism & Unreality
(1:23:36) - Blogging & Productivity
(1:28:43) - Book Recommendations
(1:30:04) - Conclusion
Please share if you enjoyed this episode! Helps out a ton!
Dwarkesh Patel
Today, I have the pleasure of interviewing Joe Carlsmith, who's a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford. Joe has a really interesting blog that I got to check out called Hands and Cities. That's the reason that I wanted to have him on the podcast, because it has a bunch of thought-provoking and insightful posts on there about philosophy, morality, ethics, and the future. I really wanted to talk to you, Joe. Do you want to give a bit of a longer intro on what you're up to?
Joe Carlsmith
Sure. So I work at Open Philanthropy on existential risk from artificial intelligence. I think about what's going to happen with AI, how can we make sure it goes well, and in particular, how can we make sure that advanced AI systems are safe? And then I have a side project, which is this blog, where I write about philosophy and the future and things like that. That emerges partly from my background. Before getting into AI and working at Open Philanthropy, I was in academic philosophy.
Dwarkesh Patel
That's a quite ambitious side project, given the length and the regularity of those posts. It's actually quite stunning. Do you want to talk more about what you're working on about AI at Open Philanthropy?
Joe Carlsmith
So it's a mix of things. Right now, I'm thinking about AI timelines and what's called takeoff speeds—sort of how fast the transition is from pretty impressive AI systems to AI systems that are radically transformative. I'm trying to use that to provide more perspective on the probability that everything goes terribly wrong.
Dwarkesh Patel
I see. And so what are the implications? I suppose it's higher or lower than I would expect. If it's higher, maybe I should work on AI safety. But other than that, what are the implications of that figure changing?
Joe Carlsmith
I think there are a number of implications just from understanding timelines with respect to how you prioritize. To some extent, the sooner something is, then you need to be planning for it coming sooner and cutting more corners or counting less on having more time. Overall, the higher you think the probability of catastrophe is, the easier it is for this to become the most important priority. I do think there is a range of probabilities where it maybe doesn't matter that much. But the difference between, say, 1% and 10% is quite substantive. And the difference between 10% and 90% is quite substantive. I know people in all of those ranges.
Dwarkesh Patel
Gotcha. Okay. Interesting. So let's back up here and talk a bit more about the philosophy motivating this. I think you identify as a long-termist. So maybe a broad picture question here is you have an interesting blog post about why the future looking back on us might think about the 21st century, given the risk we're taking. What do you think about the possibility that we're potentially giving up resources, potentially dedicating your career to building a future that, given the fact that you're alive now, you might find strange or disturbing or disgusting? To add more context to the question, from a utilitarian perspective, the present is clearly much better than the past, but somebody from the past might think that there are a lot of things about the present that are disturbing. They might not like the configuration of how isolating a modern city might be. They might find the kinds of free or cheap information you can access on your phone disturbing. How do you think about that?
Joe Carlsmith
A few comments there. One, I do think that if you took most people throughout history and brought them to the present day, my guess is they would fairly quickly come to prefer living in the present day to the past, even if there is a bit of future shock and some things are alienating or disturbing. That said, I think the gap between historical humans and the present is actually much smaller, both in terms of time and other factors, than the gap I envision between present-day humans and the future humans who are living in a radically better situation. So I do expect greater distance and possibly greater alienation when you first show up.
My personal view is that the best futures are going to be such that if you really understood them and if you really experienced what they are like—which may be a big step and might require extensive engagement and possibly changes to your capacities to understand and experience—then you would think it's really good. I think that's the relevant standard. So for me, I worry less if the future is initially alienating. The question for me is how do I feel once I've really understood what's going on?
Dwarkesh Patel
I wonder how much we should value that inside view you would get from being there. If you think about many existing ideologies—an Islamist might say, if you could come to Iraq and feel the bliss of fighting for the caliphate, you would understand better than you can from the outside view, sitting on a couch eating Doritos, what it's like to fight for a cause. Maybe their experience is blissful in some way. But I feel like the outside view is more useful than the inside view there.
Joe Carlsmith
I think there are a couple of different questions there. One is, what would the experience be if you had it from the inside? And then there is a subtly different question: what would your take on this be if you fully understood? Fully understanding is not just a matter of having the internal experience of being in a certain situation, but also understanding what that situation is causing, what beliefs are structuring the ideology, whether those beliefs are true, and all sorts of other factors. It's the latter thing that I have in mind. I'm not just imagining that the future will feel good if you're there. By hypothesis, the people who are there—at least one hopes they're enjoying it, or one hopes they're thumbs up. If the people who are there aren't thumbs up, that's a strange utopia. But I'm thinking that in addition to their perspective, there is a more holistic perspective, which is the full understanding. That's the perspective from which you would endorse this situation.
Dwarkesh Patel
Another respect in which it's interesting to think about what they might think of us is what they would think of the crazy risk we're taking by not optimizing for existential risks. One analogy you could offer, what McCaskill does in his new book, is to think of us as teenagers in our civilization's history. You think about the crazy things you did as a teenager. Maybe there is an aspect to which one would wish they could take back the crazy things they did. But my impression is that most adults probably think that while the crazy things were risky, they were formative and important, and they feel nostalgic about the things they did in the past. Do you think the future looking back will regret the way we're living in the 21st century, or will they look back and think it was a cool time? I guess this is conditional on there being a future, which takes away a lot of the mystery.
Joe Carlsmith
I doubt that they will look back with pleasure at the risks and horrors of the 21st century. If you think about how I tend to think about something like the Cuban Missile Crisis or World War II, I don't personally have nostalgia—"Sure it was risky, but it made me who I am." I also think it's true that when you look back on your teenage years, there is often a sort of nostalgia. Let's say you did something crazy, like you and your friends used to race around and play chicken at the local quarry. You survived. The real reason not to do that is the chunk of probability where you just died. To some extent, the ex-post perspective of looking back on certain risks is not the right one to use to calibrate your understanding of how to feel about it overall, especially for death risks.
Dwarkesh Patel
I see. Okay. So I think you brought up utopia, and you have a really interesting post about the concept of utopia. Do you want to talk a little bit more about this concept and why it's important? And also, why do we have so much trouble thinking of a compelling utopia?
Joe Carlsmith
Utopia for me just means a profoundly better future. I think it's important because I think it's actually possible. We could, if we play our cards right in non-crazy ways, build a world that is radically better than the world we live in today. In particular, I think we often underestimate just how big the difference in value could be between our current situation and what's available. Utopias are often anchored too hard on the status quo, changing it in small ways but imagining our fundamental situation unaltered. It's a little bit like the difference between having a crappy job or a beach vacation, and utopia is like everyone has a beach vacation. But I think the difference is more like being asleep and being awake, or living in a cave and living under the open sky. I think it's a really big difference, and that matters a lot.
Dwarkesh Patel
That's interesting because in the essay, you had a section where you mentioned that you expect utopia to be recognizable to a person alive now. The way you put it just earlier made it seem like it would be a completely different category of experience than we would be familiar with. Is there a contradiction there, or am I missing something?
Joe Carlsmith
I think there is at least a tension. The way I see the tension playing out is specifically via the notion I referenced earlier: if you truly understood, you would come to see the utopia as genuinely good. But I think the way we end up building utopia is by going through a long, patient process of becoming wiser and better and more capable as a species. It's in virtue of that process culminating that we're in a position to build a civilization that is profoundly good and radically different. That's a long process. So if I just transported you right there and you skipped the process, you might not like it. It is quite alien in some sense. But if you went through the process of really understanding and becoming wiser, you would endorse it.
Dwarkesh Patel
That's interesting to me that you think the process to get to utopia is more of a philosophical process. When you mentioned it's a process of us getting wiser, it sounds like it's a more philosophical process rather than figuring out how to convert everything to hedonium and it's eternal bliss from then on. Am I getting it right that you think it's more of a philosophical process? And why is it that you think so?
Joe Carlsmith
I definitely don't sit around thinking that we know what utopia is right now and it's hedonium. I'm not especially into the notion of hedonium. I think the brand is bad. People talk about pleasure with a dismissive attitude sometimes. Hedonium implies a sterile uniformity—people talk about tiling the universe with hedonium, and it sounds rough. Whereas I think the relevant perspective when thinking about something like hedonium is the internal perspective from which the experience of the subject is joyful, boundless, and energizing. Pleasure is not a trivial thing. I think pleasure is a profound thing in a lot of ways. But I really don't assume that's what utopia is about at all. My own values seem to be quite complicated. I don't think I just value pleasure. I value a lot of different things. More broadly, I have a lot of uncertainty about how I will think and feel if I were to go through a process of significantly increasing my capacity to understand.
Sometimes when people imagine that, they imagine we're going to sit around and do a bunch of philosophy, solve normative ethics, and then implement our solution. That's not what I'm imagining by wisdom. I'm imagining something richer that importantly involves an enhancement to our cognitive capacity. We are really limited in our ability to understand the universe right now. There is a huge amount of uncharted territory in terms of what minds can be and do and see. I want to chart that territory before we start making big and irreversible decisions about what sort of civilization we want to build in the long term.
Dwarkesh Patel
Another maybe concerning part of utopia is that, as you mentioned in the piece, many of the worst ideologies in history have had elements of utopian thinking in them. To the extent that EA and utilitarianism generally are compatible with utopian thinking—maybe they don't advocate it, but they are compatible with it—do you see that as a problem for the movement's health and potential impact?
Joe Carlsmith
Is the question something like, is this a red flag? We look at other ideologies throughout history and they've been compatible with utopian thinking, and effective altruism or utilitarianism is similarly compatible. Should we worry in the same way?
Dwarkesh Patel
Partly. And also, maybe it's still right that morally speaking, utopia is compatible with this worldview and the worldview is correct. But the implications are that somebody misunderstands what is best, identifies as an EA, and this leads to bad consequences when they try to implement their scheme.
Joe Carlsmith
I think there are certainly reasons to be cautious in this broad vein. I don't see them as very specific to EA or utilitarianism. I see them as risks that come from believing that something is very important at all. It's true that many acting from a space of conviction, especially where that conviction has a flavor of ideology—and it's reasonable to look at EA and think, this looks like an ideology—I think that's right. I think it's important to have the relevant red flags about. It's pretty hard to have a view of the world that doesn't in some sense imply that it could be a lot better. When I say utopia, I don't mean anything much different from that.
I'm not saying a perfect thing. I have a more specific view about exactly how much better things could be, but more broadly, many people believe in the possibility of a much better world and are fighting for that in different ways. I wouldn't pin the red flag specifically to the belief that things can be better. It would have more to do with what degree of rigidness you are relating to that belief with, how you are acting on it in the world, and how much you are willing to break things or act in uncooperative ways in virtue of that conviction. There, caution is definitely warranted.
Dwarkesh Patel
I'm not sure I agree that most people have a view or an ideology that implies anywhere close to the kind of utopia that utopian thinking can have. If you think of modern political parties in a developed democracy, like in the United States, a utopian vision from either party is quite banal. It's like, we'll have universal healthcare, or GDP will be higher in the next couple of decades. That doesn't seem utopian to me. It seems like a limited worldview where they're not really thinking about how much better or worse things could be. It doesn't exactly seem utopian.
Joe Carlsmith
That's a good point. Maybe the relevant notion of utopian here is something like, to what extent is a concept of a radically better world operative in your day-to-day engagement? To some extent, what I meant is that if I sat down and talked with most people, we could eventually, with some constraints on reasonableness, come to agree that things could be a lot better. We could cure cancer, cure XYZ disease, talk about the degree of abundance that could be available. But the question is whether that is a structuring or important dimension to how people are relating to the world. You're right that it's often not, and that's part of what I'm hoping to push back against with that post. I think this is a really important feature of our situation. It can be dangerous, and if you're wrong about it or acting in an unwise way with respect to it, that can be really bad. But I also think it's a really basic fact, and we need to learn to deal with it maturely. Pretending it's not true isn't the way to do that.
Dwarkesh Patel
To me, utopia sounds like some sort of peak. Maybe you didn't mean it this way. Are you saying in the essay and generally that you think there is some sort of carrying capacity to how much good things can get, or that beyond a certain point, things can keep getting indefinitely better, but at this point we're willing to say we have reached utopia?
Joe Carlsmith
I certainly don't have a hard threshold where I'm going to call it utopia. I mean something that is profoundly better. If you have a finite number of states that the affectable universe can be in, and your ranking of these states in terms of how good they are is transitive and complete, then there will be a top. I don't think that's an important thing to focus on from the perspective of taking seriously that things could be radically better at all. Talking about exactly how good and what's the perfect thing is often distracting. It gets into issues about how much suffering is good to have. A lot of this discourse on utopia gets distracted from basic facts: at the very least, we can do a ton better. That's important to keep in mind.
Dwarkesh Patel
You point out in the piece that many religions and spiritual movements have done the most amount of thinking on what a utopia could look like. There is a very interesting essay by Nick Bostrom in 2008 where he lays out his vision of what somebody speaking from the future utopia talking back to us would sound like. When you read it, it sounds very much like a mystical essay—the kind of thing that, change a few words, a Christian like C.S. Lewis could have written about what it's like to speak down from heaven. To what extent is there a spiritual or religious dimension to utopian thinking that relies on some amount of faith that things can get indescribably better in some ephemeral, indescribable way?
Joe Carlsmith
I think there are definitely analogs and similarities between some ways of relating to the notion of utopia and attitudes common in religious and spiritual contexts. Personally, I don't think it needs to be that. I don't think it requires faith or anything mystical. I think it's just a basic fact about our current cognitive and civilizational situation that things could be radically better. It's ephemeral in the sense that it's quite hard to imagine. An important source of evidence here is variance in the quality of human experiences. If you think about your peak experiences, they're often a really big deal.
You're sitting there going, "Wow, this is serious." You feel that this is something you would trade much mundane experience for. The thing we need to do is extrapolate from there. Look at the trajectory your mind moved along as you moved into some experience or some broader non-experiential improvement—your community got a lot better, your relationships got a lot better. Stare down where that is going. I don't want to call it faith. I think it requires a kind of extrapolation into a zone that is beyond your experience but is deeply worthy and important. That is often associated with spirituality and religion, and I think that's okay.
But I think there are a number of really important differences between utopia and something like heaven. Utopia will be a concrete, limited situation. There are going to be frictions, resource constraints. It's going to be finite. It's still going to be in the real world. Most religious visions don't have those constraints, and that's an important feature of their situation.
Dwarkesh Patel
Speaking of constraints, this reminds me of Robin Hanson's theory that eventually the universal economy will just be made up of these digital people, ems, and that because of competition, their wages will be driven down to subsistence levels. Maybe that's compatible with some engineering in their ability to experience such that it's still blissful for them to work at subsistence levels of compute or whatever. But it seems like first-order economic thinking implies that there will be no utopia. In fact, things will get worse on average, but maybe better overall if you just add up all the experience. This vision seems incompatible with yours of a utopia. What do you think?
Joe Carlsmith
I would not call Robin's world a utopia. A thing I haven't been talking about is what our overall probability distribution should be with respect to different quality of futures, and exactly how possible and likely it is that we build something profoundly good as opposed to mediocre or much worse. I would class Robin's scenario in the mediocre or much worse zone.
Dwarkesh Patel
Do you have a criticism of the logic he uses to derive that?
Joe Carlsmith
To some extent, my main criticism is that I think we will very likely find that competitive pressures are a source of pushing the world in bad directions. But I also think there are ways in which wise forms of coordination and preemptive action can stave off the bad effects of competitive pressures. The way I imagine avoiding stuff in the vicinity of what Robin is talking about involves that. There are a lot of complexities there.
Dwarkesh Patel
The last few years have not reinforced my belief in the possibility of wise coordination. Anyway, one thing I want to talk to you about is you have a paper on what it would take to match the human brain's computational capacity. Associated with that, you have a very good summary on Open Philanthropy. Do you want to talk about the approach you took to estimate this and why this is an important metric to try to figure out?
Joe Carlsmith
The approach I took was to look at the evidence from neuroscience and the literature on the computational capacity of the human brain, talk to a bunch of neuroscientists, and try to see what we know right now about the number of floating point operations per second that would be sufficient to reproduce the task-relevant aspects of human cognition in a computer. It's not clear to me exactly how important this parameter is to our overall picture. The way it's relevant to thinking I've been doing and that OpenPhil has been doing is as an input into an overall methodology for estimating when we might see human-level AI systems.
That methodology proceeds by first trying to estimate roughly the computational capacity of the brain or the size of an AI system in terms of overall parameter count and compute capacity that would be analogous to humans. Then you extrapolate from that to the training cost to create a system of that kind using current methods in machine learning and current scaling laws. That methodology brings in a number of additional assumptions that aren't transparent—like, "Oh, of course, that's how we would do it." You have to be a bit more in the weeds to see exactly how it feeds in.
Dwarkesh Patel
I think you said it was 10 to the 15 flops for the human brain. Did you have an estimate for how many flops it would take to train something like the human brain? I know GPT-3 is only 175 billion parameters or something, which can fit into a micro SD card, but it cost $20 million to train. Were you able to come up with some sort of estimate for what it would cost to train something like that?
Joe Carlsmith
Yeah, so my focus in that report was not on the training extrapolation. That was work that Ajaya Khatri at Open Philanthropy did using my report's estimate as an input. Her methodology involves assigning different probabilities to different ways of using that input to derive an overall training estimate.
In particular, an important source of uncertainty there is the amount of compute required, or the number of times you need to run a system per data point that it gets. In the case of something like GPT-3, you get a meaningful data point and a gradient update as to how well you're performing with each token that you output as you're doing GPT-3 style training. You're predicting text from the internet, you suggest an X token, and then your training process says, "nope, do better next time."
Whereas if you're learning to play Go and you have to play the full game out—that's hundreds of moves—then before you get an update as to whether you're playing well or poorly, that's a big multiplier on the compute requirement. That's one of the central pieces that Ajaya calls the "horizon length of training," and it's a very important source of uncertainty in getting to your overall training estimate.
Ultimately, she ends up with this big spread out distribution. I think GPT-3 was like 10 to the 24—4 times 10 to the 23 or something like that. She spreads out all the way up to the evolution anchor, which I think is something like 10 to the 41. Her distribution is centered somewhere in the low 30s.
Dwarkesh Patel
Okay, that's still quite a bit, I guess. How much does this rely on the scaling hypothesis? If one thought that the current efforts and the current approach were not likely to lead—or at least not likely to lead in a sample efficient way—towards human intelligence, it might be analogous to somebody saying we have enough deuterium on earth to power civilization for millions of years. But if you haven't figured out fusion, then it may be an irrelevant statistic.
Joe Carlsmith
Yeah, so the approach does assume that you can train a human-level or transformative AI system with a non-astronomical amount of compute and data using current methods, without major conceptual or algorithmic breakthroughs relative to what's currently available.
The actual methodology Ajaya uses allows you to assign probabilities to that assumption too. You can say, "I'm only 20% on that," and then there are other options. You can also kind of rerun evolution, which is another anchor she provides. This is often what people will say as a sort of upper bound on how hard it is to create human-level systems—doing something analogous to simulating evolution, though there are a lot of open questions as to how hard that is.
I do think this methodology is a lot more compelling and interesting if you are compelled by the available techniques in deep learning and by the scaling hypothesis views, at least as an upper bound. I think it's important. There are different ways of being interested in algorithmic breakthroughs. One is because you think deep learning isn't enough. Another is because you think they will provide a lot of efficiency relative to deep learning, such that an estimate like Ajaya's is an overestimate because we won't have to do that. We'll make some sort of breakthrough and it'll happen a lot earlier. I put weight on that view as well.
Dwarkesh Patel
Yeah, that's really interesting. That implies that even if you think the current techniques are not optimal, maybe that should update you in favor of thinking it could happen sooner. That's really interesting.
So then how did you go about estimating the amount of flops it would take to emulate the interactions that happen in a brain? Obviously it would be unreasonable to say that you have to emulate every single atomic interaction. But what is your proxy that you think would be sufficient to emulate?
Joe Carlsmith
I used a few different methodologies and tried to synthesize them. One was looking at the mechanisms of the brain and what we know about the complexity of what they're doing, and how hard it is to capture the task-relevant dimensions of the signaling happening in the brain.
I also tried to bring in comparisons with existing AI systems that are replicating chunks of functionality that the human brain has, particularly in the context of vision. How do our current vision systems compare with the parts of the brain that are plausibly doing analogous processing, though they're often doing other things as well.
Then I used a third method, which has to do with physical limits on the energy consumption per unit computation that the brain is plausibly doing. And a fourth method I gesture at tries to extrapolate from the communication capacity of the brain to its computational capacity using comparisons with current computers.
It's a triangulation—you look at a bunch of different sources of evidence, all of which, in my opinion, are pretty weak. The physical limit stuff is maybe more complicated, but it's sort of an upper bound. I think we are significantly uncertain about all of this, and my distribution is pretty spread out. But the hope is that by looking at a bunch of things at once, you can at least get an educated guess.
Dwarkesh Patel
I'm very curious—is there consensus in neuroscience or other relevant fields that we understand the signaling mechanisms well enough that we can say, "Basically, this is what it's involved, this is what the system is reducible to, this is how many bits you need to represent all the synaptic connections here"? Or is there a variance of opinion about just how complicated the enterprise is?
Joe Carlsmith
There's definitely disagreement. It was interesting, and in some sense disheartening, to talk with neuroscientists about just how difficult neuroscience is. A consistent message—and I have a section on this in the report—was how far we are from really understanding what's going on in the brain, especially at an algorithmic level.
In some sense the report is somewhat opinionated. There are experts that I found more compelling than others. There are experts who are much more in an agnosticism mode of "we just don't know, the brain is really, really complicated." They err on the side of very large compute estimates, a lot of emphasis on biophysical detail, a lot of emphasis on mysterious things that could be happening.
Then there are other neuroscientists who are more willing to say things like, "We kind of basically know what's going on at a mechanistic level," which isn't the same as knowing the algorithmic organization overall and how to replicate it. I sort of lean towards the latter view, though I give weight to both and try to synthesize the opinions of people I saw overall.
Dwarkesh Patel
Just looking at the post itself—I haven't really looked deeper into the actual paper performance that's derived—it seems like you were, to estimate the flops mechanistically, adding up the different systems that play here. Should we expect it to be additive in that way? Or maybe it's multiplicative, or there's more complicated interaction where the flops grow super linearly to the inputs? I know that probably sounds really naive, but from a first glance, that's a question I had.
Joe Carlsmith
The way I was understanding and breaking down the forms of processing that you would need to replicate in the brain made them seem not multiplicative in this way. A simple example: suppose we have some neurons signaling centrally via spikes through synapses, and then we have glial cells as well, which are signaling via slower calcium waves. It's a separate network.
You could think that if the rate of calcium signaling is dependent on the rate of spikes through synapses, then that's an important interaction. But overall, if you imagine this kind of network processing, you can estimate them independently and then add it up. They're not actually multiplicative processes on that conception. I do think there are correlations between the estimates for the different parts, but at a fundamental level, it adds up.
Dwarkesh Patel
I see. Okay. How much credence do you put in the sort of almost woo-woo hypotheses—I don't know, Roger Penrose has that thing about something quantum mechanical happening in the brain that's very important for understanding cognition. To what extent do you put credence in those kinds of hypotheses?
Joe Carlsmith
I put very little credence in those hypotheses. I don't see a lot of reason to think that. I see a good amount of reason not to think it, but it wasn't something I dug in on a ton.
Dwarkesh Patel
Okay, gotcha. All right, so you have this really interesting blog post about infinite ethics. Do you want to talk about why this is an important topic, why it's important to integrate into our worldview, and so on?
Joe Carlsmith
Sure. Infinite ethics is ethics that tries to grapple with how we should act with respect to infinite worlds, how we should rank them, how they should enter into our expected utility calculations or our attitudes towards risk. I think this is important for both theoretical and practical reasons.
At a theoretical level, when you try to do this with a lot of common ethical theories and constraints and principles, they just break on infinite worlds. I think that's an important clue as to their viability, because I think infinite worlds are at the very least possible. Even if our world is finite, and even if our causal influence is finite, it's possible to have infinite worlds. And we have opinions about them—an infinite heaven is better than an infinite hell.
Often in ethics, we expect our ethical principles to extend to ranking scenarios or acting in hypothetical scenarios, rather than just our actual situation. I think infinities come in there.
But then, maybe more importantly, I think it's an issue with practical relevance. A way to see that is that I think we should have non-zero credence that we live in an infinite world. It's a very live physical hypothesis that the universe is infinite, even if I think the mainstream view is that our causal influence on that universe is finite in virtue of things like entropy and light speed. But the universe itself may well be infinite, and possibly infinite in a number of different ways.
Max Tegmark has some work on all the different ways the universe is going to be really very large. There are a number of ways that I think we should have non-zero credence that we can have infinite influence in our actions now. The causal limitations could be wrong. It may be that in the future we'll be able to do infinite things.
I also think, somewhat more exotically, that there are ways of having a causal influence on an infinite universe even if you are limited in your causal influence. That comes from some additional work I've done on decision theory. If you try to incorporate that, if you're an expected value reasoner, it very quickly starts to dominate or at least break your expected value calculations.
You mentioned long-termism earlier. A natural argument for getting interest in long-termism is that in the future there could be all these people, their lives are incredibly important. So if you do the EV calculation, your effect on them is what dominates. But actually, if you have even a tiny credence that you can do an infinite thing, either that dominates or it breaks. And if you have tiny credences on doing different types of infinite things and you need to compare them, you need to know how to do it. I think this is actually a part of our epistemology now, though we often don't treat it that way because we're often not doing EV reasoning or really thinking about it.
Dwarkesh Patel
Yeah, that's super fascinating. If it is the case that we can only have an impact on a finite amount of stuff, then maybe it is true that there's infinite suffering or happiness in the universe at large, but the delta between the best case scenario for what we do and the worst case scenario is finite. That still seems less compelling if the hell or heaven we're surrounded by overall doesn't change.
Can you talk a bit more about your other work on having infinite impact beyond the scope of what light speed and entropy would allow us? How might that be possible?
Joe Carlsmith
Sure. A common decision theory—though it's not the mainstream, it's a contender in the literature—is evidential decision theory. Roughly speaking, you should act such that you would be happiest to learn that you had acted that way for that reason.
A way of thinking about it: suppose you are a deterministic simulation and there's a copy of you being run too far away for you to ever causally interact with it. But you know it's a deterministic copy, so it'll do exactly what you do, absent some computer malfunction. Now you're deciding whether to send a million dollars—well, it's a little complicated because he's too far away.
Let's say I have to make some ethical decision, like whether I should take an expensive vacation or donate that money to save someone's life. Because the other guy is going to act just like I do, even though I can't cause him to do that, in some sense when I make my choice, after doing so, I should think that he made the same choice. Evidential decision theory treats his action as in some sense under my control.
If you imagine an infinite universe where there are an infinite number of copies of you, or even not copies, but people whose actions are correlated with you such that when you act a certain way it gives you evidence about what they do, in some sense their actions are under your control. If there are an infinite number of them, on evidential decision theory and a few other decision theories, then in some sense you're having influence on the universe.
Dwarkesh Patel
This sounds really similar to the EPR pair experiment in quantum mechanics. The basic idea is if you have two entangled bits, you take them very far away from each other, and then you measure one of them, and you come up with some rule like "if it's plus we do this, if it's minus we do the other thing." It seems at first glance that measuring something yourself has an impact on what the other person does even though it shouldn't be allowed by light speed. It gets resolved if you take the many worlds view, but yeah.
Is this just a thought experiment, or is this something we should anticipate for some cosmological reason to actually be a way we could have influence on the world?
Joe Carlsmith
I haven't dug into the cosmology a lot, but my understanding is that it's at the very least a very live hypothesis that the universe is infinite in the sense that it's infinite in extent, and suitably far away there are copies of us having just this conversation. Even further away, there are copies of us having this conversation but wearing raccoons for hats, and all the rest—which is itself something to wonder about and sit with.
My understanding is this is just a live hypothesis, and more broadly, infinite universes are just sort of a part of mainstream cosmology at this point. I don't think it's just a thought experiment. I think infinite universes are alive.
These non-causal decision theories are actually my best guess decision theories, though that's not a mainstream view. So it comes in fairly directly and substantively if you have that combination of views. But I also think everyone should have non-zero credence in all sorts of different infinity-involving hypotheses. Infinite ethics gets a grip regardless.
Dwarkesh Patel
Taking that example, if you're having an impact on every identical copy of yourself in the infinite universe, it seems that for any such copy, there's an infinite amount of other copies that are slightly different. So it's not even clear if you're increasing—maybe it makes no sense to talk about proportions in an infinite universe. If there is another infinite set of copies that scribbled the exact opposite thing on the whiteboard, then it's not clear that you had any impact on the total amount of good or bad stuff that happened. My brain breaks here, but maybe you can help me understand this.
Joe Carlsmith
There are a couple of dimensions here. One is trying to understand what sort of difference does it make if you're in this infinite situation and you're thinking about a causal influence. What even did you change at an empirical level before you talk about how to value that? That's a pretty gnarly question.
Even if we've settled that question in terms of the empirical causal impact, there's a further question of how do you rank that, or how do you deal with the normative dimension here. That's the ethical question, and things get really gnarly very fast. There are impossibility results that show that even very basic constraints that you would have thought we could get at the same time in our ethical theories, you can't get them at the same time when it comes to infinite universes. We know that something is going to have to change if we're going to extend our ethics to infinities.
Dwarkesh Patel
Is there some reason you've settled on—you mentioned you're not a utilitarian, but on some version of EA or long-termism as your tentative moral hypothesis, despite the fact that this seems unresolved? How do you sit with that tension while tentatively remaining in EA?
Joe Carlsmith
I think there's two dimensions there. One is that it's good practice to not totally upend your life if you encounter some destabilizing philosophical idea, especially one that's difficult and you don't totally have a grip on it. Come back to it.
Dwarkesh Patel
Isn't that what long-termism is?
Joe Carlsmith
So I think there's a real tension there. How seriously should we take these ideas? At what point should you be making what sorts of changes for your life on the basis of different things that you're thinking and believing? It's a real art. Some people grab the first idea they see and start doing crazy stuff in an unwise way. Some people are too sluggish and not willing to take ideas seriously or reorient their life on the basis of changes in what seems true.
Nevertheless, I think especially things that involve "it turns out it's fine to do terrible things" or "there's no reason to eat your lunch" or whatever—things that really holistically break your ethics views—one should tread very cautiously.
At a philosophical level, the way I resolve it is I think for many of these issues, the right path forward, or at least a path that looks pretty good, is to survive long enough for our civilization to become much wiser, and then to use that position of wisdom and empowerment to act better with respect to these issues. That's what I say at the end of the Infinite Ethics post—I think future civilization, if all goes well, will be much better equipped to deal with this. We are at square one in really understanding how these issues play out and how to respond.
So it looks convergently pretty good to me to survive, become wiser, keep your options open, and then act from there. That ends up pretty similar to a lot of long-termism and existential risk. It's just that it's focused less on "the main event will be what happens to future people" and more about getting to the point where we are wise enough to understand and reorient in a better way.
Dwarkesh Patel
What I find really interesting about this is that different people tend to have different thresholds for epistemic learned helplessness, where they basically say "this is too weird, I'm not going to think about this, let's just stick with my current moral theories." For somebody else, it might be before they became a long-termist—"trillions of future people, what are we talking about here? Let's not change my mind on stuff." For you, maybe it's before the infinite ethics stuff. Is there some principled reason for thinking that this is where that stop should be? Or is it just a matter of temperament and openness?
Joe Carlsmith
I don't think there's a principled reason. I should say I don't think of my attitude towards infinite ethics as solely "oh, this has gotten too far down the crazy path, I'm out." The thing about wisdom in the future is pretty important to me as a motive orientation.
A first pass cut that I use is: when do you feel like it's real? If you feel like a thing is real, as opposed to an abstract fun argument, then that's an important signal. I generally encourage people—the mode I'm drawn to is something like: if there's an idea that seems compelling intellectually, that's a reason to investigate it a lot and think about it, really grapple with it. If it doesn't seem right to you or if it seems too crazy, ask why, and really process it. It's a reason to pay a lot of attention.
But if you've paid a lot of attention, and at the end of the day you're like, "well, I guess at an abstract level that sort of makes sense, but it just doesn't feel to me like the real world, it doesn't feel like wisdom, or like a healthy way of living," then maybe you shouldn't do it. Some people will do that wrong and bounce off of ideas that are in fact good. But overall, these are sufficiently intense and difficult issues that being actually persuaded—and not just chopping off the rest of your epistemology for the sake of some version of the abstraction—seems to me important. It's a healthier way to relate.
Dwarkesh Patel
Another example of this is that you have this really interesting blog post on ants and your thoughts after sterilizing a colony of them. This is another example of a thing where almost everybody other than—I don't know, maybe a Jain who wears a face mask to prevent bugs from going into his mouth—would say, "Okay, at this point, if we're talking about how many hedons are in a hectare of forest from all the millions of insects there, then you've lost me." But then somebody else might say, "Well, there's not a strong reason for thinking they have absolutely no capacity to feel suffering." I wonder how you think about such questions, because you can't stop living, and you're not even going to stop going on road trips where you're probably killing hundreds of insects by just driving. What do you think about such conundrums?
I have a significant uncertainty about exactly—and I think this is the appropriate position—about exactly how much consciousness or suffering or other moral properties that we associate with moral patienthood apply to different types of insects. I think it's a strange view to be extremely confident that what happens with insects is totally morally neutral. And I think it actually doesn't fit with our common sense.
So let's say you see a child frying ants with a magnifying glass. I think there is some—you could say, "Well, that just indicates they're going to be cruel to other things that matter." But I don't think so. You see the ants twitching around. As in many cases with animal ethics, I think we're a bit schizophrenic about which cases we view as morally relevant and which not. We have pet treatment laws, and then we have factory farms. So I don't see it as a radical position that ants matter somewhat.
There's a further question of what your overall practical response should be. The main thing is that, as in a lot of ethical life, there are trade-offs. You have to make a call about what constraints you're going to put on yourself at the cost of other goals. In the case of insects, it's not my current moral focus. I don't pay a lot of costs to lower my impact on animals, and I don't sweep the sidewalk for ants in particular. That's my best guess response, and it has to do with other ethical priorities in my life.
But there's a middle ground between "I shall ignore this completely" and "I shall be a Jain." It's recognizing that this is a real trade-off. There's uncertainty here, and taking responsibility for how you're responding to that.
Joe Carlsmith
Yeah, this seems similar to the infinite ethics example. If you put any credence that they have any ability to suffer, then at least if you're not going to say it doesn't matter because of the far future, it seems like there should be a compelling thing to think about. But the result isn't even like becoming a vegan, where you change your diet.
As you might know, this is used as a reductio ad absurdum of veganism: if you're going to start caring about other non-human animals, why not also care about insects? Even if they're worth like a millionth of a cow, you're probably still killing a million of them on any given day from all your activities—indirectly, from the food you're eating, all the pesticides used to create that food. I don't know how you go about resolving that.
I'd want to hear the empirical case. I think it's true there are a lot of insects. But I think if you want to say that taking seriously the idea that there's some reason not to squash a bug leads immediately to Jain-like behavior, absent long-termism, I really want to hear the empirical case about exactly what impact you're having. I'm not at all persuaded that's the practical upshot. And if it is, if that's a really strong case, then that's an interesting implication of this view and worth concern.
But I wouldn't jump to that almost out of a desire to get to the reductio. I would try to move slower and really see: wait, is that right? There are a lot of trade-offs here. What's the source of my hesitation? Don't jump too quickly to something sufficiently absurd that I can reject this whole mode of thinking, even though I don't know why.
Dwarkesh Patel
I see. Okay, so let's talk about the two different ways of thinking about observer effects and their implications. You have a four-part series on this. Do you want to explain the self-indication assumption and the self-sampling assumption? I know it's a big topic, but as much as possible.
Joe Carlsmith
Sure. One way to start is by thinking about the following case. You wake up in a white room, and there's a message written on the wall. Let's say you're going to believe this message. The message says: "I am God. I flipped a coin. If it was heads, I created one person in a white room. If it was tails, I created a million people, all in white rooms." Now you are asked to assign probabilities to the coin having come up heads versus tails.
One approach, which is the approach I favor—or at least think is better than the other—is the self-indication assumption. These names are terrible, but so it goes. SIA says that your probability that the coin came up heads should be approximately one in a million. That's because SIA thinks it's more likely that you exist in worlds where there are more people in your epistemic situation—more people who have your evidence, which in this case is just waking up in this white room. That can be a weird conclusion and go to weird places, but I think it's a better conclusion than the alternative.
SSA, the main alternative I consider in that post, is the self-sampling assumption. It says that you think it more likely that you exist in worlds where people with your evidence are a larger fraction of something called your reference class, where it's quite opaque what a reference class is supposed to be. Broadly speaking, a reference class is the set of people you could have been. In this case, everyone has your evidence, so the fraction is the same, and you stick with the one-half prior. But in other contexts, not everyone has your evidence, so SSA updates towards worlds where it's a larger fraction.
Famously, SSA leads to the doomsday argument. You imagine two possibilities: either humanity will go extinct very soon, or we won't go extinct very soon and there will be tons of people in the future. In the former case, people born at roughly this time are a much larger percentage of all the people who ever lived. If God first creates a world and then inserts you randomly into some group, it's much more likely that you would find yourself in the 21st century if humanity goes extinct soon than if there are tons of people in the future. So SSA updates very, very hard towards the future being short. That matters a lot for long-termism, because long-termism is all about the future being big in expectation.
Dwarkesh Patel
So what is the SIA take on this?
Joe Carlsmith
A way to think about SIA's story: with SSA, first God creates the world, and then he's dead set on putting you into this world. He's got your soul, and it's going in there no matter what. But the way he inserts your soul is by throwing you randomly into some set of people—the reference class. So you should expect to end up in the world where the kind of person you end up as is a larger fraction of the total people you could have been.
What SIA thinks is different. The story I use for SIA: God decides he's going to create a world. There's a big line of souls in heaven, and he goes and grabs them randomly out of heaven and puts them into the world. In that case, if there are more people in the world, then you've got more shots. You're one of these souls sitting in heaven, hoping to get created. On SIA, God has more chances to grab you out of heaven and put you into the world if there are more people like you in that world. So you should expect to be in a world where there are more such people. That's SIA's vibe.
Dwarkesh Patel
Doesn't this also imply that you should be in the future, assuming there will be more people in the future? Tell me more about why it would imply that.
Joe Carlsmith
In an analogous scenario, go back to the God tossing the coin scenario. If you substitute "being a conscious entity" for "people in white rooms," and if there's going to be more conscious entities in the future, you would really expect to be in that scenario where there are a lot more conscious beings—presumably the future. So it's still odd that you're in the present under SIA.
Yes, in a specific sense. On SIA, you wake up in the white room wondering: am I in room one, or am I in rooms two through a million? You become very, very confident that you're in the tails world with lots of rooms. Conditional on tails, you split your credence evenly between all those rooms. So you are very confident that you're in one of the rooms two through a million, not room one. But that's before you've seen your room number.
Once you see your room number, you should be quite surprised about it. But once you get the room number, you're back to 50-50 on heads versus tails. You had equal credence in being in tails in room one and heads in room one. When you get rid of all the other tails in rooms two through a million, you're left with 50-50 overall. So the sense in which SIA leaves you back at normality with the doomsday argument is: once you update on being in the 21st century—which admittedly should be surprising, if you didn't know you were in the 21st century and then learned that you were, you'd be like, "Wow, that's really unexpected"—once you do that, you're back at whatever your prior was about extinction.
Dwarkesh Patel
Maybe I'm still not sure why the fact that you were surprised should not itself be the doomsday argument.
Joe Carlsmith
Yeah, there's an intuition there: is SIA making a bad prediction? You could update against SIA because SIA would have predicted that you're in the future. I think there's something there. SIA naively predicts that you should find yourself in a situation where there are just tons of people—a situation obsessed with creating people with your evidence. This is one of the problems with SIA. You should expect to find, in every nook and cranny, a simulation of you. As soon as you open the door, it's actually a giant bank of simulations of you in your previous epistemic state. You don't see that, so you might update against the anthropic theory that predicted you would see that. There are arguments in that vein.
Dwarkesh Patel
Maybe let's back up to the original example used to distinguish these two theories. Can you help me resolve my intuitions? My intuition is very much that you knew you were going to wake up in a white room before you actually did wake up. Your prior should have been one-half heads or tails. So it's not clear to me why, having learned nothing new, your posterior probability on either scenario should change.
Joe Carlsmith
The SIA response would be that you didn't know you were going to wake up. Going back to the just-so story where God is grabbing you out of heaven, it's actually incredibly unlikely that he grabs you. There are so many people. SIA is in general very surprised to exist. In fact, that's a good argument against it: SIA says you shouldn't exist, isn't it weird that you exist? But once you're in that headspace, the way to think about it is that it's not a guarantee that you were going to be created. You are a particular contingent arrangement of the world. You should expect that arrangement to come about more often if there are more arrangements of that type, rather than assuming that no matter what, existence will include you.
Dwarkesh Patel
Can you talk more about the problems with SSA—scenarios where you think it breaks down, why you prefer SIA?
Joe Carlsmith
One of the most dramatic problems is that SSA predicts it's possible to have a kind of telekinetic influence on the world. Imagine you wake up in an empty universe except for a puppy, you, and a boulder rolling towards the puppy. The boulder is inexorably going to kill the puppy. But you have the power to make binding pre-commitments that you will execute. You also have a button that would allow you to create zillions of people, all wearing different clothes from you—so they'd be in a different epistemic state.
You make the following resolution: if this boulder does not jump out of the way of this puppy in some very unlikely way, then I will press this button and create zillions of people, all in a different epistemic state, but let's assume they're in my reference class. SSA thinks it's sufficiently unlikely that you would be in a world with zillions of those people while you have different colored clothes—since that would be a tiny fraction of the reference class if those people get created—that SSA thinks it's actually more likely, once you've made that commitment, that the boulder will jump out of the way. That looks weird. You can't just make that commitment and expect the boulder to jump.
You get similar analogs in the God's coin toss case. Let's say you wake up and learn you're in room one, but God hasn't tossed the coin yet. He created room one first, and then he's going to toss to determine whether he creates all the future rooms. On SSA, once you wake up and learn you're in room one, you think it's incredibly unlikely that there will be these future people. So now God is going to toss a fair coin in front of you, and you're still going to say, "I'm sorry God, it's a one in a million chance that this coin lands tails." That's very weird. It's a fair coin that hasn't been tossed, but you, with the power of SSA, have become extremely confident about how it's going to land. There are a number of other really bad problems for SSA.
Dwarkesh Patel
While I digest that, let me mention the problems you already pointed out against SIA. If one thinks SIA is true, one should be very confident that you're in a universe with many other people who have been sampled just like you. So it's kind of surprising that we're in a universe that is not filled to the brim with people. You could imagine Mars is just completely made up of bodies, or every single star has a simulation of a trillion people inside. The fact that this is not happening seems like very strong evidence against SIA. There are other things like the presumptuous philosopher you might want to talk about as well. Do you just bite the bullet on these things? How do you think about them?
Joe Carlsmith
My main claim is that SIA is better than SSA. I think it's just a horrible situation with anthropics. Overall, SSA is an update towards bigger, more populated universes. I think the most salient populated universes don't involve hidden people on other planets—they're probably, I don't know, maybe we're in a simulation and people are obsessed with simulating us. I think this is actually more important and worrying.
The way I see this dialectic: a big problem with SIA is that it immediately becomes certain, naively, that you live in an infinite universe or a universe with an infinite number of people. And then it breaks because it doesn't know how to compare infinite universes. To be fair, SSA also isn't great at comparing infinite universes. They both have things you can try to do in infinite ethics, where you have expanding spheres of spacetime and you count some fraction or density of people in those spheres. There's this general problem in cosmology of trying to understand what it means to have a fraction or a density of different types of observers.
My own take is that you hit infinity—infinite universes—fairly fast, and then they break your anthropics in analogous ways to how they break your ethics. That's where I'm currently at. I'm hoping to understand better how to do anthropics with infinities. Some of my work on the universal distribution—I have a couple blog posts on that—was attempting to go a little bit in that direction, though it has its own giant problems.
Dwarkesh Patel
Interesting. Do you know if Robin Hanson's grabby aliens thing probably uses SSA? Do you know if that's the case?
Joe Carlsmith
I haven't looked closely at that work.
Dwarkesh Patel
Okay. It's hard for me to think about. Maybe it'll take me a few more weeks before I can digest it fully.
Okay, so that's really interesting. You have a really interesting blog post about believing in things you cannot see. You make an interesting comment about futurism. Here's what you say: "Much of futurism, in my experience, has a distinct flavor of unreality. The concepts—mind uploads, nanotechnology, settlement and energy capture in space—are, I think, meaningful, even if loosely defined. But at a certain point, once models become so abstracted and incomplete that the sense of talking about a real thing, even a possibly real thing, is lost." Why do you think that is? And is there a way to do futurism better?
Joe Carlsmith
I think it comes partly because imagination is just quite a limited tool. It's easy, when you're talking about the future—a big thing to try to model with this tiny mind—to use extremely lossy abstractions. It puts you in a mode of having these really sketchy and gappy maps that you're trying to manipulate.
I think there's also a way in which this isn't all that unique to futurism. In general, it's hard sometimes to keep our intellectual engagement rooted and grounded in the real world. It's easy to move into a zone, especially if that zone is inflected with social dynamics, or it's an intellectual game, or you're enjoying it for its own sake, or there are status dimensions in the way people talk. Those things start to move our discourse in directions that aren't about "we're talking about the real world right now, let's actually get it right."
I think that happens with futurism, and maybe more so because it can feel like some topics are treated as "that's a real serious topic about real stuff," while other topics are treated as "this is the chance to make stuff up." My experience is that sometimes people relate to futurism that way. There are other topics where people move into a zone of "one can just say stuff here, there are no constraints." I think that's actually wrong. With futurism, there are important constraints and important things we can say. But that vibe can seep in nonetheless.
Dwarkesh Patel
It's interesting that this is true of the future and the past. I recently interviewed somebody who wrote a book about the Napoleonic War. It's very interesting to talk about in an abstract sense. But then you can, which is very seldom done, think of the reality of a million men marching out of Russia and freezing and eating the remains of horses and other people and starving. When you're not just talking about abstractions like "the border changed so much in these few decades," the concrete reality changes how you think about history so much.
Even recently, I was reading this book about the use of meth by the Nazis. There's this really cynical part where the leaders in the Nazi regime are talking about how meth is the perfect drug because it gives them courage to blitz through an area without thinking about how cold it is, without thinking about how scary it is to be in no man's land. Just this idea of a meth'd-up soldier forced to go out into the middle of nowhere, marching to Russia in the winter. I don't know if that was going to lead up to a question. I don't know if you have a reaction.
Joe Carlsmith
Yeah, I think that's a great example of the difference between relating to history as "how is the border changing" versus the concreteness of these people.
And often, I think, engaging with history is horrifying in this respect. When you really bring to mind the lived reality of all these events, it's a really different experience. I think to some extent, one of the reasons that concreteness might be often lacking from futurism is that you can't—any attempt to specify the thing will be wrong.
So you might be right about some abstract thing, like you might say, "Oh, we will have the ability to manipulate matter at this level, at this scale." But if you try to dig in and say, "Here's what it's like to wake up in the future," you're wrong immediately. That's not how it's going to be. So you don't have the ability to really hone in on concrete details that are actually true.
In some sense, you need this back and forth where you imagine a concrete thing and then say, "Okay, that's wrong, but I'll take the flavor of concreteness from that and say, 'But it will be a concrete thing. It just won't be the specific one I imagined.'" And then keep that flavor of concreteness even as you talk in more abstract ways. That's a delicate dance.
Dwarkesh Patel
Yeah. As many viewers will know, Peter Thiel has this talking point that he often brings up about how we've become indefinite optimists and that he prefers a sort of definite optimism where you have a concrete vision of what the future could be.
Okay, so to close out, one of the things I wanted to ask you about was—you said this was a side project, this blog. I thought it was one of your main projects, actually. Before you mentioned that your main work is AI, I thought this was at least part of your main work. So it's surprising. It's really surprising to me that you're able to keep up the regularity. You're basically publishing a small book every week or so, filled with a lot of insight. Unlike many other blogs on the internet, you've got great prose. How are you able to maintain such productivity on your side project?
Joe Carlsmith
I should say, a few of my most recent posts which were especially long, I had taken some time off from work and I was working on those partly in an academic context. But the first year and a half or so of the blog was just on the side, and I've gone back to having it be on the side now.
I think one thing that helps is my blog posts are too long. I have dreams of taking my long blog posts and really crunching them down into a pithy, elegant, concise statement. But that would be more work. One way I increase my output is by not doing that editing. I feel bad about that, but that's one thing at least.
Dwarkesh Patel
There's that quote—I think it's something like, "I would have written you a short letter, but I didn't have time, so I wrote you a long letter."
Joe Carlsmith
I have a friend who says the actual thing should be, "I didn't have time to write you a short letter, so I wrote you a bad letter." I hope it's not that bad, but I do think if I had more time for these posts, I would try to cut them down. That's one time-saving measure, for better or worse.
Dwarkesh Patel
At least as a reader, it often seems to me that people like you who write—maybe this describes your process—Scott Alexander says he just writes stream of consciousness and it turns out to be really readable. Your blog posts are really readable. Even the stuff I write, when I'm consciously not trying to make edits while I'm going, ends up reading much better than when I'm trying to optimize each sentence and taking two steps back for every one forward. I don't know if it's a selection effect where the things that are harder to convey require more editing.
Joe Carlsmith
I wonder. My feeling is that my writing is quite a bit better if I have a chance to edit it. It's just a time thing. But I do think people vary quite a bit.
I was recently reading this book by George Saunders, who I really admire, about fiction writing called *Swim in the Pine in the Rain*. The vibe he tries to convey—and I think this is common amongst writer types—is this obsessive focus on a sentence-by-sentence level, really thinking about where the reader's mind is right now. How are they engaging? Are they interested? Are they surprised? Am I losing them? His writing is really engaging in ways that aren't even obvious. You just start reading and you're like, "Wow, I'm really into this." But it's also a daunting picture of the level of attentiveness required. If I'm going to write everything like that, that's going to cut down a lot on my overall output. So there's a balance there. To the extent you're one of those people who can just stream of consciousness and that's close to what you'd get from editing—which I'm not sure I am—all the better. You're lucky.
Dwarkesh Patel
There's also an additional consideration. If you think there's some kind of power law to how interesting a piece is or how many people see it and find value in it, then it's not clear whether that advises you to spend so much time on each piece to increase the odds that one piece blows up, given the big difference between pieces that blow up and don't, or whether you should just do a whole bunch and try to sample as often as possible.
Joe Carlsmith
Yeah. I started the blog partly as an exercise in just getting stuff out there. I had the idea that I would one day write up a bunch of stuff I've been thinking about, but it was somehow this grand thing I would finally write up and it would be beautiful and I would take all this time. Then, for various reasons, I felt like I was approaching some aspects of my life with too much perfectionism. I needed to just get stuff out there faster. The blog was an exercise in that, and I think that's paid off in ways. I don't think I would have done it otherwise.
Dwarkesh Patel
I see. All right. Final question. I'm curious if you have three book recommendations you can give the audience.
Joe Carlsmith
Probably my primary recommendation, though this is somewhat self-serving because I helped with this project, is *The Precipice* by Toby Ord. It may be familiar to many of your listeners, but I think it's a book that really conveys the ideas that matter most to me, that have had close to the biggest impact in my own life.
Other books—I love the play *Angels in America*. I think it's epic and amazing. That's not quite a book, but you can read it. I actually recommend watching the HBO miniseries, but that's something I recommend.
Then, last year I read *Housekeeping* by Marilynne Robinson. It has this sort of numinous quality that I think a lot of her writing does. I really like that and recommend it to people. That's also a piece of fiction.
If you're looking for philosophy, a lot of my work is in dialogue with Nick Bostrom and his overall corpus. I think that's really valuable to engage with.
Dwarkesh Patel
Cool. All right. Joe, thanks so much for coming on the podcast. It was a lot of fun.
Joe Carlsmith
Yeah, thanks for having me. Oh, I'll also say, everything I've said here is just purely my personal opinion. I'm not speaking for my employer. I'm not speaking for anyone else, just myself. Just keeping that in mind.
Dwarkesh Patel
Cool. And where can people find your stuff? If you want to go over your blog link and your Twitter link and other things.
Joe Carlsmith
My blog is handsandcities.com. My Twitter handle is @jkcarlsmith. Those are good places to reach me. My personal website is josephcarlsmith.com.
Dwarkesh Patel
And where can they find your stuff on AI and those kinds of things?
Joe Carlsmith
The stuff on AI is linked from my personal website. That's the best place to go.
Dwarkesh Patel
All right, cool. Thanks for watching. I hope you enjoyed that episode. If you did and you want to support the podcast, the most helpful thing you can do is share it on social media and with your friends. Other than that, please like and subscribe on YouTube and leave good reviews on podcast platforms. Cheers. I'll see you next time.