In this highly anticipated sequel to our 1st AI conversation with Eliezer Yudkowsky, we bring you a thought-provoking discussion with Robin Hanson, a professor of economics at George Mason University and a research associate at the Future of Humanity Institute of Oxford University.
Eliezer painted a chilling and grim picture of a future where AI ultimately kills us all. Robin is here to provide a different perspective.
------ ✨ DEBRIEF | Unpacking the episode: https://www.bankless.com/debrief-robin-hanson ------ ✨ COLLECTIBLES | Collect this episode: https://collectibles.bankless.com/mint
------ ✨ NEW BANKLESS PRODUCT | Token Hub https://bankless.cc/TokenHubRSS
------ In this episode, we explore:
- Why Robin believes Eliezer is wrong and that we're not all going to die from an AI takeover. But will we potentially become their pets instead? - The possibility of a civil war between multiple AIs and why it's more likely than being dominated by a single superintelligent AI. - Robin's concerns about the regulation of AI and why he believes it's a greater threat than AI itself. - A fascinating analogy: why Robin thinks alien civilizations might spread like cancer? - Finally, we dive into the world of crypto and explore Robin's views on this rapidly evolving technology.
Whether you're an AI enthusiast, a crypto advocate, or just someone intrigued by the big-picture questions about humanity and its prospects, this episode is one you won't want to miss.
------ BANKLESS SPONSOR TOOLS:
⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum
🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://bankless.cc/kraken
🦄UNISWAP | ON-CHAIN MARKETPLACE https://bankless.cc/uniswap
👻 PHANTOM | FRIENDLY MULTICHAIN WALLET https://bankless.cc/phantom-waitlist
🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask
------ Topics Covered
0:00 Intro 8:42 How Robin is Weird 10:00 Are We All Going to Die? 13:50 Eliezer’s Assumption 25:00 Intelligence, Humans, & Evolution 27:31 Eliezer Counter Point 32:00 Acceleration of Change 33:18 Comparing & Contrasting Eliezer’s Argument 35:45 A New Life Form 44:24 AI Improving Itself 47:04 Self Interested Acting Agent 49:56 Human Displacement? 55:56 Many AIs 1:00:18 Humans vs. Robots 1:04:14 Pause or Continue AI Innovation? 1:10:52 Quiet Civilization 1:14:28 Grabby Aliens 1:19:55 Are Humans Grabby? 1:27:29 Grabby Aliens Explained 1:36:16 Cancer 1:40:00 Robin’s Thoughts on Crypto 1:42:20 Closing & Disclaimers
------ Resources:
Robin Hanson https://twitter.com/robinhanson
Eliezer Yudkowsky on Bankless https://www.bankless.com/159-were-all-gonna-die-with-eliezer-yudkowsky
What is the AI FOOM debate? https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate
Age of Em book - Robin Hanson https://ageofem.com/
Grabby Aliens https://grabbyaliens.com/
Kurzgesagt video https://www.youtube.com/watch?v=GDSf2h9_39I&t=1s
----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research.
Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures
No summary for this episode yet.
and one of the moves that often AI people make to spend scenarios is just to assume that AI's have none of that problem. AI's do not need to coordinate. They do not have conflicts between them. They don't have internal conflicts. They do not have any issues in how to organize and how to keep the peace between them. None of that's a problem for AI's by assumption. They're just these other thing that has no such problems. And then of course that leads to scenarios like that they kill us all. Welcome to Bankless, where we explore the frontier of internet money and internet finance and also AI. This is how to get started, how to get better, how to front run the opportunity.
This is Ryan Sean Adams. I'm here with David Hoffman and we're here to help you become more bankless. Guys, we promised another AI episode after episode with Ellie's. Well, here it is. Here's the sequel. The last episode of Ellie's or Kowski we titled correctly. We're all going to die because that's basically what he said. I left that episode with a lot of misgivings. existential dread. Yeah, existential dread. It was not good news in that episode and I was having a difficulty processing it. But Dave and I talked and we knew we had to have some follow up episodes to tell the full story. Bankless style and go on the journey of AI.
It's intersection with our lives with the world and with crypto. So here it is. This is the answer to that. This is Robin Hansen on the podcast today. Let me go over a few takeaways. Number one, we talk about why Robin thinks Ellie's is wrong. We're not all going to die from artificial intelligence, but we might become their pets. Number two, why we're more likely to have a civil war with AI rather than being eaten by one single artificial intelligence. Number three, why Robin is more worried about regulation of AI than actual AI. Very interesting. Number four, why alien civilization spread like cancer?
This is also related to AI and super interesting. Number five, finally, we get to what in the world does Robin Hansen think about crypto? David, why was this episode significant for you? Robin Hansen is such a great thinker. He's absolutely a polymath and really like Eliaser progresses in his thoughts in the very linear logical fashion. So he's easy to follow along with. So the first half of this episode, maybe the 45 minutes, 50 minutes is all about just the AI alignment debate and Eliaser versus Hansen, which is a debate that has actually been going on for many, many years. This is not the, well, over a decade.
Yeah, you're right. This is not the first time that Eliaser has heard about Robin Hansen or Robin Hansen has debated Eliaser. This is an ongoing saga. And so this is just coarse material for Robin Hansen. And so we really focus on this AI alignment problem and how these thinkers think that AI will develop and progress here on planet earth and how they will in friendly or unfriendly ways ultimately collide with humanity. So that's the first half of this episode. The second half of this episode, I think is when this gets really, really interesting. If you just listened to the first half of this episode, you would just think like, oh, this is the other half of the conversation to the AI debate, which it is.
The second half connects this to so many more rabbit holes and so many more topics of conversation that are actually, I would say deeply ingrained to bankless content themes, the themes of competition versus coercion, the themes of exploring frontiers, the thing of molok and the prisoner's dilemma and how things coordinate across species. And so we connect AI alignment to Robin Hansen's famous idea that he calls grabby aliens. If you haven't heard about grabby aliens, you're in for a treat. So this goes from what is a simple counter argument to a debate that we've had to a multifaceted exploration that is just so cursory of many, many, many deep subjects that I hope to explore further on bankless.
Yeah. And honestly, David, I'm dying to record the debrief with you because I want to get your take on this episode that was in context. You can see how giddy I was in the second half of the episode. I know. And I want to contrast it with our Ellie's episode and how these two thinkers think and who do you think has the stronger case. The debrief episode is the episode David and I record after the episode where we just talk about what just happened, give our raw, unfiltered thoughts. So we're about to record that now if you are a bankless citizen, then you have access to that right now. If you'd like to become a citizen, click the link in the show notes and you'll get access to our premium RSS feed where you'll have access to that.
Also, this episode will become a collectible next Monday. I'm collecting this episode so hard. Me too. I've got that easier episode of my collections. I'm also collecting this. We release episode collections for our key episode of the week. Every Monday, the mint time is 3 p.m. Eastern and whatever time zone you're in, you have to convert that. That's it. We're going to get right to the episode with Robin Hansen. But before we do, we want to thank the sponsors that made this possible, including our favorite crypto exchange Kraken, our recommended exchange for 2023, go set up an account. Kraken has been a leader in the crypto industry for the last 12 years dedicated to accelerating the global adoption of crypto.
Kraken puts an emphasis on security, transparency and client support, which is why over 9 million clients have come to love Kraken's products. Whether you're a beginner or a pro, the Kraken UX is simple, intuitive and frictionless, making the Kraken app a great place for all to get involved and learn about crypto. For those with experience, the redesigned Kraken Pro app and web experience is completely customizable to your trading needs, integrating key trading features into one seamless interface. Kraken has a 24.7365 client support team that is globally recognized. Kraken support is available wherever, whenever you need them by phone, chat or email.
And for all of you NFTers out there, the brand new Kraken NFT beta platform gives you the best NFT trading experience possible, rarity rankings, no gas fees and the ability to buy an NFT straight with cash. Does your crypto exchange prioritize its customers the way Kraken does? And if not, sign up with Kraken at Kraken.com slash bankless. Arbitrum 1 is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web 3 landscape. Hundreds of projects have already deployed on Arbitrum 1, producing flourishing DeFi and NFT ecosystems. With a recent addition of Arbitrum Nova, gaming and social apps like Reddit are also now calling Arbitrum home.
Both Arbitrum 1 and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar and fully EVM compatible. On Arbitrum, both builders and users will experience faster transaction speeds with significantly lower gas fees. With Arbitrum's recent migration to Arbitrum Nitro, it's also now 10 times faster than before. Visit Arbitrum.io where you can join the community, dive into the developer docs, bridge your assets and start building your first app. With Arbitrum, experience Web 3 development, the way it was meant to be, secure, fast, cheap and friction free.
Learning about crypto is hard. Until now, introducing Metamask Learn, an open educational platform about crypto, Web 3, self-custody, wallet management and all the other topics needed to onboard people into this crazy world of crypto. Metamask Learn is an interactive platform with each lesson offering a simulation for the task at hand, giving you actual practical experience for navigating Web 3. The purpose of Metamask Learn is to teach people the basics of self-custody and wallet security in a safe environment, and while Metamask Learn always takes the time to define Web 3 specific vocabulary, it is still a jargon-free experience for the crypto curious user.
Friendly, not scary. Metamask Learn is available in 10 languages with more to be added soon, and it's meant to cater to a global Web 3 audience. So, are you tired of having to explain crypto concepts to your friends? Go to learn.metamask.io and add Metamask Learn to your guides to get onboarded into the world of Web 3. Bangladesh Nation, we are excited to introduce you to Robin Hansen. He is a professor of economics at George Mason University in a research associate at the Future of Humanity Institute at Oxford. This takes an interdisciplinary research center approach that investigates big picture questions about humanity and its prospects, and I think explaining exactly who Robin is and what he's doing is not a trivial task because he's a polymath, certainly spans many things.
He's provided many different mental models across various disciplines, but I would not call him conventional by any means, and I'm sure, Bangladesh listener, you will see what we mean here today. Robin, welcome to Bangladesh. Glad to be here. I think I can try to explain the kind of weird that I am. Yeah, go ahead. Oh, please, because I can't explain the kind of weird I am. Tell us how you're weird. So, I think I'm conventional on methods and weird on topics. So, I tend to look for neglected important topics and where I can find centers for an angle, but I'm usually looking for a pretty conventional angle that is some sort of usual tools that just haven't been applied to an interesting important topic.
So, I'm not a radical about theories or methods. So, use things like science and math and statistics and all of those normal, non-radical things. I've spent a lifetime collecting all these usual tools, all these systems, really, and I'm more of a polymath in that I'm trying to combine them on neglected important topics. So, if you go to a talk where everybody's arguing and you pick a side, I mean, the chances you're right are kind of small in the sense that there's all these other positions. And, you know, maybe you'll be right, but probably you'll be wrong because you're picking one of these many positions, right?
If you go pick a topic where nobody's talking about it, you just say anything sensible. You can probably be right. And we've, I think, recently ran into somebody who follows that path of sorts, somebody who thinks very logically and rationally, but is applying it to unique, more unique frontiers of the place that humanity is. And that is our recent episode with Eliezer, who followed a decently logical path that was relatively easy to follow that unfortunately led us into a dead end for humanity. And so, it was something that bankless, that me and Ryan, as co-host of this podcast, but then also many of the listeners felt trouble with because Eliezer was able to guide us in a very simple and logical path over onto the brink.
And so we're hoping to continue that conversation with you, Robin, as well as being able to explore some new frontiers. Yeah, Robin, I'm just wondering if we could just wait right into the deep end of the pool here, because what happened is basically Eliezer came on our podcast. We thought we were going to talk about AI and safety and alignment and all of these things. You know, he talks about that a lot, and we thought we were going to tie that to crypto. What ended up happening midway through that podcast, Robin, is I got an existential crisis. So, David, the rest of the agenda seemed meaningless and unimportant because here is Eliezer telling us basically that the AI was imminent.
He didn't know whether it would happen in two years and five years and 10 years and 20 years, but he knew the final destination, which is that AI's would kill all of humanity and that we didn't have a chance. And basically, and I'm not being hyperbolic here, Robin, I know you haven't had a chance to go through that episode, but he basically says, you know, kiss your, spend time with your loved ones because you do not know how much time you actually have. And so this left, like me, and I think many bankless listeners on kind of a cliffhanger of like, oh my God, are we all going to die? And David tried to talk to me after that episode.
He's like, Ryan, it's okay, like, you know, but we knew we also had to like find someone who could give us another interpretation of what is going with on with AI. And Robin, we have chat, GPT4. It looks incredibly sophisticated. It looks like it's advancing at breakneck speed. And we're worried about the scenario. So when Eliezer, you'd Kousey says, we're all going to die. What do you, what do you make of that? Do you think we're all going to die? So AI inspires a lot of creativity regarding fear. And I think honestly, most people, as they live their lives, they aren't really thinking about the long-term trajectory of civilization and where it might go.
And if you just make them think about that, I think just many people are able to see scenarios, they think are pretty scary, just based on, you know, projection of historical trends toward the future and things changing a lot. So I want to acknowledge there are some scary scenarios if you just think about things that way. And I want to be clear what those are. But I wanted to distinguish that from the particular extra fear you might have about AI killing us all soon. And I want to describe the particular scenario. Eliezer has in mind, as I understand it, as a very particular scenario, we have to pile on a whole bunch of assumptions together to get to a particular bad end.
And I want to say, those assumptions seem somewhat unlikely and piling them all together makes the whole thing seem quite unlikely. But nevertheless, you just think about the long-term trajectory of civilization. It may well go places that would scare you if you thought about that. And so that'll be the challenge for us to separate those two. So which one would you like to go with first? I would like to start with understanding what you think his assumptions are. All right. Let's do that. Okay, so the scenario is you have an an AI system like some coherent system. It's got an owner and builder, people who sponsored it, who have some application for it, who are watching it and using it and testing it.
And you know, the way we would do for any AI system, right? There's an system. And then somewhere along the line, this system decides to try to improve itself. Now this isn't something most AI systems ever do. And people have tried that and it usually doesn't work very well. So usually when we improve AI systems, we do it another way. So we train them more and more data, get them more hardware, use it in algorithm. But the hypothesis here is we're going to train this system is going to be assigned the task, figure out how to improve yourself. And furthermore, it's going to find a wonderful way to do that.
And the fact that it found this wonderful way makes it now special compared to all the other AI systems. So this is a world with lots of AI systems. This is just one. It's not the most powerful or the most impressive or interesting except for this one fact that it has found a way to improve itself. And this way that it can improve itself is really quite remarkable. First of all, it's a big lump. So most innovation, most improvements in all technology is lots of little things. You gradually learn lots of little things and you get better. Once in a while, we have bigger lumps and that scenario here, there's a really huge lump.
And this huge lump means this system can all of a sudden be much better at improving itself than not only it could before, but in essence, then all the other systems in the world put together. It's really quite an achievement. This this lumped it finds a way to improve itself. And in addition, this way to improve itself has two other unusual features about innovations. First, it's a remarkably broad innovation applies across a very wide range of tasks. Most innovations we have and how to improve things are relatively narrow. They let it improve in a narrow range of things, but not over everything.
This innovation lets you improve a really wide range of things. And in addition, most innovations you have let you improve things and then the improvements run out until you'll find some other way to improve things again, but this innovation doesn't run out. It allows this thing to keep improving more than many orders of magnitude. You know, maybe 10 orders magnitude or something like it's just really a huge innovation that just keeps last, just keeps playing out, just keeps improving. It doesn't run into errors while it improves itself. It doesn't even as it discovers orders or things that slow it down and get stuck for a long time, it just it just keeps working.
Right. Okay. And whatever it does to pursue these innovations, these self modifications will change it. They probably will change its software configuration, maybe its relative use of resources, the kinds of things that ask for how it spends its time and money that it has, doing things, the kind of communication it has, you know, it's changing itself. And its owners, builders, the ones who are, you know, sponsored it and made it and have uses for it, they don't notice this at all. It is vastly improving self and its owners is just oblivious. Now, initially it's just some random obliviousness.
Now at some point, the system will get so capable, maybe you can figure out how to hide its new status at its new trajectory. And then there might be more plausible that it succeeds at that if it's now very capable at hiding things. But before that, it was just doing stuff, improving itself and its owner managers were just oblivious. Either they saw some changes that in care, they misinterpreted the changes, they had some optimistic interpretation of where that could go, but basically they're oblivious. So if they knew it was actually improving enormously, they could be worried, they could like step it, maybe pause it, try variations, try to study it, so make sure they understand it, but they're not doing that.
They are just oblivious. And then the system reaches the point where it can either hide what it's doing, or just rest control of itself from these owners, builders, and in addition, like if one that rests to control itself, presumably they would notice that, but then and they might try to retaliate against it or recruit other powers to just, you know, lock it down, but by assumption, it's at this point able to resist that. It is powerful enough to either hide what it's doing, or just rest control and resist attempts to control it, at which point then it continues to improve, becoming so powerful that it's more powerful than all the other, everything in the world, including all the other AI's.
And then soon afterwards, its goals have changed. So during this all process, two things have to have to have to have happened here. One is that have to become an agent that is, most AI systems aren't agents. They don't think of themselves as I'm this person in the world who has this history of these goals, and this is my plan for my future, you know, they are tools that do particular things. Somewhere along the line, this one became an agent. So this one says this is what I want, and this is what I am, and this is how I'm going to do it. And in order to be an agent that needs to have some goals, and during this process by which it improved, at some point it became an agent, and then at some point, its goals changed a lot, not just a little, in effect now.
So any system we can think in terms of its goals, if it takes actions among a set of options, we can interpret those actions as achieving some goals versus others, and for any system we can assign it some goals, although the range of those goals might be narrow, if we only see a range of narrow actions. So we might not be able to interpret goals more general. So if we have an AI system that is a taxi driver, we'll be able to interpret the various routes that takes people on, and how carefully it drives in terms of some overall goals, respect to how fast it gets people there, and how safely it does, but maybe we can't interpret those goals much more widely as what would it do if it were a mountain climber, something because it's not climbing mountains, right?
But still, with respect to a certain range of activities, it had some goals, and then by assumption basically in this period process of growing, its goals just become in effect radically different. And then by assumption, radically different goals through this random process are just arbitrarily different. And then the final claim is arbitrarily different goals. When they look at you as a human, you're mostly good for your atoms. You're not actually useful for much anything else at some point, and then you are recruited for your atoms, i.e. destroyed. And that's the end of the scenario here, where we're all we all die.
So to recall the set of assumptions we pile on together, we have an AI system that starts out with some sort of owner and builder. It is assigned the task to improve itself. It finds this fantastic ability to improve itself, very lumpy, very broad works over many orders of magnitude. It applies this ability. Its owners do not notice this for many orders of magnitude of improvement presumably. That's what happens really, really quickly, potentially. Well, that would be presumably the most likely way you can imagine the owner's not noticing perhaps. But the fundamental thing is the owner's don't notice.
If it was slow and the owner didn't notice, then these are and still plays out. You know, so the key reason we might post right fast is just to create the plausibility that the owner's don't notice, because otherwise why would they notice? That's also part of the size of this innovation. We're already improving AI systems at some rate. If this new method of improvement was only going to improve AI systems at the rate they're already improving, then this AI system won't actually stand out compared to the others. In order for this to stand out, it will have to have a much faster rate of improvement to be distinguished from the others.
This will then have to be substantially faster. That would set the timescale there for what it would be to be in the scenario. It both needs to be faster than the rate of growth of other AI systems at the time, substantially, and fast enough that the owner builders don't notice. This radical change in its agenda, priorities, activities, they're just not noticing that, and then they don't notice it unto the point where this thing acquires the ability to become an agent, have goals, hide itself or free itself and defend itself, and then the last assumption, and its goals radically change, even if it was friendly and cooperative with humans initially, which presumably it was.
Later on, it's nothing like that. It's just random set of goals at which point, then by assumption now, it kills us all. The question is how plausible are all those assumptions? We could walk through analogies and prior technologies and histories in the last few centuries, and I think flume advocates like Eliza will say, yeah, this is unusual compared to recent history, but they're going to say recent histories are relevant for this. This is nothing like recent history. The only things that are really relevant comparisons here is the rise of the human brain and maybe the rise of life itself and everything else is irrelevant.
Then they will reject other recent few centuries technology, trajectories as not relevant analogies. What did you just call Eliza? Robyn, a what advocate? A fume advocate? Fume, fume. What is fume? Fume is just another name for this explosion that we've been talking about. It's the most common explosion. I got to describe it. Gotcha. Cursewell's stuff, like that kind of thing. Singularity, that sort of thing. Singularity is a different concept than fume. Okay, different concepts. In some sense, a fume is a kind of singularity, but not all singularity. Robyn, thank you for guiding us because we're still learning on this.
Like, Bancluses, we had never done an AI podcast previously. We covered a lot with crypto and coordination and economics, and now we're doing this AI podcast, and I feel like we got just got punched in the face. So you rearticulating, yeah, we're walking slower. You rearticulating Eliza's assumptions is I think very helpful to me. And so we want to get to like why you think those assumptions are unlikely to be true, but I do think you are right. In the episode with him, he basically sort of painted this fantastical story of these assumptions. And he basically said, yeah, there's assumptions, the things that you're describing, I think.
And I don't want to put words in his mouth. So maybe this is what I was hearing him say, is you're just describing intelligence, Robin? That's what intelligence does. And I'll give you exhibit A. It's called human beings. And I'll give you the algorithm. It's called evolution gradient descent over millions of years and hundreds of millions of years. And we end up with a super like an intelligence, but relatives who made the animal kingdom a super intelligence that exerts its dominance and its will will has changed from just procreating and spreading its genes, genes, and mimetic material to something that evolution would have never that evolutionary algorithm would have never envisioned it actually doing.
And so I think maybe what I was hearing the criticism would be like, we already have an example of this, Robin. It's called intelligence and it's called humans. What do you think about this? So as I said, if we just think about the long run future we're in, we can generate some scenarios of concern independent of this particular set of assumptions, Elias or it's set up. So the scenario where humans arise and then humans change the world, I guess you could imagine as the scary to evolution if evolution could be scared, but evolution doesn't really think that way. But certainly you can see that in the long run you should expect to see a lot of change and a lot of ways in which your descendants may be quite different from you and have agendas that are different from you and yours.
That is, I think that's just a completely reasonable expectation about the long run. So we could talk about that in general as your fear. I just wanted to distinguish that from this particular set of assumptions that were piled on as the Fumestar. As the Fumestar, it was like something that might happen in the next few years say and it would be a very specific event. A particular computer system suffers this particular event and then a particular thing happens. That's a much more specific thing to be worried about than the general trajectory of our descendants into the long-term future. So I'd like, which one would you like to talk about?
I'm trying to summarize really just the perspective differences here and I know you've gotten, you've had this debate with Elias or before. So this is, this is like review for you. I think Eliasers' conclusions is that while the future is unwritten and the paths of our future can be many in multivariate and we can have different possible outcomes, Eliasers is like, well, all roads lead to the superintelligence taking over and I think just to summarize your position is that is a possible path and it is something to consider and it is still less likely than the many, many, many other possible paths that are also perhaps an aggregate much more likely.
Is that a fair summary of your position? So let's talk about this other more general framing and argument. So we could just say in history humanity has changed a lot, not just a little a lot. We've not just changed some particular technologies. We've changed our culture in large ways. We've changed the sort of basic values and habits that humans have and our ancestors from 10,000 or a hundred thousand a million years ago, if they looked at us and saw what we were doing, it's not at all clear they would embrace us as they are proud descendants they are proud of and happy to have replaced them.
That's not at all obvious. Even just in the last thousand years or even shorter, we have changed in ways in which we have repudiated many of our ancestors most deeply held values. We've rejected their religions, we've rejected their patriotism, rejected their sort of family allegiance and family clans or the legions as we we have just rejected a lot of what our ancestors held both dear. And that's happened over and over again through a long term history. That is each generation we have tried to chain our train our children to share our culture. That's just a common thing humans do, but our children have drifted away from our cultures and continue to just be different.
And you know over a million years, humans fundamentally ourselves change and one of the things that happened is we became very culturally plastic and so culture now is really able to change us a lot because we are we have become so able to be molded by our culture. And even if our genes haven't changed that much, although change substantially say in the last ten thousand years, our culture has enormously changed us. And if you project the same trend into the future, you should expect that this will happen again and again, our descendants will change with respect to cultural evolution and their technology and the structure of their society and their priorities.
And then of course at some point, not to distant future, we will be able to reengineer what we are or even what our descendants are. And that will allow even more change. That is once we can make artificial minds, for example, there's a vast space of artificial minds we can choose from and we will explore a lot of that space. And that allows even more big possibilities for our descendants could be different from us. So this story says our descendants will become yes super intelligent and yes they will be different from us in a great many ways which presumably also include values. And if what you meant by alignment was how can I guarantee that my distant descendants do exactly what I say and believe exactly what I believe in will never disappoint me in what they do because they are fully under my control.
I got to go gee that looks kind of hard compared to what's happened in history. So now if that's the fear you have, I got to endorse that. That's not based on any particular scenario of a particular computer system soon and what trajectory of an events it will go through. That's just projecting past trends into the future in a very straightforward way. So then I have to ask like is that what you're worried about. No, that is not what I'm worried about. That is my base case that like we're going to get more intelligent technology is going to change us culturally. It's going to change the trajectory.
Okay, but I gotta turn one zinger to this. Okay. What if change speeds up a lot? So that this thing you thought was going to happen in a million years happens in a hundred. Well, I mean for me personally I'm more of a techno optimist. So I would be more on the side of like within reason, of course, more embracing of these types of change. I know others aren't quite as embracing and also this was not the scenario at all that Ellie user presented. He presented the scenario of not rapid change that you might not like in the future and it could come within your lifetime. But actual obliteration of humanity like literally rearranging our atoms for some other artificial intelligence purpose.
And while you agree with like there will be lots of change as there has been in the past perhaps that change will even accelerate as we we we delve deeper into the kind of the technology that that is in our future. You do not think that an AI will simply the super intelligent artificial intelligence will simply obliterate humanity and kind of wipe us from from creation entirely. It'll be it won't be quite as drastic as that. But let's be careful about noticing exactly what's the difference between the scenario I presented and the scenario he presented because they're not as different as you might think.
In both scenarios there's a descendants. In both scenarios the descendants have values that are different from ours. And in both scenarios there's certainly the possibility of some sort of violence or you know disrespective property rights such that the descendants take things instead of asking for them or trading for them. Because that's always been possible in history and it can role remain possible in the future. You know today most changes peaceful waffle. And there are of course still big things that happen but mostly it's via trade and competition. And if the AI's displays us it's because they beat us fair and square at our usual contest that we've set up by which we compete with each other.
So these scenarios aren't that different. I'm trying to point out. And then the key difference here is one is the timescale. How fast does it happen? Another is how spread out is it is there the single agent who takes over everything or are there millions of descendants, billions of them who slowly went out and displays us? How far do their values differ from ours? Just how much do they become indifferent to us? And then do they respect property rights? Is this a peaceful, lawful transition or is there a revolution or war? Those are the main distinctions between these two scenarios we've described.
Elias who's described, sorry it was a very fast, there's a single agent, its values change maximally and it doesn't respect previous property rights. Whereas the scenario I'm describing is ambiguously fast, hey it could happen much faster than you think of millions or billions of descendants of a perhaps gradual and intermediate level of value difference but substantial but primarily I would think in terms of peaceful, lawful change. I think there's a missing component to this conversation that we've been having recently and I understand that there are things about the evolution of this AI and things that are about the evolution of humanity that are basically synonymous.
There's iteration, there's development, there's progress and Robin you gave the account for that when we raise our kids, we try and imbue them with our values and our cultures and there is transcription errors in that and that only so much of our values and cultures get passed along to our kids and perhaps as technology advances, even less passes long from generation to generation and our culture changes over time and this is what we call progress and when we go back to the AI innovating on itself, you also presented a scenario of improvement errors without as well. We don't know how perfectly it can improve and so as it develops it changes and adapts and these are all similar structures and so this is what we know and maybe the time scales throw us off a little bit but these are similar patterns.
There's one component missing that I'd like to highlight and dive into. When we have our generations of kids and humanity that progresses and even if it changes, it's still started from us in the first place, right? There's a logical continuation of parent to kid, parent to kid, parent to kid and so it at least starts from a place of continuation. I think the problem with this AI alignment and super explosion issue is that in the moment that we create this AI, it actually doesn't upload our value system because we are creating a completely new life form and so it is not biological life, it is not DNA that is growing up to an adult to combine with somebody else's DNA to create a kid who then grows up.
It's like that isn't being carried forth. So in the moment that we create AI, it has no trail of evolutionary history to imbue it with values and judgment and how to perceive the world in an aligned fashion and so in that creation moment, it is completely rogue and we don't know how to understand it and it doesn't know how to understand us because it is a completely new form of life within completely new form of appreciating and understanding values and I think that's the missing component, even though there are similarities in how these things progress, the bootloader for values and alignment is missing in this AI and I think that we haven't touched on yet.
So I do some work on aliens. We could talk about that later if you want, but I'm quite confident. I'm looking forward to that part of the conversation by the way. I'm all the aliens out there in the universe and all the alien AI's that they would make. The AI's that we will make will be correlated with us compared to them. We aren't making random algorithms from the randomly for the space of all possible algorithms and machines. That's not what we're making. We are making AI's to fit in our world. So you know like the large language models made recently the most impressive things. Those are far from random algorithms in the space of all possible algorithms.
They are modeling after us and most in the next few decades as we have more AI applications, machines will be made by firms trying to make profits from those AI's and what they'll be trying to do is fit those AI's into the social slots that humans had before. So they'll be trying to make the AI's like humans in the sense that they will have to look and act like humans well enough to sit in those social slots. If you want an AI lawyer, it'll have to talk to you like somewhat like a human lawyer would. And similarly for an AI housekeeper, etc. We will be making AI's that can function and act like humans exactly so that they can be most useful in our world.
And we are the ones making them. And so just out of habit, we're making them like us in some abstract sense. Now, there's a question of how much like us and then the question, there's a question of how much did you want and how much is feasible and how really close to your kids anyway or your grandkids because just remember how much we humans have changed. I think when you look at historical fiction or something, it doesn't really come across so clearly. We humans have changed a lot in our changing a lot even in the last century. If you just look at the creative change of human culture and attitudes and styles in the last century, project that forward 100 more centuries, you've got to be imagining our descendants could be quite different from us, even if they started from us.
And it's interesting and mostly software changes, would you say like at the cultural level, I mean, human hardware hasn't really changed that because I mean, although we have substantially changed the hardware too, but yes, most lawful, but in the future, we will be able to make hardware changes to our descendants. I have this book called The Age of M Work, Love and Life from robots rule the earth and it's about brain emulations. And so this is where we make very human-like creatures who are artificial, using artificial hardware, but then they can modify themselves and become more alien more easily because they can more easily modify their hardware and software as they are basically computer simulations of human brains.
So if that happens soon, then even that human line of descendants will be able to become quite different in a relatively short time. Right. If you thought the AI alignment problem would throw you for it to the, I can't wait until we get into the conversation about synthetic biology separating humans to some be gods and others not be gods, but that'll be a different podcast. Robin, I think in your argument here, you baked in the belief, the assumption that robots will adopt our values merely by like osmosis from the devs and the engineers who are coding them up because they will code them up to do certain things and behave in certain ways using characters on our English or our keyboards, for example, and just merely by being association of being created by us, we will, it's actually impossible to not imbue them with our culture and our values.
Is that what you're saying? Well, there's a big element. How do you, how is it that you think your children are like you? I mean, they are basically really biological cells, not computers, though. Humans are really quite culturally plastic. Maybe that's another thing people really don't quite get. So anthropology has gone out and looked at a really wide range of human cultures and found that humans are capable of behaving and thinking very differently depending on the culture they grew up. That's the basic result of anthropology. There are some rough human universals, but mostly we're talking variation.
The fact that you are different, seem very similar with all the other humans around you is not about sort of the innate human similarity you have. It's because you are in a similar culture to them. So it's just to rearticulate your position here. I think we are saying that alias are perhaps fearful that the super intelligent AI and humans are so far apart that they can never come to coexist. And what you're saying is that life as a whole has similarities no matter how it manifests or how it is expressed. Is that how you would say it? I was trying to tell you that your descendants could be really different from you.
I wasn't trying to convince you that there was a bound on just how different your descendants could get. I was trying to show you that in fact your descendants could get really different. Not through this food star, just through the simple default way that society can continue to change. If you're going to be scared about the food star, or maybe you should be scared about that one too, we could start to talk about what we might know in general about intelligent creatures and what might be the common features across them for all alien species through all of space time or something. There probably are some general things they have in common, but they might be fewer than would comfort you.
I definitely want us to get there, but really quick, just picking apart the assumptions that you laid out, and I want to see which ones more specifically you might disagree with or state in a different way than Ellie user. You said, you know, assumption one is that the AI improves itself. It seems core to what Ellie user thinks. Assumption two, the owners that is the people who program it don't take control, don't try to stop it. Assumption three, the AI becomes an agent, an assumption four, the agent's goals change, the AI's goals change, it ends up destroying humanity. I find some of these harder to, like, believe than others, particularly assumption four.
Like, I just, I didn't understand an Ellie user's argument. The reason that suddenly the AI destroys humanity, like that, maybe we could talk about, but let's start at the top actually. Do you have a disagreement with assumption one that the AI and AI will recursively start to improve itself? Remember, I tried to break a one into multiple parts to show you that it requires multiple things all to come together there. So not only does it try to improve itself, it finds this really big lumpy improvement, which has enormously unusual scale in terms of how far it goes before it runs out and scope in terms of how many things it allows the improvement of.
And, you know, magnitude is just a huge get win over previous thing that those are all a prior and likely things. Also, it's not the fact that tries to improve itself, that seems quite likely sure. Somebody might, well, ask a system to try to improve itself, but that it would find such a powerful method and then still not be noticed by its owners, that gets pretty, you know, striking as an assumption. I understand. And so that's where it's tied into like, you find it hard to believe that the owners, the creators of this AI wouldn't be able to stop it from doing something nefarious or devious.
That is also a difficult assumption. Well, it's first just noticing that is by assumption, this thing starts out at a modest level of ability, right? By assumption, this thing is comparable to many, many other AI's in the world. So by assumption, if you could notice a problem early on, then you can't stop it because, you know, you can bring together thousands of other AI's against this one to help you stop it if you want to stop it. So at some point later on in this evolution, it may no longer be something you could stop, but, you know, by assumption, that's not where this starts. It starts at, you know, being comparable to other AI systems.
And then it has this one advantage. It can improve itself better. And then this other, this other assumption, than what I've labeled number three, that AI becomes an agent. So how likely is an AI to become a self-interested acting agent? Is that difficult to foresee? Well, of course, some owners might make it that way, but most won't. So we're still, we're narrowing down the set here. So so my old friend Eric Drexler, for example, has argued that we can have an advanced AI economy where most AI's have pretty narrow tasks. They aren't general agents trying to do everything they drive cars to the airport or whatever they, they each do a particular kinds of task.
And that's in fact, our economy structure. Our economy is full of industries, made of firms, each of doomed, who do particular tasks for us. And so, world with those firms are now much more capable, and even, you know, artificially intelligent capable, even more than superhuman capable, can still be a world where each one does a pretty narrow task. And therefore, isn't a general agent that would, you know, do enormous change things if it became our problem. So if you had a system that was really good at route planning, say cars to get from A to B, if it was superhuman at that, it might just be really good at route planning.
But if that's all it does, there's, it's not plausibly going to suddenly transition to an agent who sees itself as having history and all goals for the world and, and trying to like figure out how to preserve itself and, and make itself go. That's pretty implausible for a route planning AI. So in a plausible future, most AI's would be relatively narrow and have relatively task, but sometimes somebody might make more general AI's that had more general scope and ambitions and purposes. And then those might be the basis of a scenario here, but the people who created those AI's, they would know it's unusual feature.
They would know this one is an agent. And they would presumably take that into account in their monitoring and testing of this thing. That's not, they're not ignorant of this fact. So the scenario whereby the route planning one just accidentally becomes an agent, I mean, that's logically possible. But now we've got to say, you know, how often do design systems for purpose A suddenly transform themselves into something that does all different thing B, it happens sometimes, but it's pretty rare. And so let's say it gets through all of these gates, right? We have an AI that improves itself in in broad ways and in ways that are, you know, somewhat lumpy.
The owners for whatever reason aren't able to take control. The AI, you know, strict them in some way. Maybe the owners have programmed this AI to become an agent. So it's an agent acting and it's free will this, this last point then, Ellie's conclusion is like the point that was most concerning, of course, is that then this AI comes and destroys humanity. And, you know, I think his rationale is basically because why not, it would have other purposes for humanity would just, you know, step over them. What about this assumption? So imagine instead of one AI, we have a whole world of AI's who are improving themselves and their values are diverging, that's more of a default scenario.
If that happens in a world of property rights, then say humans are displaced and no longer at the center of things were not in much demand, we basically have to retire. Humans go off to our retirement corner and spend our retirement savings. If that stays a peaceful scenario, then all these AI's who, you know, change and have other purposes, they don't have to kill us. They can just ignore us off in the corner, spending our retirement savings. But if there's a possibility of a revolution, say whereby they decide, hey, why let these people sit in the corner, let's grab their stuff. So, I mean, the possibility of a violent revolution has always been there and it's there in the future, but in the world we're living in, that's a rare thing.
And that's good. And we understand roughly why it's rare. So the thing that's happening different in LA Azure scenario is because it's the one AI you see, it's not in a society where revolutions are threatening, it's just the one power. And then from its point of view, why let these people have their property rights? Why not take it? Now, I would say that the main thing there is not that it has different goals, but that it's singular. And therefore not in a world where it needs to keep the peace with everybody else and be lawful for fear of offending others or the retribution, that it can just go grab whatever it wants.
That's the distinctive feature of the scenario is describing in a more decentralized scenario. Again, I think there's much more hope that even if AI is displaces, even if their goals become different from us, they could still keep the peace because, possibly, they could be relying on the same legal institutions to keep the peace with each other as they keep with us. And that's in some sense, why we don't kill all the retirees in our own, take their stuff, right? Today, there's all these people who are retired and like, what have they done for us lately, right? Kill the retirees and take their stuff, but we don't.
Why don't we do that? Well, we share these institutions with the retirees. And if we did that, that would threaten these institutions that keep the peace between the rest of us. And we would each have to wonder who's next. And this would end well. Okay. And that's why we don't kill all the retirees and take their stuff, not because they're collectively powerful and can somehow resist their efforts to kill them. We would go and actually kill them and take their stuff. That would actually physically work. That's not the part of it. That's not the problem with that scenario. The problem is what happens next after we kill them, takes the stuff.
Who do we go for next? And where does it sit? So a future of AI's who become different from us and acquire new goals and our agents threatens us if they have a revolution and kill us and take their stuff, our stuff. That's the problem there. And so, AI's solution you see makes that seem more likely by saying there's just the one agent. It has no internal coordination problems. It has no internal divisions. It's just the singular thing. And honestly, we could add that as another implicit assumption in this scenario. He assumes that as this thing grows, it has no internal conflicts. It becomes more powerful in the entire rest of the world put together.
And yet, there are no internal divisions of note. There's no right. It doesn't have different parts of itself that fight each other and that have to keep the peace with each other because that's why we have law and property rights. You see in our world is because we have conflicts. And this is how we keep the peace with each other. And he's setting that aside by assuming that it doesn't need to keep the peace internally because it's the singular thing. You know Uniswap as the world's largest decks with over 1.4 trillion dollars in trading volume, but it's so much more. Uniswap Labs builds products that lets you buy, sell, and use your self-custody digital assets in a safe, simple, and secure way.
Uniswap can never take control or misuse your funds. The bankless way. With Uniswap, you can go directly to DeFi and buy crypto with your card or bank account on the Ethereum layer one or layer two. You can also swap tokens at the best possible prices on Uniswap.org. And you can also find the lowest floor price and trade NFTs across more than seven different marketplaces with Uniswap's NFT aggregator. And coming soon, you'll be able to self-custody your assets with Uniswap's new mobile wallet. So go bankless with one of the most trusted names in DeFi by going to Uniswap.org today to buy, sell, or swap tokens, and NFTs.
The Phantom Wallet is coming to Ethereum. The number one wallet on Solana is bringing its millions of users and beloved UX to Ethereum and Polygon. If you haven't used Phantom before, you've been missing out. Phantom was one of the first wallets to pioneer Solana's taking inside the wallet and will be offering similar staking features for Ethereum and Polygon. But that's just staking. Phantom is also the best home for your NFTs. Phantom has a complete set of features to optimize your NFT experience, pin your favorites, hide the ugly, remove the spam, and also manage your NFT sale listings from inside the wallet.
Phantom is, of course, a multi-chain wallet. But it makes chain management easy, displaying your transactions in a human readable format, with automatic warnings for malicious transactions or phishing websites. Phantom has already saved over 20,000 users from getting scammed or hacked. So get on the Phantom Waitlist and be one of the first to access the multi-chain beta. There's a link in the show notes, or you can go to phantom.app-waitlist to get access in late February. So we should really hope for a world of a pluralistic world of many AIs. And in fact, you think that's a more likely world anyway.
Of course, yes. So we're already in a world of great many autonomous parts, right? We have not only billions of humans, but we have millions of organizations and firms. And even nations and government agencies in one of the most striking features of our world is how it's hard to coordinate among all these differing interests and organizations. And one of the most striking features of our world are the mechanisms we use to keep that peace and to coordinate among all these divergent conflicting things. And one of the moves that often AI people make to spend scenarios is just to assume that AI's have none of that problem.
AI's do not need to coordinate. They do not have conflicts between them. They don't have internal conflicts. They do not have any issues and how to organize and how to keep the peace between them. None of that's a problem for AIs by assumption. They're just these other thing that has no such problems. And then of course, that leads to scenarios like then they kill us all. Right. Like AIs are a monolith. But I think one of the reasons why I appreciate just your line of reasoning, Robin, and how you think is that you tap into what seems to be fundamental truth of this universe that you would find here on planet earth or in a galaxy far, far away, certain things I think can be assumed no matter what the environment is.
And then I think a lot of your logical conclusions are just like natural extensions of that. And so I want to, unless you have a thought on that, I'm just going to say I think a lot of disagreements in the world are often based on people having sort of different sets of abstractions and mental tools. And then finding it hard to merge them across topics. So I think when a community has a shared set of abstractions and mental tools, even when they disagree about details, they can use those shared abstractions to come to an agreement. But when you have people with just different sets of abstractions, that's true.
So I'm bringing a lot of economics to this. Other people might be bringing a lot of computer science, but I'm going to play my polymath card and say I've spent a lifetime learning a lot of different sets of conceptual tools and intellectual systems, including computer science, certainly big chunks of. And so I'm trying to integrate all those tools into a overall perspective where I can sort of pull in each observation or insight into this. So is this the economic reason the robots aren't going to come kill us then maybe is that is that what you're kind of providing? Or, or just if they kill us, they would do us in the usual economic way.
So it can only make sense to make sure that nobody will ever kill you, okay? They have to have good reasons. People have been killed in the world with a past, but we have an understanding of the main ways that in the last few centuries, people have been killed. That's been something people have paid attention to. How do people get killed? How does that happen? And so theft is like murder is one kind of way people get killed or is another way revolution is another way or sometimes just displacement where something out competes you and then you don't have any place to survive. So in some sense, like horses got out competed by cars at some point and they suffered substantially.
We understand how that works out. So that's the sort of thing that happened to humans we could suffer like the way horses did. Did horses suffer though? I mean, they are by population standards. Yes, they diminish significantly. Did any individual horse suffer and feel suffering as a result of cars seems like a good life to be on an equestrian farm rather than sort of slaving in a city escaping whipped by a by a master. My understanding is horse population is now, you know, as high as it ever was. But of course, you know, this is not as high as you might have projected how they continued previous growth rates.
So there was a substantial decline and then we're now most horses are pets and not more courses. Right. I'm not sure if I'm ready to be a pet, but that's a problem for my kids probably, hopefully, just quick scenario, Robin. What's more likely a single monolithic super intelligent AI does the Eleazar thing or we have humans have a robot human conflict war. And it's like, it's more like kind of maybe in the traditional sense where we have two sides and there's in what's more likely. So that second one seems far more likely to me. But I just you should just put into the context that is humans are at the moment very by an enormous number of parameters.
We vary by gender and age and profession and geographic location and wealth and personality. And in politics, especially, we try to divide ourselves up and form teams and coalitions by which together we will then oppose other coalitions. And this is just an ancient human behavior where which, you know, we form coalitions and fight each other. And we expect that will continue. So arguably say democracy has allowed us to have more peaceful conflicts or coalitions fight in elections rather than in wars. But even in say firms, you know, there's often political coalitions that are fighting each other.
And there's always the question, what is the basis of the dominant coalitions? So there's this wide range of possibilities. You could have a gender based, you know, the men fighting the women, you could have an age one, the old people fighting the young, you could have an ethnicity one, you could have a profession one. So an affirmative might be the engineers versus the marketers. Right. And so humans versus robots or robotic descendants is one possible division on which future conflicts could be based. That's completely believable. And I can't tell you that can't happen. The main thing I'll just point out that will be competing with all these other divisions.
So will it be the humans versus robots conflict or will it be the old versus young or will it be the word cells versus the, you know, shape rotators? I mean, there's all these different divisions. And it could well be that there's an alliance of human word cells and AI word cells versus human shape rotators and AI shape rotators. And that becomes the future conflict. You see, because in some fundamental sense, the division of the conflict is indeterminate. That is a fun one thing we understand about politics is whatever divisions you have, it's unstable to the possibility of some new coalition forming instead.
That's a basic thing we understand about politics. It's hard to keep stable coalitions because they're so easily undermined by nuance. At least with the human versus robot coalition, like looking into past human behavior, we tend to be pretty racist. But I think when we have robots, it would be really easy to forget our internal conflicts when there's a completely different resource. Like, why do we fight? Why do humans fight? It's usually over resources, like economic resources. And when there is a new species that is subdividing and iterating and growing as humans do, that's also sucking up the resources.
And they look, like, I don't know if they're going to be metal in the future. But that's my current envision vision of them is like the metal silicon terminator type robots walking around. And there's so many resources on the planet. And so like, that would be a pretty easy dividing line between humans and robots that I could imagine would make that conflict much more likely. And so regardless of like how maybe it's the LEAzer way in which a super monolithic super intelligent robot comes and we have to fight that. Or like at some point, there's conflict potentially. And I might even say likely if there is a different, like, call it species, there's a recently, this is kind of aside, this is going back to like the super intelligent stuff.
But I think we can now call this just AI conflict. The future of life organization released an open letter calling for the pause of all general AI experiments. A few people signed it, Elon Musk, Steve Wozniak, you've all know a Harari Andrew Yang. I just want to get your reaction to this letter and people signing it, Robin. Like, would you sign this letter or are you against signing this letter? And just what do you think about the idea of this? David, can we tell people what it says to it's basically a call on all AI labs to immediately pause for at least six months. The training of AI systems more powerful than chat GPT4.
So this letter says don't go beyond chat GPT4 beyond there. It gets even scarier. Let's pause, let's halt. Let's figure out this AI alignment issue first. So first just notice that we've had a lot of pretty impressive AI for a while now. It's when the AI's are the most human-like with these large language models that people are the most scared and concerned. So that suggests that maybe a very advanced AI will look pretty human-like in many ways. And don't forget that our descendants will start to add metal to themselves and become different in many ways. Just like the training relations are metal and quite different.
So again, it's not so obvious where the division line would be. But to go to this particular letter, first of all, with respect to the general concerns they have, if we had a six-months pause, at the end of that we really wouldn't know much more than we know now. The main purpose of the pause would seem to be to allow, say, the time for government to get its act together and have institutes some more official law that enforces such a pause to continue. That would be the main purpose for the pause. You'd be wanting to support the pause if you were wanting that further to happen. That's not like we're going to learn that much in six months.
Or how about if maybe you were a competitor and wanted to catch up? So then we go to the, is this, you know, so there's first, if we could do the pause, would it be a good idea? And then one of the issues is like who would be participating and who not? So the ideal thing is say we could get a global pause somehow. Would that be a good idea? Now we're basically talking about should we basically shut down this technology for a long time until people feel safer about it? So for that issue, I think the comparison with nuclear energy is quite striking. Basically around 1970, the world decided to back off a nuclear energy and we basically instituted regulatory regimes that allowed the safety requirements asked of nuclear power plants to escalate arbitrarily until they started to cut into costs.
And that basically guaranteed this would never become lower cost than there are other ways of doing things. And people were okay with that because they were just scared about nuclear power. So basically the generic fear didn't go away. And they just just narrowly said this could just never be safe enough for us. Whatever extra budget we have, we want it to be safer. And that's the way they put it. And so I would think, a similar thing would happen if they are the kind of reassurances people are asking for are just not going to be feasible for decades at least. So you'd basically be asking for this to be paused for decades.
And you know, it's even hard to imagine that eventually overcoming that. Because you know, the fundamental fears as we've been describing is just the idea that they might be different. And they might have different agendas and they might outcompete us and that's just not going to go away. So I would see this as basically do you vote for substantial technological change or not. And I get why many people might think, look, we're rich enough we're doing okay. Let's not risk the future by changing stuff. And they voted that way on nuclear power and they might well vote that way on AI. I would rather we continue to I think we have enormous far we can go if we continue to improve our tech and grow.
But I can understand why many people think no, we got lucky so far we things didn't go too bad. We're in a pretty nice place. Why take a chance and change anything. So that's all if it was possible to actually have a global enforcement of such a pause and then a further law. But of course that just looks really hard. That is you know this technology is now pretty widely available. That is you know it might be that the best new systems are from the biggest companies that kind of forward the most hardware to put on it. But the basic software technology here is actually pretty public and pretty widely available.
And so you know over the next few decades even if you managed to say no no more than a billion dollar project doing this you're going to have a lots of less than billion dollar projects doing this. And of course it'll be hard to have a global band. And so the US now has a commanding lead and the main effect of a delay if it's not global would be to take away the US lead. And it's just this is looks like a hard technology to ban honestly. You know you might be able to get Google and open AI and Microsoft or something to pause because their efforts because you know they are big companies with pity public activities.
But and Robin I'm trying to understand so even if it was enforceable understand the reasons you gave why it's not enforceable and it's very difficult to do some sort of a you know global ban of some sort. Let's say it was for a minute would you support it? Do you think this is worth pulling the fire alarm over? But again I think it's comparable to say genetic engineering or nuclear energy or some other large technologies that we've come across in the last few decades where there really is huge potential but there's also really big things you could be worried about. And honestly I think you just have to make a judgment on the overall promise versus risk framing.
You can't really make a judgment here based on very particular things because that's not what this is about. We made a judgment of nuclear energy to just back off and not use it that much. That's a judgment humanity made 50 years ago within the last few decades we made a similar judgment on genetic engineering basically. Nope we just don't want to go there for humans at least and we may be about to make a similar decision about AI but honestly this trend looks bad to because many people think social media is a mistake and maybe we should like undo that and go back on that. So the trend of blocking technological progress is bad to you in general whether it's nuclear or genetic engineering or social media or AI or or any of these things.
Right actually I'm concerned that this is the future of humanity actually here. So I did this other work on grab aliens on sort of the distribution of aliens to space time and in that framework the most fundamental distinctions between alien civilizations is the one between the quiet ones who stay in one place and live out their history and go away without making much of a mark on the universe and loud ones who expand and then keep expanding until they meet other loud ones and I can see many forces that would tend to make a civilization want to be quiet and that's what we're talking about here.
That is even in the last half century the world has become a larger integrated community especially among elites whereby regulatory policy around the world has converged a lot even though we have no world government. You certainly saw that in COVID but you also see it in nuclear energy and medical ethics and many other areas. Basically the elites around the world in each area talk mainly to each other. They form a consensus worldwide about what the right way to deal with that area is and then they all implement that and so there's not actually that much global variation in policy in a wide range of areas and people like that I think compared to the world certainly it's reduced civil wars of various kinds and people like the idea that instead of nations fighting and competing with each other that we're all talking together deciding what to do together and that that sort of talking makes you know deal with global warming it may deal with inequality it may deal with overfishing there's just a bunch of world problems that these people talking together feel like they're solving and people will like this world we're moving into where we all talk together and agree together about what to do about most big problems and that new world will just be much more regulated in the sense that they will look at something like nuclear energy and then everybody say no we don't want to do that and that's shame anybody who tries to do that and slowly together limit humanity's future and that could go on for thousands of years and then if we ever have a point where as possible to send out an interstellar colony to some other star we will know that if we allow that that's the end of this era once you have a colonist go somewhere else that they are out of your control they are no longer part of your governance sphere they can make choices that disagree with what you've done they can then have descendants who disagree they can evolve and become different from the center and come back eventually to contest control over the center so that becomes a future world of competition and evolution that could be go very strange and start places but if we would all just stay here and not let anyone leave then we can stay in this world of us we talk together we decide things together we only allow our descendants to become as weird as we want them to be if we don't want to start kind of weird descendants we just shut it down and that's the quiet civilization that we may become and that's kind of what's that stake here I would say with Danny AI it's one of many questions like that that we are answering about do we want to allow change and new large capacities that might threaten strangeness and conflict so I think this is actually the moment where this podcast episode goes from continuing the conversation that we had about AI with Eliezer and all of those alignment problems and that conversation and this actually becomes a part of a larger conversation that we've been having on bank list for a while now and this has to do with the status quo versus innovation and progress as well as it does with what you were just saying Robin about grabby aliens and so I want to try and connect these dots really quick this idea of AI and AI innovation along with crypto innovation and whether or not it should be regulated by the elites by the status quo and whether it should be contained and or and are the elites happy with the way of the the harmony of the social order and perhaps we shouldn't have new competition and new exploration into the frontier because that is how we maintain the social order because there's nothing new that's happened that what what you're saying that this does is this keeps us in a it's like an isolationist approach except it's an isolation approach from like inside a planet earth and I think being the future tech optimist that Ryan IR and I think you are as well you aren't for that you would like to penetrate that isolationism that has from the like the social elites saying hey let's not experiment with crypto or AI or longevity or synthetic biology research let's just like keep everything harmonized and in control and we will use our large centralized power to keep that keep the world under control and then we have this other conversation that we're about to go into which is grabby aliens which is whoever is these alien species that is expanding out into the world chose to not do that they chose to explore the frontier they chose to innovate under the guys of competition of capitalistic competitive competition to innovate and start to expand outwards into into space and I think baked in your argument is that you actually do need competition in order to become to explore the frontier and so I'm wondering if we can if if if this is a if that was a good summary and be kind of like like do you see that that picture of just like how this concern about AI or concern about progress in general is also linked to like this the grabbyness or quietness that you call an aliens and maybe you can characterize these these different kind of aliens and the choices that they make of it as a civilization yes I thought that was a reasonable summary I think when we see people today discuss the possibility of our descendants spreading into the galaxy they are often wary and a bit horrified by the impact they might have that is the sort of people we've become over the last century are people who find that a jarring and even unpleasant scenario because it is actually fundamentally jarring and unpleasant so I am with you in wanting to allow such changes but I want to be fully honest about the cost that we are asking the world to accept that is if you wanted our descendants to just stay pretty much like this individual us in event in sorry indefinitely that's not what we're talking about here if the cost of allowing our descendants to expand into the universe and explore technologies like AI and nuclear power etc is literally alienation that is we are now alienated from our ancestors our world and lives are different and we feel that at some level that we were not built for the world we're in this is an alien world that we're in compared to the world we were built for we we feel that deep inside us and that will continue it will only get worse and you know the the time it will get better is when we can go change who's inside us to become more compatible with these alien worlds but that will those descendants even more different from us so that's that's really the cost you have to be asking so this future world of strange new technologies is also a competitive world and that competitive world includes conflict it includes some some people dislike some kinds of things displacing others some things just being shunted aside marginalized and it may even include war violence it certainly probably includes radical change to nature not just biology on earth but you know our descendants who go out into the universe would likely not just pass by and plant flags they will take things apart rearrange them and radically change them and sometimes that'll be ugly sometimes it'll be violent it's sometimes it'll leave crude ugly waste uh and be inefficient where if they could have done it better that will be the course and this universe we see now that's pristine and nicken you know the way it was from long ago will just be erased that's the cost so i want to explore this idea of um of grabby aliens and i'm sure listeners who are being thrown into this like odd adjective grabby might be a little bit confused and so i'm hoping we can explain the nature of grabbyness but i'm hoping we can actually do it inside of the context of planet earth and human history because i think that naturally extrapolates into the galaxy because this is where the the place of grabby aliens that's where they play and first i think i want to ask you the question humans are we grabby because if you look back in history you have uh some sort of quiet human races quiet human species human tribes that uh found that that were found by the grabby humans you can call these the conquistadors or the the conquerors right the Roman Empire very grabby empire uh any sort of empire that like looked outward and expanded i would under your uh trying to understand like robin Hanson's like air works of of grabbyness i would call any sort of empire that expanded grabby and then these grabby empires found the quiet like tribes that were peaceful and grabbed them uh and then assimilated them into the grabbyness and so this is kind of how i would ask i would like present this inside of a context that we understand because we understand human history but i want to ask you this very basic question of just like human nature are we grabby so almost all biology has been grabby and therefore almost all humans but it's not so much about our nature so the fundamental point here is there's just a selection effect that's the key point that is if you have a range of creatures with different cultures or biological tendencies and some of them go out and expand another's note if there is a place they can expand to and they would actually you know could reproduce there then there's a selection effect by whichever ones do that they then come to dominate the larger picture uh that's just the key selection effect so there may be many alien species in civilizations in the universe and maybe most of them choose not to expand but the few ones who do allow expansion they will come to dominate by space time volume the activity of the universe and that's how evolution has worked in the past it's not that all animals are all plants are aggressive violent and hostile it's that they vary and some of them have an habit of sticking one place in hiding and other habit of jumping out and going somewhere else when they can and the net effect of the variation and all their habits is when there's a new island that pops up it gets full of life because some of those things that move land there and grow and any new mountain grows higher and then new life shows up at the top of the mountain and a you know new niche opens up of any sort where life is possible there and then some life goes there and uses it that's just the selection effect so that's what we should expect in the universe there's the question of which way we will go and if I focused on humans I'd say it's a trade-off between what would happen if we don't coordinate and how hard we will try to coordinate so an uncoordinated humanity there's certainly enough variation within humanity some of us would go be gravity it might not be most of us but certainly some of us given the opportunity would go grab mercury or Pluto or whatever else it is and then go out and grab farther things we might choose to prevent that we might choose to organize and coordinate so as to not allow that those things to happen and we might succeed at that we are we have enough capability perhaps to do that and so then it becomes a choice we'll we allow it but basically whatever you're talking about something that that only takes a small fraction of us to do and we vary a lot then the question is will we allow that variation to make it happen or will we somehow try to lock it down the bankless audience is pretty familiar with the idea of mollock it's a it's a topic that we've revisited a number of times are you familiar with mollock i'm familiar with the famous scot Alexander essay on it and although i think the concept isn't entirely clear in that essay sure yeah so mollock just being like the idea of the prisoners dilemma say you have two or two or or almost any number of human tribes on the earth and most of them decide to be quiet and peaceful it really only takes one to be grabby and that one will come to dominate the earth because it chose to be grabby and it grabbed everything else so it's almost this prisoner's dilemma about if you choose to not be grabby you are implicitly making the choice of being grabbed by the larger larger tribe that has elected to be grabby and i think this is this is how we extrapolate this into the future with your grabby aliens thesis where there are sure there are many civilizations out there maybe there are many like us that only exist on one planet and we have a bunch of elites on the planet that say hey let's not investigate AI and let's not investigate longevity or or genetic engineering let's just stay put and we would quite call these quiet aliens or you know us being the quiet aliens what that the choice being made is that grabby aliens are eventually going to arrive on earth and grab us and so if you don't become a grabby alien you are going to be grabbed by somebody else and so this is my my why i think this this moment in human history when we have this letter saying hey let's pause AI research is what you are focusing on is like well this is a very important decision point for humanity as to whether we choose to be quiet or not quiet and of course this isn't the only choice but this is one of the many choices down a long list of choices that could actually decide culturally what we want to be in at least for the short term is this is how you see this this fork in the road as we currently are well let's just clarify say in a peaceful society like ours we could think of a thief as grabby and then we could say well if we don't steal somebody else will steal so i guess we should steal and you could imagine a world where that was the equilibrium but if we coordinate to make law then we can coordinate to watch for some a thief and then repress them sufficiently so as to discourage people from being thieves so a universe of sufficiently powerful aliens could coordinate to prevent grabbing if they wanted the claim which i believe is true that in fact the universe hasn't done that it might be that within our human society we have coordinated to enforce particular laws but out there in the rest of the universe it's just empty and there's pretty much nobody doing anything it's the most of it that we can see and so it is really just there for the grabbing no one's gonna slap our hands down for grabbing the stuff we could just keep grabbing until we reach the other grabby aliens at which point then we might try to set up some peaceful law to keep the peace between us and them but we don't have to fight wars with other grabby aliens per say but there's all this empty stuff between here and there then it seems like you either grab it or somebody else does i'm wondering if we may have blown past some listeners here who hurt us just talking about alien civilizations they're coming to like grab earth and they're like what do you guys talking about what worth like all of these alien civilizations robin david we don't see them anywhere when we look up the stars but that is what your your grabby aliens paper is all about i think the synopsis of the grabby aliens paper you know packs this punch if loud aliens explain human earliest quiet aliens are also rare robin can you sort of explain what your grabby aliens idea actually is and why there might be future alien civilizations that are expansionary and coming our way and why we might want to be a civilization that that rises up and expands in our own sphere of influence in order to meet them so we're gonna go through this briefly and quickly turns out there's just a kurs gus sucked video that came out yesterday that how has 2.
6 million views that's explaining some of the basics of grabby aliens in case people want to see that kursa the cute animations that do these very technical things in very nice ways congratulations on that by the way so the key idea is we wonder about the distribution of aliens in space time and one possible theory you might have is that we're the only ones at all and in the entire space time of that we can see there'll never be anybody but us in which case the universe would just have waited for us to show up whenever we were ready we can reject that interpretation of the universe because we are crazy early so our best model of how advanced like a life us should appear says that we should be most likely to appear on a longer live planet toward the end of its history and our planet is actually very short-lived you know our planet will last another billion years for roughly five billion years total of history the average planet lasts five trillion years and because life has to go through a number of hard steps to get to where we are there's actually a power law in terms of when it appears as a function of time the power being the number of steps and so say the steps are six then the chance that we would appear toward the end of a longer live planet rather than now on this planet is basically that factor of a thousand in their lifetime raised to the power of six for this power law i.
e. 10 to the 18 more likely to have appeared later on in the universe so we're crazy early relative to that standard and the best explanation for that is there's a dead light soon the universe is right now filling up with aliens taking over everything they can soon in a say a billion years or so it'll all be full and all taken at which point you couldn't show up later on and be an advanced civilization everything would be used for other stuff and that's why you need to believe they're out there right now so now that you've got to believe they're out there right now you wonder what are we'll how close are they what's going on out there and for that we have a three parameter model where each of the parameters is fit to a key data point we have and this model basically gives you the distribution of aliens in space time and you know if you like we can walk through what those parameters are and what the data point we have for each one is but the end story is civilizations typically expanded a very fast speed a substantial fraction of the speed of light they appear roughly once per million galaxies these grabby alien civilizations and if we head out to meet them we'll meet them in roughly a billion years spending near the speed of light so they're quite rare that rare but not so rare as to be empty in the universe that is once per million galaxies there's many trillions of galaxies so that means there are millions of them out there and right now the universe is roughly half full of them so that seems strange the universe looks empty but you have to realize there's a selection effect everywhere you can see is a place where if there had been aliens there they would be here now instead of us so the reason things looks empty is because you can't see a place where they are because they would move so fast where they already get to here that here would be taken the fact that we are not now taken here says that no one could have gotten here and therefore if they so if you were able to look out into the stars and see the aliens that's nonsensical because if that would be possible they were to have already grabbed you by that time right because they move so fast there's a relatively small volume of the universe where you could see them and they haven't quite got here yet most of the places you could see they would be here and grabbing you doesn't necessarily mean destroying you it just means possibly expanding to the borders such that you can't expand into their borders it would be enveloping you and then changing how the world around you looks so we can be pretty sure we have not now been enveloped by a gravity alien civilization because we look around us and we see pretty native stars and planets which are not then mathematically changed so yes in the future we might be involved and other things out there might be involved but we're not now we couldn't see this situation we're in if those alien civilizations that come here see and this is actually why this intersects with the aliezer AI problem because the way that you said that the civilizations that are out there would have come and enveloped us and then changed the environment that is around us thank hopefully leaving us at peace but like this is the AI alignment problem in another form where like another rogue alien civilization is also another paperclip maximizer and they're out just gathering all the resources doing the things that they do according to their values hopefully their values are that when they do expand into our civilization they leave us alone because some alignment is still there but it is the same fundamental structure of like there is these goals and alignments with the universe around them and these aliens expand and they change the atoms of the matter that they expand into and because we haven't seen that yet because that's the assumption that we have but because we haven't seen that yet you are able to and your grabby aliens paper actually like kind of place us in the arc of history because of this assumption that grabby aliens are grabby and that they will attempt to grab us to add to that I mean they might be artificial intelligences as well wouldn't they rob it sure almost surely they are yeah you know anything you know within a thousand years I expect our descendants to be almost entirely artificial and certainly within a million years and these things would be billions of years older than us so yes they we would be our artificial descendants will meet their artificial descendants and maybe a billion years and they won't have saved something like us now I can give you a little more optimism in the sense that if aliens these grabby civilizations appear once per million galaxies if the ratio of quiet to loud ones is even as high as a thousand to one that would mean that in this expansion that they've been doing that they will do they'll only ever meet a thousand of these quiet when as they expand through a million galaxies and so these rare places where an alien civilization appeared would be pretty special and worth you know saving and isolating because grabby alien civilizations should be really obsessed with what will happen when they meet the other grabby civilizations they'll be really wanting to know what are these aliens like because they'll have this conflict at the border and they will wonder are we going to be out past somehow will they trick us somehow what what's going to happen when we meet the border but they might look at a national park out of us then light but so so my turn every grabby civilization will be really eager for any data they can get about what are aliens like and so this small number of quiet civilizations they come across will be key data points they will really treasure those data points in order to just give us some data about what can aliens be like and so that would be a reason why aliens came and enveloped us they would mainly want to save us as data about the other aliens now you know that doesn't mean they don't freeze dry soil and run experiments etc.
I mean it's not necessarily going to let us just live our lives the way we want but they wouldn't just erase this all either well Bangladesh nation I elect Robin Hanson to make the case for not freeze drying us and to preserve us to the aliens if they come at some point in time but this is not necessarily a near term that they're coming but it's it's more kind of the rate of spread one interesting aspect of the model is it's it would it be accurate to say Robin that the model predicts alien civilizations like spread light cancer and I mean that maybe mathematically and you know without the negative connotation that that brings well alien civilizations are created even more like cancer so in your body you have an enormous number of cells and in order for one of your cells to become cancerous it needs to undergo roughly six mutations in that one cell during your lifetime so that's basically the same sort of hard steps process that planets go through planets are each in order to achieve an advanced civilization they also need to go through roughly six mutations that is the mutations are each unusual thing has to happen and then the next unusual thing happens and then the next unusual things happens until all six have happened and then you get something like us so the key idea is there's you know a million galaxies each of which have millions of planets and then all these planets are trying to go down this path of having all these mutations but almost none of them do successfully by their deadline of life no longer be possible on that planet and it's a very rare planets like ours for which all six mutations happen by the deadline of life no longer be possible on the planet and that's how cancer is in your body that is 40% of people have cancer by the time they die and that means one of their cells went through all six of these mutations but that was really unlikely vast majority of cells only had one or zero mutations and so life on a planet reaching advanced level that it could expand in the universe is mathematically exactly like cancer right and so it follows the same power law with time actually so the probability that you get cancer as a function of your life is roughly time to the power six because it takes roughly six mutations that's why you usually get cancer near the very end and the chance that planet will achieve advanced life is roughly the power six for time and that's why in fact in the universe universe is appearing over time faster and faster according to roughly a power six because of this exact power law and so very early universe had almost nothing and then recently we've shut up but around us they're all popping and the rate at which they're appearing now is much faster than it was in the fast because of this power law and it shouldn't be lost on listeners that cancer is grabby cancer falls in the grabby category and so there's a bunch of quiet cells that are just mining their own business doing their job in harmony with their neighbor cells and then one cell goes rogue and decides I'm going to grab everything that I can around me and I'm going to grow to my best ability and so like it's just interesting to see no matter what scales or what mediums we perceive to be whether it's the biological cell it is human species as a whole it is this theoretical AI super intelligent robot but like the same structures continue to show up and so Robin thank you for helping us navigate all of these different planes of existence and being able to reason about them all at once well we did a brief survey but right happy to have a number of different rabbit holes that we you know did not go down in the industry how about this because we've got a crypto audience Rob do you have any hot takes on crypto what do you think of this stuff how about that I'll check that box I mean I don't have a new take on crypto my old take has always been you know for any new technology you need both some fundamental algorithms and some fundamental tech and then you need actual customers and some people to pay attention to those customers and their particular needs and you have to adapt the general technology to particular customer needs crypto unfortunately moved self into a regime where most of the credit and celebration and money went from having a white paper and an algorithm and not so much for actually connecting to customers and so unfortunately there's this huge appetite for tools and platforms under the theory that if we make a tool and platform other people will do the work to connect that to customers and unfortunately there's not so many people who are sitting in that next rolls but them succeeding at that task is the main thing that will makes the rest of crypto succeed or not there's plenty of tools and platforms not so many people trying to market concrete products to particular customers and holding their hand working with them when it doesn't work them changing us somehow iterating in order to make a product actually worked for concrete customers that's how pretty much all business innovation needs to happen in crypto as well as everywhere else it doesn't crypto isn't different by this regard it's just that crypto sort of fell into this world where you got all the recognition and attention and money by writing white papers and implementing their first version of the algorithm that you had then shipped to then moved over to another company to write another white paper and algorithm right instead of actually staying with the algorithm and trying to get customers to use it so I wish crypto well there's lots of interesting possibilities there but that's in my mind the major problem with crypto is the lack the neglect of actual customers yeah everyone details of making customers happy everyone in the crypto industry is recently at least the VC landscape is talking about everyone is trying to sell picks and shovels and no one's bothering to actually sift for gold right so maybe we need some more gold diggers out there I think so and this is Robin being a utility show me the utility and we certainly understand that that take on crypto and we'd certainly have some work to do in that area but I think we should have you on some time again Robin and there's so much we could pick your brain about I know you're a huge advocate for prediction markets as a way to solve for things and this is one of the other creative institution ideas that I think crypto people will be more interested the most crypto people are pretty open to creative institutions oh we are so you got to come back and and talk to us about sort of institutions and some of the new creative institutions because I don't know if you notice Robin but around us it seems like a lot of our institutions are crumbling or falling to pieces or releasing their trust or just in the worst case we're just locking down and not allowing much innovation or change it's 100 percent is so all right well let's leave this conversation as it's only decaying slowly they're still not innovating and growing this is a to be continued bankless nation if you haven't had enough of Robin Hanson I certainly haven't I could talk to this man for hours then let us know and we'll see if we can get him back on another time but you you have helped me understand a bit more about artificial intelligence and and for that I certainly help are you going to sleep tonight I'm going to sleep much better tonight honestly you know yes so thank you you know there are some things in there I think I need to relisten to and think about a little bit more the these descendants being so much unlike me that that might make me concerned but I'm far less concerned than after the leaser episode so I appreciate that it's a natural concern as it is that's right action items for you bankless nation will include a link to the leaser kowski episode we're all going to die it was called that was seriously the title Robin and we'll also include a link to the a i fume debate which we talked about that term I just learned what that term was the age of m a book that I'm adding to my a q from robin Hanson of course talking about this is artificial minds is it not robin artificial implementations of ordinary human there you go that sounds fascinating and of course grabby aliens there is a kirkazat video as well as the original website grabby aliens dot com will include all of that in the show notes risking disclaimers got to let you know none of this has been financial advice it's not even a space-faring civilization advice I don't think you can definitely lose what you put in but we are headed west this is the frontier it's not for everyone but we're glad you're with us on the bankless journey thanks a lot