In this episode, we delve into the frontier of AI and the challenges surrounding AI alignment. The AI / Crypto overlap at Zuzalu sparked discussions on topics like ZKML, MEV bots, and the integration of AI agents into the Ethereum landscape.
However, the focal point was the alignment conversation, which showcased both pessimistic and resigned optimistic perspectives. We hear from Nate Sores of MIRI, who offers a downstream view on AI risk, and Deger Turan, who emphasizes the importance of human alignment as a prerequisite for aligning AI. Their discussions touch on epistemology, individual preferences, and the potential of AI to assist in personal and societal growth.
------ 🚀 Join Ryan & David at Permissionless in September. Bankless Citizens get 30% off. 🚀 https://bankless.cc/GoToPermissionless
------ BANKLESS SPONSOR TOOLS:
🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://k.xyz/bankless-pod-q2
🦊METAMASK PORTFOLIO | TRACK & MANAGE YOUR WEB3 EVERYTHING https://bankless.cc/MetaMask
⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum
🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle
👾POLYGON | VALUE LAYER OF THE INTERNET https://polygon.technology/roadmap
------
Timestamps
0:00 Intro 1:50 Guests
5:30 NATE SOARES 7:25 MIRI 13:30 Human Coordination 17:00 Dangers of Superintelligence 21:00 AI’s Big Moment 24:45 Chances of Doom 35:35 A Serious Threat 42:45 Talent is Scarce 48:20 Solving the Alignment Problem 59:35 Dealing with Pessimism 1:03:45 The Sliver of Utopia
1:14:00 DEGER TURAN 1:17:00 Solving Human Alignment 1:22:40 Using AI to Solve Problems 1:26:30 AI Objectives Institute 1:31:30 Epistemic Security 1:36:18 Curating AI Content 1:41:00 Scalable Coordination 1:47:15 Building Evolving Systems 1:54:00 Independent Flexible Systems 1:58:30 The Problem is the Solution 2:03:30 A Better Future
----- Resources
Nate Soares https://twitter.com/So8res?s=20
Deger Turan https://twitter.com/degerturann?s=20
MIRI https://intelligence.org/
Less Wrong AI Alignment xhttps://www.lesswrong.com/tag/ai-alignment-intro-materials
AI Objectives Institute https://aiobjectives.org/
------
Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research.
Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures
No summary for this episode yet.
Welcome to Banclists, where we explore the frontier of internet money and internet finance. And today, on this episode of our Zuzalu series, we are exploring some new frontiers, new frontiers and new technologies, all of which are poised to completely revolutionize the world and change everything about the operating system that society is currently running. Banclists nation today, we are exploring the frontier of AI, which is actually a frontier that we've already been exploring on Banclists. So if you've been listening to our other AI episodes, these will make you feel right at home. AI had a big week at Zuzalu.
The AI crypto overlap, everyone knows this huge, and it seems like such a massive frontier that people don't actually know where to start with it. ZKML or machine learning models and data that's verified by zero knowledge cryptography was a huge topic of conversation, and you'll hear about that in our cryptography episode with Daniel Short. Philip Diane at AI Week gave a killer talk titled MEV for AI people, which was this gigabrain presentation about how MEV bots in aggregate kind of presents this omnipotent omnipresent artificial intelligence. And since MEV has been decently corralled and contained, maybe we can learn a thing or two from the MEV industry in our approach to managing AI risk.
There were conversations at Zuzalu about how AI can put the autonomous back into DAO's and how AI agents could soon be roaming the Ethereum landscape shoulder to shoulder with all the human players out there. But mainly, as Zuzalu, the AI conversation inevitably converged into the alignment conversation of which you'll find two flavors here in this episode, one strongly pessimistic, and then the other characterized by this resigned optimism that is prevalent throughout all of Zuzalu's frontier tech challenges. Imaginable rewards blocked by seemingly insurmountable obstacles. Of first in this episode, we have Nate Sorries, who is the executive director at MIRI, the machine intelligence research institute of which Eliezer Yudkowski founded.
Nate's perspective on AI and AI risk is definitely downstream of Eliezer. So we pick up where Bankless left off with Eliezer, and Bankless Nation, it's dark. But nonetheless, Nate admits that it's less dark than it was a few months ago now that the world is waking up to the potential risk that AI brings to this world. Following the conversation with Nate is Dager Turan, who is charging into the AI frontier with his head held high with a clear path forward for himself. Dager believes that the AI alignment problem is actually just downstream of human misalignment, and that we actually won't be able to align AI until we align ourselves.
This conversation has to do with epistemology, what is truth? Individual preferences and how AI models can help us become the best versions of ourselves because if we become the best versions of ourselves with the assistance of some AI tool, we can collectively produce the best versions of our communities. And if we do that, then our communities can coalesce into the best versions of society, all aided by truth telling AI agents who can help humans navigate through our chaotic world of social organization and politics and social media. Really a fascinating conversation that is actually pretty proximate to our conversation with Tim Urban that we had not too long ago.
I'm really excited for you to listen to these conversations, bankless sessions, so let's go ahead and get right into it. But first, I'm going to talk about some of these fantastic sponsors that make this show possible. Kraken Pro has easily become the best crypto trading platform in the industry. The place I use to check the charts and the crypto prices, even when I'm not looking to place a trade. On Kraken Pro, you'll have access to advanced charting tools, real-time market data, and lightning fast trade execution, all inside their spiffy new modular interface. Kraken's new customizable modular layout lets you tailor your trading experience to suit your needs.
Pick and choose your favorite modules and place them anywhere you want in your screen. With Kraken Pro, you have that power. Whether you are a seasoned pro or just starting out, join thousands of traders who trust Kraken Pro for their crypto trading needs. Visit pro.krakken.com to get started today. Metamask has something new. Introducing Metamask Portfolio. Metamask Portfolio is the best way to view your crypto portfolio from a holistic level. See everything across all the chains all at once. In your portfolio, Metamask will report the aggregate value of all the assets in your Metamask wallets.
And even the other wallets you import too. But Metamask Portfolio isn't just a passive portfolio viewer. It is a place to do all of the money verbs that make DeFi so powerful. You can buy, swap, bridge, and stake your crypto assets. So not only is Metamask the easiest place to see your wallets in aggregate, but it's also a powerful battle station for all of your DeFi moves. So go check out your Metamask Portfolio because it's waiting for you to open it up. Check it out at portfolio.metamask.io. Arbitrum is accelerating the Web3 landscape with a suite of secure Ethereum scaling solutions. Hundreds of projects have already deployed onto Arbitrum 1.
With a flourishing DeFi and NFT ecosystem, Arbitrum Nova is quickly becoming a Web3 gaming hub. And social dApps like Reddit are also calling Arbitrum Home. And now Arbitrum Orbit allows you to use Arbitrum Secure Scaling Technology to build your own layer 3, giving you access to interoperable customizable permissions with dedicated throughput. All of these technologies leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM compatible. Faster transaction speeds and significantly lower gas fees. Are you a dev, but you don't know solidity?
With Stylus, Arbitrum's upcoming proposal for a programming environment upgrade, developers can write smart contracts in Rust, C, C++, and many more coding languages. Arbitrum empowers you to explore and build without compromise. Visit Arbitrum.io where you can join the community, dive into the developer docs, bridge your assets, and start building your first app on Arbitrum. Thank you so much. We are here with Nate Stories, and we are starting the AI week here at Suzanne. And Nate, is AI researcher? Is that how you would call yourself? Alignment researcher? I would say I'm the executive director of the Machine Intelligence Research Institute.
These days, it's last research that I'd like. I have done alignment research before. Okay. Can you explain the institution? The institution? What is that? So I didn't found it. It was founded by Elias Yeard Kewski. I don't know exactly when, maybe around 2001. Fun fact, it was originally founded. It was originally called the Singularity Institute, and it was founded because Elias were wanted to make a GI as fast as he could. And then along the way, he realized that it doesn't go well by default, and it doesn't go well for free. And so then the organization pivoted to trying to make this AI stuff go well.
And for many years, the institute did some research, did some field building, did some awareness raising, and so on and so forth, until around 2012, 2013, when they pivoted to peer technical research. And this was related to some of the field building, some of the awareness raising, moving to other groups as the field got a little larger. And I got involved when they pivoted to the technical research more exclusively. And so I was originally involved as a technical researcher, and then when the previous executive director left, I was the air apparent. Okay. Sorry. The machine, what's the name of the institute?
Machine Intelligence Research Institute. Okay. A. Mary. Mary. Okay. So it sounds like Mary has its own trajectory of itself that probably runs in parallel with like human understanding with machine learning at large. Perhaps, but also perhaps quite compressed. I think I don't know the exact years I wasn't around then, but I think it was only a couple of years of work before Elias was like, hey wait, this could get tricky. Right. Okay. So I actually didn't know that about Elias or that first he was like AGI as fast as possible. And then he was like, whoa, whoa, whoa, AGI as slow as possible. Yeah, I think I think I'm not sure as slow as possible, but like AI done correctly, AGI done correctly.
I think we were hoping for a long time. Like one of the reasons we do technical research is that like often your lever is just like solve the problem, like screw slowing people down, that like this is people off. It's like slowing people down is sort of a last resort. The original hope was can we just like solve the problem in time? It doesn't look like we're on track to solve the problem and it looks like we have less time than, you know, I was hoping back in 2014. And so I think it is with great sadness that people like Elias are and I are now saying we need time. We need more time. Can you kind of walk us through your own trajectory?
How do you how do you become the executive director at Mary? Well, if you want to go back far enough, I at a pretty young age realized that the world was not very well organized and wasn't I was I was in a civics class and up until that particular civics class, I had some intuition that like there were lots of problems in the world but people were sort of trying to fix them and the reason there were still all these big problems in the world was that we didn't have the technology. We didn't have the like we were still a young race. We were still a young species. We hadn't like matured to the point where we could fix these issues.
This was sort of like an implicit wordless intuition rather than a conscious belief and then, you know, I started learning how the US government works and I was like, oh god, it's like run by a bunch of monkeys. Like it's like monkeys invented like monkey systems and it's all like working about as well as you'd expect if it was like invented by people at no idea what the hell they were doing and then like a lot to run for like hundreds or sometimes thousands of years and just like carrying off into various and so it was like okay, like obviously from the MoLock and crypto world we would call this coordination coordination failure.
Totally. Yeah so so I was very interested in solving coordination failures and more generally in making the world better place and I tried my hand at various versions of that while I was like pretty young and bad at things and you know it's it's a hard it's a hard problem none of it worked and the sort of security circuitous route is that ultimately I got a job in tech while I was like trying to find ways to really move the levers on that problem decided to donate a decent amount of money to good causes to sort of keep me honest about actually trying to make the world a better place was trying to research where the best places to put money were donated to some like global poverty type charities and then started bumping into these arguments that like maybe this AI stuff is actually one of the biggest places to intervene on the world.
I sort of read up on it there are a couple other factors in my life that were also causing me to notice this this AI thing I read up on it and I was like oh geez like this is just obviously you know I was looking at the wrong problem like the coordination problems are big and they're real but like this AI thing you know like humanity like lives or dies and gets like a great future or no future depending how this AI thing goes and I just like completely missed this problem for like eight years of thinking myself as trying to make the world a better place and going for the for the heart of the problems but so when you went into the AI topic did you see for did you first see AI is a solution to all of our coordination issues or AI is a problem for all of our coordination issues or did you see it simultaneously at the same time a little bit of both I was I was maybe somewhat primed towards understanding some of the issues with AI due to my work on coordination problems it's like slightly embarrassing but I was like working on like various coordination mechanisms that could address the sort of concerns people had at the time like how can a well coordinate society without coercion address for example like concentration of wealth in ways that the society as a whole doesn't like and you can you can set a various coordination mechanisms of like like yeah you can sort of try to think about like what a non-course of ways that a society as a whole can like try to both have a market system and not let it get out of control in certain ways and while while like messing around like toy models this and like attempts to like prove certain theorems I just like couldn't get some of the results I wanted and it turns out that I couldn't get some of the results I wanted because nothing stops one actor from being powerful enough that they can just run away with everything right and this was sort of like it was one of those issues where I was sort of like well you know I can get it to work in a lot of cases but I can't get it to work in all cases and then with the AI stuff I was like oh that's why like can you can you elaborate on that like why why does the AI why is the AI like the kernel of the issue I mean AI is a is a version of that particular issue but like fundamentally no matter how good your market coordination are or your market coordination systems are on earth like if somebody has the raw technological power to like for example get what we call in the in the business a decisive advantage so like like maybe the easiest thing to imagine given that we already know that you know trees are machines that turn dirt and sunlight into more trees by stripping carbon out of the atmosphere and building wood we know that like nanotex possible if you imagine something that just like gets to nanotex before everything else and it can just like reassemble you into a more willing trade partner that asks for less of the gains from trade suddenly all of your coordination mechanisms that were like market-based and non-core server whatever melt before this thing and like am I saying that literally happens or literally like is in a market framework not particularly but you can sort of see how like I was sort of like trying to put a like collaborative agents interacting framework on a like physical reality where it's just a fact with physical reality that like things with a sufficient technological edge just uh can work the table with you if if they have too much of an edge and if they don't care about you simply put is this kind of are you just combining MoLock problems and the bankless nations pretty familiar with MoLock problems we've we've done a lot of content on MoLock you combine MoLock problems with exponential technology and then you arrive at some sort of like logical end gains where humans get their atoms repurposes that more or less the the simple articulation uh it's not a bad summary um I wouldn't use exponential in particular I'm not I make no strong claim that it's an exponential curve my guess is that uh it's it's not and it's worse um and uh you know much of the issue here is if you make something that uh is optimizing the world and it's optimizing the world towards some target that doesn't have concern for you in it um like I would have I would have much fewer qualms about like uh I would have basically no qualms about uh like human technological development I sort of am very optimistic about uh humanities better natures and humanity like being able to figure out uh like how to make the world more like we would want it to be upon reflection and if we were wiser and rather than like locking ourselves into totalitarian dystopias which I think totally could happen but like if we can just like ramp up human like intelligence and capability and so on without like accidentally killing ourselves I'm like pretty bullish on humanities prospects and so it's not it's not so much like oh no technology's coming just coming too fast we want to be able to handle it it's more like oh no we are like on the brink of building optimization processes that optimize the future much harder faster better than we can and they're optimizing it to a place that has no room for us and so it sounds like AI is one way and that might happen but you're also saying the way that you're talking it sounds like there's other ways in AI or not when the same opt hyper optimized future that's not optimized for humans could play out without AI like I sort of expect we're going to get to super intelligence one way or the other AI looks to me like one of the like basically the only feasible route modulo like if if humanity can't come together and coordinate to take some other route I think other routes like whole brain emulation are probably preferable where to be clear I'm no carbon chauvinist and I very much want to live in a future with like artificial friends where those artificial friends have like very different sorts of like desires and goals and objectives from me I'm not like humanity must keep an iron grip on the future I want like space for aliens I want space for artificial minds I want space for other kinds of life the like concern here is like building a mind that doesn't care for life that doesn't care for fun that doesn't care for like diversity of experience and like interesting arcs and like cosmopolitan value and like broad inclusive like good times and I think that we are in fact barreling towards that cliff edge of making something that fills the universe not with weird valuable stuff but with non valuable stuff okay so you've been thinking about these problems for a long time when did you start it Mary what year was that uh that was uh 2014 but I was hired um I once I noticed that the problem existed uh I donated I think $16,000 which I think at the time put me in the top 10 public donors list which uh and they were like congratulations you're now in the top 10 public donors list and I was like what and they're like you know we're doing we're doing I don't remember the exact amounts but they're like we're doing our fundraiser for $200,000 for our yearly budget this year uh $100,000 of which we're trying to raise in the community and like $100,000 of which is like matched from another donor uh and you know it's like three weeks into the four week fundraiser and they raised like 20k of it and I was like oh god like this is worse than I thought oh my goodness hilarious that and I was in 2014 or was that that was 2013 uh yeah so I donated to your revival actually working at Mary that's right so I donated more money um because I had no nose that bad uh did you end up funding yourself for your own salary uh I took I took a very big pay cut uh moving from Google to Mary um uh and uh I sort of you know I I was expecting not to be very skilled at working on these issues and you know maybe I'm not there there have been a lot of people uh on these issues but at the time uh they uh uh I was like how how can I help and they're like oh maybe if you go to the math you can like come to somewhere workshops and so I I came to one of their workshops and then uh a few months later they were hiring me and then a year later they were saying can you run the place um so uh it uh I I am largely in this field by dent of and this position by dent of showing up early uh and like for the love of god people more skilled than me uh like biomeans come replace me right so that that's kind of what what I was uh leading us into so that was 2014 when you started it's now 2023 so you're almost there for a decade now um now AI is having a moment um very much spurred by chat GPT all of sudden crypto podcasts are talking to AI people um what is that trajectory like so as somebody who was immediately compelled by the problem at its very essence so far long ago now fast forward to where we are now and like kind of the problem seems to be on the horizon I don't know how close it is I don't think anyone does it's kind of the problem but like here we are nine years later and now many many many people are talking about it can you just talk about that experience yeah uh it's it's heartening um uh one one thing that I have really enjoyed about it is uh I've spent many years having conversations with people in the field uh many of whom sort of don't really want to hear that uh their work by default is uh like barreling towards destruction uh and so I have like these long conversation trees like I have rejoinders to all sorts of counter arguments uh and when I go into these discussions with with uh you know people on the capability side of things uh I have like all sorts of responses prepared and I sort of I'm like ready to like go down this long decision tree and then I sort of like nowadays many more people are noticing the issue and uh you know I was invited I was invited here um and I think crypto people would actually pretty much really resonate with that we're like we have to explain you know Bitcoin uh 21 million hardcap we have to explain all these things like proof of work and like the conversation trees that we have to go down you we've all we've built out those like in naive responses those like spinal reflexes and then lately move moving into 2022 and 2023 fewer of those things we have to explain especially as we just printed out a bunch of money and for covid semi checks like all of a sudden we have to explain the concept of scarcity a little bit less and so it kind of sounds like a similar experience that the AI people have yeah totally like now I go to people who aren't in the field and I'm like ready to go down all these decision trees and they're like what's the issue and I'm like well and the most basic sense here's the issue and they're like oh yeah that seems rough and I'm like oh man this is such a different conversation I mean that's the first step right the first step is education yeah and then also acceptance of the problem okay I could imagine for so long you were saying hey like people ask you hey what are you working on Nate and you'd be like oh I'm working on AI alignment and then people are like why the why the hell are you working on that yeah they're like oh that's weird or they're like is that some weird termnator thing yeah uh and it's been it's been nice to sort of uh I sort of I sort of think that uh like a lot of the basic issues have been pretty obvious the whole time and that we're now seeing people who like don't have uh distort incentives noticing the issues um but it's really quite hardening to see uh I don't know where it will go but uh it's been it's been it's been nice to see people starting to notice uh that you know this is a real thing it's really on the horizon like you said I don't think we know how far it's very hard to it's very hard to predict um at least with precision uh but it it has looked to me like one of the biggest issues facing humanity for a while and it's very nice to see other start to notice that as well so that that leads me to the the question of just like how optimistic you are and I'll ask that in in two phases first the the same question that will we ask both LEA's or and also Paul Christiano's like all right what what are your chances of doom what what are your what are your chances of the worst AI problem being the worst the worst the version of itself uh I mean worst version of itself I think is very is very hard to get but the version we're like we all die like there are faced worse than death but like the version where we all die I think this is pretty likely uh I think this happens by more than 50% oh definitely um Paul Christiano gave us send to 20% so you're you're saying uh more than more than 50% uh my understanding of Paul is that he has 10 to 20 on the scenarios that I think are like AI takeover and higher probabilities in that on like humanity completely disempowered um I'm definitely I'm definitely more pessimistic than Paul on these counts uh like I would say that on my models and visualizations on my understanding of the problem there is very little hope uh and most of my hope comes from me being wrong somehow uh and so my probabilities on this destroying everything I know and love are like as high as my probability like they're about as high as my probabilities can go given uh like the the fact that I may just be totally wrong and hopefully I'm okay so you're you're pretty close to the LEAzer side of things which is like 95 to 99% doom yeah I mean I think uh 99s are hard to get uh uh but like uh but there there's also a difference between like what does the world look like as I see it the world the world looks as I see it like like the place the like as things seem to me or just like you know within a rounding over 100% uh and the the difference between that and my betting odds is then like hopefully the world's not as it seems right yeah so you know what you're saying is like we don't the nature of the AI problem is just a lot of we don't know and so what you're saying is like the reason why you maintain some level of optimism is because there's like a white swan event that's possible that could save us yeah and you know I have a bunch of I've thought a lot about various parts of this problem and uh I have you know various guesses as to where white swans are more or less likely and for instance uh it looks to me like the white swans are less likely in my unknowns about AI and more likely in my unknowns about how humanity is going to react to the problem um although there are still some unknowns in how AI goes where there could be white swans okay do you remember when you were first um working on this problem uh I know you weren't as skilled or as knowledgeable back in 2014 to 20 like 17 when you were first working on this problem but what was your level of optimism or pessimism back then and like how has how has your attitude towards the problem shifted over the last almost decade that you've been working on this uh you know it's gone up and down uh I've rarely had double digit odds of survival um but I have I have had double digit odds of survival when I've been explicitly quantifying and you know most of these numbers are like coming straight out of my butt I would like one put too much but by definition everyone's numbers are coming out of their butt and that's kind of like yeah but some people are all trying to yeah you're uh and I don't I don't spend a lot of time worrying about specific numbers like you know once once it's less than 3% chance we get good outcomes it doesn't it doesn't affect my day to day I'm not like staying up trying to calculate significant digits here I'm like man humanity does not look like it's up to this sort of task that doesn't you know I've seen humanity try to coordinate um and here we kind of so one thing we have not figured out yeah uh and for the record the reason that uh my uh the the way that I managed to have like high probability this is tricky is not that there's any one part of the puzzle that looks to me insurmountable uh that you know humans are pretty good at solving problems when they put their minds to them uh the way that like the reason that I'm like pretty pessimistic here is uh it looks to me like there's a bunch of different ways for things to go wrong and there's a lot of things that need to go right for things to go right like um like you not only need to solve various technical challenges you need to have uptake of the technical solutions in uh the relevant organizations those organizations need to be able to be able to be radically recognize the difference between a real solution and a fake one uh you need to have them like caring at all which is not even a fight that like we've won yet there are you know you have like the the heads of labs that like Microsoft and Facebook like poo pooing a lot of these issues um and so there's like five six seven needles and like this so I want to combine two metaphors where like the stars need to align except the stars are needles that we also need to thread and like that's and that we need all of those things to happen yeah and you're saying is like that's that that window is small that's that's where you get the uh the difficulty from and to be clear um uh I think it would be a fallacy to say like look I can give you like six things you need to do and like what's the chance you can get all of them that sort of reasoning doesn't really work like if if I line up all six and then the one that I assigned least likelihood two happens probably I underestimate this likelihood probably underestimated the correlation like uh like these are not independent right like if we can solve the hardest of these issues whichever one that turns out to be probably it's because we turned out to have coordination skill or competence and so on like I'm not saying you can drive the probabilities arbitrarily low by the fact that I can like line up a bunch of hurdles I'm more saying it sure seems to me like there's a bunch of hurdles man and like each of them like like like well not all of them but like many of them have a character that like humanity hasn't really faced before and this all adds up to me being like man I'm like single digit uh probabilities of survival here so with this new or maybe first first surgeons of interest from to the AI problem now thanks probably thanks to chat GBT thanks for the problem itself how has that shifted your optimism at if at all uh it's uh I mean it's I feel a little hopeful about it I feel like a spark of hope here uh it doesn't uh it doesn't like shift my probabilities on the ground too much um like uh this is like a really dumb model but like if you imagine having like three variables each with a 100 one in a hundred chance uh and success is like uh multiplying them all to get all together then raising one of the variables from like one percent to a hundred percent doesn't change your overall probability too much right but it is the first needle that is threaded so like if you go if you're like I know it sounds like you don't really like these like specific numbers but if you are like 99.
99 point 99.9% doom and then because people are now optimistic you go down to 99% doom it's still an order of magnitude right right so like it does feel like uh like some orders are magnitude uh on the models that say we're screwed right so like like my my the parts of my models that are like maybe we're fine are like maybe I'm just wrong about some stuff and the parts of my models that say we're fucked these models are basically saying you have 0% and like you know it's it's really like you know uh zero points blah blah blah and then you know uh a one or whatever and there you're getting some orders of magnitude which does feel hopeful and like enthusiastic about that and it like ups the probability that I'm wrong about something that matters like humanity's general ability to coordinate as would like so it does it does like it does I'm like heartened by it it doesn't notice the stars we've we've need we need to align one one is started to show that it's moving in the right direction it's moving in the right direction like there's a whole bunch like I'm not like oh it's only like the populace is going to realize these problems and react sanely there's sort of there's like so many more steps that people can still get off the train here there's like you know if you look at the national response to covid uh people noticed that there was a pandemic going on and that didn't make them respond in any sort of reasonable way to it uh like maybe uh AI will be different uh you know they're sure were more movies about like robots gone wrong than about pandemics beforehand at least I would be my guess um but like many of the movies don't really understand the the issues there's like ample rum like like uh like I there's just there's I've seen politicians try to respond to issues before and I would not say that the like social awareness needle has been threaded it's like showing it's showing promising signs and that like gives me some spark of hope uh but we haven't like cleared a whole obstacle in a way that would make me be like wow like humanity is bringing much more confidence to this issue than I expected not not to keep on on this one particular line of conversation for too long because it's only one part of the overall figure but just like we don't actually need I want to present the argument that we actually don't need all of humanity I guess we do need governments to coordinate and so we need the leaders and we definitely need those key figures yeah but like if the bottom half of the IQ of humanity is like it's not a problem and then the top half of the IQ of humanity is like this is a real problem I'm going to count that as a big win because like we don't need all of the we just need the smart people to focus on the problem uh I think that's mostly right uh uh modulo the issue where like which regulations happen and how the how the government systems move I think does depend a lot on the public uh or at least it can uh I I do think you write that like you know there's some like if we there there are ways that we could resolve the technical issues with such like uh resounding success uh and it could turn out that the resolve technical issues were sort of like so like obvious in their uh property being solution or like otherwise very beneficial to capabilities such that like you just get very big uptake and that maybe could be done with like uh a relatively small handful of geniuses who can do much better the problem than I ever could um and like maybe that would be a way to just solve the whole issue without going through like various other social obstacles or political obstacles or so on and in that sense sure you just need like maybe even the one like maybe maybe there's like one bright person somewhere in the world who would find this problem easy who like hasn't had the opportunity to see the problem yet because the world's really bad at like getting resources and problems right um I wouldn't bet on it at this point um but in that sense I would agree that like in some sense we just need the right minds and the problem um okay so pivoting to the looking at the problem as a whole you said your models are effectively close to zero at being able to solve this problem on model yeah on on model like like like within yeah on my models cool within your models uh why why do your models say that um it's it's again due to sort of like a destructive argument like uh many paths lead to doom here uh and much of the reason is uh the way that like one one of the forks is the way that people don't seem to really take the problem too seriously uh and there's sort of it feels to me like there's sort of gradations of this of like uh like people I don't know like back in the day the a lot of the argument centered around like is artificial intelligence even possible is uh significantly smarter than human intelligence even possible uh and like once you convince people of this then they're like well can it be solved in the next like hundred years or whatever and once you get past this they're sort of like uh well maybe it'll just be like more moral as it gets more smart and then like once you get past this there's like well maybe you can just sort of like train it and it's fine and like the train keeps going and like people can always find a reason to get off the train at the next stop and you then can put in like a bunch of painstaking effort to try and like uh lay down the arguments says to like uh like why the issues are like maybe harder than this um and we just seem like very far from like the people at these labs running these labs like they're they're they're they're getting off the trains at pretty early stops um and even if reality starts beating them over the head with various things I expect them to like only move one more stop or only move like as far as reality is forcing them and then like separately my models say that uh there are issues that predictably arise uh only once the AI uh gain significant capabilities and can be a real threat to you uh and if you're in this regime where you're sort of like need to drag people along and they sort of like only start believing things when they empirically see those things uh now you have an issue where like if there's issues that don't empirically arise until the AI can wipe the planet with your civilization or wipe the uh wipe your civilization off the off the planet uh that's too slow that's too slow right I give like other models for like visual to like understanding this if they're solving the AI alignment problem is that there's a bunch of decision trees that we need to go down so I'm imagining literally a tree and say there's like big tree big big tree big oak tree and there's a single fruit on this tree oak trees don't go apples but let's say say this oak tree grows apple there's one apple and that is the solution apple and it's very high up in a very far branch away and we're at the trunk and we need to find our way to that one single apple that one single fruit on the tree except that tree branches eight times and then when you have each branch each branch branches itself eight times and so like it's a it's an exponential problem because you have to choose the right path without knowing where the solution is without knowing where the gazette golden savior apple is and we need to make the correct path towards the apple without falling down any sort of the dead ends and so maybe one of the ways to articulate why your models are basically close to zero is that you're just bearish on humanity picking the right branch fork to lead to the solution apple for like the for the number of times that we actually need to do that yeah that's that's a pretty decent analogy um I would say that like um like you've got to be a bit careful arguments like this because if humanity has like managed to choose the past seven forks correctly you're probably not like thinking that there's an independent probability on the eighth uh and like uh so like I don't I don't actually but like I'm not actually getting my probabilities here from like these sort of like well sex adventure look at all the independent guesses like once you've been wrong about humanity picking the right branches a few times you should no longer think they're independent right you can start to be optimistic like hey maybe we're doing something right here right um uh and like another thing I would add to the analogy is that like a bunch of the branches are like full of apples that give you money until their apples that give you death right uh and so like and everyone's like making arguments as to why they can like detour into this money branch uh which and they do give you real money right up until they like give you death right uh and then you add that to the fact that humanity like has seemed in practice to be like very interested in wrong branches right and very susceptible to the money apples very susceptible money apples the money death apples right now you're getting like a bit closer right uh I will note one place where I'm commonly misunderstood uh is that people think that I think the technical problems of alignment are like super duper hard for some reason uh this is like basically not the case uh my my stances much more like the technical problems of alignment are basically underserved there's been like you know uh a few dozen people working on these things for not terribly long like if you were if you meant like humanity has spent spent much more effort trying to like solve physics uh at least I think uh not actually terribly familiar with how many scientists there were in uh like pre-Newton era and the pre-Newton era and like whatever club Newton right eventually was in uh but like humanity just really hasn't put much of an effort towards these these issues and one thing that makes the problem tricky is that like there's much less room for trial and error or some of my models predict given these like issues that empirically only show up right around the time that your uh civilization is getting wiped um and that that raises the difficulty level but it's not like we have turned the best minds of three generations to these issues and they have come up empty handed empty handed it's like we've turned like uh like a few dozen like weirdos and nerds who were like able to be compelled by these arguments 10 years ago to these issues and you know now we're turning like a few more people uh to these issues but like I am not saying like and there's some great technological feat um than these people doff I'm saying there's like a normal technological feat and meanwhile everyone's like scurrying around doing something else instead mm-hmm okay where the normal technological feat does have these extra difficulties of like you can't do as much empiricism which may be enough to push it over the edge of like humans couldn't do it but like in in large part it's just underserved sure sure when uh the last few moments of our paul christiano episode we asked him like why well what are the bottlenecks what are the constraints to solving this problem and his answer was uh interestingly not funding it was talent it was supply of of brain power would you agree with that uh yeah um right at least funding was a lesser problem like you can always yeah like money's fungible uh and although it's tricky because it's only fungible to a degree uh like you do have issues if you try to put a lot of funding that you like start to distort the incentives and get people who are like showing up who grifters yeah grifters and you also have like legibility issues where uh like are you distorting the field towards the legible work um uh and away from the like less legible but potentially more important work i basically think you shouldn't worry about that at reasonable monetary scales right now just to really dive into that problem you're saying legible versus illegible where the illegible it's just like hard to understand hard to comprehend but actually technically correct and then uh or in particular like hard for a grant maker or a like funder to evaluate whether you succeeded right right like if you can read the paper and it's simple to understand the grant maker might like oh let's fund that but it could actually just be a wrong path on the on the the death apple tree uh yeah and like uh i basically think you shouldn't worry about this you could with sufficient amounts of money get into these situations where like i start to worry that you're distorting the field in that way um uh but i i do think i think i think talent's lacking i think there's maybe two kinds of talent that are that i consider to be relatively different that i think are lacking uh one is just like more hands on deck trying to understand how these AI systems that we have today work like we're starting to make like fledgling minds they're doing stuff that we can't do by hand we don't know like like we couldn't program similar capabilities by hand we don't like know what the like algorithms data structures type stuff we don't know like what uh like we don't know how these things are working we know how we built them we know how we got them to work we don't know like internally how they're working understanding that would give us quite an edge in figuring out like how to point minds at things on purpose uh and uh i think we sort of want all available hands on deck trying to do that stuff uh and then i think that separately there are questions of like uh like i sort of the the alignment problem i think can basically be factored into this is like not quite true but it's like a fine first approximation can basically be factored into one challenge of like how do you sort of make an AI that wants x for some x of your choosing and then separately question of like what x can you put in there such that like you're happy with what happened uh there's like a bunch of additional issues with additional issues are like how do you make it uh be able to like do one thing without a ton of side effects you didn't want and like then shut down uh rather than like you know prevent you from turning it off so it can verify forever that it's successfully completed as task wherever like there's there's issues of like how do you uh and that's sort of like a whole separate packet concerns but uh uh you know many people sort of think the problem is like what would you ask an AI to do such that the results would be good but it seems to me the problem is much more like how do you get an AI to do to like care about what you want it to care about right in the first place uh and that seems to me like it takes a a difference and often less legible type of research uh that I think can totally be informed by understanding uh how the the current AI's work uh these these sort of like directions go hand in hand but uh one of the big things I think we're missing talent wise is the sort of person who's like uh like has the like uh like ambition in the gall to say like I just can take a swing like figure out what the hell is going on with minds how the minds like end up caring about things pursuing things or like having preferences uh or like having targets how does that work like I think that I can like figure out how that works and how to direct it uh like humanity does not have a theory of minds in this way we do not have a theory of like minds that can be pointed and it's probably not for lack of that theory being possible uh it's probably for lack of like just having gotten there like science wise uh and you can sort of come at that from one end which is just like figuring out other things in front of us work and then like trying to learn what you can about minds and learn what you can about aiming them uh I think there are also other ways to come at that um that take much I think less eligible research and like more like independent visionary sort of research although I think you need a bunch of a vision to uh make progress and figuring out how these how the how the current systems work but um that's that's another place where I feel like we're really hurting for talent is those like ambitious visionaries who just think they can take a swing at the whole like alignment challenge okay so when you say we fast forward to the future and we've solved the alignment problem um because that's the only future that we'll have to be able to reflect upon this question in the future uh is there is there like going to be a statue of a person who's gonna be like they solve the alignment problem or is it going to be like a team of people or is it going to be not even like a moment where like AI alignment solved and it's just like the alignment problem just dies by a thousand cuts do you have any sort of mental model for this um like I think these things uh I think it's like pretty unclear like and and part of the question doesn't come down to how does it happen but it comes down to like how do humans attribute things uh like uh uh like it's it does seem to me like historically a lot of like big theoretical insights a lot of like paradigm shifting theoretical developments uh end up attributed post-talk to individuals um like uh Newton or like Einstein uh I think there's like a bunch of truth to this uh I think you know we also want to count like Raymond and Laplace um and that we don't know and that's kind of the point but that helped out like uh Newton and Einstein um uh that I'm getting my like L names mixed up I not not actually sure it was Laplace um but also at the point yeah uh but uh like surely it will be an effort that requires lots and lots of people surely will be an effort that requires like many insights from many camps surely there will be huge amounts of like uh individual labor much of it probably thankless from like uh people who show up and can work on like the shovel ready projects that scale with labor whether there will also be like critical insights that come from like uh geniuses that like change the paradigm uh I think my guess if we like condition on getting to a future where the problem was solved my guess would be yes but that guess is like a little bit distorted by like how did we get out of the hole that we seem to be in and like well probably there was like some force that like made things go a lot better than it looks like they're on track to go and like loan geniuses are the sort of force that can do this um if you're just like looking for pop bills that it takes loan geniuses seems hard to call so the bankless audience is sufficiently large to the point where I'm gonna say that there's at least one person listening to this conversation who's gonna be like I'm ready to dedicate my life to solving the AI problem what advice you have for that person uh where should they start how should they get started it's tricky uh there's um like a lot a lot of the people that I'm most excited about have come in from very different angles and have their own sort of like novel perspectives on the problem uh I'm like generally much less enthusiastic about like some people some people come into this problem and they're like let me read everything everyone's ever written about the problem and like try to synthesize it and get a sense of like where things are and then like work from there and other people come in and they're sort of like you guys are obvious idiots you haven't checked like the obvious things the obvious things are these things like let's try looking at it this way let's try doing like and like you know maybe maybe maybe I'll like send my research around and people can tell me where I'm like being obviously dumber like retreading past mistakes uh but like I'm starting with the assumption that like no one here previously was like on the ball I tend to be more optimistic about people in the latter category right uh I tend to think you need a creative solution like creative solution and like someone who's more like I can obviously have ideas about this myself rather than like let me make sure to integrate everyone's previous ideas so far like everyone's previous ideas so far haven't solved the problem and also many of them are kind of dumb some of them mine uh like uh like uh like uh like at the same time they're sort of like a bunch of context that I do think people need and it's sort of hard to get um like what are good intro resources uh like um my guess is that like the less wrong wiki has a AI alignment intro resources page and if you google like a alignment inter resources less wrong you'll find like a collection of a bunch of different intros people are written uh and then you can maybe like find one of those intros that like resonates with you uh it's hard I don't I don't really know how to onboard people I really know where the people who come in and are like I have a chance of solving this whole damn thing come from uh but I think I think my advice would be like maybe look around on the internet for some resources that that you like and also maybe like well it sounds like it's a dark force it sounds like it's uh there's no guide there's no university path there's a dark force that you have to get through and if you get through the other side to be able to be competently talking about this thing congrats you're there but also the whole problem itself is also a dark force that's yeah that's it's definitely a part of the issue is that and it's not for lack of like a guide someone wrote on the internet you know there's probably like at least half a dozen of those I sort of don't think any of the guides are very good uh like this field is like like like a yeah you're totally right someone who can solve these problems needs to sort of be able to like go off to the frontier where things haven't been done before uh and part of that is like even getting to the to the beginning of the problem at all like it's a skill you'll need although I'd love to be able to like get people to beginning the problem more rapidly but it does feel to me a bit more like um trying to like figure out physics pre-Newton rather than trying to figure out physics in the days where we have physics classes right like there's not there's not there are people who've tried to write intros there are people who've tried to write like open problem lists I don't think they're great you can find them on the internet like try to develop your own intuitions and approach the problem as if like we know very little because we do in fact know very little okay so that's that's if they want to direct the solve the problem head on um I know that I don't have the mind to to apply my skills to this so what do I do I do podcasting what about some secondary skills and secondary efforts that people could uh to to help solve this problem um I uh like there's I think there's a place for uh regulation on these matters which pains me deeply to say given how well regulation has uh done on various issues in the past and you know I think I think many harms in society come from uh over regulation but uh like it seems to be humanity's only tool for uh going to a field that like self-professes largely that has a decent chance of killing literally everybody and saying like hey maybe like back off on that until like we understand what we're doing well enough to like do this job properly I would love humanity to have tools that weren't regulatory for this um uh and like if I was trying to design the coordination mechanism I would be like uh trying to handle it with like uh like liabilities rather than laws uh but well the problem is sufficiently large that the cost of regulation are acceptable uh that's what it seems to me uh I'll say this with sadness and so uh like that's a whole track where you know I'm no expert in uh how to get regulations to be actually like not really targeted and good but um you know I think there's a bunch to be done there I think most people will also be like well I don't have like the mind or the ability to go into like politics where matters at the moment um and I'm not sure going into politics is is the right thing there but that's another area where people uh where I think there's like work to be done um that takes a different that draws some different skill set uh uh I'd like if if people think they have an edge in like the education problem uh in like uh like writing up the basic arguments in a way that like reaches a different sort of audience or is like more compelling to a different group of people or is like uh more modernized or something I think these are all like uh find things to be doing um like uh like for example you seem to me to have noticed you have an edge in like talking to folk like uh helping uh uh like the the sort of arguments and the recognition of the issue about our audience I think this sort of stuff is great draws on a different set of skills um for a lot of people who don't have like one of these three opportunities I think they're often isn't an easy way to help out you know reality does not need to give everybody a like like like laws of physics don't care about you they can just like drop you in a in a world that is like uh under serious that a destruction while not giving you an easy thing to do about it um and I think like there's a skill to sort of like not lose a bunch of sleep over it not getting terribly depressed about it like looking around for where you can help and like uh if you can and you want to uh because you're like believe that there is like uh a big threat here and you care about averting it then like hell yeah I respect that and if like you look around and you can't find good ways to help out such as life uh keep an eye out and like no need to get depressed about it psychologically how do you deal with uh this this looming problem like when you wake up in the morning are you like uh shit we're gonna die or like how do you deal with this mentally uh I mean I have for a long time not had much faith in humanity's ability to coordinate and so most of the emotional blow most of the update for me uh was in late 2012 when I became persuaded on these issues uh you know it's it's it's it's not like uh I was like oh I was interesting and then like over time as I saw humanity like go down paths that seemed to me like quite derpy and like fail to take the problem seriously and handle appropriately my probabilities went down I sort of like I think correctly just like guessed early on that probably humanity was going to be pretty derpy about this and go down false paths uh like derpy as a technical term uh like I sort of like try not to make predictable updates and so like there was a day in late 2012 where I was like oh jeez like I was wrong about a lot of like I was wrong in my previous pursuits I missed like the biggest problem heading towards this planet it's like kind of embarrassing that I wasn't able to figure out myself like I thought of myself as like trying to go for the world's biggest problems but when I was like 14 I instituted what seemed to me like the world's biggest problem and never like sat down and tried to make a list of like what other like problems might be bigger I was just like obviously coordination's the biggest one all like go for the throw on throw on that and like needed some other people to come along and be like hey have you noticed this intelligence thing and how it's like is the primary factor determining the future and how humans are not at the maximum of intelligence and like are on their way to make like other intelligences that won't by default care about anything nice and so there was a day when I sort of like that argument hit me I like my probability of a wonderful future dropped from like you know like mid 90s to mid 10s or like mid zeros I guess and I mourned and then I like didn't feel a need to like psychologically focus on it a bunch after morning like you mourn and then you try to save your civilization uh there are still many times like it's not on my mind when I wake up there are still many times when I'm sad there's still many times one like I see something particularly beautiful or particularly moving or that I really quite like about like this planet and my species and like sentron life more broadly and I like shed tears about it but it is not a dominant psychological factor as opposed to like it's a deep source of sadness but I'm not like constantly wallowing it I don't really have a question here but I will say that say there's a 1% chance of solving the problem that doesn't just mean that we don't die though it means that actually a the all of the negative side of the ailing problem to kill the kill us side inverts and it saves us and produces the inverse the level of bad turns into a level of good absolutely that we've never seen before totally and so there's something there about just like maybe that 1% of that chance is so small but the good that comes out on the other side of that is really really good I mean that's what we're fighting for but no no stronger reflections other than that uh I do think people often underestimate just how good things could get uh like uh like one one place where I prefer talking to most people in AI versus talking to people from more general population at least uh in uh like America and especially like more blue tribe is it seems to me like there is a big meme of misanthropy uh especially like among the blue tribe of like maybe humanity isn't worth saving maybe like uh like humanity has done too much of the source of evil maybe we're the source of evil and I do think that we are like the uh like basically the only source of evil around like mosquitoes are still edging out humans for the top killer of humans while the mosquito malaria alliance really uh which uh I am personally offended by and think we should uh white mosquitoes off the map uh so that we can be number one for killing humans um and also the number one in killing mosquitoes and also number one in killing mosquitoes that'll show them uh but uh like I would not dub malaria evil I would dub like the holocaust evil uh like uh we are the source of evil we are where all of these ills come from um you know we are like destroying large swaths of the environment and uh like uh that is sad um but we are also like the source of uh like love and beauty and friendship and art and like these things also aren't like uh universally compelling there isn't like a stone tablet in the stars that says like love is great uh the reason that we have uh the reason that we care about like love and friendship and like uh hope and fun and enjoying ourselves is because these were the correlates of fitness in the ancestral savannah where our species evolved uh and the particulars of these emotions and feelings and things we care about those particulars depend on uh the specifics of our uh of our development hopefully some of them overlap with various aliens uh but probably not exactly and it's very unclear how much it's very unclear how other evolved aliens how alien they will be um but like those those things are also in us uh and they're also from us and we might be the only source of that in the universe uh and we might be we are we are very likely the only source of that within you know a hundred million light years uh and uh like we also know about ourselves that we appreciate the the fun and though we don't like the misery we can look at ourselves and be like wow like we don't like the evil uh we and you know it's subtle we're like we it would probably be a tragedy to remove like sadism from humans entirely but we're like well we want ethical sadism right like peri or sadists with some masochists have it all be like uh like consensual and within the bounds of like ethics uh like uh it's it's not just like we don't want to like there is there is some of our inheritance some of our inheritance is in is like very adjacent to the parts of ourselves we don't like but we also can look at ourselves and see that we don't like that in aggregate we like have these negative consequences these negative externalities that no one intended we can look at ourselves and say we don't like uh the impulses in us that lead us to like great atrocities uh and humanity like the future I think does not look like a similar mix of humanities virtues and humanities vices as we get smarter as we get more capable as we get better at self-incordination problems as we get more time to think as we get wiser as we we become more who we wish we were like we on purpose uh uh like promote our virtues and demotor vices uh and like is it tricky yeah and do we know what a wonderful future looks like no it's it's like very subtle you can't just go around giving everyone everything that they want this like uh or like solving all the problems like part of life is having like uh like obstacles to overcome and having like real choices that are meaningful uh and so on and so forth but like we can uh we can make a world that is kinder where the obstacles are like more meaningful where you don't have like terrible things happening to good people for like where the only reason is that's the laws of physics like everyone like many people people say like everything happens for a reason they sort of believe they live in a just world but things don't happen for a reason here like we can build a world where like your trials uh like tend to give you things that were worth the trouble that like pay off later and that you were glad for and we can build a world where like uh children are dying unnecessarily and needlessly because they were like we're born in the wrong part of the world and got some terrible disease we have installed yet uh and we can do so much better than that like if we if we like you know transcend the the bounce of humanity with like like the technological limit is huge uh like i'm an old school transhumanist and like think we should do something at least as cool as building dyson spheres although maybe there are like better ways to put your stars to use um like i'm i'm looking forward to the matrochka brains if we like decide that's worth the effort uh like there's there's so much potential there's so much potential that this uh species has as one of like uh perhaps the only uh source of like love friendship happiness fun in the universe um aliens may have other things and we'll care about that too but it might not be quite ours and if there are aliens are probably distant uh and like we can solve so many problems especially if we have smarter friends who are trying to help solve them with us if we can get you know artificial minds that are significantly smarter than us significantly more capable than us and that also are into this great project of like the glorious like transcendence future full of like flourishing happy civilizations having good times there's like i can't i can't describe the future specifically for you because i expected to look like foreign and weird and strange to me and be full of people like pursuing desires that like i don't recognize uh but there is so much upside and like yes humanity is a lot of darkness in it too but like that's just one more obstacle on our way to the glorious transcendence future that is easier to overcome if you have smarter friends that was that was beautiful um Nate uh thank you for just articulating what we get out of solving this alignment problem because not only do we get to not die but we get what seems to be kind of the inverse of that uh so thank you for for walking us through um everything you're doing and why the fight is worth fighting totally cheers mantle formerly known as bitdow is the first dow led web3 ecosystem all built on top of mantle's first core product the mantle network a brand new high performance a theorem layer two built using the op stack but uses eigenlayers data availability solution instead of the expensive a theorem layer one not only does this reduce mantle networks gas fees by 80% but it also reduces gas fee volatility providing a more stable foundation for mantle's applications the mantle treasury is one of the biggest dow owned treasuries which is seeding an ecosystem of projects from all around the web3 space for mantle mantle already has subcommunities from around web3 onboarded like game seven for web3 gaming and buy bit for tvl and liquidity and on rounds so if you want to build on the mantle network mantle is offering a grants program that provides milestone based funding to promising projects that help expand secure and decentralized mantle if you want to get started working with the first dow led layer two ecosystem check out mantle at mantle dot xyz and follow them on twitter at zero x mantle introducing polygon 2.
0 the value layer for the internet for too long the limitations of blockchains have held back app development and stifled user adoption the internet allows anyone to create an exchange information what's missing is a value layer that lets anyone exchange store and program value that's where polygon 2.0 comes in polygon labs has unveiled a series of innovations that will radically alter the polygon ecosystem and web3 as a whole by leveraging groundbreaking zk innovations such as polygon zk evm the next iteration of the best in class plonky 2 proving system and a first of its kind zk powered interoperability layer polygon 2.
0 will give users and devs unlimited scalability and unified liquidity right now there is a polygon improvement proposal regarding a potential zk powered upgrade of polygon proof of stake if approved polygon proof of stake would become a layer to zk evm validity so make your voice heard on this proposal by joining the polygon discord today you have a chance to help the polygon community give the internet the value layer it deserves are you planning to launch a token is your token already live and are you granting your employees and contractors vesting token words and are you trying to figure out how to take care of taxable events for your team toku makes implementing a global token incentive award simple with toku you will get unmatched legal and tax support to grant and administer your global teams tokens toku will help you navigate across the life cycle of your token from easy to use pre-launched token grant award templates to managing post-cliff taxable events with payroll for legal finance and hr teams it's a huge complex task to have to comply with labor laws payroll and tax obligations tax reporting and crypto regulations in every country that you employ someone it's difficult time-consuming manual and costly and it's drawing more attention from global regulators and governments toku makes it simple for leading companies in the space protocol labs hidera get coin and many more so if you want some help navigating the complex world of token compliance go to toku.
com plus bankless or click the link in the description below bankless nation we are here as your dollar and i'm talking to dare teran dare welcome the shop thank you good to be here dare you want to just explain for people who don't know who you are uh where you are and what you're up to so my name is dare i am currently leading the ai objectives institute we are a non-profits research lab focusing on the question of alignment and uh we are interested in building what tools would be able to add value to the current ecosystem to bring the future that we want to be in so an institute that is focus on uh solving the alignment problem sounds like that people who are at this institute believe that the alignment problem is existential am i am i on the right track here yes very much so okay so we've been beginning all of our ai interviews with just asking the guest what is your percentage of ai doom of percentage likelihood my personal percentage of ai doom is fairly low for human civilization to end totally would be quite low i would give it to two to five percent while for us to end up in a future that is not desirable more so existential risk rather than extinction risk would be very very high given the current dynamics we're living in right now okay so existential risk can you measure so we uh chance of death low chance of significant disruption and displacement in the hierarchy of life high yes right yes okay and so like maybe you could illustrate like what that looks like what what is your likely scenarios here the core principle that has brought the ai objectives institute together was that the ai revolution as we call it can bring an unprecedented level of flourishing to human civilization but the current systems we are living it in do not place us on that default path we currently have a lot of incentive gradients that cause power to be concentrated we have misaligned incentives in the form of nation stage to corporations to attention economy that distract humans from what we actually want to focus on the ai systems right now are learning and copying these behavioral patterns causing much more large scale disruption in the landscape and this propagating further the doom scenarios that i am most concerned about do not look like nanobots crawling all over us all of a sudden but look much more like economic failures institutional failures environmental failures as we know today at a much higher much more unpredictable rate and we already have mechanisms to deal with these things but we do not know the scale at which we will be coordinating for this okay so you let's set off a bunch of existential crises right and yours i think you're saying that just like ai is going to accelerate crises that are already in existence rather than creating a net new crisis although you're leaving some small amount of room for that uh the bigger issue is that there are already human crises that we have that we don't have solutions for and ai is going to accelerate exactly i think the main crisis that i am most afraid of is in the shortest term within the next two to five years we will have massive disruption to the institutions and systems as we know today we see ai as an optimizer and we have already other misaligned optimizers in the landscape they do look like markets and corporations they look like nation stage they look like you know misaligned incentives that has driven mass scale invasions that we're experiencing in the last you know two years with russian Ukraine to massive bank collapses to voting systems that end up in gridlock these systems will be exacerbated in a pace that's we are not yet familiar with and that's to in our perspective from the ai objectives institute's perspective there is a very strong continuum rather than a sharp break between human misalignment as we experience today an ai misalignment until we solve human alignments talking about a purely ai alignment system feels superfluous in my opinion so you're saying that solving alignment there's needs to be a correct order of operations there and before we look to solve ai alignment humans need to first look inward and solve our own alignment first that's kind of the take i think ai tooling in fact can be quite helpful for us to solve alignment at its base and there is a lot of cross pollination that i think is necessary between understanding how we as humans so far before ai have been able to keep certain systems in check be it you know being able to give corporations a legal entity that can be interacted with or the way nation states have different systems that are checks and balances for each other there's a lot of value to be driven from how ai alignment research can feed into current systems that have been experiencing different levels of misalignment and how these systems and how we have dealt with this can feed into ai alignment we see this as one central problem that's ai is learning from to put it in another spotlight solving ai alignment being able to align a g i to a specific set of human values and perspectives actually doesn't solve human alignment problem it just pushes the problem from a silicon substrate to dissociate technical substrate in which case it might be much harder to control that's a really interesting could you could just elaborate on that because like there are some human values that we want that i think i can claim without evidence that we think are good as in like don't kill me and things like that and so like but i think what you're saying is like if we go further down like what we think are good we'll start to get into the very subjective realm and if we start to align ai without defining what humans think are good we can run into a problem right okay uh so my next question is like it's still a problem if the ai's come and kill us right right and so like maybe i'm a little bit lost with how to proceed here but like there seems to be an order of operations problem which problem do we solve first right and how do we even define the problem space the problem space yeah so there's this concept of differential technological development what is most important here is to decide on the order of operations of which problems can be solved first so that they can shed light into the next problem this is like developing cars before seat belts are invented is more risky than understanding if we are to develop cars we should have seat belts the question of differential technologies are what are the technological pieces that make sense to tackle first so that we have the right substrate on which we can build a future that we want to build towards so this is what we focus on at the ai objectives institute what are the pieces that will be able to yield a safe aligned ai downstream and what are the pieces that are necessary right now to build first the center of all of this is a coordination problem i actually define existential risk as a failure to coordinate at the face of an existential risk is what makes existential risk come together so there's a lot of tooling that we're focusing on right now on scalable coordination there's a lot of tooling we're focusing on epistemic security and systems of the alignment these are three avenues that i think need to have much more research so that we can mobilize together we can identify the loopholes in our thinking as individuals as collectives and as systems so that we can bring a level of systems alignment to in line with how we want humans to proceed only thereafter we can start contemplating the scale of agi now i think it is quite a ways away for we still have enough time to be able to have by the time agi arrives for us to have built in institutions that are built on the backbone of scalable coordination incorporation that is why i think there's a lot of hope for us to be able to avoid total catastrophe and i'm interested in thinking of it from that angle which i think is quite necessary in the current alignment landscape what are tooling that we can build right now that will bring an incremental net good rather than talking about what we should avoid what we should not do i would like to bring light to the world what we should do what we should be focusing on today okay so it's your position that agi in the short term is going to produce immensely powerful tools we can use these tools to help humans with their human problems that they had before ag ever came on the scene one of those things is human coordination and so we can apply ag to solve human coordination hopefully this all happens before the ag doom alignment problem manifests that aliasa talks about and the idea is that we race to use ag tools to align humans amongst other existential crisis including our own ability to coordinate and then we will be able to solve the ag alignment problem more head on is this more or less your roadmap yes i would say that the ag alignment problem is more or less the same problem it is the natural extension of the human alignment human coordination problem so because we cannot coordinate as humans we can't coordinate on the ag alignment problem we do not have mechanisms to identify what are the sets of values that an AI system should be aligned to and i think it is quite short-sighted to say let's just pump in more text to large language models and at some point they will be able to figure out a more detailed you know i'm going to the weeds a little bit solving single single alignment is technically easy and saying you know if we have one unitary agent that is able to be superhuman and super intelligent they will figure out what is best for us so we should fully focus on that i think this is quite faulty i think by the time we get to this stage there will be many narrow AI applications that will be strong enough to actually put humans in a catastrophic risk so we need to actually start from there what will these narrow tools look like at the hands of misaligned state actors what will these look like at the hands of you know exploit the corporate behavior how can we make sure we can have safeguards around these as humanity as a civilization and will those tooling actually bring a better AI future that is the intersection that we are interested in i think the answer is there okay so the idea is that like modern day late-stage capitalism has produced large scale corporations that are misaligned with humanity in general and then you give them super powerful narrow artificial intelligence and they just become misaligned faster and that's like one one model one application of where this could go wrong and there's perhaps like five or six or seven more examples like this i mean i would actually argue that we already live in this scenario this is not the future this has been happening within the last ten years i mean this couple simple examples insurance companies probably at this point have an AI system to decide which claims they should reject immediately because they are least likely to be followed up on one could say well this is the insurance companies job and they are rightfully doing this to me this is actually a fundamental alignment problem we already have an optimizer system inside another optimizer system that is the insurance company that is rejecting claims that is causing human lives to be potentially at risk and we have devised a society that has normalized this behavior we have devised ways in which you know a corporate company like companies are able to hide the environmental externalities that they are building to the world to the landscape the questions i ask is are AI systems able to share world models with us that will be able to have us understand these externalities better to be able to incorporate that into fundamental decision making to put it in more fluffy words can AI systems and our understanding through these tools elevate our sense of what humanity wants to be so that we know where we want to go that is why i am hopeful about being able to build a future this requires a lot of coordination this requires a lot better epistemic security this requires much more thinking about how do we want to envision the institutions of the future so maybe you could just illuminate some of the strategies that you are working on at the machine objectives institute the AI objectives institute one line that we had on our website for a long while was our objective is better objectives which is it is almost tongue and cheek as we do not think AI systems can have fixed objectives similarly humans do not have fixed utility functions in fact the relaxedness of these is what gives flexibility to human evolution of thought of our coordination of our ethics so in some ways the name is tongue and cheek on that front but the goal is to come up with better objectives continuously right okay so what are the if we could drill down into the details of what it takes to come up with better objectives for artificial intelligence like just the details like if you will right for sure so we think of the society let's look at the societal stack from an individual level to a collective level and then to a systems level on the individual level the core work is finding individual autonomy and sovereignty through bringing better epistemic security the world that we are living in right now especially in the western world with democracies a lot of this heavily relies on information transfer people vote people coordinate based on the information they receive around the world now we are entering a new paradigm in human communication where most of the content is about to be generated by AI systems in this world are we able to use AI systems to bring a different level of epistemic security and confidence to have us understand is the content i am engaging with with a latent agenda how can i stay through to my objectives as an individual how can i stay through to my alignment with the continuous flow of information that we are interacting with that is constantly fighting to hijack our bandwidth this at the lowest level is the most important level a lot of mechanistic approaches to alignment assume that individuals have a level of autonomy and sovereignty they do know what they want we actually start from that question we do not know what we want we do not always take actions in line with our incentives due to bounded rationality due to myopia due to pure distraction how can we use tooling that comes from the a islandscape for that so this is the first avenue of research there's we have come up with a research agenda that has some specific avenues that we would like to explore that we believe is a net incremental good i'll go into some of the details on that please but first just to really just make sure i'm understanding here you started with the individual yes and i think that was an intentional choice yes saying like starting with optimizing for individual freedom and autonomy is a high level goal a higher level goal than the rest of the stack which i think we're about to go down right but could you explain just in the library on why we start with the individual and why that's important individual is the building block of the society every decision that we are making ultimately comes down to an individual's ability to understand the world that we are interacting with we have devices the systems that we are operating in right now assume individuals ability to give feedback to a corporation to a government through our behavior on purchasing or through our vote and we coherent around that assuming that AI or democracy or media AI is just one form of super intelligence we have devised many other forms of super intelligence is in human history assuming that the individual maintains a level of autonomy and sovereignty throughout this interaction as we live in the world is what has caused the crisis that we already are in this is not in the future we already are in this scenario let's look at social media it's the archetype example in 2008 we thought you know this is going to be a revolution that brings us a level of connectivity of mutual understanding that will heal democracy instead we ended up with echo chambers we ended up with massive epistemic fracturing with respect about facts people believe in we found people that get locked in more to their own bubble echo chambers we could have foreseen some of these stuff this all sheds light into it's ultimately the individual autonomy and sovereignty that is the core building block of civilization then the question we ask is is the AI tooling of today able to bring a different level of epistemic security our answer is yes and this is very worthwhile to be tried now epistemic security so epistemic can you just define that here yes the information that you are receiving you know where this is coming from you know you have a sense on what this information is trying to accomplish or whether it is true or false how you relate to this information how you want to relate to this information and how you want to participate in the world given this new information currently a lot of these systems are actually quite shaky in this society we live in today and we are entering truth world right and we are instead of securing this we are saying let's come up with a level of generative AI capabilities that can flood even more information and then we are talking about what will AI be aligned to the question is do humans even have van with to be able to share this information right okay okay so epistemology the study of knowledge epistemic security is just like securing the ability for the individual if so they have the choice to sometimes people just want to live in the cozy comfort of being fed the information that they want but importantly giving the individual the choice to have access to truth is a basal building block you're saying to talk about the rest of the societal stack exactly and we're going to use AI tools to improve that right so a couple of projects we are working on right now on this front that is the core building blocks is can we use AI technologies right now AI techniques right now to be able to inform certain patterns of perception for example i'm not even going to go into whether or not the content you're interacting with is true or false i will instead start from the side is the content you're seeing designed to elicit anger from you is the content you're interacting with designed to trigger an addictive loop we're giving you a dopamine high really fast is it eliciting anxiety is there a latent agenda in this content or is there a subcultural affiliation in the content that you're interacting with is it using language that is geared towards a certain subgroup is it repetitive is there evasiveness is it stating beliefs or is it a response to something else turns out large language models are actually really good at detecting these kinds of patterns because what they are doing is pattern matching i'm predicting the next token so a language model is able to say oh this next token is unusual it looks like the next token the difference here is guiding me towards a different future so this is one building block that on its own is able to add a lot of value to the language so we're building a prototype called lucid lens which evaluates the content that you're interacting with continuously to be able to guide you into hey it looks like you have been on a dopamine loop lately do you want to shift what you want to do or looks like you're engaging with content that is extremely repetitive in this discourse can we bring a level of intentionality to your objective alignments personally okay so you're building that system right how do you make sure that that system isn't biased right because like maybe that maybe the repetitive loop of iterative content is the correct thing and then you're saying this AI suggests to humans that hey maybe you should get out of your repetitive loop how do we know that that's not an unbiased thing to do right there's a couple approaches here that we can take one of them is can we ground language models to an individual's own affiliations there's some people working on this front the I'll give a fairly simple answer a heads up like something like a chimney cricket on your shoulder that can point out hey you might be stuck in you know a rage baits you know click scroll for the last while that check in is in the right context is not harmful to an individual you might say yes I acknowledge that I want to continue I want to proceed I like where I am at rather than have there be a sharp judgment of the nature and the quality of the content but another part that lambs are able to do very well is to being able to fetch further information that can give you a larger picture around the content that you're interacting with yeah this is just one of the many avenues I will zoom back from lucid lens another one is can language models replicate an individual's affinity to the points that they are able to help us stay grounded in a set of objectives that we want to be in this we say mine this we call mindful mirror is a different avenue of research you can think of it this way the first one is about a machine helping humans stay grounded to their objectives the second one is helping a human stay grounded to their sir a human to give the same feedback to a machine to say here are things that I would like to prioritize these are what is good to me having an individual individualized personal language model that is secure that is not living in a large company servers but that is completely owned by you having this to be able to secure your understanding of the landscape of emotions of content that you're dealing with that can help you ground yourself in a moment where you are lost this is an incredible piece of technology that can actually create net value to the society okay so I think I can categorize your mechanisms into two different camps one and the crypto world we use this idea of credible neutrality quite frequently right and there are some mechanisms that are credibly neutral which aren't to say like hey what you're doing is bad but they're just little alerts saying like hey you've engaged in very repetitive YouTube rabbit hole type behaviors right I'm just going to let you know that that is what's happened right without saying anything negative or positive or suggesting a rerouting just like a neutral mechanism to perhaps bring you out of the hole and let you know that this is perhaps perhaps a danger zone or perhaps not right and just for you to be able to step out of your consciousness tunnel right and zoom out and just like a just like a like that in meditation sometimes they ding that gong right right and just like hey if you're lost in thought like ding the gong right and so like it's a way to just to get you to snap back out of it if you are in it right and so they credibly neutral mechanism right which we love and then the other one is being able to customize your own personal llm to align with your desires and since you are the one implanting your biases into the llm we also feel that since we are not imposing that upon others we're only imposing that upon ourselves that also checks checks the box of credible neutrality right and i would like to underline one thing which is that the large language models the goal isn't to create a locked-in version like there is there's many fault lines that we can fall through here as we're building these which is why the goal isn't to launch this as a product at widespread use right away but to actually approach this from a question of rigorous study to the point that we make sure these tools are doing the things we want to do what are their failure points for example a very terrible way to do this would be to cause individual value lock-in in which case you know a language model constantly reaffirms things to you from your past state to the point that it prevents you from moving forward i am more interested in a language model that is able to give me awareness of my drift through my own evolution of thought these are some of the aspects that are actually very fundamental to the question of alignment how was the landscape of values shifting for an individual then how am i changing as a person right what was important to you there's a series of simple questions are my beliefs consistent with my beliefs are my beliefs consistent with my actions are my actions inconsistency with the community that i'm living in in their beliefs and their actions being able to have visibility into these these systems are incredibly fractured right now when we're building even more walls i believe ii tooling can actually help us overcome some of these to be able to understand all looks like what i wanted to do i am not able to do it given the incentive gradients i am existing in right now the crypto world has suffered from this quite a bit how can we have this be more visible to everyone i believe questions of contemplating what values do we want ii systems to be aligned to et cetera really require a level of rigorous understanding of self first so this is the first step then we move to the next category which is how can we scale this up to the question of a collective which is the next tier in human society right and and before we get to there i just really want to drive a drive something home it really sounds like we're trying to allow a i for humans to become the best versions of themselves right right if we want to talk about some like some some people in the psychology realm uh niche a would call this the uber bench right like the uber man uh the superman the best the literally becoming the best version of yourself because that also scales up in society right if you as an individual become the best versions of yourself that makes you a better community member and that makes communities better which i think is kind of where this idea goes i call it extended cognition if i am able to have an ability to understand myself cross-sectional through my own history based on what i have engaged with based on what i have thought what i have written that is powerful we can act somewhat like the scariest applications are also the most worthwhile if this data is compromised in any way that also creates you know a mirror of me to exist in the society so then the question becomes how can we do this in the most secure manner that is really within your own autonomy and sovereignty because we already live in a world with very difficult attention economy that everything is competing for your attention for your beliefs so that you can vote or so that you can purchase a certain way it's very important for us to be able to bring a level of autonomy to the individual as AI tooling proliferates in this landscape okay let's move up the the social stack a community comes next yes collective collective question here is scalable coordination i this is where i am most passionate about i think there is incredible value to be added here in some ways you know there's many alignment labs we do share you know our concerns are in line with a lot of the other spaces like meary or redwood where there is massive challenges that are coming in and the question is why have we not yet been able to coordinate at a scale where we have lined up our incentives as a society so we can tackle these AI problems instead we ended up in a race dynamic where you know entities that are spun up to counter end up participating in the race dynamic for this being able to level set the understanding of what is true across every participant being able to bring visibility into what are different perspectives that exist in society right now how can i engage with these more effectively how can we come up with collective decision-making systems so that a collective can find its alignment the first category was about an individual finding their alignment the second category is how can a collective find their alignment so you you corrected me when i said community into replace it with collective i think the reason why you did that is because like a community seems to be like a handful of people right a hundred people or a thousand people in a town but a collective is like the hive mind of these people yeah and that is the thing that we are trying to produce alignment for is that my that's my interpretation yes and i would say i am in a collective with a lot of people that are not in my community necessarily a community is a more intimate collective um depending on how and you can say you know well there is different ways to cut the social strata that we live into say you know these are different categories all of these are valid the thing i am interested in is say we are able to align age i to one human what do we align age i to now how can we come up with definitions for collectors yeah we recently did a podcast with a guy Tim urban on the subject of liberalism yes and he had this great illustration of um higher mind thinking versus lower mind thinking right and then higher mind is like a genie and lower mind is like a golem right like just like a golem is just like dominant punches and a genie is like magical and higher and then when you have collectives if you have a society that is a society dominated by lower mind thinking primitive mind thinking like reptile brain you have a collective golem right but then if you have a society based on higher mind thinking higher order thinking using their uh more recent parts of their developed brain their their prefrontal cortex then you have a collective genie and this thing as it can actually even if it engages with a different collective genie so you have like the republican genie and the democrat genie two genies can actually make progress together right they can actually come together and and produce a road map whereas two golems just come and fight and so this is actually a similar subject that we've had on the podcast and maybe a way to illustrate this i see some parts of this to be in line with our thinking um the way we have been developing AI systems are much closer to a golem right now um i'll get it i'll get into that uh the downstream problem of AI alignment it's human alignment right um i have thoughts on how to make AI systems be more like genies as well we can get to that in a little bit but um to not to uh change course of the discussion um one of the projects we have on this front is called talk to the city and this is a digital town hall that you can summon out of unstructured feedback that you have collected from a polity we currently have voting systems where you're sending one bit of information to the government every four years and you're hoping that they will be able to represent this the best we have been used to categorical information on voting multiple choice uh and referendums the question is am i actually able to share my perspective in its real uh more true to my own perspective through human language and have a central entity receive this information and make decisions based on this information talk to the city is a prototype that collects different unstructured text feedback from the entire community that we're looking at synthesizes into a set of different perspectives that exist in the community and train conversational language models for each of these perspectives so that you can have these perspectives talk to each other or as an individual be it you know you're a policy maker or a journalist or just a citizen in this city um be able to engage with all of these reasons why the goal here is not to find consensus the goal here is to understand different viewpoints so you can make sure you can address these in some ways one of the problems that we would like to solve and this again goes back to the real alignment problem it is easy to find the lowest common denominator across everyone and that causes a lot of short-sighted policy making that causes a lot of i mean political theory seeing like a state explores this in depth um i am listening i read that by the way great book it's incredible it's very helpful for shaping our thinking i am interested in being able to understand a more in-depth policy how is this going to actually impact people what are the reasons why this may be bad for some groups yet still creates a perito improvement over the current state these require a level of sophistication that isn't about saying okay it looks like everyone agrees on do not kill humans so let's codify that can we actually understand okay but i am interested in having more resources for my village and this requires a compromise with something else if the proposal is you know should we build a road from A to B yes or no the answer might be we should build a road from B to C and being able to give a community to express this and how this be received systematically is something AI tooling can actually bring to the landscape right now that can bring a different level of collective coordination capability so okay so it sounds like at the individual level we have these language models that can be perhaps perceived as like our personal assistance our personal like our series our our Google or Microsoft Cortana's right to help us think and help us know and help us learn right and then that can um amalgamate to a higher order right well LLM right that like you said it doesn't come to consensus on our behalf but allows us to see when you aggregate everyone's personal assistance what does everyone believe and it processes that data of individual beliefs into uh something for us a reason about and understand and move forward yes uh now that we understand right i am not interested in the genie that you're talking about to make policy for us i am interested in this system to be able to show us these are the considerations we need to take into account that people have voiced this is something that AI tooling can help us with today right now which is why we are building it and that brings a level of scalable coordination and cooperation capability another aspect here is to finding positive some outcomes that individual groups may not have been able to see you know the solution to you know should we build the keystone excel pipeline the right answer might be well a hydroelectric time will still bring the same energy and workforce to the region without actually having an environmental pollution cost so it can produce emerging um emerging optionality right and these are things that the systems we are building are actually really good at um i would rather not yield the level of reasoning and decision-making to an AI system but in the near term we are able to use these as building blocks to improve the capability for humans to cohere with each other now what's really interesting here is we are getting tubers with one stone if we build these pillars the first one is these tools are net good for humanity we need these right now to be to solve human alignment to take a step towards human alignment but also these tools will create data sets for AI alignment as well it will be able to show how humans have cohered around decision-making on specific world models or have humans say yes this AI system actually was able to represent me through a time we currently don't have this data yet we're talking about solving AI alignment don't get me wrong I do think you know mechanistic interpretability and a lot of machine alignment research is incredibly important as we also need to look at what does this mean for humanity otherwise we will just push the problem of alignment downstream and make it much harder to solve right right okay so so what you're saying is like when we have this collective consciousness that we are able to reason about via all of our native large language model assistants coming together and pow-wowing about what everyone believes will start to be able to lock in some beliefs um we'll probably lock in the idea that killing humans is bad uh that'll probably happen pretty first everyone will come will be an agreement about that so then we can use that shared understanding that killing humans is bad as a way to codify that into law about uh AGI right and then we perhaps can go higher from there and be like okay now that we understand that everyone believes that killing humans is bad we can also lock in that theft is bad and then we can start to get higher up the stack of what we believe and then use that to operationalize more powerful AI is that the path here i very much disagree with the framing oh no okay please help me help me understand um value lock in is a concept that is well explored in AI alignment and effective altruism and similar landscapes the goal is not to lock in values the goal is to build systems where people the polity is able to continuously give feedback and participate into an ever evolving consciousness in sort of one AGI that has learned the moral code and then can proceed what is necessary is a system that can continuously take feedback from people as the value landscape shifts as more unpredictable events happen this actually the crypto world has lived through this many times as you know the incentive shift as speed of you know progress shifts all the way from gas fees to coordinating you know how purchases can be made we need systems that are actually resilient towards updating based on how the landscape is changing so i am i would be quite worried about building systems that can learn and fix something in place perpetually much more so i would be interested in systems that are trained to fetch new information to understand how the ground is changing throughout time most humans i mean some values i hope we perpetually agree with such as do not kill do not go harm but then again what does harm mean what does this mean in case of you know euthanasia that you know might be an opt-in from the individual how does one proceed we already are living in a world where we are exploring these kinds of questions before AI the AI systems i think it's quite dangerous to force AI systems into you need to find what the optimal version is and enforce it to the rest of the world the right way to go is build an AI system that can learn from humans on where humans want to go at any given point and bring us towards there and have there be a level of courage ability so like there's a rule of thumb that i've come to understand in the crypto economic world which is called no magic numbers as in when you build a crypto economic system if you just pick a fixed number that is a point of rigidity and fragility and i think that's perhaps what you're saying about when we train our AI to be aligned with us any sort of rigid or fixed parameter can create fragility and longtail consequences that we don't understand right okay so we're in agreement there right and so okay so you didn't like when i was saying like hey all the agreements all the humans agree that killing is bad let's lock that in right maybe i'll rephrase and say like in this one moment of time all of the humans of a local collective all in that one moment of time agree that humans are killing is bad so the AI that's reading that data will in that moment of time choose to not kill humans is that a better way to describe it and i would rather have us have systems that don't necessarily give AI the ability to be able to kill humans in the first place but more so CDs as the tools that they are they are super intelligent things we can consult and learn from and iterate from there but yes the AI systems values should be able to evolve as human systems evolve and it's really downstream of human autonomy and sovereignty into collective decision-making that can bring the systems to be able to be aligned okay so okay so we started the individual we moved up to the collective is there a next step what's higher up the last step is systems level systems okay systems are more complex than a collective in a system we don't only have multitude of people but entities that have their own capabilities that have their own agenda for example a corporation consists of individuals but it has goals it has affordances that go beyond what any of these individuals can do furthermore we have developed systems in the world that you know don't hold the individuals accountable for the failures of corporations this is the side where we have seen massive problems with misalignment throughout human history both in terms of you know states governance and we have toppled many systems you know divine right of kings was impossible to overcome yet here we are communism was the same there is a lot of systems that we evolve we currently operate under a capitalist system that is heavily governed by incentive gradients that have caused a lot of shifts for how even you know nonprofits that are developing AI have shifted their priority towards monetization and productization so the question then becomes how can we design systems that can stay aligned to the betterment of the collective to the betterment of the individuals everything goes back to you know is this actually producing well-being for the participant or is it good-hunting something good-hards law as in when you pick a measure that becomes a target it ceases to be a good measure we live in systems that already do this this is precisely why we don't want an AI to lock in and create rigidity but design systems that can continuously evolve through that window okay so this seems like the hardest problem yes it also seems like the frontier of coordination problems that humans have arrived at in our grand in the grand scheme of things and we still haven't tackled that problem right it sounds like perhaps understanding that that's the the foundation that we were at right we actually might need AI to solve that problem and not be able to solve that problem without it we haven't solved this problem right one thing I say is our work the AI objectives institute in in some sense the word AI is not that relevant this work was relevant 300 years ago I think this world would be relevant this work will be relevant post-AGI as well this is a question of how do we design structures institutions that can stay aligned to the collective this is a millennia old problem AI is just the newest building block in this story it's a very critical building block in this story that can cause a lot of damage if we don't do it's right which is why I have high fears of existential risk if not extinction risk but that is possible as well if we don't coordinate I believe we would be able to coordinate and build better institutions which is what I want to work on well I do do you think that actually solving the point of like like at the systems level we have borders and that's kind of like a coordination breakdown the fact that different countries operate by different rules and coordinate differently yes and so and then there's also different economic systems right different different systems are they're disparate they're disconnected it would be better if there was a single global system along with all the other problems that we've had with Stalinism and other like atrocities throughout the 1900s my question to you and I'll just reiterate it is like that's the frontier of human coordination that you know we've solved it at the tribe we solved it at the at the community solved it at the city we solved it at the nation state level haven't solved it above that and honestly haven't really solved it completely at the nation state level either and we haven't solved it on an individual or collective level either but we are taking steps towards all of these it's solved it it solved more or less a different person in the stack yes and my question to you is like do we need AI to take the next step in solving it at a more systemic level I actually see it from the opposite way around given we are building AI in AI-driven institutions given the workflow that we have right now will yield institutions that have AI in it we have to look into how AI will interface with this right if we say you know let's turn off AI so we can solve this problem for another couple centuries or millennia sure we can do that as all we don't live in that hypothetical AI is here it is present I would ask how can this be helpful for institution design yeah so it's would you agree that AI all these language modules everything we're talking about here is both the problem and the solution at the same time it is the problem right now because we haven't yet been able to solve human alignment hence any super intelligence any super competent entity that's can excel human capacity can be dangerous such as misaligned you know state actors or exploitative corporations right of course an AI will be very dangerous as well so we need to tackle this problem right so we have a couple experiments on this front that are also incredibly important and these are the monoliths that require much more coordination and help than one group to solve so our goal is to foster an ecosystem that has many approaches all the way from you know crypto is super important zk proof is super important on the spiller we are interested in building a proof of concept of an open agency architecture system that can help an institute our goal is to showcase that an institution can make decisions based on feedback from a larger collective based on expert opinion in a level that is transparent and visible and interpretable rather than a complete black box and we believe that it is incredibly important for AI tools building institutions and AI building institutions to follow this and also it is important for the AI systems to be built on top of this principle as well yeah and open agency is a concept that director has pushed forward the simple explanation would be can we shift our thinking from agents which are singular monoliths that have fixed goals that have low visibility that have their own ways of doing things towards agencies which has different faculties and different tasks that are being passed around as it reasons about the world and as it takes actions into the world building more open agency systems rather than closed monolithic systems is net good designing institutions that operate this way is a net good and similar to the question of human alignment we have been doing this for a couple millennia we have iterated through different governance structures towards more visibility towards democratic systems we are going to continue with these paradigms as no AI tooling enters the picture as well yeah I really just want to kind of drill down on like my understanding of how AI fits into this stack because I think your big message that you have is AI is yet another thing that we need to figure out how to align along with all the other things that we need to figure out how to align but it's quite it's a very urgent one and my mental model is that like if we can AI is unique from the other systems that require alignment in the fact that we can use it it's special it stands out from the rest of the problems in that if we can align AI we can align everything else and we actually might require AI to align everything else so that we can also align AI I would say it stands out in some ways for it has certain capabilities that hasn't been emerging from others I highly doubt if we can align AI we can align everything else we can we can solve a version of AI alignment where you can align a single AGI towards a single set of human values that then can go and rampantly destroy half of the world and we end up in a Thanos scenario or we end up in a malloc scenario where you end up an aligned AI towards a certain set of values that exploit towards building more and more resource takeover these are not good cases of alignment a multi multi alignment case where we're looking at there is a polity that has different perspectives how and these can be represented as different agents these can be represented as different general AI systems I don't only like we are more likely to end up in that scenario already we have multiple tools we have independent groups that are building many tools these tools require a level of coordination between each other these tools need to be able to have their own interpretability to the people they are accountable to so I highly doubt we will end up with one monolith but the question is just like how we are trying to solve coordination today with many if not AGI more narrow tools that are still capable of massive damage to human existence how do we create alignments across all of these systems okay so if you would there can you like speed run us through the version of the universe that you hope to see I give you and everything that you want to see happens what does that universe look like over the next five to 50 years sure I am excited about the value AI tool and can bring to the universe we will have systems that will help us discover ourselves better we will have systems that help us give visibility to our priorities and see how this is acknowledged by the rest of the world we will have systems that elevate humanity towards what it wanted to be rather than avoid what we afraid we would become I am interested in a world in which I understand the participation that I want to make how does this contribute to the world that I have bandwidth to explore and play I would like there to be a world where everyone is able to have bandwidth towards the hobby source the joys that they're bringing in to pursue I want a world in which we are able to see how our opinion counts in a larger system that is making decisions that is interpretable and accessible I want there to be more human connection ultimately it's around being able to have humans interface with each other more not less that's why the core of this all week comes down to the coordination problem can we have humans see eye to eye understand each other can we have the AI tooling reduce the barriers reduce the incentive gradients that are shaping up right now that prevent humans from finding more agreement more shared values more shared joys with each other this is what I am most preoccupied with the world I'm afraid of is one that we talk about this all in our team numbers that we have decided to care about are going up and up and up while we don't necessarily have more human flourishing that is what I'm afraid of AI tooling can bring us an unprecedented level of human flourishing the default systems do not place us on that path and I would like us to go towards there and I think this is possible I think the tooling that can be built today already takes massive steps towards here this tooling we're interested in building this tooling for humans to use we're interested in building this tooling so that the AGI labs can adopt an open agency architecture so they can make more grounded decisions on what the collective is interested in what is safe what is interpretable there's a lot more in there that we don't go into in detail on how we can have systems that can make decisions that are by design safe and verifiable all of these will be a net plus to the world we're living in I don't think we will solve human alignment but I think AI tooling can take a massive step towards that direction that is the world I'm interested in there I get the intuition that you are an optimistic person is that correct I would say so yes I think we need more optimism in this landscape so that we can see what we want to do I think there's a lot that can be done and I have many fears as well but I think these fears can be solved if we get to the level of coordination there if people are peaked by this conversation then they want to learn more about aspects of this conversation where should they go check our website objective that is IS that is send an email to our team hello what is it at message to me I guess digger at objective that is would love to chat if any of you are interested in helping on creating this vision come along we need many many folks to bring this together and make it a possible truth for us so yeah thank you so much yeah thank you