Did Ethereum break or what actually happened?
We brought on Preston Van Loon & Terence Tsao, two Ethereum Protocol Developers, to help us answer that question.
------ 📣 CONSENSYS | DILIGENCE FUZZING https://bankless.cc/diligence-fuzzing-pod
------ 🚀 Airdrop Alpha is waiting for you on Bankless.com https://bankless.cc/Alpha
------ BANKLESS SPONSOR TOOLS:
🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://k.xyz/bankless-pod-q2
🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask
⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum
👾STADER LABS | ETHX LIQUID STAKING https://bankless.cc/Stader
🗣️TOKU | CRYPTO EMPLOYMENT SOLUTION https://bankless.cc/Toku
🎮IMMUTABLE | GAMING ECOSYSTEM https://bankless.cc/Immutable
🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle
------ Timestamps:
0:00 Intro 7:53 Did Ethereum Break? 8:22 What Happened? 14:50 Slot Visualization & Connectivity 20:30 Did Users Notice? 23:57 Why Was This Worrisome? 26:19 Preston & Terence's Emotional Response 35:55 Ethereum Resilience 39:30 Was This Good For Ethereum? 41:20 Solana vs. Ethereum Liveness 44:40 Closing & Disclaimers
------ Resources:
Is Ethereum Breaking? https://twitter.com/BanklessHQ/status/1657122512516575232
Preston Van Loon https://twitter.com/preston_vanloon
Terence Tsao https://twitter.com/terencechain
----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research.
Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures
No summary for this episode yet.
You In the night with finality on the Ethereum blockchain last week and we want to go all we want to go over that today With some experts who can help us answer the question as to whether a theorem broke or not What does it mean to break what happened? What does non-finality mean? I think we'll talk about that and what are the risk conditions that we are exposed to as a result of this How did a theorem handle this event is is really the central question in my mind David? Who do we have on today? We have Preston Benloon and Terence Tao of the co-founders of prismatic labs now in Arbitrum now part of off-chain labs And we brought breast press in and other members of prismatic on to the show before to understand the world of clients Because it is a complicated world But not so complicated that we won't be able to understand it here on the show So some very important questions with some very smart people who have been with the Ethereum proof of stake effort Since it was once upon a time called Ethereum to but before we get into the show We've got to talk to our friends and sponsors at consensus because we've got to talk about this thing called diligence fuzzing It's a new fuzzing auditing tool just fuzz it.
Yeah with a sick new name called diligence fuzzing So what's diligence fuzzing for smart contracts? solidity devs auditors enterprises engaging with web3 that need to ensure that their smart contract is secure before proceeding to main net diligence fuzzing it's an audit gray security tool that serves as an automated way to find cone code Funeral abilities for people without hosting their own infrastructure or writing complete complex test cases So there are many ways to do automated testing, but buzzing is a way brand new way with Quite a lot of sophistication according to consensus that you can go and access right now So this is an extra layer protection for your smart contracts in addition to and above and beyond typical code audits Humans are messy and can miss the things, but diligence fuzzing is diligence by robots Fuzzy robots apparently David can you imagine releasing a smart contract not fuzzing it?
I just can't even imagine I prefer fuzzy smart contracts It's fuzzing all right, so There was an event that happened last week and this was an event that happened after the rollup And I'm wondering if we could kind of tee this up before we bring Preston and Terrence on so what happened? I think maybe the first sort of many event was was on Thursday and then again Friday if I'm recalling correctly all all my weeks are blurred David But give us the recap before we get in here Yeah, it happened twice and so ethereum failed to finalize Which is an interesting sentence what does that mean? Blocks were still propagating transactions were still going through but finalization was not happening And just at a high level finalization is just something that happens when Enough clients enough validators have given the thumbs up about a certain epoch epoch Which are I believe Preston's gonna come in here and give me the whip.
I believe 64 blocks But it's a collection of blocks and then if enough validators give the thumbs up then that that epoch finalizes and that is Permanently embedded into the blockchain when a when a epoch is finalized it requires Sleshing conditions to go back And unwind those blocks and so if a block isn't finalized it's not as permanently embedded into the blockchain as otherwise could be Except with this event which happened twice Ethereum failed to finalize because blocks were being missed validators were not doing their jobs I'm only able to explain it to some degree at a high level And so we're gonna go talk to Preston and Terrence to get a deeper dive into what this world of finalization means should do a disclosure before we get in Of course now prismatic is owned by off-chain labs Which is a sponsor of the bankless podcast very supportive of what they're doing layer two and client development where it also David and I are notorious holders of the asset called eth So you should know that also podcast that we are bullish On Ethereum generally and are interested in finding out what happened today So guys we are going to get right to the episode with Preston and Terrence to talk about whether Ethereum broke or not But before we do we want to thank the sponsors that made this episode possible including our number one recommend Exchange Kraken go check it out Kraken Pro has easily become the best crypto trading platform in the industry The place I use to check the charts and the crypto prices even when I'm not looking to place a trade on Kraken Pro You'll have access to advanced charting tools real-time market data and lightning fast trade execution All inside there's spiffy new modular interface Kraken's new customizable modular layout Let's you tailor your trading experience to suit your needs pick and choose your favorite modules and place them anywhere You want in your screen with Kraken Pro you have that power whether you are a season pro or just starting out Join thousands of traders who trust Kraken Pro for their crypto trading needs visit pro.
Kraken.com to get started today mantle is a brand new high-performance ethereum layer 2 network built differently from the other layer 2s You may be familiar with mantle is a modular layer 2 built on the op stack But uses eigenlayers data availability solution instead of the expensive ethereum layer 1 not only does this reduce mantle's gas fees by 80% compared to other layer 2s But it also reduces gas fee volatility mantle It has a decentralized sequencer set eliminating the risk of downtime and censorship on the network And because mantle implements multi-party computation notes layer 1 settlement execution is shortened from seven days to as low as just one or two mantle is the first layer to built by a Dow and is backed by one of the biggest Dow treasuries in the world bit down mantle already has subcommunities from around web 3 on boarded to help the growth of mantle like game 7 for web 3 gaming Or edu Dow for the world of D side and buy bit for tvl liquidity and on ramps check out mantle at mantle dot xyz and follow them on twitter at 0x mantle Immutable is at the forefront of web 3 gaming on a mission to bring digital ownership to every player offering the world's best games and game development platform Immutable lets game builders and players focus on great gaming experiences So build your next web 3 game on easy mode with immutable's leading full stack web 3 gaming platform It's built in ux features like the immutable passport are designed for games to scale to the next billion players coming to web 3 with Immutable players can sign up with an email pay with a credit card and experience a frictionless purchase flow inside of games So discover your next favorite game and explore a network of 150 games building on immutable including such titles as Gawson chain guilds of guardians eluvium embersword and metal core so join web 3's largest ecosystem of games and players build play and connect at immutable dot com Bankless nation press and ban loon and Terrence Tao are the co-founders of prismatic labs one of the organizations that maintains the prism Ethereum consensus client Prismatic labs was one of the earliest client teams that was formed to start building what was then called Ethereum to which we now just call Ethereum and they've been very key to Ethereum development every step of the way Preston Terrence welcome to bankless They happy to be back Thanks for having me Yeah, okay, I'm gonna look forward to being educated about some of the nuances of finality and security, but first Did Ethereum break Preston?
No, it did not break, right? We did see Ethereum actually work this We have a time where Ethereum has difficulty finalizing And we got to see it actually work in action, right? So this is not Like a problem with Ethereum that specification did not break it actually it actually worked and it worked in a Unfavorable condition and we got to see that happen in main net for the for the first time So how would you summarize what actually happened and so I did my best to explain it in the intro How would you describe to the layman explain it like I'm five what actually happened here? Sure, and how many blocks are there in an epoch Preston most importantly There are 32 blocks in epoch not 64 at least you got it the power of two so that's nice What happened was Multiple clients prism specifically and tech who had this problem where If it received these sort of old attestations attestations being what the validators are producing to say This is what I think the head of the chain is they received these old attestations which caused prism in particular to replay a bunch of states, right?
It had us to go back in time to evaluate whether or not this attestation was valid or Or how to count this vote You know to being a pretty expensive operation that somehow bypassed or filled the caches that we had so We weren't able to you know Keep that work in the client had to recompute it every time so essentially it was An accidental I guess like kind of denial of service where the clients were so busy Doing these computations they weren't able to respond. This is the beacon node part of the client The beacon node was not able to respond to What the validators were asking to do the validator said here's the block I want to produce here's the attestation I want to produce these weren't getting out the door because the client was so bogged down with trying to process these Valid but untimely attestations That's the gist of it This happened to two clients Preston yeah, well Two clients have released updates specifically to handle this scenario That's prism and tech who lighthouse already does it really well so A lot of us have Sort of taken in inspiration from their model of how they handle these old attestations in short They they basically dropped them to say This attestation is not going to be very valuable to how we're going to update the Fork choice It's not worth re-computing all this all this effort.
So we use for a copy their model of Just excluding that attestation from consideration not doing all the work. And so why did this happen so suddenly Well We're still trying to track down the exact you know trigger to the event But what we think had happened is that some clients namely lighthouse when They were when they lose a connection to the execution layer the layer that The client that handles transactions and the the state of ethereum as we think of from you know the application layer That being like go ethereum or or whatever you use for your execution client when We lose access to that some clients will behave well.
They'll just withhold these attestations and then later produce those Just sort of later release them. So what prison does is they just don't We just don't attest during the time because we can't really come up with We can't advance our node. So we're not it just wouldn't really be attesting there Other clients will produce this sort of old valid, but you know not valuable attestations And that was what actually triggered the event. It's not a common path that happens But what we have learned is that it wasn't Something that was nefarious. It was just some sort of unexpected behavior valid behavior But unexpected that prism and perhaps tech who didn't consider or handle very well I think a good like a question here.
It's like why it didn't happen on a test net right like why it didn't happen before I think there are several recent to it the first reason is that there's a lot of validators now there is 600,000 validators and validator matters Like the number of validator matters the more validator you have the more hash you know you have to do on a state and that's slower And our test net only has 400,000 validators. So even the test net is lower than the main net Which is obviously a flaw here that's something that we're taking back and we're gonna revisit how do we do this type of test net The second thing is there's a lot more deposit now right because we just now we enable withdraw and then the deposit key with and then people are kind of Getting their top off if they're gonna really pass it and then with so many deposit There's a lot more hash in that you have to do they also increase the latency So those are the two big factor and once we reach those two bowling points and we can't just create this that's That's Okay, so to recap what I'm what I'm hearing both There were our new conditions about the state of the beacon chain a higher number of validators are increasing the resource requirements of nodes And then also from what from what Preston was saying prism in lighthouse shared similar properties because prism just borrowed from lighthouse's code base to do this one part of the client That in prism and because that part of Was shared this showed up this problem that was triggered by the increased computational requirements and and bandwidth or networking requirements of the more and more Validators coming in that was triggered But then it oh because of the shared property of lighthouse and and prism that showed up in two clients Which is how it actually showed up as a non-final event in ethereum is that is that recap accurate?
Sort of so we we the fix was to replicate what lighthouse handles this behavior It was prism and teku that did handle this scenario very well So when the outage happened right there were Something like 60% or more blocks were missing so that's an indicator that more than one client is having issues And we believe that to be prism and teku because both clients have really just sort of emergency updates or critical updates Since the incident on friday and thursday Okay I have a question in general about what this looks like for ethereum to not finalize so here is um a Beacon chain visualizer we've shown before and on bankless I think it's specifically when that you know hard fork upgrades happen You're sort of look for these things to go from from red to green right these are Proposed blocks that we're looking at so when the blockchain doesn't finalize what does When ethereum didn't finalize on I guess it was thursday once for a period of time And I don't know how long that was and then also friday what first of all what does this chart look like?
Well the the main indicator is that the The green bar you see at the bottom of each epoch justifying a finalize that's not gonna be Greater than two-thirds you needed to be greater than two-thirds to justify or finalize Okay, in the case of The incident last week we were missing a ton of blocks So you can see a couple of red spaces here in the blocks those are missing blocks This is a proposer whoever they may be failed to To produce a block at that slot And because there were so few blocks you know more than half of them were missing It wasn't possible to pack in all of the attestations and also very likely the same nodes that we're suffering from Not being able to produce the blocks we're not able to produce the attestations either So two parts to it you have to have enough blocks to Include attestations and you have to have enough attestations That agree with each other that being more than two-thirds to finalize and just to are just to finalize an epoch And how long were these periods of time where this sort of degraded finalization service was happening On the first time on Thursday, I think it was for four or maybe maybe five epochs that we had A gap in finality usually we're seeing every epoch is finalizing You know two epochs later.
So in this screenshot you can see epoch 63 is finalized when the Current epoch 65 finishes. We're going to see that 64 will finalize and we just have this sort of continuing rotating Um, you know the epoch two epochs ago is finalizing Well when it wasn't finalizing on Thursday it took five epochs to get back on track and then on friday it took even longer I don't recall the specific number maybe terns knows but what was significant about friday is when it goes Longer than five epochs. We start to incur a additional penalty that everyone gets penalized and it's called a a Alive list liveness leak.
I think um terns you want to explain Indeed sure, so we we have this inattivity deep so this inattivity deep is essentially one make ethereum world three Resilience right so imagine we cut the internet in half and then there's this Say asia and then USA and they cannot talk to each other right in that case we want We want either side of a validator to dig their balance faster to lose their balance faster So we can finalize faster on each side. So you begin to lose balance faster If we don't finalize for more than four to five epochs and then and then eventually after 17 days you will loss Probably around 16 Eve then you will get ejected from the pool.
So this is the first time this inattivity um Yeah, inattivity deep happened. So I did some math earlier So we ended up burning about 28 Eve and I don't know if this is something that we should be proud of just everyone looks for Eve But it's slightly different. So on average every validator if today if you are offline at that time you lost about one dollar. So yeah Okay, just to double down on that mechanism the idea here is Ethereum's philosophy is to withstand world war three So if we imagine the internet as a gigantic cell and that cell gets cut in half Ethereum should find a way to to bridge the gap between the division of the internet and the way here is that If can if there's can't be consensus between the nodes across the world There are the assumption is that there's going to be at least a few nodes that can span that gap and for all the nodes that cannot span that gap They will get inactivity leaked and down until they are kicked out of the validator set And then the few remaining nodes that can span east and west or whatever division of the internet They will be the ones left standing or not getting inactivity linked because they are the ones still propagating the blocks And that's the mechanism that that entered.
Is that correct? Yep, that's right Well, at least we got to test the inactivity leak right guys. There's a silver lining here I don't think we've ever tested that in the wild and so this inactivity leak Came to play for those who are staking with what was it as it just prism Antecu and lighthouse kind of carried on was it just a sort of a subset of client users? I think it's hard to tell I think every client had some issue at that time because of this high CPU usage So they're sending their attestations, but their attestations may not be received may not be processed So I think like every single client unless today you run your client on a very very very powerful machine That that you were likely put even or you're losing a few cents I got it.
Okay, so what happened that that's the view sort of from the from the client and high level blockchain Perspect and we've seen that What about the average user Were what happened during this time were they able to get transactions through or what happens to users using the using the network did they even notice? So I guess put it this way right if today you wasn't for pitching chainsplorer if you wasn't for that side Will we ever find out the issue maybe yes, maybe not, but I don't think we'll find out the issue at that time right We'll probably find out the issue like a few hours later when someone was using this finality notion before right so So with that it basically you can say that yeah, it probably doesn't affect most people if today you're You're doing a swap on unit swap you're buying something at least those still goes on But if today say today you're actually using finality, which is a pretty rare set of application I will say maybe there too Maybe like maybe like it's changed does a big amount of withdraw then that would be effective But that's a kind of a small set of the use case.
I mean was the acceptable? No, I don't think it's acceptable for Ethereum to be This global internet of money which we cannot have this right, but I'm really happy that I'm really happy that life This still goes on because that's probably like I would say close to 95% of the Use case all the transactions were going through completely It wasn't as if there was a pause and sort of user transactions. It was just this one Finality piece that some sort of you call them settlement Applications might use it might be a layer two or it's an exchange maybe doing a big batch Process with a lot of money and they they want to wait until full finality is achieved before they before they settle it Those are the kind of apps that were maybe affected even though that's even hard to see because this is so brief but users weren't affected at all like the regular users and Liveness was preserved through this event Press and looked like you were going to add something to this Yeah, I wanted to add that the real impact to for the end users was that there was a diminished amount of block space, right?
So when we're packing transactions into blocks what drives the gas price of Ethereum is the Demand for block space and the availability So I did notice that at least on Thursday gas prices were slightly higher than the daily average It was like something like 120 quay at least on the epoch that I checked Which you know in terms of the scale of Ethereum gas pricing is kind of a negligible increase Maybe it was a hundred epoch before and out 120 we've seen a lot of You know Daps push the gas price much higher for different reasons So it didn't seem like anyone really would have noticed right it wasn't like astronomically high It's just slightly higher than average during that time due to Less available block space for transactions, but everything was still going through pretty in a timely fashion You may have noticed your your transaction took Up to a minute, but where normally takes you know 12 seconds But that was really really yet you didn't really feel that it was down or broken in any way So in that case, what's the big deal here like why do we care about finality?
What would what would have been what could have been a worse condition? What could we play out a scenario or maybe this happens again and then something actually bad happens like what's the big worry here? Well The concept of finality in proof of stake is is one of the main features, right? It means that we have finality in the blockchain and that we can be assured that that Point of time is is final and can never be Rewritten or we can have go back in time Build on a prior block and sort of exclude this history that's happened As Terrence of saying that's very important for some transactions So if you're doing maybe a very large purchase, let's just say You know, we're at a point where you can purchase property in the real world.
You say I want to close on this property I'm going to send you my eth and then I'm or whatever it might be my tokens And I'm not and we're not going to consider done until there's finality, right? So there there is that Aspect where we wanted to be final cannot be rewritten So it's also part of the entire security of ethereum, right? If ethereum aims to be the global settlement layer, you need to have this consistent finality in a timely fashion And it can also affect, you know, layer two's or other Networks that are leveraging ethereum security I was imagining a scenario or what if this played out for A very long time, right?
We have optimistic rollups out there with the A seven-day I think a seven-day window for the challenge period So you can imagine that if this incident went longer than seven days, what does that mean for the challenge period Even though it has resolved it's not entirely final because It's not final until it's finalized And I thought about that as being sort of like the The medium-term worst case I mean the the obvious worst case this goes on longer longer until we have so much in activity Leaked that a lot of validators are ejected eventually ethereum recovers L2s would recover and those challenges would eventually be finalized But you could see that would be a major disruption if it went on longer than seven days We're can you like put us into the headspace of press and when this was happening like were you freaking out Where you're like oh that's a curiosity Just like sometimes yeah, we can talk about like the technical details of Finality but what about the emotional response like how I can give you guys the response of people on twitter Which was mass panic freak out.
Oh my god Ethereum is broken that was the response on twitter, but yeah response Yeah, so this was when I happened on Thursday it was around like 4 p.m. My time and I'd been starting my day very early and I was like I'm gonna go I'm gonna go take a nap like I'm kind of tired right and then I just check right before I go to make sure nobody needs me for anything And I see this message. I never want to see again from nishant. He's on our team nishant does If you send me these a couple times He says maintenance stop finalizing and I said oh my god what there's no way I'm gonna be able to sleep for days You know is this three words those three words maintenance stop finalizing and This is how I found out about it the first time so this isn't the first time meena has had a finality Delay or finality incident happened a few years ago where prison was unable to produce blocks and prison was a super majority Thanks to client diversity.
We're not Have that risk anymore, but he is same message. He sent me he said prison of maintenance stop finalizing And that time It took us I think three days to figure it out to figure out what the issue was And we were sleeping in rotating shifts like maybe a few hours at a time So I when I thought okay, I'm gonna go take a nap and then he said those three words I said great. I'm just never gonna sleep again. So emotionally that was that was challenging Well, I think this does a very good job illustrating of why client teams and are so important And why client team maintainers are heroes that should wear capes So press in Terrence.
Thank you for for always wearing a cape even though You don't actually legitimately literally wear it wear a cape There's a lot more that I want to ask about but first we have to talk about some of these fantastic sponsors That make this show possible especially metamask if we have used jargon in this episode that is confusing to you I bet you metamask learn has an explainer for you on your product. There is a link in the show notes here we go Learning about crypto is hard until now introducing metamask learn an open educational platform about crypto Web 3 self-custody wallet management and all the other topics needed to onboard people into this crazy world of crypto Metamask learn is an interactive platform with each lesson offering a simulation for the task at hand Giving you actual practical experience for navigating web 3 the purpose of metamask learn is to teach people the basics of Self-custody and wallet security in a safe environment and while metamask learn always takes the time to define web 3 specific vocabulary It is still a jargon free experience for the crypto curious user friendly not scary Metamask learn is available in 10 languages with more to be added soon And it's meant to cater to a global web 3 audience So are you tired of having to explain crypto concepts to your friends go to learn dot metamask dot i oh And add metamask learn to your guides to get onboarded into the world of web 3 hiring people worldwide paying them in crypto Providing them access to benefits it all so complex But it doesn't have to be complying with labor laws payroll rules tax obligations and crypto regulations in every country that you employ someone is Difficult time consuming manual and costly and is drawing more and more attention from regulators and governments But there is good news Toku is here Toku is the first employment and compensation platform for the crypto industry that makes this easy Toku helps you hire employees or contractors and pay them in fiat or crypto legally Compliantly and with all the taxes handled in over a hundred different jurisdictions So whether you're an early stage company with just a team of two or you're an enterprise with 200 Toku has a solution that meets your needs Toku is already working with the leading companies in the space protocol labs Hedera get coin and many more so transform your employment and token payroll operations with Toku You can reach out to Toku at toku.
com slash bankless or click the link in the show notes Introducing eth ex from stator eth ex is a liquid staking token designed to maximize rewards all while securing ethereum With stator you can run in an ethereum node with just four eth and 85% lower capital requirement versus the 32 Eath required for solo stake with stators for eth nodes You can get a 35% average higher yield since your charge fees are those who use your node to stake their eth by running a node with stator The eth ex staking derivative token can get minted on your validators and can flow into the world of defy Which stator is actively building integrations and partnerships into to increase the utility of eth ex stator allows for both permissioned and permissionless nodes to join the network Maximizing its potential scalability for eth ex while preserving the values of decentralization and openness behind its liquid staking token Go to stator labs.
com slash eth and sign up to get access to the stator staking protocol Arbitram 1 is pioneering the world of secure ethereum scalability and is continuing to accelerate the web 3 landscape Hundreds of projects have already deployed on Arbitram 1 producing flourishing defy and NFT ecosystems with a recent addition of Arbitram Gaming and social daps like reddit are also now calling Arbitram home both Arbitram 1 and Nova leveraged the security and decentralization of ethereum and provide a builder experience that's intuitive familiar and fully evian compatible on Arbitram both builders and users will experience faster transaction speeds with significantly lower gas fees With Arbitram's recent migration to Arbitram Nitro It's also now 10 times faster than before visit arbitram.
io where you can join the community dive into the developer docs bridge your assets and start building your first app with Arbitram experience web 3 development the way it was meant to be secure fast cheap and friction free And we're back now Karen's I'd like to turn to you What was your story what what is it like from your perspective to hear the words main net stopped finalizing as a client team dev Yeah, it's definitely not great here in that Thursday afternoon Typically I stop all my meetings After Thursday, so I was excited to get to coding have some personal time with some paper But um, so I heard yeah, so like present this did I heard from the news I heard from telegram and to lose 65 percent of participation.
It's it's a little crazy like when I first read that I just couldn't believe that I was like Just how that happened right it's like imagine like 65 percent of the node just disappear right it has to be like This sounds very like catastrophic if you really think about it So in my mind, I just started piecing everything together quickly right because like 65 percent then I know it's not just one client because like Thanks to client diversity like every client is pretty spread even my presence Sorry, a prison is 33 percent my house has that 33 percent. So I know it's not just one client So it's not a client bug then I felt okay, maybe it's like a Maybe it's like a target attack right and then Maybe it's like they really like someone is attacking the network Finally someone is actually taking a network and then but actually like the whole thing like went back to normal after like 2 to 4 epoch which is like 20 minutes so they cannot give us more time to think about what the issues is right and then I thought Well, if it's a target attack like why did this stop if you I mean if they could have done for longer like why stop there So I so I didn't really think it was a target attack at that point then I thought maybe there's like a really big It's change or staking pool that runs something on a GCP and AWS and then they shot down or something like that So that was another reason, but I luckily I think the whole thing took about like 4 to 5 hours until we Came to the geese of geese of it.
That's basically our software was little bit Um under of under of optimize So how hard of a problem was this to fix as soon as so it sounds like diagnosing wasn't too difficult But what was the actual solution process like It wasn't hard to fix We could have fixed the problem within the first 30 minutes But we kind of wanted to do the perfect fix right because everything is trade-off client software is a trade-off If you optimize one thing you cannot lose another thing and this is why client diversity is nice right So I'm gonna shut out client diversity again just lie house What fine because of the trade-off they they made right today Network is under another scenario like house may underperform Life versus prison just everything is a trade-off here, right?
So we kind of We kind of picked the fix that we thought had the right trade-off that where we like and then again This is a team effort like like probably like 5 to 10 of us like we got together We talked about it and then without okay. This is the fix that we want We also share our fix our idea with five other awesome client teams as well, right? And then because we're all learning from each other. We are or study each other's design. So yeah Well one interesting observation I have about this is this is um this was sort of a Finality event, but it was but it's also like a non-event Right like uh when we asked the question at the beginning of this episode Presson did Ethereum break and you were just emphatically no it didn't break And so this is maybe a demonstration of the resilience of the Ethereum network and I will What came into play?
Like how did the Ethereum network show resilience during this event like that? Terrence you just talked about Multiple clients was was that the key thing? Is that the thing that that saved our ass? I guess I would say or there other elements that demonstrate the The resilience of the theorem first to you Terrence That's one thing right if today it was three years ago when prison was the majority client I will be really really worried right but today we're just like a piece of the pie and I'm so therefore I feel like I can steep a lot better The second thing is that we really need to thank like this This thing is called lmd ghost This is designed by beta dig and a few breathing researchers from the ethereum foundation So lmd ghost is this consensus mechanism is this for choice rule that we use for ethu or Ordeal beacon chan basically this is so essentially The beacon chan ethereum proof-of-stake consensus is made up to part the first part is lmd ghost Essentially if there is likeness it basically ensure today if we don't have safety We don't have finality the blockchain keeps going and this is very important right as it turns out because before I I always question what do we need this?
Why can we just have safety? That's it. What can we be like a tandem in chan? What can we be like maybe like the other blockchain that just have safety right? And this is nearly the reason this this is why because like if things goes wrong If we don't have finality, we don't want to affect the use we don't want to affect the UX right One people still be able to buy on that t's they'll be by coffee still to make trade So essentially to answer your question This lmd ghost for choice rule was the one that saved our butt It prioritized the liveness that's what it did specifically pressing what would you add to that what saved us Yeah, I think tern said a well.
It's the specification of ethereum lmd ghost and these things that make ethereum The way it is combined with client diversity right like The specification worked here it when there's an incident it can recover it can live on And because there was an incident and we're having client diversity It was minimized to a subset of the network right it could have been worse if it was a more of a majority prism It brings me back to the days when we had these early test nads Like medasha which was 100% prism and it was just on fire. It was it was terrible When there were incidents and you know Thankfully client diversity we've done a really great job over the last two years With three years however long to Smooth that out And we could see you here today that there was an incident and it recovered on its own without intervention within minutes And that we were able to Release mitigations and hot fixes and client updates within days to say okay now.
It's not gonna happen again So I think that went really well. It's a great test in main net So I'm gonna pull in something from my Bitcoin or world They always love to say this is good for Bitcoin And the spirit of that that response is that Anti-fragile crypto-economic systems what doesn't kill that makes them stronger. So pressing Was this good for Ethereum? Yeah, I think you know like we're saying this is the first time we've seen it in May net in production. We knew it would work. We've done it in testnets, but actually see it in in action It just goes to show the resiliency that when things are going horribly wrong It can still be okay and it's still Eventually finalizes and that the mechanisms and the specification and the design of Ethereum proof of stake is resilient So I thought that was great And also yeah So like Ethereum is evolving protocol as we know right it is still constantly heavy under heavy R&D And then we definitely came out here with like a very better understanding of just like very little detail of things that we want to improve of even like Future research spec as well.
Just I heard your podcasts on MEB burn right that's a very good podcast, but because of this right we're kind of looking at MEB burn we look at PBS to see how we can do better there as well because everything is interconnected Like they always they basically all touch each other and I definitely feel like we definitely got all of this very strong And not only that but clients have discovered a weakness that is now removed correct Yeah Cool nice. I love anti-fragile systems My last question is this and this the ghost back to Terran something that you were saying about kind of the The design of Ethereum proof of stake with LMD ghosts.
It's like weighing the weeds for people, but like the Priorities that the Ethereum network makes as we all know it's not high transactions per second on The base layer on main net right instead a theorem has made other trade-offs And one of those trade-offs is Liveness a liveness trade-off and I want to just focus for a minute on Some other outages that have happened in crypto with other layer one blockchains And we don't have to name names. It's like I don't want us to Like anyone to feel like you know the theorems punching down that kind of thing Okay, so like but like just just think of the first name that comes to your mind about a blockchain But but also just like just the facts.
Okay, so like um, I think When people were talking about and by the way bankless included we're sorry Everyone listening for the the tweet headline all right. We've made a mens on that That happened this week, but when people equate um ethereum breaking To sort of the the downtime and the breaking or temporary halting of other blockchains It's much different. So I'll just throw the name salana out there Just just for a minute. So we can talk about that as a just the facts kind of use case here um When salana was down because it did not prioritize Liveness when it had some some issues. It was actually offline right like you couldn't get a transaction through for a period of time I'm wondering if you guys could just just weigh in on that briefly because it is a useful contrast point Okay, Ethereum doesn't have the highest transactions per second on the main net, but um, it does have this feature of Liveness Terrence could you talk about that for a minute?
Yeah, so I just wanted to add right like there's nothing like what we are doing today, right We're working on opens for software and this is the open network That means anyone can read any code that we write where like chefs were cooking the kitchen right and anyone can join the network right So it makes you very like vulnerable to attacks Therefore, right? therefore I think favoring lifeness is very important just because like you want your life to keep going right say today You have a country that's built on top of Ethereum. I want to buy coffee and then I I mean if today if doom is done We cannot process transaction they I cannot buy my coffee then I will be very upset, right?
So I would say lifeness is a very important property probably the most important property if you want higher throughput There is layer two right and that's what their two is for but if you must base there definitely a lifeness Yeah, you know, I think that's what makes a blockchain great is to have liveness right It's not useful if you can't use it so to have liveness is I think the most one of the most important Aspects of Ethereum more so than transactions per second right we have solutions for that We can't abstract too layer two or or other clever mechanisms But if you don't have whiteness then none of that works you have zero transactions per second.
So it is the most important Aspect of Ethereum You're welcome. Thank you guys Thanks for summarizing this. Thanks for jumping on the bankless podcast explaining this to the bankless community and the crypto community Did a fear and break the answer is no this was maybe whoever treats that is done Yes This is maybe Ethereum's biggest live-ness test yet though And so it was good to come out on the other side and see a resilient network Pretty unique. I think something that's that's very special in the space and worth preserving and that's always got to thank you Whenever we have client devs on on the bankless podcast We would be remiss David if if you and I did not express our thanks both to you Terrence and to you Preston and also to any other client dev that is doing this hard and essential work to build this incredible freedom technology network That we care so much about thank you sincerely open source superheroes for We appreciate you guys coming on Yeah, it's our pleasure to get to work on it really it is an exciting opportunity We know that Ethereum is going to change the world and to be working on it at the protocol level There's nothing better that we could imagine doing so it's our pleasure to be here Plus one to one person said thank you for having us We're sitting disclaimer skies uh, of course, gotta let you know Ethereum is resilient But uh, if the asset is risky so is all of crypto so is defy you could definitely lose what you put in But we're headed west.
This is the frontier. It's not for everyone, but we're glad you're with us on the bankless journey. Thanks a lot You