Sharding what? Don't worry. By the end of this conversation, you’ll know how sharding has evolved over time, why danksharding is possible, how proto-danksharding will get deployed, and so much more.
------ 📣 ALCHEMIX | Get a self-repaying loan today! https://bankless.cc/Alchemix
------ 🚀 SUBSCRIBE TO NEWSLETTER: https://newsletter.banklesshq.com/ 🎙️ SUBSCRIBE TO PODCAST: http://podcast.banklesshq.com/
------ BANKLESS SPONSOR TOOLS:
⚖️ ARBITRUM | SCALED ETHEREUM https://bankless.cc/Arbitrum
❎ ACROSS | BRIDGE TO LAYER 2 https://bankless.cc/Across
🏦 ALTO IRA | TAX-FREE CRYPTO https://bankless.cc/AltoIRA
👻 AAVE V3 | LEND & BORROW CRYPTO https://bankless.cc/aave
⚡️ LIDO | LIQUID ETH STAKING https://bankless.cc/lido
🔐 LEDGER | NANO S PLUS WALLET https://bankless.cc/Ledger
------ Topics Covered:
0:00 Intro & Danksharding Explained 7:00 Panelist’s Backgrounds 8:34 How Sharding Has Evolved 15:56 Why Danksharding is Possible 23:40 Proto-Danksharding 27:15 Data Availability 35:40 Data Availability Sampling 48:00 Cryptography for Noncryptographers 56:10 How Does All of This Get Deployed 1:00:05 The Big Open Questions in Sharding Land 1:05:30 Closing Thoughts
------ Resources:
Vitalik https://twitter.com/VitalikButerin
Dankrad https://twitter.com/dankrad
Protolambda https://twitter.com/protolambda
Tim Beiko https://twitter.com/TimBeiko
Proto-Danksharding https://www.eip4844.com/
----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research.
Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://newsletter.banklesshq.com/p/bankless-disclosures
No summary for this episode yet.
Hey, Bankless Nation, welcome to our special live stream. This is going to be a panel addition. I think, David, this is going to be our dankest panel, maybe the dankest episode we have ever recorded because we are talking about dank sharding today and I got to be honest going to this episode, I'm not entirely sure what it means. And what is dank sharding? What will ask some of the participants this and maybe David, you could describe the setup of this panel and who's on it and how we're going to handle this episode. Yeah, of course, you guys all know Tim Beco, he's the guy on the bottom of the screen.
Tim manages the all-core desk call and is coordinating the lead into the merge and beyond. And we've had Tim Beco on before, he let a very technical EIP1559 panel and asked the questions that Ryan and I are just not smart enough to ask. So this is one of those panels, we're going to get as technical as possible. We have three fantastic panelists who are behind the scenes, we've got Vitalik, Dankrad and Proto, who are the minds behind dank sharding and sharding in general and other Ethereum related technologies. And Tim is going to be able to ask questions that are the technical questions, the smart questions.
But before we get there, Tim, we just want to cover some high level stuff. What is dank sharding? How to get its name and just like, what does it mean for users? Right, well, first, yeah, thanks for having me on, guys. So dank sharding and Proto dank sharding, which we'll also get into our iterations over the sharding design for Ethereum. And we'll spend, you know, the bulk of this panel discussing what they are, what the trade offs are and what not. But at a high level, sharding is a way for Ethereum to have more data pass through the network. And because layer two solutions, like ZK roll ups and optimistic roll ups, they produce it on a data, if we have a way for them to post that to the Ethereum network more cheaply, it immediately reduces how much they need to charge users for transactions.
So all of these kind of flavors of sharding all have kind of the same end goal, which is to create a cheap place for layer two solutions to post data on Ethereum. And the impact of that is that the transaction fees that end users and the pay on layer twos is is lowered by a lot. So this is all about getting transaction fees down, particularly on layer twos, gas fees down on layer twos. And Tim, is this related? You mentioned Proto dank sharding and dank sharding. You guys will talk about all that in the panel. But is this related to EIP4844 because that's another EIP we've heard a lot about. Yes, so EIP4844 is Proto dank sharding, basically.
And then what do I have to think about it is like Proto dank sharding is like maybe the first step we get towards sharding and then dank sharding is the simplification that we've had on the previous roadmap. So like the more prefixes we have, it's like the sooner we get them. Yeah. All right, and this has been, like you said, an iterative cycle, an iterative process to get to where we are today. And dank sharding comes from dank rad, who's on the panel. And Proto dank sharding comes from Proto lambda, who's also on the panel. So these guys' names have been baked into the name of this EIP itself.
Tim, is this the new EIP1559? Is EIP4844 going to be the new EIP that we focus about going into the future? So there's a lot of stuff we're working on right now. So you know, it's definitely one of the big ones. And I think for end users, it's one of the most impactful ones because it directly affects like the gas price they pay. Hopefully it's much less contentious than 15559 was. But yeah, it's definitely an EIP. You're going to be hearing more about over the next few months. And just one last question before we hand it over to the panelists and we bring the panelists online. There's been a bunch of conversations about, you know, when EIP1559 came and go, people were like, is this going to reduce gas fees and the answer is no.
And then people are like, the Ethereum merge, is that going to reduce gas fees? And the answer is no. This EIP reduces gas fees. This reduces transaction fees, not for the layer one, but for the layer two, correct? Yes, on the other two, yes, that's correct. And basically, 4844 is a way for us to get some of the reductions of sharding quicker. And then the full dank sharding rollout gives us even more reductions. But because 4844, aka Proto dank sharding is simple to implement, we can just get that first. Fantastic. All right. I think that is all of my questions. And I think that's all of Ryan's questions.
So with that, we're going to ask the panelists to come in from the shadows and turn on their cameras and me and Ryan are going to actually duck out of here. Excuse me. We're going to dank out of here and let Tim take over this stream. Good luck guys. Welcome to the panel. And Tim, thank you for doing this and then absolutely just take it away. Awesome. Thanks. Okay. Sweet. It's just us now. So I guess, yeah, before we get into it, do you want to just take a minute and kind of talk about like what you work on and who you are? Yeah, Vitalik, we can start with you. Yeah, so hi. I'm Vitalik. I'm the co-founder of Bitcoin Magazine.
I write a blog. I contribute to specs once in a while. Thank you. Thank you. Yeah. Hi. Hi, I'm dank hat. I'm an Ethereum researcher since 2019 and I'm working on among other starting and what else are we're going to improve custody. Statelessness. Yeah, some project on the on the roadmap of Ethereum. Nice. And Proto. I used the work at the Ethereum Foundation on research there. Now I do the same thing, but I'm not optimistic. I helped with Shardin year-to-year-on, and I'm contributing back to layer one while working at optimism and layer two. Sweet. Okay, so just to kind of get into Shardin generally.
Over the past couple of years, what Shardin means that Ethereum has changed a lot. And I think the biggest one is this shift from full execution Shardin, the only data Shardin. The topic, do you want to give us just an overview of how that shift happened in a research roadmap and why we've landed on just doing data Shardin? Yeah, so I think there's been this ongoing simplification of the Shardin roadmap that started really sometime in 2016. So for people who have been in the Ethereum ecosystem for a long time, you might remember some of the scaling docs that we published back in 2015, back in our 2016, some of the blog posts thinking that I came out in 2014.
And the stuff that was there in those earlier periods was in a lot of ways really complicated, right? Like there were these ideas around hypercubes and hub and spokechains and in protocol supported cross-shard transactions that would be routed between like one corner of a hypercube to another corner and where the protocol would help them move from like one Shard to another to a third to a fourth. And there was even thinking about super quadratic Shardin, which is basically saying like, instead of just having Shards, you have Shards on top of Shards and potentially like an infinite hierarchy of Shards inside of Shards.
So actually the sort of stuff that the telegram time project ended up by incorporating into their paper, though I guess that never really ended up coming close to going live, unfortunately. But that was the kind of thinking that we had back in 2015 and in 2016. And I think after that, the progression has just been this like big slow process of, I guess increasing pragmatism, increasing appreciation of how complicated it is to develop and actually bring the production just about anything. Like what feels like 10 lines of code actually becomes like hundreds of lines of code once that you had all of the year.
A couple of cities that clients have to inevitably have and deal with that, you know, how some particular thing interact with the sinking process, how some particular thing interacts with the fork choice how. Some particular thing and interact with the database and the need to store it and optimize formats and that sort of stuff. So the process of simplification, I think the big first step was definitely the decision to not bother with anything beyond quadratic sharding and just say we're doing quadratic sharding right. So not bother with with doing ever any kind of shards on top of shards and just saying we have to be can change we have shards shard headers or connected to the beacon chain.
So that was the first step, then the second step was the move from this concept of like chains that have regular commitment blocks in the beacon chain. I think there was another word for them, I forget what, what, what, what that word is a cross links was the word, right. You have shard chains that cross link into the beacon chain. So move it from that to a system where you just have every shard walk directly get included in the beacon chain. So that was the second simplification, I forget exactly one that happened, I think that might have been around 2019 or 2020 or so. And the big benefit of that simplification is that it meant that we didn't have to worry about shard chains anymore.
And then after that, there was the idea that we're going to do data sharding first. This was when I started talking about the roll up centered road map, right, basically instead of shard walks actually containing transactions that would be executed at the ethereum layer. And then the blocks would just contain big blocks of data. And it would be the responsibility of we were to roll up protocols to use that data space in order to create secure and more scalable experiences for their users, right. The system would provide non scalable computation and scalable data and what a roll up does is it converts scalable data and non scalable computation into a scalable computation.
So we have a somewhat more performically or one that has extra data space and then we combine that with this way or two ecosystem and the way or two ecosystem ends up like really bringing all of the scalability to life. So that's the roll up centered road map and at the beginning, I think the roll up centered road map was phrased in this ambiguous way where basically said, well, look, the roll up centered road map realistically, like data sharding is the obvious prelude to full sharding anyway, right. Like if we're going to implement full sharding with EVM's all the shards, it's just an obvious first step to have data shards first, but it turns out the data shards are actually really good for roll ups already.
And so we might as well run with that we might as well realize the role of sort of best hope for short term scalability and and just take that direction and try to make the best of it. And that still leaves open the door for adding EVM execution shards in the future, but it basically says, well, actually, you know, Ethereum will be fine, even if we end up never actually keep really doing that right. So that's roll up centric road map since another simplification basically saying we don't have to bother with execution on shards and then that also allowed some other simplifications, like it made it even more possible for shards to not bother with fortress rules, for example.
And the next simplification after that is think sharding where basically we said that there is actually this merge proposal mechanism where there's only one proposal that chooses all of the yeah short blocks and all of the shards that appear within a particular beacon block. And that simplifies things massively in a whole bunch of ways that basically means we don't have to deal with like the whole short proposal bureaucracy, which simplifies that can set this couple of city a huge amount. It simplifies some some of the economic properties a huge amount to basically makes the system feel much more similar to like just what a non scalable channel look like except it's just more scalable right and that extra scalability happens in the background.
We're dig shorting finally like that's not a simplification that's more a step on the way to fault dig shorting that gets us maybe half the benefits of shorting but at a points that's maybe like halfway along the timeline to actually getting full sharding out there so we actually get some of the benefits sooner right so. So that was a general progression like basically more complexity to less complexity more more of a theory i'm trying to do everything to less theory i'm trying to do everything and more willingness to work with way or two protocols and those things together and that's where we are now.
Yeah, thanks that was that was comprehensive. They're like talking about bank sharding like can you kind of walk us through how. Like this idea that it's okay to assume that the block proposers are the block builders have to track all of the shards because of this separation between. Proposers and builders that we've seen kind of over the past few years, especially with the rise of mv so yeah just talk us like why is it possible to do something like bank sharding and not sacrifice the decentralization properties of the chain. But yeah so I guess I mean maybe like if we go a bit into the history of mv like or maybe think about how it has been recently so like it started with maybe like maybe some mining pools doing some stuff like to exploit mv and like to like.
Yeah, to to make to get some more than just the transaction fees and over over the course of time this has become more and more professionalized like nowadays most of them work with some other entity, for example flash stock flash spots that. sells them complete bundles of or yeah maybe selling is wrong word here that buys complete slots for bundles to be included in blocks that exploit a certain amount of mv and then and then the block producers which are currently the miners they just get the payment for that they get like. So yeah so there's those searches are now like lots of independent entities that try to find the best mv and and mining pools now don't have to bother with us anymore and and basically it turns out that if you want to properly decentralize those which is something we really want to do like so right now there's like maybe a few mining pools like.
tens of them or so and so like the way it works right now is that basically flash spots has a business relationship with each of them like they they basically have a trusted relationship and if like one or one of the two sides did something naughty like for example the. The miner could start looking into the strategies and instead of just executing them they could like exploit the strategies themselves so for example if you ever an arbitrage transaction then like you can often do much worse things with the individual trades if you if you want to exploit them or like they could try to figure out what the strategies are by by what they're being sent and so on.
So this this doesn't scale to a system where instead of like a few tens of mining pools we have probably tens of thousands or so of individual validators because you can't have like a trust relationship with each of them that's not going to work. So the only way to translate this world of MEV into into the future of proof stake is is by having some form of proposable that separation the way proposable that separation works is that instead of having traditionally like I guess like five years ago we all thought of it as the same thing if you propose a block you build that block right. And with proposal a builder separation that's not true anymore what we do instead is someone a builder builds a block and the proposal just proposes oh yeah i'm going to propose that block that this guy build.
So we separated the two roles and in that way we can have a professional role of block building building which is this role that will extract MEV or work with all the searches and so on. And we can have the proposal and that is just a normal validator and the good nice thing is proposing is extremely simple and cheap because it's basically just selecting the highest bid and saying yep you get to build the block. And whereas building is complex process where you have to manage lots of searches and they have to trust your system that their strategies won't get exploited and so on so that that is more suited for complex and more capitalized entity and so.
So it's it's good that's not not everyone has to do it and basically this is this is the I guess this is how we're seeing the future of block building on ethereum there's not there's not really currently a viable alternative to this world. And and what we also know is that sort of building these building scalable system, especially also building this massive data available to system becomes a lot easier once you assume that there is someone who can handle these massive amounts of data. Once you put that entity into the system and say well there's someone who can compute this encoding who can distribute all this data and so on then many things become a lot simpler and so in the past I guess we didn't really think about these designs because we were like well we want ethereum to be extremely decentralized.
And now with the with this proposed a building set builder separation coming into the design space due to me, it's also become available to think about for other things and basically this is how we. Or like I first thought at the end of last year while let's use this let's use this idea where you have these entities that can handle for example large amounts of data is not really a problem if you are. Are like running some large machines anyway it's not like an absolutely insane amounts or data sent center kind of amounts just like large machines with a good internet connection kind of amount and.
And yeah exploit these entities and and let them basically do this building and that allows us to get to a much simpler and more efficient charting design. Got it and so am I right in thinking like because it's basically very hard to build an optimal block that becomes a specialized industry but once you do have a block that is it's very easy to verify its validity right like so right. But you know finding the right block the build is it's hard and so you need you know tons of machines to do that but once you have found it then anyone can verify it's almost analogous to proof of work in that way.
Yeah yeah that's absolutely correct like basically very fine I mean that's that's the that's the crux of data availability checks the this idea that. There is a way to check that this amount of data is available that needs much less work than actually downloading all the data so there's this asymmetry where like there's someone somewhere we need to do this work of encoding the data and distributing the data but then verifying it as much easier got it. Okay and to get us all the way to proto dank charting now and so recently the three of you have have written an EIP IP4844 that's now colloquially known as proto dank charting which helps kind of lay the foundation for this full charting design will out for cry without requiring this entire kind of.
Shard data network to to be live and proto do you want to walk us through like what are the things that 4844 does to get it stored shardings and also like how does that help player to like how do altos then use 4844 to provide their users with lower transaction fees. Right so we just end up with how dank sharding basically introduce a data availability sampling and his other new advanced tech features to try and distribute this job across the network better whether this comes with additional complexity will work take more time to properly test and introduce it to be very. So instead of waiting for the full version of dank sharding we can reduce the features at and go with a model data in between these things we can offer additional data and we can look at layer two's like what kind of security properties they need and optimize for that and he can already make a big winter and then later with the additional features we can go to fill them.
And so what we start with here is the changes to pay for the data as a layer to this type of transaction that introduces the data to the network and we need some changes to distribute the data across the network but it won't be as much data just yet so it's manageable by the own network to download and so we don't need sampling yet and we can have everyone download the data. Got it and how yeah, how does a layer to then use that data like from say optimism's point of view, how do you actually leverage that is it just changing where you post the data that's currently posted in normal transactions that's called data is they're more involved.
You need to optimize for the layer to really needs you can take a part of all the things that the layer to use us one of those things is publishing the data making sure that is almost minority that protects the layer to is able to get the data in the first place. And then there is this other property that the layer to uses right now of getting the data long term, but these are very different data availability is this property that ensures actors are able to get the data and this can be for a more limited amount of time. And so you do need to ensure that even with downtime and even with censorship and whatever other.
Inferencing circumstances is actors on a network on a layer to are able to get the data. But then after some amount of time this should be sufficient to guarantee the security of the layer to because you want the actors to be able to reconstruct the state because on you have a full state on you have the full history they're able to challenge the operator, challenge the sequencer of the road up. Got it. I think we've mentioned like a few times already and I want to make sure we kind of clarify for folks is this idea of data availability. And I think this is something that at least to me was not like clear until I spent way more time looking at the sharding is like what when we say data availability.
What exactly do we mean and how is that different than like the data we store on like the Ethereum blockchain today. I don't know if it's like they want to give a quick overview. Sure. I think it's definitely a very important and the subtle topic like I think even the really big point of comparison a lot of people have is like what's the difference between what we're doing and IPFS right. And this like IPFS is a platform where if you publish data then that like presumably you know if the incentives are right or different people care about the data. The data gets broadcasted and then anyone who wants to download the data is able to download the data.
The difference between that and what Ethereum is doing and going to be doing is that Ethereum is and will be providing consensus on data availability basically so you just you can always have a hard consensus on. The question of whether or not a piece of data with a particular hash or a particular commitment actually is available and what we mean by available basically is what did the data goes through this publication process where it got broadcast on a public network and anyone who wanted to actually download the data actually did have a lot of time during which they could have downloaded the data.
So basically the difference between that and something like IPFS has to do with the case of a malicious publisher right. If I'm a malicious publisher then like on IPFS I could potentially do something where you know I get control some small number of servers and then I publish to the small number of servers and then a small number of servers might respond to and say data is available to some people but they might not respond to other people. So some people get the data other people don't get the data but you never actually like get this kind of very binary consensus on whether or not the data was actually published.
Now the reason why this kind of concept of consensus on data availability is necessary has to do with a lot of these layer two protocols where those layer two protocols depend on data being out there and use like not downloaded by everyone by default but downloadable by anyone in the case that they want to download it for some security properties right. One very simple example is a ZK roll up right and a ZK roll up you have a sequencer that sequencer accepts transactions that sequencer publishes these blocks that contain state delta as they can get a proof and that's a sequencer also contain like basically manages this kind of internal state like what is the state of the ZK like you know the balances contracts whatever inside of that ZK roll up.
Now the difference between a ZK roll up and the validity right is that a ZK roll up has like state delta as or inputs on chain in a validity and we only have the proofs on chain and you have everything else off chain. From a point of security point of view of like can they force invalid stuff to go into the system both roll ups and validity is both protect against that right because of the ZK stock prevents you from actually bringing anything in valid. The place where they're different is what happens if the sequencer disappears right what happens if the sequencer becomes malicious and they just basically shut off from the network never talk to anyone again and the reason why they do this is basically that because they want to.
To make just make the make it not possible for someone else to interact with that system going forward and so if people have money inside the system that money gets stuck now in a validity i'm. This is actually a problem if the quality of operator does this then. They can't steal but they can make people's money stuck and so you know if they're really mean they could potentially like extort a major. Basically say like hey you know if the whales don't like stands at 20% of their money to a ransom address then they're just going to make everyone's money stuck forever in a ZK roll up on the other hand there's this guarantee that because.
The either the inputs or the state delta is get published to the chain if the original sequencer disappears you can always have a new sequencer come in read the data from the chain and basically initialize the exact same state that the original sequencer has and so be in the exact same position and have the exact same capability. To then be able to continue providing ZK roll up blocks processing was draws and processing transactions right so because that data is on chain and so someone else can come in and be constructive and like. Basically swap themselves into the same role you don't have this same security problem that validity is half right so the difference between the two basically is data like in this like on chain or is it off chain now why does it matter if it's on chain because on chain is a very simple convenient medium where.
If you see the data is on chain, even if you don't personally process it, even if you personally don't care about it, you still know that. If something terrible happened and you needed to recover, then you will be able to like actually go on chain and grab that data right so what proto dink shorting and like and that eventually full dink shorting try to do. Is they basically try to like really zero in on providing your platform that provides. Exactly that capability right so the beacon chain would actually only contain hashes of data and so if you're a client, then you would base you be just downloading the beacon chain and you would get hashes of everything but what is a hashes here I mean like hashes of like easy to commitments and the.
You know Bob like complicated math but like think of them as hashes like actually yeah it is a key to G is a cryptographic revalent hash once you buy by yeah you know definitions of collision resistance and prematureness and so forth but. Basically yeah the actual full data would instead of live in this like shorted system where or the inside of shards that we inside appeared for some networks and the point of all of this machinery around data availability sampling and that sort of stuff is to basically provide a wave. Kind of checking and guaranteeing that the data actually has been published through this mechanism where if in the future you need it you will be able to get it without actually requiring everyone so I directly download all all of the data themselves right now.
The chain does not have to store the data that or the shards do not have to store that data forever right so like the plan is for them to delete that data after some period of time like it's a you know numbers have been thrown around of like somewhere like could be 30 days could be a couple of months and then. Basically yeah but but the point is to give enough time that any mechanism or like anyone that wants to be able to download the data will be able to download the data and like for there to be enough time that for all of the people that would be making backups for what we're related to a particular application to actually have the time to do that right so.
Basically you can create the system that's like really optimized around this idea of like how can we just provide this exact guarantee of like data availability like proof that the underlying data behind a particular hash has actually been published to this public notice board where if people want that they can get it. So that roll up similar to it can take advantage of that for scalability without incurring the complexity costs of like actually trying to short and like full on EVF execution or whatever. Right right and so it's like the guarantee of the Ethereum L1 protocol is quite tight it's like we guarantee that this data will have been published on the network for this amount of time beyond that obviously there's still ways to retrieve that data they're just not guaranteed by the Ethereum protocol right then they could very well be on IPFS but that's not a guarantee that the protocol can make.
Right exactly yeah and and we've we've touched kind of on this a couple of times already like this idea of data availability sampling and only verifying like having each their each valid or verify parts of the data to make sure that like overall entirety has been published. Denkrat do you want to give us like an overview of how does data availability availability sampling work for calling an intermediate audience you know not a photographer yeah yeah yeah so. So the idea between the behind the data availability sampling is that you somehow want a scalable way to to ensure that some amount of data is available like is basically and available means you could download it if you wanted to right and so the obvious obviously we know if we don't know that which is what we do now it's available that simple right because if you could download it then you could download it.
Okay but but how do we make it scalable so scalable means we have the same amount a constant amount or maybe. Increasing a logarithmic amount but not like a linear amount of resources that we need to do this amount of work so we need to find some way of doing this in a more efficient way and the way data availability sampling does this is what what nodes do is that they sample they pick these random parts of the data and they say I want this and this and this. And and I'll try to get it and only if I can get all of these or maybe a vast majority of them then I will consider the data to be available and naively like if you just do this on blocks as they are now then it doesn't block and why because.
If there's say just one piece of the block missing the probability that you catch it is tiny because you would have to request exactly that piece of the block and you want to only request a really small part of it so the probability that you catch it is more. So that doesn't work because we know like in blockchains basically the thing with blockchains is like the bad things could happen anywhere like even one single missing transaction could screw up the whole system so you cannot allow any part of the data to be missing like just sampling directly doesn't work so what you have to do instead is you have to first encode the data and you encode it in such a way and that's just this is called.
Read Solomon codes you encode it in such a way that that any any fixed fraction of for example you can pick 50% if any 50% of the data are available then then you can reconstruct the whole from that so you encode it in that way and now. Now the scaling becomes different because now you don't have to ensure that all the data is available you don't have to know that all the samples are available but you have to know that 50% are available and that's a task that you can do statistically because if you download 30 samples or well I mean the correct way is like saying like this if someone is trying to hide this is the attack we're trying to defend against right someone is trying to hate hide the data somehow.
If they do that they have to make less than 50% of the samples available if they make less than 50% of the samples available and you don't know it for example 30 then the probability that. That you all of those will be available is true to the minus 30 which is one ability in a billion and so it's really small and by downloading 10 more you decrease it by another factor of 10 so that that is a scalable way of ensuring that availability and that's the principle of hard works. Right right and basically building this entire this entire system is why shipping bank sharding is going to take a while.
Proto can you walk us through like in the meantime in like the 48 44 world how do we like sidestep that why why can we get away with not not having all this already live. So for background here first of the merge we separate Iferia into a constant slayer and execution there and we are not throwing more data at the execution there but rather we continue to scale the consensus there and even then we are only doing so with a limited amount of data. So we're talking about a month or maybe three months some amount of data that is retained after the period we start to prune the data so we can ensure that it is available for their tools for system for for a sufficient amount of time for them to secure their network.
But then at the same time it doesn't grow infinitely like it doesn't grow independently where now you have a bounded amount of data to store consensus not we can distribute this between the different speaker notes. Right and and I guess just to clarify this for the for the listeners the amount of data that we make available in proto dank sharding is less than the amount that we make available in full dank sharding correct. Right so we fill dank sharding the disability job of storing and propagating the data between all the notes on the network between the different fellow data. Whereas if you for it for for we still require of the consensus notes to acquire of the blood data that we limit this we don't make it grow independently so we can increase the throughput.
Got it and can one of you give me an estimate like you know how much do we lower the cost of storing data for their tools with for it for foreign and how much do we lower it further with a full sharding deployment. Roughly sure so currently very blocks are anywhere between like 50 maybe tops like a hundred kilobytes it's very variable like worst guys it could grow a lot larger but you are paying for call data data that is going through the if you am that's available forever. This is very different type of data that a roll up really needs so instead you can try and optimize you can have this different type of data called block compared to call data and we can grow it from this order of magnitude from like 50 kilobytes to maybe like a megabyte per block.
And this is obviously this is already a huge increase that roll ups could benefit from reduce cost of it and then the full dang sharding they can go another order of magnitude larger because now we don't have to store off data on one note that we can distribute it across 64 notes we could have a multiplier here and like how we do the data apart. Got it so it's like we get an order of magnitude increase with just four for four terms of how much data we can have and then we get another order of magnitude would full dig sharding. And then one thing that's that's been interesting to me to the learner as I've been spending time on this is the idea that like the demand for storing data in these blobs or in the full sharding system is like independent or at least decordated from the demand to use a theorem gas right like there might be people who are willing to pay a lot to execute computation and people who are willing to pay a lot to store data but they don't necessarily.
Overlapped and it's like it creates two different markets and so can one of you kind of walk us through like how we're like designing these two different markets and and isolating them from each other to an extent. Well, so it starts with the transaction type very at this additional fee parameter that with this fee you create a different market and so if you really want to you could separate the transaction pool and the capacity and the type of resources is very different. I think Vitalik already wrote a post about a multi-dimensional EIP 1559 where we can try and think of all the different resources in Ethereum as different markets and I believe don't cut already has a post on how EIP 1559 could work for this type of bloop data instead of regular gas.
Right so it's like there's two auctions happening in parallel one is people bidding for like transaction computation and the other one is people bidding for storage and we can use kind of the same mechanism which we have already for gas and call data which is like weirdly bundled to then to then separate it and have one 1559 that works for gas and one 1559 that works for shard data. Another thing we haven't touched on this a lot but Vitalik I think you mentioned them earlier on but that the starting design requires a production of case in g commitments and they're kind of like a hash but not really I know dankrod you have like a great post about them.
Do you want to explain again and sort of to non cryptographers what these are and you know how they resemble the cryptography that's currently in Ethereum and differ from it. Yeah I mean so I mentioned earlier at the end of explaining sampling how the data has to be encoded in this way that we call a read Solomon code which is a way to ensure that any 50% of the data can be used to reconstruct the whole data. So I mean I guess data is a bit misleading here because there's like the original data which is the actual payload that we're talking about but then the code expands this data so it becomes twice as large in the process as well.
And so and so what is a read Solomon code so a read Solomon code is. Well so yeah we call it it's a polynomial so basically what what what it means is you you've learned about polynomials maybe in mathematics classes it's it's a certain type of function and and basically. This type of function has the property that that when you when you know it at a certain number of points which we call the degree of the polynomial then then you know the whole polynomial so basically we use that property. In order to put this polynomial function through the data and then if you if you have like a certain number of samples which is like half the amount of the full encoding then you can get all of them.
And the reason we need KZG commitments is this like when you just sample the data there's one thing that you can't decide from those samples and that is. Whether they encoding us correct what if someone just like read Solomon calls have a certain structure right they have a certain structure that allows us to reconstruct the whole thing but what if someone encoded it in a different way what if they just put garbage in it then every 50% of the samples would give you different data and of course that's not acceptable because the data has to be unique it has to be that all 50% of the samples give you exactly the same data.
And and the way we we do that I mean that different approaches but basically over the years we have ended up here where we just found this amazing type of commitment the carty or Katie's KZG commitment that you can basically see as a hash it's similar to a hash of. Data but with a property that instead of hatching to just data it hashes a polynomial so it's a way to hash a polynomial function and and basically reveal any point on it and so that guarantees that this this correctness of the encoding and that's why we need these KZG commitments. Got it and as I was reading about KZG commitments one of the first things you stumble on as a non cryptographer is there require a trusted setup and.
You know I because it's for the viewers to walk through like what what actually what are the trust assumptions in a trusted setup you know like what are the things that like we are trusting in that setup and as we as we make one. To kind of enable this on an Ethereum is there things that like end users can do so that they can have kind of a higher assurance that's the setup was performed correctly and like minimize the trust assumptions they are making individually. Right yes yeah so basically the trusted setup is like basically what we what we have to do is we have to generate these elliptic curve points that have a certain relation that that's like one of the fundamental inputs of the of the KZG commitment scheme and the trusted setup is basically away and okay and in addition to that property that they have a certain relation is nobody is allowed.
To know the actual relation between them so this has to be a secret and that's why it has to be this trusted setup and it's called trusted set up because like one of the ways to do a trusted set up is just to say hey we all trust him Tim you do it and you give us the outputs and then it's done and you throw it away and everything will be fine. But the problems of course that's not really sufficient for the Ethereum community people would be like well what's the what if so instead we have this way of distributing this trust and saying like we let many many people participate in this trust set up and and we can design trusted setups in a way so that if even a single one of these.
People that participated in it and did it according to the protocol and protocol means that you execute this whole thing you run this program send your output and then you destroy your data like you destroy the secret that you've used to do it you don't you don't keep it and if even a single person out of the potentially 1000 set up going to set to participate did this properly then set up is completely safe so even like let's say. 1000 people do this 999 colluded and they all kept their secret and they come together and try to reconstruct it but one person that properly and they don't have it.
Even in this case these 99 99 people know absolutely nothing that helps them to break it so that's the security guarantee which we call like n minus one so even if n minus one collude they can't get anything on that last person that participates properly. And yes I mean obviously this property has the nice property that if you are really really worried about this and I like oh my god how can I trust these people and then you can just participate so like one obvious way is if you participate and you know you did things properly. Then you don't need to trust anyone because you're part of it and and yeah it's it's done so like you you can consider it secure obviously.
There are also all the parts of like making sure that all the software works properly and so on that we're all very familiar with in our blockchain systems like all of this we're relying for the functionality theme as well but so obviously it has all to be very well audited and we need several implementations of this as well. But yeah so that's that's I guess like the other trust assumption that we have to make sure that all the software safe and works properly. Right so there's like kind of different levels it's like either you know you don't care at all and you just trust that somebody somewhere has been honest.
And maybe you just some that you don't care but you learn about Ethereum 10 years from now which is basically what you have to do because you can't participate. If you're if you're part of the Ethereum community today there's going to be an opportunity for people to participate individually. So then as long as you're confident that like your participation was was correct then then the whole output should be correct. And then if you're even kind of more concerned there's a specification from which we can write different implementations. You know if you didn't trust the existing ones you could write your own and also produce an output or at the various kind of review the code of the different ones and make sure that they they match up.
And yeah you would kind of get a high level certainty that things are correct. And I guess that we're coming up on time here. The last thing I didn't want to talk about is like how do we actually get this deployed, which is a part I'm usually get involved in. So you started prototyping 48 44 along with some other folks can you give just kind of a quick summary of like what was done so far in terms of prototyping and what do you think are like the next steps that the community can expect. So earlier this year it started with an initial right of what could be like then during the hackathon in Denver it transformed into this software where we have an actual implementation of the proposal.
And over time we have been improving that and testing that. And what we need to go and do from here is there are two these two different branches right where we want to further develop the client software to be able to make a test network. And we want to continue the development on this trusted setup so that we do have the cryptography everything there that's that all set as well for when we do want to deploy this. And then once we have both ready, you can make larger and larger business and then eventually included as an ERP via the algorithms process into if you're a minute. Got it got it so we have some initial prototypes we want to product like grow them make them more robust make sure the trusted setup is working according to plan and then once we have that it kind of becomes a normal IP we need to shepherd through the process.
And then one thing that that's worth noting about 48 44 is what's very neat about it is from the execution layer point of view so like the kind of smart contract and end user transaction generation point of view. Sharding is basically done then like there's no more changes that will do for people to interact with this blood data what what will happen is then we need to deploy kind of this entire dank sharding infrastructure on the consensus layer. But from applications point of view that that's kind of just happens in the background. But at the consensus layer then crowd like what are what are the steps to get this deployed like how many how many hard forks do we need to get there you talked a lot about like proposal builder separation before so like what do you see like the logical set of stepping stones to get us to the full full sharding on on a consensus layer.
Like so clearly like I mean this is the reason why we chose this stepping stone of proto dank sharding that it's a it's something that gets a substantially closer to the full implementation so it will become simpler it will like the interface for example will stay the same the execution layer changes will be minimal once that's implemented. So that's why it's really nice if we can get that done in the Shanghai hard fork and then I mean I think it's very unlike that will be the next hard fork after that but hopefully relatively soon we will get yeah we'll get this I yeah I don't know it's currently still there are so like some things we definitely need to work on like there's there's a lot of work on the network working to be done.
To have yeah full sharding rolled out I I currently have no exact estimate but I would hope that there can be done in one hard fork. Okay I sympathize with not be able to give direct estimates about complex projects so I won't push you on there and maybe yeah to kind of close this off like the topic like if people want to contribute from like the from a researcher like engineering. Point of you like what are like the big open questions in sharding land that they should spend their brain cycles on. I think one. Problem is definitely like during out the networking of data availability sampling make there are designs that we have that work in theory right like there's doing it based on sub nets there is the yeah approach of trying to make a DHT much faster there's a couple of other techniques in the middle but like really taking those ideas and from an engineering perspective like just trying to optimize it really hard like how do you.
Actually make a yeah basically it's like a specialized is scalable DHT or we are publishing and downloading can happen extremely quickly. So that's one problem and I think in general like in the ethereum ecosystem and the working side is one of the sides that's talked about the least I mean possibly just I think a bit of an accident history that the ethereum core research community just happens to like never really have people who. Like spent a lot of time thinking about networking stuff it's generally like you always spend a lot of time thinking about I'm going to cryptography and get sent us economics but it's still a really important problem and it's a problem where I think it will be amazing if we can have more very active networking.
So that's a short term or that's like a very clear problem another problem is that with folding shorting there's this issue of like how to combine it well with proposal builder separation and there there's some like economic challenges there's still the challenge of look well how do you actually make a good proposal builder separation protocol how do you add like censorship resistance lists so you can bypass a censoring builders. And once again there like we have ideas on each of those things but there's still the question of digging into the details combining a PBS design with the design of ethereum's future proof of stake which is something that at some point we'll probably start.
Having to kind of talk and think more about right like there's been this increasing effort within the ethereum research and the protocol community of basically thinking through what would a better proof of stake design look like in the long term like you know do you want to have single slot finality how do we achieve single slot finality what other benefits can we achieve. How can we offer more in protocol of what wider offers to people extra protocol to try to reduce some intersticking pull centralization incentives and just generally increase its simplicity right so i've written a bunch on that like if you just.
Google for single slot finality can probably find it but the do it that and the intersection of that and propose builder separation the intersection of that and. Sharding is you know going to be another research area going forward and then the other one is also adjacent to dig charting stuff but and also adjacent to 4444 stuff which is very critical to. prototype charting and dig charting actually being viable and just generally ethereum scaling well is like. Creating the like as decentralized as possible and as robust as possible systems to give people the same guarantee is that they come to expect out of ethereum in terms of history retention but without requiring participants in the core ethereum consensus protocol to make all be retaining blocks forever right so.
There's a team working on portal there's things like the graph like there's this big a long list of projects right and trying to figure out you know how to make those better or even create better alternatives to those I think also an important area. Oh also one last one I really wanted to throw in this is important the switch the layer to is I think one where it's very important for the ecosystem to try to maintain and even improve its decentralization going through that switch and so you know we need light clients that can poke into optimism and poke into arbitraven poke into us dark that. And that's something that there's been a bit of like theoretical thinking work on but it's the sort of thing that I think there is a lot of room for people to swap themselves in and like really try to improve that ecosystem a lot right.
Like basically trying to like really think through if everyone's really going to migrate over to over the next that's well to 24 months especially as protein shorting comes alive and the rule of costs go down even more like how do we really make sure that transition goes well and that it preserves all of the decentralization properties and even improves on those properties that we've come to expect of ethereum. Yeah that's a really good list. We have like five minutes left so I wanted to leave a bit of time for the three of you if there's anything you wanted to share that you think is important about char day in your four and four four or ethereum generally that like we haven't talked about the four is yours.
Whatever you want to rant about or get people to pay attention to. And. When the last week has been kind of chaotic to say the least I think this is like the time of market where if you feel a little bit down like maybe read a post try and getting both with news on projects. It's not a beer market it's a builder market read the specs for the ep there's the site called ep for it for for calm the opportunity started and then there are simple diagrams and all the way down to like very elaborate post about the cryptography info and yeah just reach out and get building stay optimistic. I love it. They cried Vitalik anything else you want to share.
Stay optimistic but I'm in the long term hopefully stays your knowledge. Wow. Anything you heard that crowd. Yeah I mean like I mean I think I think the week has shown that. That you need solid designs and that we need to like build things that can actually last and that are yeah build build to last and. Won't go away and that that's what we're trying to do here and yeah so I'm optimistic on this but yeah I am clearly also pessimistic about many other things that are happening in the ecosystem so we just have to be better and build better. Love it. Yeah then that's basically a wrap the bank list guys did ask me to end with a disclaimer so here it goes and I'm reading for the screen now risk and disclaimers crypto is risky you could lose what you pull in but we're headed west.
This is the frontier it's not for everyone but we're glad you're with us on the bank list journey. Thanks a lot and yeah thanks a ton Proto Vitalik Bankrad for coming at a bunch of different hours across your respective time zones. I think this has been really helpful to explain the entire shari robot to people. Hey we hope you enjoyed the video if you did head over to bankless HQ right now to develop your crypto investing skills and learn how to free yourself from banks and gain your financial independence. We recommend joining our daily newsletter podcast and community as a bankless premium subscriber to get the most out of your bankless experience.
You'll get access to our market analysis our alpha leaks and exclusive content and even the bankless token for airdrops raffles and unlocks. If you're interested in crypto the bankless community is where you want to be. Click the link in the description to become a bankless premium subscriber today. Also don't forget to subscribe to the channel for in-depth interviews with industry leaders ask me any things and weekly roll ups where we summarize the week in crypto and other fantastic content. Thanks everyone for watching and being on the journey as we build out the bankless nation.