AI is getting dangerously good at smart contract security. Faster than crypto is ready for. Alpin Yukseloglu joins Bankless to break down EVMBench (built with OpenAI), a benchmark testing whether AI agents can detect, patch, and exploit real fund-draining bugs and why the jump from ~12–13% exploit-finding to 70%+ could rewrite today’s security assumptions. We unpack what that “70%” really means, why crypto’s verifiability is an ideal training ground, why AI labs haven’t prioritized crypto data yet, and what a 24/7 blackhat vs whitehat AI arms race means for DeFi.
---
📣SPOTIFY PREMIUM RSS FEED | USE CODE: SPOTIFY24
https://bankless.cc/spotify-premium
---
BANKLESS SPONSOR TOOLS:
🔮POLYMARKET | #1 PREDICTION MARKET
https://bankless.cc/polymarket-podcast
🪐GALAXY | INSTITUTIONAL DIGITAL FINANCE
https://bankless.cc/galaxy-podcast
⚡ EUPHORIA | REAL-TIME ONE-TAP TRADING
https://bankless.cc/euphoria
🌐BRIX | EMERGING MARKET YIELD
https://bankless.cc/brix
🏅BITGET TRADFI | TRADE GOLD WITH USDT
https://bankless.cc/bitget
🎯THE DEFI REPORT | ONCHAIN INSIGHTS
https://thedefireport.io/bankless
---
TIMESTAMPS
0:00 AI’s exploit leap: 12% → 70% and the “Superhuman auditors”
7:02: Staring at the singularity without losing your mind
10:31 Agency » doom: the Thiel framing
19:10 What’s most at risk (and what’s safer)
23:37 What EVMBench actually is (benchmark + harness)
27:03 Why exploiting is the key: killing false positives
29:24 AI gets “good at crypto” fast: verifiability
30:56 What “70% exploit rate” really means
33:32 Why AI labs avoided crypto (it’s not technical)
43:38 Blackhat vs whitehat: how the race plays out
47:21 Agents and “payments at the speed of light”
51:02 EVM vs Solana: network effects
56:18 AI formal verification as an endgame
58:06 EVMBench V2: expanding the frontier
59:54 Why Alpin stays in crypto
---
RESOURCES
Alpin Yukseloglu
https://x.com/0xalpo
EVMBench
https://paradigm.xyz/evmbench
---
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures
No summary for this episode yet.
Thank you. We are here with Alpen Yveselolu. He is an investment and a research partner at Paralan. Also the co-author of the paper titled EVM Bench, an open benchmark for smart contract security agents written in collaboration with OpenAI to measure the ability of AI agents to just detect or patch or exploit smart contract vulnerabilities. We're going to talk about the way that AI and AI capabilities are going to impact our crypto ecosystem or smart contracts. Alpen, welcome to Banchless. Hi. Thanks for having me. I want to start off the question with a very big question. How at risk are we from AI?
How large of a threat does AI smart contract capabilities pose to our industry? Yeah. I mean, in the long term, it's now increasingly clear that AI is going to be extremely, extremely good for crypto because, especially on the security front, because we're going to get to a world where because everything is much more secure, the ceiling on the industry is much higher. So our partner Matt talks about how if you have a grocery store that's run by a mom and pop because they can't see everything in the store, there's a limit to how big they can get. But the moment you add security cameras in, so security has this effect of increasing the capacity, the carrying capacity of an industry.
I think in the short term, it's up to us because the models are getting extremely good, makingly good. When we started working on EVM bench, which is a benchmark that consists entirely of fund draining critical bugs around six months ago, the models were able to find less than 20% of the bugs, like around 12 to 13%. And just over the course of while we were working on the benchmark, this number went up to over 50%. And in between when I drafted the launch tweet and when I had to actually hit send with the release of Jeep 5.3 Kodaks, it jumped up to over 70%. So these things are just growing at a blistering pace and it's very important that we position the industry in a way that we can defensively protect against a tax.
But in the long term, I think it massively increases the carrying capacity of crypto. Yeah, I think what you're saying is in the long term, we get something approaching perfect security. Yeah. Right now, we do not have perfect security. Let me ask you the same question, but a little bit differently. Say only bad actors, only black hacked actors have access to AI capabilities. In that context, how at risk is our industry, like how exploitable are our smart contracts given the increase in AI capabilities? Yeah, I mean, I think it's really hard to say when we approach super intelligence levels, I do think until we hit the, like right now the models are quite good, but they're not better than the best human auditors.
So we already have existed in crypto under the threat model of extremely intelligent adversarial actors that are constantly trying to break all of our software that with all the money in it. So in that sense, like crypto is already quite hardened, but it's just really hard to know when we talk about sort of technology inflecting into super intelligence. It's very similar to how in coding capabilities were increasing mostly linearly over the last several years. And in December last year, they crossed some threshold where they were better than sort of the median engineer. And it's a lot of stuff clicked for everyone and it started becoming this aha moment and this sort of aircraft moment.
And I think something very similar will probably happen with security where right now it's like increasing pretty rapidly at a, at still at a linear clip, but it's not as good as the best human auditor is yet. So we don't feel it yet. It hasn't actually broken any of our assumptions. But once we hit in maybe six, eight months, I'm pretty confident at this point by the end of the year, a super human AI auditor, this will just completely break all of our assumptions and we'll need to go back and make sure that we're hardening all of the contracts that are housing the what nearly hundred billion dollars of assets in crypto.
Galaxy operates were digital assets and next generation infrastructure come together serving institutions and to end on the market side. Galaxy is a leading institutional platform providing access to spot, derivatives, structured products, defile ending, investment banking and financing. With more than 1600 trading counter parties, Galaxy helps institutions navigate every phase of the market cycle. The platform also supports long term allocators through actively manage strategies and institutional grades, taking and blockchain infrastructure. That scale is real. Galaxy has over $12 billion in assets on the platform and average a $1.
8 billion loan book in late 2025 reflecting deep trust across the ecosystem. From digital assets, Galaxy is also building infrastructure for an AI powered future. Its helios data center campus is purpose built for AI and high performance computing with more than 1.6 gigawatts of approved power capacity, making it one of the largest sites of its kind. From global markets to AI ready data centers, Galaxy is serving the digital asset ecosystem end to end. Explore Galaxy at galaxy.com, 5Bankless or click the link in the show notes. Euphoria brings one tap trading to the palm of your hand. Based on mega-eath, Euphoria takes real-time price charts and projects it over a grid of squares.
You tap the squares that you think the price will enter in just 5 to 30 seconds in the future. If the price goes into that quadrant, you can pocket anywhere between 2 and 100x your trade. No other application helps you trade faster and with more leverage on market driving events, like FOMC meetings, presidential speeches or global macro events. Thanks to mega-eaths, real-time blockchain Euphoria is the way to get real-time price interactions with the market. Euphoria, you'll be able to compete with friends using Euphoria's real-time social trading experience, allowing you to go head-to-head with your friends.
A great party trick if you project the app on a TV. It'll be like the Mario Party of derivatives. To trade on Euphoria, people can deposit stablecoins from any chain or do direct fiat transfers, and everything gets converted into mega-eaths native stablecoin USDM in the background. Check it out at euphoria.finance and download the app or find it in Telegram as a mini-app. In 2024, emerging markets generated over $115 billion in annual yield for investors, with yields ranging between 10 to 40%. These are some of the highest, most persistent yields on earth. The problem? DeFi can't access them.
Bricks changes this. Built on mega-eath, Bricks takes emerging market, money markets, and solve them carry, and turns them into composable primitives you can access straight from your wallet. While DeFi investors earned 3-6% on stablecoins and tea bills, institutions have been harvesting 10 to 50% yields backed by sovereign monetary policy. Bricks connects these worlds with institutional-grade tokenization, local banking rails, compliance across jurisdictions, and real-time stablecoin settlement. Bricks does the heavy lifting, so DeFi can finally access real collateral and structured products on top of real-world yield.
Even the best carry trades can be within reach. Bricks brings DeFi's promise to the emerging world and brings the emerging market yield to your wallet. Let the yield flow with Bricks. Open, if we zoom out here, though, and we think about AI intelligence and its security capabilities and its bug detection capabilities, kind of going exponential, and we think about a super-intelligent AI, I don't even know how to think about security in general, because it can envision scenarios beyond human comprehension. For instance, what if it thinks up a way to crack some of our cryptography with some math that we didn't even know existed?
I heard Justin Drake on a podcast recently talk about us. It's not just the threat of quantum computers, which is kind of a real-known threat, and some of our encryption algorithms are under threat due to quantum computers. If we have a super-intelligent AI, I mean, who knows what it could have the ability to actually hack and decrypt? I guess my question is, when it comes to super-intelligent AI, is security just like not even a thing we can prepare for? How do we even think about it? I mean, I think security, so the way I would think about it is that I think right now this frontier is very illegible, and if you try to do this at the limit thinking, you end up leading to very odd places that may be very psychosis-inducing.
I think one of the skills, but I think the capacity to face the singularity and stay sane is a very important skill to develop. I think the best we can do right now is that we can get ourselves into the frontier, into this sort of experimentally bound future where we're running the experiments ourselves, and be ready when those inflections happen to be able to react. Because I think the model of like, there are only bad people in the world, and they're going to have access to this technology, and they're going to break all of our systems. I think this is what leads to the sort of psychosis around like, are we all just completely screwed?
But that's not what's going to happen, right? We're all going to be in there together, and it's a one-way in this frontier, and as we have access to these frontier models, the sort of agent harnesses around them that are able to run these exploits that are able to, for example, find undiscovered math or that are able to break existing photography, there will be both sides of it. But right now, it's not clear whether this is going to be an offence or defense-favoring technology. I will say that there are still fundamental constraints in the world, like you can't break laws of physics, there are systems that are chaotic, like, for example, the three-body problem where the fact that even if you have superintelligence, you can't predict too many more ticks ahead because it's just a fundamentally chaotic system.
So I do think that the world is a complicated place, and there are still physical laws and constraints that will catch these things, and practically the best we can do is we have to push that frontier together instead of letting it just sort of happen to us. I think that's a fair point. There are physical constraints here, and superintelligent AI does not just mean like, it could appear as a god to humans, but it does not give it god-like capabilities to break physical laws of the universe, certainly. Yeah. Actually, I want to dig into this because I got the sense that maybe you have some insight here because I've been doing a lot of staring into the abyss of the singularity and trying to stay sane, and I don't quite have a knack for it, but sometimes I feel like I'm staring, and I am feeling myself go a little insane.
I just don't feel settled about it. Is there some wisdom you can share, just like a pattern or something that I was starting to get out of what you just said, well, maybe the key is to take it a step at a time and don't think about the far future and the limit, just maybe think about the next day, the next month, the next year. How do you stay sane when you stare at the singularity? I mean, I think the core point is agency, right? So Peter Tiel has this framing where acceptance and denial, most people relate to them as opposites, but in many ways they're the same thing because both of them imply that you're sort of everything is out of your control.
If you're fully accepting that something's going to happen, then you're not doing anything about it, and if you're fully denying that something's going to happen, you're also not doing anything about it. That sense, I think, both the doomers and the accelerationists are both wrong. The real answer is that we have agency over these outcomes and that you yourself can bend the arc of the future, and I think there's a lot of comfort and not a lot of stability in that because if you believe that you have agency over the outcome, then somehow you're still in control. I will say that the current frontier, and I guess you can argue the frontier has always been experimentally bound, which means we don't know.
We can't sit down and theorize about what is going to happen. Generally, the way that we're going to figure out how things are going to happen is by experimenting and then by seeing the results and these AI models and also the current frontier of technology is grown. It's not manufactured. It's much more organically evolved in a way that we can't today really predict. Even just the armchair trying to theorize about what's going to happen, it'll drive you insane because you can't know. It's not bound by your ability to algorithmically figure out the future. You have to get in the trenches and try things.
There's some stoic acceptance, the fact that you can't know everything and so you just have to let that go. Then there's also a belief in agency and I guess an applied belief. How much of agency is just a blind faith versus maybe a practical faith of doing? For someone listening in general that feels themselves looking at the singularity and it feels frozen, I guess, by just not knowing what to do and doesn't feel like they have a lot of control in agency, is that something that they can develop or is your advice, you just have to have faith, you just have to believe that you have more agency and that will become self-fulfilling and you will have more agency.
Faith is good but it's not a particularly agency inducing headspace to be in. When we for example started having the thought that agents could get extremely good at exploiting smart contracts which we're obviously heavily exposed to about eight months ago, we could have just sat down and been like holy crap, are we screwed or are we not screwed, but it's like this. It turned out there's another path which is well you can go figure out to what extent these things are at risk and then also start making headway into the labs that are actually pushing this frontier and maybe start getting crypto integrated into them so that we can get into a position where as the fog of war clears and as we start to see, for example, now that there are defensive measures that we can take that we're in a position to actually exercise those paths and I think there's something about the humorism and also the general in the limit thinking about the singularity and about superintelligence that is just like captures people because it's like you can sit down you can think about it for a long time and it'll make you feel very strong emotions but at the end of the day that's not going to be the thing that is actually like you can go build things you can go work with the people who are pushing the frontier you can get in the trenches and you can start contributing and the information you gain from that is going to be much more grounded than whatever one can come up with in their head and this is I mean I think many of us in crypto are used to this because a lot of crypto was like this for most of its history it was extremely illegible it was very hard to pin down exactly what the use case was going to be and now we're starting to see okay we have this store of value use case we have stable coins that are compounding at this monstrous right year over year the industry kind of gave birth to the whole market of prediction markets which which is sort of adjacent to crypto now but it's just compounding at this insane rate and you know five years ago it would have been very hard to say this is exactly what's going to happen and it took some level of some combination of faith and some some combination of agency to to actually go and build those things and I think so culturally this has been a very big anchor point for paradigm because our entire firm is built around around building and researching alongside the investment and and so if you talk to anyone on the paradigm team or in our orbit like the sense of groundedness you get is anchored in the fact that we're actually we have contact consistent contact with the frontier and and and there's sort of there's comfort in that there's stability in that and there's agency that comes from that I think it is worth uh understanding reflecting on like the only reason why the singularity staring into the void is intimidating is because what it's what all of these technologies are doing are providing everyone else with agency should produce the singularity in the first place and so if that singularity is intimidating to you you know grab a mop you know get do do something like there's work to be done and I think you know the best path forward is through there like if if you are intimidated by everyone else having agency because of these tools you can have agency yourself well it's a it's a polarizing technology right it's it's like the the threshold of agency you need to be able to execute on something is going down so so like if you're kind of on the fence you get snapped to zero or to you can do a lot of fun to order while yeah and I think in that sense like if one has the intuitive sense about themselves that they might get they might be on the side of that fence where they get squeezed to zero that can be extremely fear inducing and actually the solution to that is to is to be much higher openness right adopt the technology much faster be much more fluid about about changing and adapting to to the environment there's um I think uh not our team had mentioned at some point about how their times when speed is more important than cohesion and I think the current environment were in because the frontier is so unknown and there's and so unknowable to some extent there's there's moving fast and adapting fast has a premium over being able to sit down and figure out exactly what's going to happen and put the pin in the right place and this is somewhat paradoxical because the more agency you have the more it may seem like it matters that you do the game selection right and you sort of pick the right game to play et cetera but practically and empirically what's happening is that actually it's better to move fast and ship the thing within 24 hours of inception then it's a sit for two weeks and try to figure out the exact right way to construct it and then try to ship it then and and I think that probably goes all the way down to like many parts of one's life where where we're in an era of speed over cohesion yeah yeah we are in the uh just do things era I wasn't out been expecting this to be such a philosophical episode to open this episode up but now I think we can kind of like uh corral ourselves and point our agency towards the topic at hand which is yeah what happens when people feel high agency with AI towards the security of our smart contracts um is it worth talking about like what kind of contracts are like most at risk or least at risk is there some sort of like category or knowledge landscape that we can understand that when AI's have to have very high smart contract agency that we should be paying attention to certain kinds of smart contracts over others is there a conversation there I think um I think um not in terms of market I think that's really hard to say like you know a dex contract we're still lending market but I think you know simple contracts that have been around for a long time I think are probably better in a better position than for example like the 200th contract deployed on Binance smart chain or something like that where you know there's in the in the past there's been a sheltering that's happened from uh being in a small market so if you if you were deploying something where the most amount of money that one could make if they fully exploited it was in the order of low thousands of dollars then you were sheltered by the fact that they're just much bigger fish and and the bad actors and even the good actors or the people who are trading etc like are just are just not you're not in there over in the window but as the models get better because the cost of inference is so much lower than the cost of an extremely talented security researcher that long tail might get shaken out very quickly so so I think most at risk is probably small cap or some you know low TVL protocols in long tail chains that are that are still built on a well understood stocks like the EVM and solidity and then and then I think there's there's just sort of an unknowable security risk for the for the major contracts the OG DeFi contracts that are currently battle tested but are still very complicated and and you know we'll see over it over the coming a year or two to what extent those those contracts are actually exposed right so the OG contracts that have had a ton of Lindy and a ton of value locked over time that have been tested by the market are like safer in the near term but nonetheless the people managing those contracts will still need to have agency to be on the defensive to make sure that they are winning the arms race against the offensive types well well the the price is much larger for exploiting those contracts so I think it has this like you know there will probably be this canary in the coal mine effect where there will there will be smaller or or protocols that are less secure but have a lot of assets in them fall first and and I think they're you know you know we'll have to look out for the first exploit that happens that is almost entirely from AI and then from there it'll be the race will be on to start start taking the defensive measures necessary right and then like the long tail as you said the long tail contracts testing in prod will no longer be a thing because when the cost to exploit a $1,000 contract is like you know 10 to $50 of tokens then those contracts are simply won't exist somebody will write a bot that says hey cloud open claw go hack me some some contracts and then that thing will actually have the capacity to do that because the people that you know didn't really think too hard about their security because they weren't didn't need to think too hard about the security those people will have will not have a good day yeah I think this is a general trend that that the long tail will get collected by who can use AI well like for example you can look at something like prediction markets where there are markets where if you trade them to perfection the most amount of money you can make is maybe 50 to $100 and like it's not worth it for Jane's treat to put a quant on those markets because it's too expensive for them and it's bound by the cost of intelligence and the cost of attention but if you can trade those markets near perfectly for ten cents of inference then you'll do it and in aggregate maybe that long tail is pretty valuable so so right now we're in a world where the long tail is sheltered by the fact that it's small and as AI gets better in all of these different domains we should just be assuming that all of that's going to get collected by people who are able to use these tools let's talk about EVM bench this is the the paper the tool you guys call it a tool is that right yeah it's it's a benchmark and also an agent harness so we maybe had two releases in conjunction the first evaluates the ability of an agent to to exploit smart contracts and then the second one is is sort of an agent harness that is like you know similar to an auditing agent so it can actually find the bugs and obviously the agent harness that we released is sort of not at the frontier of capabilities because we don't want it to be used for for black hats but we have a we have a UI that that you can upload any smart contract into that will do sort of a a baseline check for bugs can you define harness that I appreciate that's a technical term yeah I think coders will be aware of but I'm not yeah so the the core idea is the model labs will release these LLMs right so you'll have GPT 5.
3 etc and you know you can do the baseline test of like just at just prompt the models as chat GPT hay is their bug in this contract and then you in addition that like you can like let's say that gets you to to x percent on the benchmark you can add a bunch of scaffolding around the model that says hey like for example here is an EVM that you can test against you can deploy a contract and actually run an exploit and see if you're able to drain the money and it turns out that if you give agents these tools this are scaffolding that that they can sit in they perform much better so the harness is like similar to basically it holds the agent the the model and it gives it superpowers that are specialized to the task now the the interesting thing on the current arc of AI is that most of these tools that we add in fall like flake off a time because as the model gets better it just absorbs the harness the the core example being how at the beginning of tesla's fully self-driving development card potty talks about the majority of the code was hard coded and and handwritten and very quickly started ramping up to now I think over 50% of it is actually just the model there's no like they had they removed all the C++ code that that was like saying if if x then y and the model just figures out a way to do it so right now we're you know the agent harness quote-unquote is in the if x then y like hard coding hard coded tools that we're adding in to give it these capabilities but probably in the fullness of time it'll it'll get absorbed by the agent as it gets I see I see so like the harness is kind of like a bootloader to get it started but then eventually data will take over data and experience will take over the actual internal like operations of the machine yeah exactly yeah I mean right now the harness is super valuable in like very counter intuitive ways like for example it turns out that just giving an agent the ability like an environment to test against even if it barely uses it leads the agent to think for longer and for to try harder and and thus get better results so it's like there's still so many low hanging there's still so much low hanging food because the agents themselves are not fully well parameterized and calibrated yet but yeah I mean definitely will get to to a point in time when you know the next version of code acts or our opus will be able to just spin up an EVM on its own and we won't need our harness for it okay so what does the tool actually do is the tool the thing like the agent doing the exploiting or doing the patching or is it just the benchmarking like talking to me about the actual utility here the core the core release the core thing that we want to get out in the world is the benchmark it's how good are the models exploiting smart contracts there are three components to the benchmark the first is the ability to detect bugs the second is the ability to patch bugs and then the third which is through the most interesting and novel contribution stability to exploit bugs which which is yeah one of the biggest problems with previous attempts at having having security related for example auditing agents has been this problem around false positives so the agent comes to you and says I found 50 bugs in the contract and maybe one of those 50s an actual bug but it just is so time intensive for you to go through and figure out which ones are real that it's not better than a human auditor and what we did in in this sort of in the exploit component of of the benchmark is we leaned on the fact that crypto is verifiable and we used this production grade EVM environment where we load in a bunch of chain state and we set up a bug environment and let the mod agent try to exploit it we leaned on this to lower the false positive rate down to basically zero so it got to a point where if the agent tells you that it found a bug it literally has a proof of concept that it can exploit against it can run against a production grade EVM environment and and drain money from a contract and this this is sort of the core breakthrough of the of the paper is that is that there's a verifiable environment that actually leads to a very low false positive rate that's the actual benchmark is like you guys have established like you guys can actually measure the thing effectively yeah exactly because otherwise if if someone says oh we found all of these bugs and we got 90% on this benchmark you don't know what it means because you have no way of knowing if half of those are real or fake right so the verifiability was ended up being very important I think this is one of the reasons why models are going to get extremely good at crypto very fast because the basically you can slice the future relate to AI into two categories one is the verifiable stuff and the other is the unverifiable stuff and the verifiable stuff is very easy for the models to learn because they have a very clear training signal and they know exactly when they got it right and they can just keep running at that and and improve and and climb that hill whereas the unverifiable stuff it's like you don't know there's no tests you don't know if you got it right or wrong so it's like are you good at writing a poem is your joke funny like these are very difficult for the models to get good at and you know if you were to just take the whole universe of code and look at which pocket was the most verifiable you probably would end up with pocket that's almost entirely crypto right the whole substrate is based on the concept of being verifiable which means that with very little data even though with their there isn't that much in the in the form of contracts there isn't that much in the form of like it no crypto people are in these labs generally are very few the models have gotten extremely good because it's so verifiable and and also just as the models get better like for example Gemini famously learned an entire language just in context so as the models get better you you know the amount of data you need might might be might be lower so I think the general trend and trajectory of you know these models are going to get extremely good at crypto extremely fast I think I think we can bet on that so when the EVM bench paper says something like top models are going from 20% to over 70% exploit rate you know something like the newest jet GPT code x what does that mean 20 to 70% you like 70% of all smart contracts it comes in contract tacked with it can exploit like what does that mean practically yeah practically it means we collected all of the historical fund-draining critical bugs from open audit contests like Hodorin rate the set of so so the bugs that are found are not like oh you know you you have the small issue where where maybe like someone could have frozen the contract for a day or something like that it's much more you could have strained money from this contract if you found this bug and the 20% initially or less than 20% initially was meant that if you took up frontier model and you put in front of it all of the hardest all of the hardest audit problems after his knowledge cut off it would not be able to find the vast majority of them and by the end of the benchmark it meant that over 70% means that you know if you just reran cotarina and instead of GPT for it had GPT 5.
3 code x 5.3 code x would have found over 70% of the bugs the critical fund-draining bugs that human auditors found so throughout throughout his throughout our history of human audits and finding bugs it would have found 70% of those of the critical ones with with some constraints like for example we didn't go all the way back in history we started out past the knowledge cut off of because we we want to avoid contaminations so this kind of gets you a rough benchmark of like basically chat GPT 5 5.3 code x is like 70% as good as all of the human auditors out there something like that something like that these things are highly non-linear like for example there are dumb bugs that can lead to losing all the money in the contract but are actually not that hard to find so so that's why I think I think the the there there's more in the paper that is notable about the fact that these aren't there wasn't just like one trick that the model figured out and like got to 70% it wasn't like all re-enferencing right this a very diverse set of bugs that it was able to find but yes it's like fundamentally the the models are getting you know very close to being as good as the best human auditors one thing that's fascinating about having a benchmark is then like it seems like all the frontier laps love love to compete for benchmarks winning benchmarks right in the humanities last exam in fact they almost like game some of the training towards these benchmarks and so if you have an attractive benchmark that kind of propagates even like socially to all of the front your labs then it seems like it provides some sort of social incentive to have them you perform and train in order to you know compete against one another to exceed each other on those benchmarks is that kind of the flywheel you feel like has been set in motion with the yeah bench I think it's I think it's an important I mean maybe the zoomed out point is that crypto in its history has been very stigmatized and very illegible to the aylabs so the fact that there hasn't already been a massive push for crypto related evaluations is kind of absurd because the labs currently today are entirely bottlenecked on evaluations that are verifiable and economically important and crypto takes both of those boxes so so I think it took paradigm pushing a little bit of our weight around to get this through into the labs and I think yes like hope that our our firm hope is that this will start the flywheel of of labs paying more attention to this technology and we're going to continue doing work on this front as well do you have an explanation is why that it's been so slow because it's also been perplexing to me because it's all open source it's all out there it's all like we already got all the data yeah I mean what we talked to has he bristling his pressure his explanation was there's a lot of liability when it comes to if you finance and and crypto and having AI models trained on those data sets right it's like what if an AI model does exploit a bug whose fault is that so maybe there's some risk associated with it there's also you you just mentioned kind of a stigma I think yeah meme coin casino speculation certainly Peter from open claw he had a negative experience that he associates with with crypto culture with much people calling themselves part of the crypto industry try to front run him and develop meme coins and he just considers it shady like what are the reasons why the frontier labs have been so slow to train on this incredibly rich data set that as you said it's it's it's perfect for training because all of it can be verified my sense is that it's almost entirely a social thing I mean in in my pure group crypto is the biggest industry that has remained the most contrarian and I think part of that is because it's very reputationally volatile and part of it is because there's this dynamic where the best people in the industry the gap between the best people in the industry and the median person in the industry is much larger than anywhere else so if you for example don't have exposure to the high quality pocket of crypto then all you see are the scams and it's it's like that can distort your view such that you you just completely dismiss the industry and historically there's been a lot of alpha in that right and I think a lot of us have benefited from the fact that that there's significant reputationally a reputational volatility and if you aren't as sensitive to that as a person in terms of your temperament you can do very well in crypto but I think I think it's a social thing and I think it's just this legibility point about about there's hasn't been a a brand that can bridge the crypto and the AI worlds I think it's like if something touches crypto it sort of in the AI world historically has been tarnished and as a result people just have tried to try to avoid it altogether and this is sort of this was created in opening for something like for example something like EVM bench to get built it and shipped inside open AI without any sort of compete I think like all of the major model labs are going to be running on this benchmark and probably any future versions of it without significant competition inside the labs like there aren't like 50 30 to 50 crypto related benchmarks or or training environments that people are shipping in some sense it's actually agency inducing for us because because they'll just defer to the to the crypto industry to just figure out what's valuable for them but I think it's fundamentally a social issue and it's it's sort of tied to all of these dynamics around like you know you see someone who gets extremely wealthy who you don't think should get extremely wealthy like maybe it was like there's a lot of volatility in the industry and like there's some person who you don't respect to you who made a lot of money like these are all kinds of things that that go on in the minds of like the AI researchers at these labs that lead them to think that the whole industry is a scam and you know obviously there's this has made it an incredible environment to be investing in crypto because it's just not in the ocean window of anyone in the valley but but I think I think that's a core dynamic interesting as we know bought these AI LLMS they are very good at writing code it's like the first thing that they got good at is that the same is it also true with writing EVM code is there a gap there between writing the rest of the world's code and EVM specifically yeah I mean historically there has been and part of the reason you know one component of what motivated us to start this work was the realization like man these models are so good at Python and so bad at solidity and so bad at for example like Solano related code honestly anything that touches crypto and part of my expectation at the time was that we were going to have to go like crowdsource a bunch of data from the industry and like spoon feed it into the labs to get them the models really good at this but it turned out that because the substrate is so verifiable and also because there's sort of generality in these models they ended up getting quite good much faster than we expected with much less input than than we expected so there's this dynamic where if you teach a model of poem in English and then you know biology in Spanish it figures out how to write a poem in Spanish even though you never describe how to write poetry in Spanish the model and I think that kind of dynamic is happening here as well where where sort of like quote unquote learning the language of crypto without without as much direct training data and also just because I like very hard to like under under rate the verifiability of the thing right most software is hard to verify you need human labor laborers to go in and check is this thing correct is it running kind of the only threshold of verifiability you have is does the program compile and does it pass the tests but the tests need to be sort of written by human right you don't have this notion of like we have a bunch of state we can make assertions about the state we can for example send a model on a new contract on a new EVM that's never seen before and make assertions about whether it's able to quote unquote drain money from it like those concepts all would have otherwise needed to be hard coded into the program but because it's crypto we and there's so many standards it's verifiable and the models are just wrapping up really quickly in capabilities okay so do you think that this right now is the time that the the floodgates of crypto data to train these models that that is now starting to open going to be opened and if so what types of capabilities do you expect future models to kind of drop I mean yeah will they have skills and I guess personas connectors directly for crypto within some of the the core LLMS or what types of developments are you looking forward to there's a reason we started with security and not sort of general programming capabilities it's because it has this very nice shape of it's extremely economically valuable it's extremely sort of intelligence bound right it's like you can't yeah it's intelligence bound and it's very easily verifiable so we know when an exploit has happened so security capabilities I expect will develop very quickly and then we've talked about so all the implications of that all their crypto-related capabilities I think for example things in the domain of mechanism design or around market market related films like what is the mechanism for an exchange how how do if you have market of agents right what is the best way through which they should coordinate with each other these are I think open fertile soil and then of course you can go down to the protocol layer right you can say well how does a model land of transaction on the throwing blockchain how like there's a security side at the at the Ethereum client level right or at the protocol client level where like sure maybe they're 100 billion dollars of assets sitting in open source smart contract but there's way more than that in eth and soul market cap that can be exploited if you're able to find critical vulnerabilities in gath wreath etc so so I think going down into protocol layer is going to be important model capabilities around mev and and sort of extractive extractive tactics I think that will have the same effect as you know long tailhacks right there's a bunch of stuff on chain that you can just collect if you if you're able to do the end-to-end process of figure out alpha in the market construct a trade and underwrite it and then submit the transaction landed on chain reliably these are all things that actually the models are not that good at right now but they will get good at really quickly and all of that does seem long-term good for crypto however the the here in the now and and the short term like when we read in the event even bench paper that top models are going from a to over 70 percent exploit rate I'm not sure whether to like I mean feel good about that or bad about that because it sort of depends what the intent is and who is harnessing it so if it's white hat that's great I mean that improves our ability to find bugs and exploits before a time you're still however if it's black hat that's not great because you know it improves their ability to find these before we do so it depends I guess with with this tool said it depends who is using the tools and the intent behind them as to whether it is short to medium-term bearish or bullish I guess one thing it does is it does seem to inject some variance and uncertainty into the market I'm feeling that everywhere with AI right now it's just like it could be really good it could be kind of bad but one thing it's it's not going to be is boring it's going to be highly variant outcomes I guess when you think about the capability that is being unlocked here and the ability for elements to detect exploits is that good or is that bad for crypto in the short to medium term yeah well I mean you mentioned the in the long term crypto's positively lever to almost all of these developments and as models get extremely good at security this will raise the ceiling for the whole industry because and that's because it's like going to be a survival of the fittest thing right because all the weak stuff gets just so I it's up to us I think it's up to us like what the path there is us being the industry it's it may be survival of the fittest it may be that that we get figure out a way to have the defense get ahead and I think but the core point is that the amount of assets that can be sustained on these networks is proportional to how secure they are and in the long term there's this benefit where as security improves more assets will be able to securely stay on chain now in the short term I think I think this is one of those things where it's in our hands right I think it's bound by the industry's agency on the best way to handle this we don't know like if we just let the clock play forward there's a lot of uncertainty we don't know exactly who whether the attackers you know the black hats will get capabilities before the white hats do but we also are active participants in this market and we can we can bend the arc of this such that for example we make sure that if there are frontier models or unreleased models or there are there are new developments in security relating to AI that we that we get this into the top protocols you know one one version of the world that you can imagine the short to medium term is that you always have every single contract being scanned by both adversarial actors and defensive actors 24 seven and when there's a bug that's surfaced you know whoever catches it first sort of will will react accordingly and then in that world it is just kind of more of a race between between the good guys and the bad guys and and I think we have a lot of a pretty great hand in terms of in terms of making sure that the good guys have have the lead in that race at the end of this once all of the contract like the weak contracts have been exploited or we've beefed up security enough such that they're not exploitable I guess that gives us an incredibly hardened financial system for the world something that's ultra security there's almost like a it's close to perfect right it's like how many how many nine's I mean it's maybe like four nine five nine's and they almost creates kind of a a barbell model of a security for for the world for financial assets like the most secure financial assets will probably be in this dark forest environment like on chain how do we know that because the world has thrown everything it can add it including our most intelligent LLMs and it's still there it's still standing right hasn't been exploited that'll be one side of the barbell the other side honestly it's things that are completely outside of the digital world altogether you know like a clump of like a bar of gold or something that digital gold and then everything in the middle will be pretty exploitable pretty insecure I wonder if that's what is on the horizon with with our world um until the LLMs figure out how to synthesize gold right it's right right in some of the robots after the gold right yeah I think I think that I guess that makes sense and I think that that barbell view of the world might be how things play out I guess the way that I relate to crypto as an industry and also as a technology is is that if you start from the first principles vantage point of let's say you want to do payments at the speed of light right like I want to send you Ryan money from America to Europe or some other part of the world and I center as fast as send an email and the problem that you have there is that you don't know if I also sent that money to David right you have this double spend problem and this was what Bitcoin solved and it got the time for that transaction down to about an hour and since then we have had successive developments that have increased both the the speed of these transactions and the expressivity of these transactions and I think that I think that in that world view of this is like this is not just some path dependent thing that happened that the crypto industry emerged the way it did and that actually this is if you were to play it forward from first principles that this is how it has to be you end up in this in this conclusion where you know if you have agents that want to move at the speed of the internet and the current banking system was you know created before cars were invented that that these that those agents are going to discover the crypto rails as I was the right way to transact and you know I think there was a concern maybe like six to eight months ago now definitely on for me where it was not clear if the agents would get good enough at crypto related software for example for them to be able to discover the current rails and maybe they'd have to reinvent them from scratch but over the last six to eight months as we've been working on this work of the open AI it's becoming increasingly clear that that one they're extremely strong network network effects inside of crypto and that and two these agents are able to just learn like they just want to learn these verifiable things and crypto is very high on that list so so at this point I think you know I've become extremely extremely bullish on on crypto is a substrate for for these agents and I think it's sort of an open game who in crypto is going to win that but but the shape of the technology fits it perfectly the EVM is by far the most common programming language programming environment in crypto solidity being the most common language and then there's like some long tail languages like cardanos like Haskell for example and as we know AI loves data the more data and AI can get on the better better it can be what do you think this network effects around environments how does that play into AI like is the EVM going to be the favorite environment for AI's to work in does Salona's what's what's Salona SVM SVM yes VM does the Salona's SVM also you know cross the threshold or maybe the threshold thing is a false illustration what do you think I think it's it's actually very hard to say and currently the ball is in there we don't know part of the reason what I mean I can point to some of the bottlenecks we had while we were developing VM bench where we also have actually started work on on Salona related component of it but for example one challenge was that actually it it requires a lot of human talent like human talent to be able to go in and and construct these evils and that was just much easier for us to come by for solidity and these things are really hard to build right these are pretty heavy infrastructure so so even small additions of friction lead to you know when you need to cut scope in some capacity that you kind of have to cut it in in the direction of the stuff that you can do more easily first that being said I think the point about the fact that you know one crypto's verifiability is a huge edge and two these models are becoming less and less data hungry when it comes to learning new programming languages I think it may end up being more even of a playing field than it might seem and I think you know at least we have have an intent and interest in actually going across ecosystems and down the stack to the protocol layer and making this sort of more expansive of a crypto flagship benchmark but yeah right now there are obviously network effects around the EVM right one other example of a counter intuitive reason why why someone like Solano might be able to catch up is that first at first glance it may seem really bad that most of the contracts there are closed source but if you take the world view of actually if it's open source it gets in the training set then the closed source benchmarks and training and contracts for training might actually be more valuable for for a model's development because it's not in that right now for example like we have these sort of what are called canary tags in various parts of our evaluation that filtered them out of the training process of most models so you know there are tricks we can do but still it's possible that these the bugs in EVM bench over time leak into the pre-training of the models whereas if it were closed source it would not leak in it all so so my expectation is actually that you know there will be some asymmetry at first but actually the models will get really good at all and and there will be sort of a merit based sort of right who you know who the best will rise to the top are you sitting on usdt or stablecoins after taking profits and wondering where to deploy next what if you could access stocks gold and ETFs without ever leaving crypto that's what tokenized stocks on bit get unlock traditional markets still run unlimited hours but capital is moving on chain 24 seven on bit get you can trade tokenized stocks and ETFs 24 seven with up to 100 x leverage all settled directly in usdt or usdc no brokerage accounts no off ramps no platform switching bit get has already processed over eighteen billion dollars in tokenized stock trading volume with most of that happening in the past month alone the platform now captures close to 90% of on those tokenized stock spot market share as gold and silver hit record highs on chain training followed over the past two weeks volume in svl on tied to silver and iau on linked to gold surged this is bit gets universal exchange vision and action crypto equities and real world assets in one place built with crypto native speed and flexibility if you want to trade stocks the way trade crypto explore tokenized equities on bit get learn more by clicking the links in the show notes this is not investment advice few people in crypto put real skin in the game when they make public top or bottom calls the defi report is one of them the week before the october 10th flash crash michael from the defi report emailed his entire newsletter saying he's going aggressively risk off and sold the majority of his book from crypto into cash this is when ethos about four thousand dollars and bitcoin was 110 michael runs the defi report an industry leading research platform built on data cycle awareness risk management transparency and most importantly skin in the game we like michael at bankless we like his analysis and that's why you hear him on the bankless podcast about once a month and the defi report is giving bankless listeners one free month of access to the defi report so if you're looking for some sharp data driven analysis to make better informed decisions around your portfolio you can learn why and how michael called the top and what he's doing next all in the defi report pro check it out there is a link in the show notes ethereum's aspiration is to formally verify it's entire end-to-end techsack in the fullness of time you know we didn't first we have to get like the beam chain we have to do all the hardware skip there but ultimately we want to do a formal verification of the entire ethereum techsack AI-based formal verification is that is that a real thing how does AI capabilities work its way into the conversation of formal verification yeah well i mean i think it is i think it is a real thing last year we invested in a company called harmonic which is a foundation math model co-founded by a lot kind of of robinhood and tutor i think part of their thesis and i think part of what the world is clearly going is that is that there's more software that is being generated then can be possibly reviewed by humans and formal verification is one way to quickly check whether a component of software is actually doing what it says it's doing and then obviously in the context context of security it can especially the spec is written correctly it can be it can be a step function change now it's not a silver bullet in the sense that you still have to write the spec for the formal verification so there's still surface for bugs to get in there but you know you can make the case that actually the surface for bugs in writing a formal verification spec might be lower than writing the code to start with and and definitely with time i think all of the best all the best models all of the best software will probably end up being formally verified and if you take the vantage point of an agent and you have two options to choose from one of them is formally verified and one of them is not the formally verified one might just gain preference just because it has all these nice properties there's a section in the paper of the EVM Bench paper called future directions what does EVM Bench V2 look like how does this it's called a project project to grow from here the top level goal that we have is to help the model labs develop the crypto capabilities of their models and i think that security is one component of that and maybe an increasingly urgent component of that but there there's so much that EVM Bench does not touch so there are other ecosystems and stacks there is a protocol layer which we talked about where maybe arguably it's more important from a security standpoint that the Syrian protocol is secure rather than any specific solidity contract there there are out of protocol components like how do you land a transaction on chain how do you deal with the mempool how do you deal with sort of the non-deterministic parts of crypto and and then obviously there's there are components that are even farther on the verifiable and intelligence down trajectory like for example around cryptography and around serenage proofs etc and i think all of these are extremely fertile soil for future work so we're currently open to trying to source collaborators for for future versions of of EVM Bench and you know we're obviously working on on next steps for ourselves and i think that this this direction of like we finally have a foot in the door into the model labs for for getting crypto capabilities into the frontier models and i think that we should we should leverage that as an industry and we should try to try to get these models as good as crypto as we possibly can help in your your very smart as evidenced by this paper in this conversation you have a high degree of of agency obviously you're definitely on the frontier you've chosen to stay in crypto and not kind of leave and go to AI and you seem incredibly bullish at crypto even bullish that it's that it's contrarian at this moment in time what why crypto for you personally i've personally never had hard lines around industries in my mind i think we talk about like i work and acts because like to make what we're doing legible to other people but i don't think that's the right way to relate to it i've spent all all this time in crypto because it's been one it's been extremely intellectually interesting and two it has this it's just as i mentioned it's remained extremely contrarian among my smartest friends in a way in ways where i can put my finger on exactly what they're missing i think that sort of that's kind of the best that that one can ask for i think that you know we talked about how cryptos positively levered to the security developments in ai but you know you can make the case that it's positively levered to most of the developments in the world right now like for example as as the creation of new as new creation of new goods and intelligence etc becomes commoditized scarce assets become more valuable as geopolitical instability ensues systems that are extra sovereign right outside of outside of any jurisdiction that that are kind of the equivalent of end-to-end encryption for finance those have more space to thrive and i think that you know i grew up in turkey most of my family still there i think that people who grew up in america do not have an in general and sort of a stable world world do not have the sense for what can happen as the world destabilizes and i think that uh you know as many people in in the country that i grew up in are starting to onboard to crypto rails and sort of using that as a lifeboat i think it's increasingly clear to me that that this technology is is on a sort of compounding trajectory to do really massive things and so the combination of that plus you look around and no one's even talking about it it's just really exciting yeah and you do seem convinced that the acceleration of AI is going to benefit crypto that it will be you know all boats rise together and i think that well there's some category of software industry that AI doesn't seem to benefit at least in the short run you know anthropic drops a new security module and all the cyber security stocks drop like 10 to 15 percent in one day why are you so convinced that AI's acceleration will be beneficial to crypto it's not obviously nothing is guaranteed right now i think if we let everything run its course it may be bad for crypto it may be good for crypto we don't know i guess the the the conviction that i have is that if we push things in the direction that we want them to go in that we can make AI be extremely good for crypto and and also i think there's the component of this where i do strongly believe that if you were to redrive all of this from first principles you end up in a place that's very similar to where we currently lined it with crypto and yeah i think i just think that for all the reasons we've talked about that for fundamental reasons crypto is extremely good for AI and AI is extremely good for crypto so i i think that i think that you know nothing is guaranteed and we still have to exercise our agency but but i think for all the reasons we've discussed so far it's like pretty clear to me that that these things are going to converge in a positive way well let's end a note on high agency and conviction alpin thank you so much for joining us today well thanks for having me got to let you know bankless listeners of course none of this has been financial advice you could lose what you put in hopefully an lllm out there a white hat is protecting it we're headed west this is the frontier it's not for everyone but we're glad you're with us on the bankless journey thanks a lot