--:-- Host (Erik Torenberg): So we've been through all of these tech waves and we've learned how to have this discussion in a way that for the United States interest balances these two things. And if we're going to make a departure from a posture that was developed from 40 years, better have a pretty damn good reason.
--:-- Host (Erik Torenberg): Today, a new frontier of scientific discovery lies before us. You can sometimes judge a book by its cover, and I think this is a strong start. The conversation around AI regulation in the U.S. has changed dramatically. Just a year ago, the loudest voices were calling to pause or shut down open source AI. Today, the U.S. is pushing to lead the global race. So what changed? And what does it mean for innovation, competition, and the future of open source?
--:-- Host (Erik Torenberg): I'm joined by A16Z general partners Martin Casado and Anjney Midha to unpack the new AI action plan, the politics behind it, and the implications for builders and policymakers alike. Let's get into it.
--:-- Host (Erik Torenberg): As a reminder, the content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.
--:-- Host (Erik Torenberg): So we're talking a week or two after the action plan has been announced. Looks like we've come a long way.
--:-- Guest (Martin Casado): Yeah. You guys have been on the front lines for years now in this discourse, fighting to make this possible. Why don't we trace where we've been so that we could then understand how we got here and where we're going?
--:-- Guest (Anjney Midha): I mean, under the Biden administration, we had the executive order, which was basically the opposite of what we're seeing today. I mean, it was trying to limit innovation. It was doing a bunch of fear mongering. But to me, what was even more striking was not regulators being regulators. You'd expect that. But if you remember, Ange, and this is why we got involved, is you'd have these politicians making recommendations, which is fine, you'd expect that. But nobody was saying anything. Academia was silent. The startups were silent. And if anything, the technologists were kind of supporting it.
--:-- Guest (Anjney Midha): So we were in the super backwards world where it was like, innovation is bad or dangerous and we should pause it. There was this discourse and it was somewhat fueled by tech. And then nobody was going against it. So I think today we should definitely talk about the action plan is great, but we should also talk about how the entire industry has kind of come around to say, listen, we need to keep these things in check, we need to be sensible.
--:-- Guest (Martin Casado): Yeah, I mean, pause AI. That was two years ago, remember? The big sort of all the CEOs signed this petition.
--:-- Host (Erik Torenberg): Oh yeah, I think that was the last AI action summit, right? The one before Paris.
--:-- Guest (Anjney Midha): Guys, there's been so many of these.
--:-- Guest (Martin Casado): Yeah, let's try. I'm not sure.
--:-- Host (Erik Torenberg): No, no. Remember, what was the Dan Hendricks' CAIS? What was California AI?
--:-- Guest (Anjney Midha): The Center for AI Safety.
--:-- Guest (Martin Casado): Center for AI Safety. That's it. The nonprofit. Yeah, yeah, yeah. And then they got all of these people to sign this list, like, when you need to worry about the existential risk of AI. And that was the mood. It was almost like, can I just do something by contrast?
--:-- Guest (Martin Casado): I was there during the early days of the web and the internet. And at that time, you actually had examples of the stuff being dangerous. Robert Morris let out the Morris worm. It took down critical infrastructure. We had new types of attacks. We had viruses. We had worms. We had critical infrastructure. We actually had a different doctrine for the nation. We said, the more we get on the internet, the more vulnerable we are. So instead of mutually assured destruction, we have this notion of asymmetry. There were all of these great examples of why we should be concerned. And what did everybody else do? Pedal to the metal. Invest more technology. This is great.
--:-- Guest (Martin Casado): We still wanted the internet. We wanted to be the best. We wanted to build it out. The startups were all over it. And coming into this AI stuff two years ago, it was the opposite. There were the concerns with new technology, which you always have, but there were very few voices that were like, actually, it's really important we invest in this stuff. So that's kind of the bigger change. This is more cultural change.
--:-- Guest (Anjney Midha): I think that's right. There was a moment last summer where somebody sent you and me a link to the SB 1047 bill. And I remember Marty and I reacting like, there's no way this is going to get any steam. What was absurd to us was that it made it through the House and the Senate. And it was on its way to a final vote and would have become law, one signature from the governor later.
--:-- Guest (Anjney Midha): I think there was this escalation where I realized that technologists like to technology and politicians like to policy. And we pretend like these two things are in different worlds. And as long as these two worlds don't collide and the engineers get to build interesting tech and there's no sort of self-own too early, we generally trust in our policymakers. That changed completely last summer. It was a really weird cultural shift. A lot of the policymakers who actually were quite open about the fact they didn't know much about the technology because it was moving so fast still felt like something had to be done. Therefore, this is something, therefore it must be good. And this was the most egregious example of this being adversarial: SB 1047.
--:-- Guest (Martin Casado): I totally agree. But that culture shift was one from "let's let the tech mature and then decide how to regulate it later" to "let's try to regulate it in its infancy." That was a massive shift in my head. But let's just talk about how bad it was. You had VCs. Their entire job is investing in tech, talking against open source. It was absurd. The Node, Founders Fund, they're like, open source AI is dangerous. It gives China the advantage.
--:-- Guest (Anjney Midha): Right.
--:-- Guest (Martin Casado): There was just some sort of prognostication that if we didn't do open AI, the Chinese would somehow forget math and not be able to create models. And then you forward by a year and they've got the best models by far and we're way behind. So the people that are supposed to be protecting the U.S. Innovation Brain Trust were somehow on the side of "let's slow it down." And I think now there's this realization that China's really good at creating models and they've done a great job. We've kind of hamstrung ourselves from whatever discussion we were having.
--:-- Guest (Martin Casado): I think you're right. It's good to be concerned about the dangers and job risks, but it has to be a fulsome discussion. You need both sides. When you and I jumped in, it just didn't feel fulsome at all. It was like one side was dominant and there was almost no one on the pro-tech, pro-innovation, pro-open source side.
--:-- Guest (Anjney Midha): I just think it didn't feel grounded in empirics.
--:-- Guest (Martin Casado): Well, certainly not that. It came from...
--:-- Host (Erik Torenberg): So what is the steel man of the critique of open source that they were making a couple of years ago?
--:-- Guest (Anjney Midha): That this is like a nuclear weapon. Would you open source your nuclear weapon plans? Would you open source your F-16 plan? So the idea was that this was like nuclear weapons are not dual use. Nuclear energy is dual use. An F-16 is not dual use. A jet engine is dual use. But a lot of the analogies that were used at the time were something that, if you squint one way, parts of it are dual use. They could be used for good or for bad. But the examples were clearly the weapons. And that's what they would say. They would say, listen, these things are incredibly dangerous. Would you open source the plans for an F-16?
--:-- Guest (Anjney Midha): And then the other side, which slowly decided this conversation is ridiculous, we got to go ahead and set it up, says, no, you would not do this for an F-16 because that is a fighter jet. However, a lot of the technologies used to build it, yes. This is fundamental. It's not like people aren't going to figure it out anyways. And we need to be the leader just like we were the leader in nuclear. And by the way, in nuclear, when that came out, we invested incredibly heavily in it. The things that we thought were proximal to weapons, of course, we made sensitive. But all the universities were involved. The entire country had the discourse. And that just wasn't what was happening.
--:-- Guest (Martin Casado): I think that's true. There was a substantive argument against open source, and there was an atmospheric one. The substantive one was the one Martin mentioned: the technology was being confused for the applications. And all the worst case outcomes of the applications or misuses were then being confused. But they were also theoretical too.
--:-- Guest (Anjney Midha): It's even worse than that. It was like, this could potentially create bioweapons. It was funny, we got a bioweapon expert and he's like, well, not really. The difference between a model in Google is almost nothing. But that was used as this straw person argument. And then it could hack into a whole bunch of stuff. Nobody had ever done it before, but it was theoretical. So it was these theoretical arguments that were very specific versus broad technology. That was one.
--:-- Guest (Anjney Midha): And then the atmospherics. There was a famous former CEO who went up in front of Congress and literally in a testimony said, the US is years ahead of China. And so since these are nuclear weapons and the misuses were being confused with the technology and we're so far ahead, let's lock it down so we can maintain that lead and therefore our adversaries will never get their hands on it. Which were both just fundamentally wrong for the reason Martin said. AI was not introducing new marginal risks.
--:-- Guest (Martin Casado): Well, at least not identified at the time.
--:-- Guest (Anjney Midha): Not at the time. You would go to Dawn Song, who is a safety researcher, MacArthur Genius Fellow at Berkeley, and you'd say, what are the marginal risks of AI? She'd say, great question. We should research that. That should be a good research problem.
--:-- Guest (Martin Casado): Yeah.
--:-- Guest (Anjney Midha): The world expert on this question was like, this is a very important, but it's an open research statement. So no empirical evidence at the time that AI was creating net new marginal risks. And just factual inaccuracies that we were ahead of China. Because if you just paid attention to what's happening, DeepSeek had already started to publish a fantastic set of papers, including DeepSeek Maths v2, which came out last summer. And you're like, okay, obviously these guys are clearly close to the frontier. They're not years behind.
--:-- Guest (Anjney Midha): And so when R1, DeepSeek R1 came out earlier this year, a lot of Washington was shocked. "Oh my God, how did these folks catch up? They must have stolen our weights." No, actually it's not that hard to distill on the outputs of our labs. Have you actually looked at the author list of any paper in AI? Where do you think these people come from? So I think those two things were, it felt like we were being gaslit constantly because both the content and the atmosphere were just wrong.
--:-- Host (Erik Torenberg): Maybe one question for the smartest people or the most sober people who were against it is, maybe they were asking where should the burden of proof be? Because it's hard to prove that there is risk, but it's also hard to prove that there isn't risk. So this question of what's risky: is it riskier to just go full steam ahead? Or is it riskier to kind of slow down until we better understand these models' interpretability, etc.?
--:-- Guest (Anjney Midha): I mean, I think it's really important to ground these hypothetical discussions on what we've learned as an industry. The discourse around tech safety has been around for 40 years. And we went through it with compute. Remember when we were like, okay, Saddam Hussein shouldn't have PlayStations because you can use GPUs to simulate nuclear weapons? That was actually a pretty robust and real discussion. But that did not stop us from having other people create chips for video games.
--:-- Guest (Anjney Midha): We went through the internet. We went through cloud. We went through mobile. We've been through all of these tech waves. And we've learned how to have this discussion in a way that for the United States interest balances these two things. We've had areas that were very sensitive to national governance. Think about Huawei and Cisco, for example. And we as a nation did start to put in import and export restrictions as a result. So I just feel these almost platonic, polemic questions like the one that you just posed aren't rooted in 40 years of learning.
--:-- Guest (Anjney Midha): All I ask is, if we're going to make a departure from a posture that was developed from 40 years, we better have a pretty damn good reason. And if we don't have a good reason, then I think we should probably learn from that experience.
--:-- Guest (Martin Casado): Yeah, I think extraordinary claims require extraordinary evidence. The burden of proof should be on the party making the extraordinary claims. If there's a party who's going to show up and say, AI models are like nukes and California should start imposing downstream liability on open source developers for open sourcing the weights, that's a pretty high claim to make. You should have exceptional proof if you want to change the status quo. And the status quo is you do not hold scientists liable for downstream uses of their technology. That's absurd. That's a great way to shut down the entire innovation ecosystem and start throwing researchers in jail. We don't want that. We want them to be trying to push the frontier forward. I just don't think that the tall claims were being followed up by tall proof.
--:-- Host (Erik Torenberg): When we're talking about open source, are we all talking about the same thing? Meaning are there degrees of open source or is it kind of just like a binary?
--:-- Guest (Anjney Midha): Open weights, I think, was the primary contention. That if somebody put out the weights of a model and a bad guy took those weights, fine-tuned it, did something really terrible two years later, the SB 1047 regime proposed that the original developer of the weights that they put out basically as free information should be held liable. Which was absurd.
--:-- Guest (Martin Casado): I just want to make sure we're very clear because people jump on top of these things. What he's saying is correct. So basically if the weights were over a certain size and there was a mass casualty event... I think catastrophic harm was the word used, but there were no real...
--:-- Guest (Anjney Midha): No, it was mass casualty. There were so many versions. Actually, I don't know which version you're talking about.
--:-- Guest (Martin Casado): Yes. But I remember we actually looked it up. The legal definition was three or more people were killed or the medical system was overwhelmed. There was actually precedence of this, including a car crash.
--:-- Guest (Anjney Midha): Right.
--:-- Guest (Martin Casado): And there was actually precedence of this happening in a rural area, which basically doesn't have any sort of capacity. So basically it would move the conversation to the courts and outside of policy. Which is, again, historically, we've taken a policy position on these things, which follows precedents that we understand, to make sure that we don't introduce externalities, like allowing China to race ahead with open source, which has happened.
--:-- Guest (Martin Casado): The key thing is by moving it to the courts, even if you don't... One could argue, oh, Ange, but sure, it's moving to the courts. That means it's open for debate. It's not clear that open weights are going to be regulated with liability. The point is that that creates a chilling effect. The chilling effect is the idea that when our best talent is considering... I could be sued. I'm a random kid in Arkansas developing something. I don't want to be in a world where it can be resolved in the courts. I can't even afford that.
--:-- Guest (Martin Casado): And in a situation where you have an entire nation-state-backed entity like China actually doing the opposite of a chilling effect, encouraging a race to the frontier, why on earth would we want... There's this meme of a guy on a bike and he picks up a stick and puts it into his front wheel and topples forward. That's the effect of a chilling effect. That is what chilling effect is, right? At a time when your primary adversary is racing.
--:-- Host (Erik Torenberg): So let's trace how the conversation has changed because we don't see Vinod tweeting about open source anymore. Obviously, open source changed his tune, especially right now. Is it really just DeepSeek? Or how do you trace how the sentiment shifted on open source? Let's go through a few theories.
--:-- Guest (Anjney Midha): I'm not really sure what happened. I almost felt like it was culturally in vogue to be a thought leader on the negative externalities of tech. It kind of started with Bostrom, but it was picked up by Elon. It was picked up by Moskowitz. A bunch of these intellectuals that we all respect and still do. They're the titans of our industry in our era. They were asking these very interesting intellectual questions around, do we live in a simulation? What happens if AI can recursively self-improve? That created whole cultures and online social discourse around this stuff. To no small part, that became a bit of a runaway train. And it's just catnip to policymakers.
--:-- Guest (Anjney Midha): So I think part of it is people didn't really realize that this has become so real because GPT-2 comes out and then 3 comes out and all this stuff's amazing. And somehow it got conflated. So I think part of it is just a path dependency on where we came from, which is the legacy of Bostrom. I think that was part of it.
--:-- Guest (Martin Casado): I think the ungenerous approach would be that there was a lot of discourse, but a lot of the people pushing the discourse were first order thinkers. They weren't doing the math on, wait a minute. If policymakers who have no background in frontier AI—which by the way, nobody does because this space is only three, four years old—start to take discourse as canon, which is a big difference, then what happens? What are the second and third order effects? The second and third order effects are that you start making laws that are really hard to undo. And you start mistaking interesting thought experiments as the basis for policy. Once that happens, law is basically code. Code is hard to refactor. Law is impossible to refactor.
--:-- Guest (Martin Casado): The second and third order effects were of a lot of well-intentioned folks in the existential risk community saying, look, if you're intellectually honest about the rate of progress of AI, it's not crazy to say that there are some existential risks on the technology. It's non-zero. Sure, yes, that is true. But then to say that that threshold is high enough to start introducing sweeping changes in regulation to the way we create technology, that leap, I don't think a lot of the early proponents of that technology realized they would do that. In fact, I think Jack Clark, who runs Policy for Anthropic, literally tweeted towards the end of the SB 1047 saga, he was like, I guess we didn't realize the impact of how far this could have gone.
--:-- Guest (Martin Casado): And I think to those of us who had interacted with DC before and regulation before, the second and third order effects were much more discernible, illegible. And then I think what DeepSeek did was just made it super legible to everybody else. I think DeepSeek was the catalyst, but it wasn't like there was a step. It didn't change the reality that the second and third order effects of policymakers confusing discourse for fact were always going to be terrible. I just think it brought to light something a lot of us were already saying, which is we're in a race with adversaries. And that should be the calculus we should be working backwards from.
--:-- Guest (Anjney Midha): There was always this prevailing view, which just turned out to be so wrong, from really well-intentioned people, which was like, it's going to be regulated anyways. If it looks like we're self-policing, we can dictate how that happens. And unfortunately, that just turned out not to be true because whatever self-policing we seemed to be doing scared the shit out of people. And then, of course, I would say very opportunistic elements in tech decided to use that for whatever agenda that they had. And so it kind of got away from us.
--:-- Guest (Martin Casado): Mark had this sort of the Baptist bootlegger.
--:-- Guest (Anjney Midha): Yes, I was just going to say, exactly. True believers and then sort of people who use that thinking to support their own ends. And it seems like that's changed, even just on the company. But the reality is I think the majority of people are neither. The majority of people are pragmatists that are not trying to take advantage of the system. They think, well, maybe if we have this discourse, it's an honest discourse and then we'll self-police. I just feel like the silent majority was not part of the discussion.
--:-- Guest (Anjney Midha): Maybe the biggest change now is those people are there. The founders are there. Academia is there. VCs are there. Now the people that are not either Baptists or bootleggers are driving the discussion. Independent of the action plan itself, I feel much more in a better position now. For example, there's still a bunch of stupid regulation that's popping up, but I'm not calling Ange at night saying we have to do something now because I feel like there's actually representation that's sensible. At the time, there was none.
--:-- Guest (Anjney Midha): And I think to move to the action plan, if you read the first page, what a marked shift. The fact that the co-authors include technologists. I think that was the core problem. DC is a self-contained system. The value is a self-contained system. A lot of the people here were assuming best intentions over here and vice versa. What happened is a few bad actors essentially used that arbitrage opportunity to represent Silicon Valley's views incorrectly in DC. When we saw some of the legislation, we had policymakers calling us up and saying, wait, you guys aren't happy with 1047? But the other tech people were calling us and saying, you'd love more of this kind of regulation. We said, what other tech people? And it turns out we are not one homogenous group. Little tech is extraordinarily different from big tech, which is extraordinarily different from the academic communities.
--:-- Guest (Anjney Midha): One of the things we had to contend with was we used to be one shared culture. And then when tech grew, there are some major differences in the Valley between parties. We are not one tech ecosystem anymore. We have different interests. And DC hadn't updated that. What's amazing about the action plan is it's written by people who have bridged both with enough representation across the four or five different subcultures within tech who have different interests. I think that's new.
--:-- Host (Erik Torenberg): Going back to open source, why don't you talk a little bit about how different companies have thought about it from a business strategy perspective? Maybe we saw Meta with the first big open source push. OpenAI has sort of evolved there too. I've seen even Anthropic seems to be evolving their dialogue a little bit. How should we think about open source as a business strategy in terms of what's changed here and why?
--:-- Guest (Anjney Midha): Oh, look, I don't think this part is actually playing out beautifully along the same trend lines of all previous computing infrastructure: databases, analytics, operating systems, Linux. The way it works is the closed source pioneers the frontier of capabilities. It introduces new use cases. And then the enterprises never know how to consume that technology. When they do figure out eventually that they want cheaper, faster, more control, they need somebody like a Red Hat to then introduce them and provide solutions and services and packaging and forward deployed engineering around it.
--:-- Guest (Anjney Midha): The arc generally in enterprise infrastructure has been closed source wins applications. And open source tends to do really well in infrastructure, especially in large government customers, regulated industries where there's a bunch of security requirements, things need to run on-prem, the customer needs total control over it. Broadly, you could call that the sovereign AI market right now. Lots of governments and lots of legacy industries are going, wait, this open source thing is really critical to us.
--:-- Guest (Anjney Midha): Whereas two, three years ago, open source was viewed as this largely philosophical endeavor, which it is. Open source has always been political and philosophical by definition. But now there's an extraordinary business case for it. That's why I think you're seeing a lot of startups and companies also changing their posture because they're going, wait a minute, some of the largest customers in the world, enterprise customers, happen to be governments and happen to be legacy industries and Fortune 50 companies, and they want stuff on-prem. And that's when you go adopt open source. So I think there's been a business shift as well.
--:-- Guest (Martin Casado): I don't know if you do agree.
--:-- Guest (Anjney Midha): Yeah, this is great. So I totally agree. I do think it's interesting to have a conversation where it's the same and where it's different. Everything I've said is exactly right, which is we have a very long history with open source, and it's a very useful tool for businesses, but also for research and academia. But let's just talk about businesses and startups. It's a great way to get a distribution advantage. It's a great way to enter a market where you're not an incumbent and you're a startup. So it's just one of the tools for building in software that's been used. And open source has been used in a very similar way. You can use it for recruiting. You can use it for brand. You can use it to get distribution. And we see all of that.
--:-- Guest (Martin Casado): There's something that's unique about AI that software doesn't have. We're seeing very viable business models come out of it that don't have the limitations of traditional software. This is for two reasons. One of them is open weights is not the ability to produce the weights. But open software is the ability to produce the software. If you give me open software, I can compile it, I can modify it, whatever. But giving open weights, you don't have it. You don't have the data pipeline when you're talking about open weights. So you don't actually enable your competitors in the same way open software enables it. So that's one.
--:-- Guest (Martin Casado): The second one is this is a very nice business model. It's kind of a piece dividend to the rest of the industry. You open source your smaller models that anybody can use, but the larger model you keep internally, which is actually also more difficult to operationalize for inference.
--:-- Guest (Martin Casado): I mean, there's kind of good reasons to do this. And then you charge for the largest model and then the smaller open models you use for brand or distribution or whatever. And so I feel like it's actually almost an evolved version of open source from a business strategy and industry perspective for these reasons. I think it's the AI flavor of open core, which was historically supposed to be a theoretically sustainable model for open source software development. That was really hard to implement because of the reasons I said: once you gave away the code, it was really hard to protect your IP. But with weights, you can contribute something to the research community, give developers control, and allow the world to red team it and make it more secure. Meanwhile, because of the way distillation works and some of the post-training techniques, you can still hold on to some of the core IP, which allows you to build a viable, sustainable business.
--:-- Guest (Anjney Midha): And that is unique about open source AI. But also you have the data pipelines, you have the data. Just because I give you the weights doesn't mean you can recreate the model. You could distill it to a subset model, there's a bunch of stuff you can do, but not necessarily recreate it. Having been a student of open-source business models for 20 years and watching how it shaped the way the industry has adopted and built software, I actually think the AI version is more beneficial to the companies doing it for sure. As a result, we're going to continue to see a lot of it. I think we should just assume that open source is part of it and every country is going to do it. One of the best things about this current AI Action Plan is it acknowledges that and wants to incentivize the United States to be the leader in it. That's such a dramatic shift from where we were this time last year.
--:-- Guest (Martin Casado): There's an ecosystem mindset. People who have worked in any kind of developer business—which Marty and I unfortunately have spent way too long doing, working on dev infrastructure and dev tools—internalize this idea that you have to often trade off short-term revenue for long-term ecosystem value. I think what the action plan shows is that yes, in the short term, it may seem like we're giving away IP to the rest of the world by open sourcing weights and showing them how to create reasoning models. But in the long term, if every other major nation is running their entire AI ecosystem on the back of American chips, American models, American post-training pipelines, and American RL techniques, then that ecosystem win is orders of magnitude more valuable than any short-term give of IP. Anyway, as we saw with DeepSeek, that marginal head start is minimal.
--:-- Host (Erik Torenberg): Okay, so just to close the loop on open source. Over the next several years, how do you predict open source and closed source will intersect? What will the industry look like?
--:-- Guest (Anjney Midha): Well, I think these are two different markets. The requirements of the customers are completely different. If you're a developer building an application and you need the latest frontier capabilities today, you have a different set of requirements than if you're a nation state deploying a chat companion for your entire employee base of 7,000 government employees. The product requirements—how you provide them, how you deploy them, the infrastructure, the service, the support, and the revenue models—are completely different. Often people don't realize that closed source and open source are not just differences in technology, but completely different markets altogether. They serve different types of customers. If you believe AI is this explosive new platform shift, then there will be winners in both. I do think we need to contend with the fact that it's getting harder and harder to be a category leader if you don't enter fast. The speed at which a new startup can enter the open source or closed source market and create a lead is absurd. We both have the chance to work with founders who are literally 20-something-year-olds out of college, two years out, building revenue run rate businesses in the tens to hundreds of millions of dollars serving both of these markets. The biggest mistake is to confuse these two markets as one and to do the classic "let's wait to see how they evolve." The pace at which a new entrant can create a lead in the category is quite stunning.
--:-- Host (Erik Torenberg): Let's go into the action plan. What are our biggest reflections from it? Where are we most excited?
--:-- Guest (Anjney Midha): If you look at the quote they start with—I wanted to read it out because I thought it was poignant. It was: "Today, a new frontier of scientific discovery lies before us." I thought that first opening line was fantastic. Out of all the things they could have said, they could have said we're in a nuclear arms race. Sure, the first page title says "Winning the Race." But if you actually start reading the document, the first sentence is a quote from the president that says, "Today, a new frontier of scientific discovery lies before us." I love that they led with something inspirational. Because ultimately, the technology has to confer some benefits on humanity. I personally love the fact that we are just starting to explore what these frontier models mean for scientific discovery in physics, chemistry, and material science. We need to inspire the next generation to want to go into those areas because it's hard. It's really hard to do AI in the physical world. You have to hook up wet labs and start doing experiments in an entirely new way. You need people who are excited not only about machine learning work, but also the hard work of being lab technicians, running experiments, and pipetting new materials and chemistry. That was missing in a lot of the discourse under the previous administration. You can sometimes judge a book by its cover, and I think this was a strong start. Now we should dive into some of the bullets.
--:-- Host (Erik Torenberg): The other one that I thought was a huge omission is there's basically no real mention of investing in academia. There are some oblique references to it, but it's been such a mainstay of innovation in computer science for the last 40 years. Not having a major part of it is a shame. I understand there's a standoff between higher ed and the administration right now. I get it, and I actually think both sides have fairly reasonable points. But to have a major tech initiative without including academia feels like fighting a battle with a hand tied behind our back.
--:-- Guest (Martin Casado): This is a good problem to have, which is that I think it's extremely ambitious. It's a little bit light on execution details—that's what happens next. A good example: I do think directionally it was great that they said, let's build an AI evaluations ecosystem. I love that because it acknowledges that before we start passing grant proclamations about whether these models are risky or dangerous, let's first agree on how to measure the risk in these models. That part, in addition to the open source bullet, was probably the most sophisticated thinking I've seen in any policy document. The reality is America leads the way. Within 24 hours of this dropping, Marti and I were getting texts from folks in other governments around the world asking, "What do you guys think?" It was not hard for me to endorse it and tell them to look at it as a reference document. There are things here that are arguably more sophisticated than policy experts even in Silicon Valley would recommend. Building an AI evaluation ecosystem is not easy, and they lay out a pretty thoughtful proposal on why that's important. Now the question is how, and that's what we have to help DC with—the hard work of implementing this stuff. The vibe shift from "let's not jump the gun on saying these models are dangerous" to "let's first talk about building a scientifically grounded framework on how to assess risk" was not at all a given. I was really excited about that.
--:-- Host (Erik Torenberg): There's been a lot of focus in the last few years by several companies, but also by the brand industry, around this idea of alignment. Have we made any progress on alignment? What is your assertive perspective? What are they trying to do? Is that a feasible goal? Help us understand what they're trying to solve for.
--:-- Guest (Martin Casado): At an almost tautological level, alignment is an obvious thing we'd want to do. I have a purpose, I want to align the AI to this purpose. These models are problematic, generally unruly, chaotic—whatever adjective you want to use. Understanding how to better align them to any stated goal is very obviously a good thing. I think we'd all agree that alignment to whatever the goal is—to make it more effective at that goal—is good, especially given these models tend to have a mind of their own. The subtext that I bristle at is that the people doing the alignment are somehow protecting the rest of us from whatever they think their ideal is, in terms of dangers to me or thoughts I shouldn't have or information I shouldn't be exposed to. That's why I think we need to be careful, even when we come up with policy, not to impose a different set of ideological rules on top of these. Alignment is something we should all understand. Actually aligning them to me is where I take issue from any sort of top-down mandate.
--:-- Guest (Anjney Midha): I agree. There's a quote from a researcher that I think is very accurate: you've got to think about these AI systems as almost biological systems that are grown, not coded up. They express software, but in many ways, when you're training a model, you are actually growing it in an environment of prior history and training data. Often empirically, you don't know what the capabilities of the model are until it's done training. I think that's a useful analogy. Where I think it falls down is when people say, "If we can't align it because we don't know—it's a biological mechanism until it's grown up—then we can't deploy these AI models in mission-critical places until we've solved the black box problem, the mechanistic interpretability problem." That's the ability to trace deterministically why a model did something. We've made a lot of advances in the last few years, but it remains a research problem. That doesn't mean just because you don't understand the true mechanism of the system, you don't unlock its useful value. Most general-purpose technologies in history—electricity, nuclear fusion—there are many examples where we knew they were complex systems and we didn't truly understand at an atomistic or mechanistic level how they worked, but we still used them. We don't understand how the internet works. There's a whole research field of network measurements trying to find out what the internet was going to do. Any complex system has states you just don't understand. These models more so than many, and the implications are very real, but we know how to deal with ambiguity.
--:-- Guest (Martin Casado): We don't even know how our own brains work.
--:-- Guest (Anjney Midha): Or consciousness. But we don't stop working with other human beings.
--:-- Guest (Martin Casado): Unfortunately, we're stuck with them. We have no option on that one.
--:-- Guest (Anjney Midha): Totally. To extend that analogy, what do you do? You say, "I don't know how a brain works. It's got a bunch of risks. This person may be crazy, but I still want to unlock all the beautiful benefits of big, beautiful brains that humans have." So you develop education. You send kids to school. You teach them values. Then they go to college, learn something specific, and get tested in the real world. They get a resume, work experience, and prove they are within a risk-based framework that's manageable. That as a society has unlocked human capital—arguably the greatest technology we've had in 500 years of modern industrial innovation. What I hate about the alignment discourse is it sometimes confuses the fact that we don't understand the system with the idea that we can't use it. Mechanistic interpretability—reverse engineering why a model does something—is still a research problem. But that doesn't mean we haven't made progress on how to use unaligned models or improve alignment to a point where they're useful in massive ways, like software engineering.
--:-- Host (Erik Torenberg): I think what the smartest might say is it's not that, it's really just "what's the rush?" Maybe let's focus on integrating all the capabilities we already have before pushing the frontier. There's a risk of slowing down too that maybe isn't fully appreciated.
--:-- Guest (Martin Casado): Until we've solved cancer, every month that we're not rushing to the frontier of accelerating biological discovery or scientific progress is a month that millions of people are suffering from disease that we could be solving with AI. We don't talk about the opportunity cost of slowing down the frontier. This is the thing with all of these questions on innovation. There's the Bostrom urn experiment—the whole urn hypothesis is there's an urn of innovation, and you pull out balls. One of them is a black ball that destroys everything. So eventually you'll draw that ball. Why would you ever do innovation? That's the thought experiment. The answer is so simple: it turns out it's much more dangerous not to pull out balls than to pull out balls. That's always the answer. When people ask about PDOOM—probability of doom—the answer is not 0.1 or 0 or 100. The answer is the PDOOM without AI is actually quite a bit greater than the PDOOM with AI. And "what's the rush?" The answer is the same thing. If you ignore the benefits of technology, you would say if it's all negative, no rush at all. The reality is the benefits are so dramatic, and they're clearly economically beneficial now. Thank God we've got a year's worth of data on this stuff. They're clearly beneficial in a number of areas of core science. The rush is getting to the next set of solutions, as opposed to being afraid of a set of problems we still can't clearly articulate. As soon as we do understand marginal risk and we do have those problems, we absolutely should address them directly. The action plan does a great job of penciling this out. It wants to explore implications on jobs, defense, and alignment. That's exactly where we should be—in the exploration phase.
--:-- Host (Erik Torenberg): Do we have a definition of marginal risk or a perspective of how to think about that idea?
--:-- Guest (Martin Casado): Let's be clear what we mean by marginal risk. Computer systems are risky. Network systems are risky. Stochastic systems are risky. We've got decades of ways of thinking about measuring, regulating, and changing common behavior based on this type of risk. The question is, can you take all that apparatus that's been hard-won and apply it to AI? If so, we know it's effective because we've used it before and we've got a lot of experience with it. It's ready to be done. Or is there a different type of risk that's not endemic to those systems? In which case, we'll have to come up with something new. You go down that exploration—maybe it works, maybe it doesn't. That's what marginal risk is. The problem is, if you don't know what it is, how are you going to define a solution?
--:-- Guest (Anjney Midha): I think that's right. Philosophically, the idea is if you're going to say we need new solutions, then you need to articulate why the problem is new and why our existing solutions that work really great are no longer sufficient. It's almost obvious when you state it, but this was the state of the world a year ago. We were having to look around the room and say, "Can I raise my hand? Why are we introducing net new liabilities and new laws that we've never had to do before if you can't articulate why there are new problems to solve?" If it ain't broken, why are you trying to fix it? Marginal risk is a slightly more technical way to say we have the tools to manage risk. We don't need new ones. If you think we need new ones, then take a minute to articulate why.
--:-- Host (Erik Torenberg): Is there anything else you wanted to make sure we got to? Otherwise, let's wrap up. I think it was great. Time to put the action plan into action. Excellent. Martin, Anjney, thanks so much for coming to the podcast.
--:-- Guest (Martin Casado): Thank you, that was great.
--:-- Guest (Anjney Midha): Thanks for having us.
--:-- Host (Erik Torenberg): Thanks for listening to the a16z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com slash a16z. We've got more great conversations coming your way. See you next time. Thank you.
