--:-- Host: Before we dive into the substance of the California Senate Bill 1047, can you start with giving your high-level reaction to the bill and then maybe give listeners a sense of why it's such a big deal right now?
--:-- Anjney Midha: Shock, disbelief. It's hard to understand just how blindsided startups, founders, the investor community that have been heads down building models, building useful AI products for customers, paying attention to what the state of the technology is and ultimately just innovating at the frontier, these folks, the community broadly feels completely blindsided by this bill.
--:-- Anjney Midha: When it comes to policymaking, especially in technology at the frontier, the spirit of policymaking should be to sit down with your constituents, startups, founders at the frontier, builders, and then go solicit their opinion. And what is so concerning about it right now is that this bill, SB 1047, was passed in the California Senate with a 32 to 1 overwhelming vote, bipartisan support. And now it's headed to an assembly vote in August, less than 90 days away, which would turn it into law.
--:-- Anjney Midha: And so if it passes in California, it will set the precedent in other states. It will set a nationwide precedent. And ultimately, that'll have rippling consequences outside of the U.S. to other allies and other countries that look to America for guidance and for thought leadership. And so what is happening here is this butterfly effect with huge consequences on the state of innovation.
--:-- Host: There's a lot to get into with the proposed law and some of its shortcomings or oversights. But the place I want to start is both SB 1047 and President Biden's executive order from last year established mandatory reporting requirements for models that are trained, and this is a little difficult to speak, but bear with me, listeners, that are trained on 10 to the 26 integer floating point operations per second, or FLOPs is the acronym of compute power. So can you explain to listeners what FLOPs are and why they're significant in this context?
--:-- Derrick Harris: Right. So flops in this context refers to the number of floating point operations used to train an AI model. And floating point operations are just a type of mathematical operation that computers perform on real numbers as opposed to just integers. And the amount of flops used is a rough measure of the computing resources and complexity that went into training a model.
--:-- Derrick Harris: And so if models are like cars, flops might be the amount of steel used to make a car, to borrow an analogy. It doesn't really tell you much about what the car can and cannot do directly, but it's just one way to kind of measure the difference between the steel required to make a sedan versus a truck. And this 10 to the 26 flop threshold is significant because that's how the bill is trying to define what a covered model is.
--:-- Derrick Harris: It's an attempt to define the scale at which AI models become potentially dangerous or in need of additional oversight. And this all starts from the premise that foundation models trained with this immense amount of computation are extremely large and capable to the point where they could pose social risks or harm inherently, if not developed carefully. But tying regulations to some fixed flop count or equivalent today is completely flawed because algorithmic efficiency improves, computing costs decline, and so models that take far fewer resources than 10 to the 26 flops will match the capabilities of a 10 to the 26 flop model of today within a fairly short timeframe.
--:-- Derrick Harris: So this threshold would quickly expand to cover many more models than just the largest, most cutting edge ones being developed by tech giants. It will basically cover most startups in open source too within a really short amount of time. And so while today in 2024, realistically, only a handful of the very largest language models like GPT-4, Gemini, and other top models from big tech companies are likely to sit above that 10 to the 26 flop threshold. In reality, most open source and academic models will soon be covered by that definition as well. This would really hurt startups. It would burden small developers and, ironically, it's going to reduce the transparency and collaboration around AI safety by discouraging open source development.
--:-- Derrick Harris: What we see frequently is people in labs going out there and saying we going to build big state models that cost less to train that use fewer resources that use more data or different types of data. There are all these different knobs to pull to get performance out of these models. It seems like you could have this sort of performance for a fraction of the cost in a small number of years. Right.
--:-- Anjney Midha: So that all comes down to two key trends. One, the falling cost of compute, and two, the rapid progress in algorithmic efficiency. Empirically, the cost per flop for GPUs is halving roughly every two to two and a half years. And so this means that a model that costs about $100 million to train today would only cost about $25 million in about five years, and less than $6 million in a decade, just based on hardware trends alone, just Moore's Law. But that's not even the whole story, right? Algorithmic progress is also making it dramatically easier to achieve the same benchmark performance with way less compute rapidly.
--:-- Anjney Midha: And so when you look at those trends, we observe that the compute required to reach a given benchmark of reasoning or capability is decreasing by half about every 14 months or less. So if it takes 100 million worth of flops to reach some given benchmark today, in five years, it would only take around $6 million worth of flops to achieve that same result, just considering the algorithmic progress alone. Now, when you put these two trends together, it paints a pretty stunning picture because the cost to reach any given benchmark of reasoning of capability is dropping by about 50 times every five years.
--:-- Anjney Midha: And so that means that if a model costs $100 million to train to some benchmark in 2024, by 2029, it will probably cost less than $2 million. That's well within a startup budget. And by 2034, a decade, that cost will drop to somewhere between $40,000, $50,000, putting it within the reach of literally millions of people. And despite these clear trends, the advocates for the bill seem to be overlooking or underestimating this rapid progress. Some folks are suggesting that, oh, these smaller companies might take 30 euros or more to reach this 10 to the 26 flop threshold. But as we've just discussed, that's a pretty serious overestimation.
--:-- Anjney Midha: So even assuming a model costs a billion dollars to train to that level today, it's going to cost as little as 400,000 in just a decade. This is easily within the range for most small businesses who are going to then have to grapple with compliance and regulation and so on. And so look, the bottom line is that given the breakneck pace of progress and compute costs and efficiency, we can expect smaller companies and academic institutions to start hitting these benchmarks in the very near future.
--:-- Derrick Harris: Yeah, I think it's a relevant touch point to remind people that a smartphone today, like an iPhone 15, has more flops, more performance than a supercomputer did about 20 years ago. Like the world's fastest supercomputers. Your iPhone can do more flops than that. The Apple Macintosh G4, I think back in 1999, had enough computing power that it would have been regulated as a national security threat.
--:-- Derrick Harris: So these numbers, these are very much sliding scales to your point. That's right. That's right. That's a great historical example. I think there was this 1979 Export Administration Act, right? That the U.S. had written in the Cold War era in the 70s. And the definitions for what was dangerous and the Cold War became obsolete so fast that a couple of decades later when the Macintosh launched, it was technically a munition.
--:-- Derrick Harris: And so we've been here before and we know that when policymakers and regulators try to capture the state of a current technology that's dramatically improving really fast, they become obsolete incredibly fast. And that's exactly what's happening here.
--:-- Host: The other thing is, at the time we're recording this, there are some proposed amendments floating around to SB 1047, one of which would limit the scope of the bill to applying, again, only to models trained at that compute capacity. And additionally, that also cost more than $100 million to train. So what's your thought on that? And again, if we attach a dollar amount to this, doesn't it make the compute threshold kind of obsolete?
--:-- Anjney Midha: Yeah. So this $100 million amendment to train might seem like a reasonable compromise at first, but when you really look at it, it has the same fundamental flaws as the original flop threshold. The core issue is that both approaches are trying to regulate the model layer itself, rather than focusing on the malicious applications or misuses of the models.
--:-- Anjney Midha: Generative AI is still super early, and we don't even have clear definitions for what should be included when calculating these training costs. Do you include the data set acquisition, the researcher salaries? Should we include the cost of previous training runs or just the final ones? Should human feedback for model alignment expenses count? If you fine-tune someone else's model, should the costs of the base model be included? These are all open questions without clear answers.
--:-- Anjney Midha: And forcing startups, founders, academics to provide legislative definitions for these various cost components at this stage would place a massive burden on these smaller teams, many of whom just don't have the resources to navigate these super complex regulatory requirements. Plus, when you just look at the rapid pace of model engineering, these definitions would need to be updated constantly, which would be a major drain on innovation.
--:-- Anjney Midha: So when you combine that ambiguity with the criminal and monetary liabilities proposed in the bill, as well as the broad authority they're trying to give to the new frontier model division, which is sort of like a DMV for AI models that they're proposing, which can arbitrarily decide these matters, the outcome is clear, right? Most startups will simply have to relocate to more AI-friendly states or countries, while open source AI research in the US will be completely crushed due to the legal risks involved.
--:-- Anjney Midha: So in essence, the bill is creating this disastrous regressive tax on AI innovation. Large tech companies that have armies of lawyers and lobbyists will be able to shape the definitions to their advantage, while smaller companies, open source researchers, and academics will be completely left out in the cold. It's almost like saying we've just invented the printing press. And now we're only going to let those folks who can afford $100 million budgets to make these printing presses decide what can and cannot be printed. It's just blatant regulatory capture.
--:-- Anjney Midha: And it's one of the most anti-competitive proposals I've seen in a long time. And what we should be focusing on instead is regulating specific, high-risk applications and malicious end users. That's the key to ensuring that AI benefits everyone, not just a few.
--:-- Host: Now, you've mentioned that the purported goal of some of these bills, 1047 in particular, is to prevent against what you might call catastrophic harms or existential risks from artificial intelligence. But I'm curious, do you think, I mean, are the biggest threats from LLMs really weapons of mass destruction or bioweapons or autonomously carrying out criminal behavior? I mean, if we're going to regulate these models, I mean, should we not regulate use cases that are like proven in the wild and can actually do real damage today versus hypothetically at some point these things could happen?
--:-- Anjney Midha: Absolutely. I mean, basically what we have is a complete over rotation of the legislative community around entirely non-existent concerns of what is being labeled as AI safety, when what we should be focusing on is AI security. These models are no different than databases or tools in the past that have given humans more efficiency, better ways to express themselves. They're really just neutral pieces of technology.
--:-- Anjney Midha: Now, sure, they may be increasing or allowing bad actors to increase the speed and scale of the attacks, but the fundamental attack vectors remain the same. It's spear phishing, deep fakes, it's misinformation. And these attack vectors are known to us. And we should focus on how to strengthen enforcement and give our country better tools to enforce those laws in the wake of increasing speed and scale of these attacks.
--:-- Anjney Midha: But the attacks themselves, the attack vectors haven't changed. It's not like AI suddenly has exposed us to tons of new ways to be attacked. And that's just so far off and frankly unclear, and today largely in the realm of science fiction. And so the safety debate often centers around what is called existential risk or these models autonomously going rogue to produce weapons of mass destruction or the Terminator Skynet situation where they're hiding their true intentions from us.
--:-- Anjney Midha: And sure, maybe there's some theoretically tiny likelihood that that happens many, many, many years from now, but exceptional claims require exceptional evidence. And so the real threat here is from us not focusing on the misuses and malicious users of these models and putting the burden of actually doing that on startups, on founders, on engineers.
--:-- Derrick Harris: Right. And to your point, even if a model made it marginally easier to learn, let's say, how to build a bioweapon, like one, people know how to do that today. We have all of these things. We have labs dedicated to all of these things. You still need to get materials to carry out these attacks. And there are regulations around acquiring those materials and databases around who's buying what.
--:-- Derrick Harris: Exactly. Exactly. I think what we really need is more investment in defensive artificial intelligence solutions. What we need is to arm our country, our defense departments, our enforcement agencies with the tools they need to keep up with the speed and scale at which these attacks are being perpetuated, not slowing down the fundamental innovation that can actually unlock those defensive applications.
--:-- Derrick Harris: And look, the reality is America and her allies are up against a pretty stiff battle from adversarial countries around the world who aren't stopping their speed of innovation. And so it's almost an asymmetric warfare against ourselves that's being proposed by SB 1047.
--:-- Anjney Midha: Yeah, I'm certain there are governments that would in fact fund those $100 million models. Well north of that, right? Yeah. And we have increasing evidence that this is happening and that our national security actually depends on improving and accelerating open source collaboration.
--:-- Anjney Midha: So just two months ago, the Department of Justice revealed and published a public investigation, the conclusion of which was that a Google engineer was boarding a plane to China with a thumb drive with frontier AI hardware schematics from Google. This was a nation-state-sponsored attack on our AI ecosystem. And the only defense we have against that is actually making sure that innovation continues at breakneck speed in the country, not adding more burden to model innovation.
--:-- Host: The other thing that SB 1047 would do, which we haven't really touched on, is impose liability, civil and in some cases criminal liability, on model developers for the civil liability part. If they build a model that's covered by this bill, they need to be able to prove with beyond reasonable assurance or whatever the language is that this could not possibly be used for any of these types of attacks. And also they have to be able to prove that no one else could come along and say fine tune their model and use it for some sort of attack, right? So that's a whole new level to be on the hook for money as an individual or jail time as an individual for building this model and not making it quote unquote safe enough.
--:-- Anjney Midha: Oh, no, you're absolutely right. The idea of imposing civil and criminal liability on model developers when downstream users do something bad is so misguided and such a dangerous precedent. First off, the bill requires developers to prove that their models can't possibly be used for any of the defined hazardous capabilities. But as we just discussed, these definitions are way too vague, ambiguous, and subject to interpretation. How can a developer prove a negative, especially when the goalposts keep moving. It's an impossible standard to meet.
--:-- Anjney Midha: Second, the bill holds developers responsible for any misuse of their models, even if that misuse comes from someone else who's fine-tuned or modified the model. It's ridiculous. It's like holding car manufacturers liable for every accident caused by a driver who's modified their car. So it's an absurd standard that no other industry is held to.
--:-- Anjney Midha: And the practical effect of these liability provisions will be to drive AI development underground or offshore, no rational startup founder or academic researcher is going to risk jail time or financial ruin just to advance the state of the art in AI. They'll simply move their operations to a jurisdiction with a more sensible regulatory environment and the US will lose out, period.
--:-- Anjney Midha: The worst part, these liability provisions actually make us less safe, not more. By driving AI development into the shadows, you lose the transparency and open collab that's essential for identifying and battle-hardening vulnerabilities in AI models. What we need is more open source development, not less.
--:-- Anjney Midha: So while the bill sponsors may have good intentions, imposing blanket liability on model developers for hypothetical future misuse is the exact opposite of what we need. Supporters might argue, well, we put some behind bars for lying to the government for lying about the capabilities of their models. But again, you might not know the capabilities of your models or what a downstream user could do with that.
--:-- Host: I wanted to ask you too, because you've built startups, you invest in startups. I mean, can you walk through the kind of wrench this type of compliance would throw into, whether it's the finances or the operation or just the general way that startups and innovative companies work?
--:-- Anjney Midha: Oh yeah, look, I love California and that's why I'm fighting so hard for this. I did my undergraduate and graduate work here in the Bay. I founded my first company here. I sold that to another California company. And over the last decade plus that I've been here, it's only become more and more clear to me that a huge part of what makes the entire startup AI ecosystem even work is the ability for founders to take bold technology risks without having to worry about the kinds of ambiguity and liability risks that this bill is proposing.
--:-- Anjney Midha: When we first started Ubiquity 6, my last company, the goal was to empower developers to use our computer vision pipeline for all kinds of new use cases that we hadn't even imagined. We had some idea of what people would do, originally augmented reality applications. But after we launched it, we found millions of users who used our 3D mapping technology for entirely new kinds of uses, from architecture and robotics to VFX and entertainment that we hadn't even considered.
--:-- Anjney Midha: And so the whole engine and the beauty of platform businesses is that developers can focus on developing general and highly flexible technology, and then just let the market figure out entirely new niche use cases at scale. And this is true of almost every great AI business I've either worked with directly or invested in, right? Whether it was mid-journey and image generation and anthropic and language models or 11 labs and audio models.
--:-- Anjney Midha: Great technologies always find their way into downstream uses that the original developers would have had no way of knowing about prior to launch. And to burden that process with the liability of this bill of saying that developers have to somehow, prior to launch, demonstrate beyond any shred of reasonable doubt, which is, again, a completely ambiguous definition in the bill, that these users were known about, their risks were understood, that exhaustive safety testing had been done to make sure none of these things would be possible, just completely killed that engine.
--:-- Anjney Midha: If we went back in time and this bill passed, as much as I hate to say it, there's no chance I would have founded my company in California. Speaking of startups that to say nothing about open source projects and open source development which have been like a huge driver of innovation over the past couple of decades We talking about very very bootstrapped skeletal budgets on some of these things but hugely hugely important.
--:-- Anjney Midha: Fundamentally, I don't think the current wave of modern generative scaling laws-based AI would even exist without open source. If you just go back and look at how we got here, as formers, the atomic unit of how these models learn, was an open source widely collaborated on development. In fact, it was produced at one lab, Google, and allowed another lab after open publishing and collaboration, which was OpenAI, to actually continue that work.
--:-- Anjney Midha: And there's no chance we'd be here without open source. The downstream contributions of open source continue to be massive today. When a company like Mistral or Facebook open source models and release their weights, that allows other startups to then pick up on their investments and build on top of them. It's like having the Linux to the closed-source Windows operating systems. It's like having the Android to the closed-source iOS. And without those, there's no chance that the speed at which the AI revolution is moving will continue.
--:-- Anjney Midha: Certainly not in California and probably not in the United States. Open source is kind of the heart of software innovation, and this bill slows it down, has a chilling effect on open source by putting liability on the researchers and the builders pushing open source forward.
--:-- Host: Yes, and the other thing about open source is, I guess this is true of any model, theoretically, but the idea is someone takes it and builds on it, right? In AI, in generative AI or foundation models, you would call that fine-tuning, right? Where you kind of retrain a model to your own purposes using your own data. And again, this bill would, as written, impose liabilities, again, on the original developers if someone is able to fine-tune their model to perform, theoretically, some sort of bad act, right?
--:-- Anjney Midha: I mean, how realistic is it for someone to even build a model that would be resistant or resilient against these types of fine-tuning attacks or optimizations, for lack of a better term?
--:-- Derrick Harris: Yes. So this is another can of worms as well. Again, a symptom of the root cause of this bill's flawed premise of regulating models instead of misuses. So in the current bill draft, the language says that these restrictions and regulations will extend to a concept of a derivative model, which is a model that is a modified version of another model, such as a fine-tuned model.
--:-- Derrick Harris: So if someone makes a derivative model of my base model that's harmful, I am now liable for it. It's akin to saying that if I'm a car manufacturer and someone turns a car I made into a tank by putting guns on it and shoots people with it, I should get thrown in jail. The definition of what a derivative model is also super vague. And so now the bill sponsors are considering an amendment that says, oh, let's add a compute cap to this definition.
--:-- Derrick Harris: And they've decided to pick 25%, which is quite arbitrary. And to say, if somebody uses more than 25% of the compute that the base model developer used to fine tune a model, then it's no longer a derivative model and you're off the hook for it as the base model developer. Well, that's absolutely nonsensical as well.
--:-- Derrick Harris: As some great researchers like Ian Stoica at Berkeley have shown, it takes an extremely small amount of compute to fine-tune a model like Vicuna, where with just 70,000 shared GPT conversations, they fine-tune Lama to become one of the best open source models at the time, showing it really doesn't take much computer data to turn a car into a tank, to borrow an analogy.
--:-- Derrick Harris: And so like with the 10 to the 26 compute threshold issue we discussed earlier, This is just another arbitrary magic number the bill authors are pulling out of thin air to try and define model layer computing dynamics that are so early and changing that it's absolute overregulation and will kill the speed of innovation here.
--:-- Host: All right. So you've alluded to this, but I wanted to ask directly, if we say not all regulation is bad, if you were in charge of regulating AI, how would you approach it? Or how would you advise lawmakers who feel compelled to address what seem like concerns over AI? What would be your approach?
--:-- Anjney Midha: Non-negotiable really here should be zero liability at the model layer. What you want to do is target misuses and malicious users of AI models, not the underlying models and not the infrastructure. And that's the core battle here. I think that's the fundamental flaw of this bill, is it's trying to regulate the model and infrastructure and not instead focus on the misuses and malicious users of these models.
--:-- Anjney Midha: And so over time, I think it would prove out that the right way to keep the U.S. at the frontier of responsible, of secure AI innovation is to actually focus on the malicious users and misuses of models, not slow down the model and infrastructure layer. We should focus on concrete AI security and strengthening our enforcement and our defenses against AI security attacks that are increasing at speed and scale. But fundamentally, these safety concerns that are largely science fiction and theoretical are a complete distraction at the moment.
--:-- Anjney Midha: And lastly, we have no choice but to absolutely accelerate open source innovation. We should be investing in open source collaboration between America and our allies to keep our national competitiveness from falling behind our adversarial countries. And so the three big policy principles I would look for from regulators would be to regulate and focus and target misuses, not models, to prioritize AI security over safety, and to accelerate open source.
--:-- Anjney Midha: But the current legislation is absolutely prioritizing the wrong things and is rooted in a bunch of arbitrary technical definitions that will be outmoded, obsolete, and overreaching fairly soon. One might say we should regulate it the same way we regulated the internet, which is to say, let it thrive. It really is tantamount to saying we've barely just invented the printing press, or we've barely just invented the Model T Ford car. And now what we should immediately do is try to rush and prevent future improvements to cars or to the printing press by largely putting the responsibility of any accidents that happen from people irresponsibly driving the car out on the streets on Henry Ford or of the inventors of the printing press.
--:-- Host: So then the final question here, taking everything into account, what can everyday listeners do about this? I mean, if I'm a founder, if I'm an engineer, if I'm just concerned, what can I do to voice my opinion about SB 1047, about frankly any regulation coming down the line? And how should people think about making their voice heard?
--:-- Anjney Midha: Yeah, so I think three steps here. The first would be to just read the bill. It's not very long, which is good. But most people just haven't had a chance to actually read it. Step two, especially for people in California, the most effective way to have this bill be opposed is for each listener to call their assembly rep and tell them why they should vote no on this bill in August. Right. This is less than 90 days away. So we really don't have much time for all of the assembly members to hear just how little support this bill has from the startup community, tech founders, academics.
--:-- Anjney Midha: And step three is to go online, you know, make your voice heard on places like Twitter, where it turns out, you know, a lot of both state level and national level legislators do listen to people's opinions. And so, look, I think if this bill passes in California, it sure as hell is going to create a ripple effect throughout other states. And then this will be a national battle. If you like this episode, if you made it this far, help us grow the show. Share with a friend or if you're feeling really ambitious, you can leave us a review at rate this podcast dot com slash A16C.
--:-- Anjney Midha: You know, candidly, producing a podcast can sometimes feel like you're just talking into a void. And so if you did like this episode, if you liked any of our episodes, please let us know. We'll see you next time.
