AI Revolution: Top Lessons from OpenAI, Anthropic, CharacterAI, & More

Host: Look, this is the beginning of something amazing because there's no limit. This is right now an inflection point where we're sort of redefining how we interact with digital information. These are the fastest-going open-source projects. These are the fastest-going products and some of the fastest-going companies we've seen in the history of the industry.

We, for a long time, really focused on building our own infrastructure. We have hundreds of thousands of servers. He said, well, I think we can get by with like 500. I said, okay, I think we can find 500k somewhere. And I remember you deadpan saying, dude, I'm talking about $500 million.

The internet was the dawn of universally accessible information, and we're now entering the dawn of universally accessible intelligence. The AI revolution is here. But as we collectively try to navigate this game-changing technology, there are still many questions that even the top builders in the world are grappling to answer. And that is why A16Z recently brought together some of the most influential founders from OpenAI, Anthropic, CharacterAI, Roblox, and more, to an exclusive event called AI Revolution in San Francisco recently.

And in today's episode, we share the most important themes from this event, starting with the economics of AI, but we also touch on broad versus specialized models, and which ultimately may win, the importance of UX, and also whether we can expect scaling laws to continue. By the way, several founders comment on what they're seeing there, including Noam Shazir, lead author of the preeminent Transformer paper from back in 2017.

Now, I won't delay us any longer, other than saying we've got a lot more coverage of this event coming, including how AI is disrupting everything from games to design, how two important waves in machine learning and genomics are colliding, and what we can expect from the enterprise. But in the meantime, if you would like to listen to all the talks in full today, you can head on over to a16z.com/airevolution.

As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.

All right, let's start with Martin Casado, general partner at A16Z, giving the why now, and also how the economics of the space may finally be coming together. I will give you the punchline up front. The punchline is if you've ever wanted to start a startup or join a startup, now is a great time to do it. But how early are we in the trajectory of this technology? For example, the microchip was invented in the late 50s. But it wasn't until the turn of the century when Steve Jobs famously put a thousand songs in your pocket. So just how much opportunity is still on the table?

Okay, so what has the narrative been for AI over the last 50 years? The narrative is this episodic thing with summers and winters and all of these false promises. I remember when I joined for my PhD in 2003, for my cohort that joined, I would say 50% of the people were doing AI. This was like when Bayesian stuff was super popular. And then within three years, everybody's like, AI is dead, right? And so it's kind of been this love-hate relationship with it for a very long time. But if you look at all of the graphs, we've made a tremendous amount of progress in the last 70 years. And along the way, we've solved a lot of very real problems, right? Like way back in the 60s, we were doing expert systems, which are still used for diagnosis, right? Like we're very good at beating Russians at chess. You know, we're doing self-driving cars, we're doing vision. There's just a lot of stuff that we've solved. And so much so it's become a cliche that every time we solve a problem, we're like, oh, well, that wasn't really AI, right? So we just keep moving the goalposts.

So we've had steady progress. We've solved real problems. And not only that, it's been a while now that we've been better at humans for some very important things. For example, like, you know, perception or handwriting detection. It's been about 10 years since we've been better than humans at entity identification. And not only that, we've actually gotten very good at monetizing this particularly for large companies, right? And so as we all know, there's been a ton of market cap that's been added to companies like Meta and Google and Netflix by using AI. So I think the question we should all ask ourselves is why hasn't this resulted in an actual platform shift? And by platform shift, I mean, why has the value accrued to the incumbents? And why hasn't there been a new set of kind of AI-native companies that have come up and displaced them, which we've seen in many other areas, right? We saw that in mobile. Obviously, we saw it with the microchip, et cetera.

What I'm going to argue is that the capabilities have all been there, but the economics just haven't for startups. So if you step back and you look at the standalone case for AI economics, not like what a big company can extract from it, but a startup, it's actually not been great. I mean, to begin with, a lot of the sexier use cases are pretty niche markets, right? Like, you know, it's great to beat Russians at chess. Like, maybe it's a useful tool that you can apply to solving bigger problems, but that not itself is a market. I actually think the second point is the most important point, and it's pretty subtle. Many of the traditional use cases of AI require correctness in the tail of the solution space. And that's a very hard thing for a startup to do for a couple of reasons. One of the reasons is if you have to be correct and you've got a very long and fat tail, either you do all of the work technically or you hire people. So often we hire people, right? And for startups to start hiring people to provide solutions is a variable cost. And the second one is because the tails of these solutions tend to be so long, think something like self-driving, where there's so many exceptions that could possibly happen, the amount of investment to stay ahead increases and the value decreases, right? You have this perverse economy of scale. So we've actually done studies on this, and it turns out many companies that try to do this as startups end up with non-software-like margins. They're lower margins, and they're just much harder to scale. Of course, with robotics comes the curse of hardware, classically a very difficult thing for startups to do. And if you really think like, what is the competition of most use cases of AI? It tends to be the human. And traditionally, it's stuff like the human brain is really good at like perception, right? Like the brains that we have have evolved over 100 million years to do things like whatever, pick berries and evade lions or whatever it is. And it's incredibly efficient at doing that. So this leads to something that most investors know, which we call the dreaded AI mediocrity spiral. And what is it? It's very simple, which is, let's say a founder comes in and they want to do an AI company and they're going to use AI to automate a bunch of stuff. Of course, correctness is really important and they want it to look good at first. So they hire people to do it instead of the AI. Then they come to us, we invest in them and I joined the board. Then I say, listen, this is great. You need to grow. And they're like, oh man, we need to grow. This AI is hard, like the tail is very long. I'm going to hire more people. And now you're on this treadmill of continuing hiring people. And this is one of the reasons why so many startups that have tried to do this just haven't had kind of this breakaway economics and the value accrues to large companies that can actually sink these perverse economies of scale. But market transformations aren't created when economics get 10 times better. They get created when they're 10,000 times better. So what is the learning from the last, say, 70 years? It's not that the technology doesn't work. It's not that we can't solve the problems. It's not even that we can't monetize it. Big companies are great at monetizing it. It's that it's very, very hard for startups to break away. And if startups can't break away, you don't get a transformation. But what about the current wave, where the everyday consumer can prompt LLMs with natural language and have it output a variety of things, from conversations to images to even 3D models? So this wave is very, very different, and we're already seeing productive, viable businesses, right? Like, I like to call them the three Cs. There's creativity, like any component of a video game you can automatically generate. There's companionship, which kind of more of emotional connections. And then there's a class that we call Copilot, which will help you with tasks. These are already emerging as independent classes. So remember the properties of AI previously that made it difficult to build a startup company. So none of these really apply to this current model. The first one, obviously, these are large markets that this is being applied to. It's like arguably all of white-collar work. Even just like video games and movies, it's like $300 billion market. These are massive, massive markets. The second one, again, I think is, you know, the most important point and maybe the most subtle. In this domain, correctness isn't as much of an issue for two reasons. One of them is when you're talking about creativity, the first C, there is no formal notion of correctness, really. And what does it mean to be incorrect for like a fiction story or a video game? I mean, for sure, like you want to make sure like they have all their fingers, but even then, do you really in sci-fi? And so we have absolutely adapted to use cases where, you know, correctness is not a huge issue. The second one is a little more subtle, I just think it's so important, which is the behavior that's developed around these things is iterative. And so that human in the loop that used to be in a central company is now the user. So it not a variable cost to the business anymore. The human in the loop has moved out. And as a result, you can do things where correctness is important like for example developing code because it iterative. So like you constantly getting feedback and correction from the user. And I want to talk to you about this brain portion because I think it's so interesting. I'm not a neuroscientist, but for these types of tasks, the silicon stack is way better than the carbon stack, right? So, if you think about it, like, traditional AI, a lot of it is doing stuff like the 100 million-year-old brain is doing, right? The one that's been fleeing predators or like picking strawberries or whatever it is. And that's very, very hard to compete with. Like remember, if you have the CPU GPU set up a self-driving car, some of these kits are like 1.3 kilowatts where the human brain is 15 watts. So economically, that's very tough to compete with. The new gen iWave, it's kind of competing like the creative language center of the brain. It's like 50k years old. Like it's much less evolved. And it turns out it's incredibly competitive, so much so that you actually have the economic inflection we look for for a market transformation. So let's just go ahead and break down the numbers very quickly. So let's say that I, Martin, wanted to create an image of myself as a Pixar character, right? So if I'm using one of these image models, the inference cost, let's call it, you know, a tenth of a penny. It's probably less than that, actually. Let's say it takes one second. If you compare that to hiring a graphic artist, let's say that it was a hundred bucks in an hour. I've actually hired graphic artists to do things like this. It tends to be a lot more money than that, but let's conservatively say that. You've got four to five orders of magnitude difference in cost and time. So these are the type of inflections you look for, certainly as an economist, when it's like there's going to be actually a massive market dislocation. I'll give you another example from Instabase. So let's assume that you have a legal brief. It's in a PDF. You throw it into this kind of unstructured document LLM, and then you ask questions for that legal brief. Again, the inference costs, say a tenth of a penny, maybe it's a little more, maybe it's a little less. Time to complete maybe one second, maybe a little more, maybe a little less. But as someone who has actually spent a lot of money on lawyers' hours, I want to point out a couple of things. The first one is it takes more than one hour to iterate on this for sure. And the second one is they're not always correct. In fact, built in for any interaction I have with a lawyer is cross-checking their work and double-checking their work. So again, we have four to five orders of magnitude difference in cost and time. And if you want to have like an example of like how extremely nutty that this can get. Like, I see no reason why you can't generate an entire game. There are companies today working on it, the 3D models, the characters, the voices, the music, the stories, etc. Like, there are companies that are doing all of these things. And if you compare the cost of, like, hundreds of millions of dollars and years versus, you know, a few dollars of inferences, now we have, like, internet and microchip level asymmetries in economics. Now, listen, I'm not saying this happens soon. We're not there yet. What I'm saying is this is the path that we're on. And these types of paths are what you look for with big transformations. So it's little wonder why we're seeing so much take off the way that we have. And these are the fastest-going open-source projects. These are the fastest-going products and some of the fastest-going companies we've seen in the history of the industry. And it's because, again, it's less the capabilities and it's much more that the economics work. So listen, this may sound hyperbolic, but I really think that we could be entering a third epoch of compute. I think that the first epoch, of course, is the microchip. Before the advent of the computer, you actually had people calculating logarithm tables by hands. Like, that's where the word comes from. They were computers. They would compute. Then we created ENIAC, along with other machines, but let's look at ENIAC. So ENIAC was 5,000 times faster than a human being doing it, right? There's your three to four orders of magnitude, and that kind of ushered in the compute revolution, right? And this gave us a number of companies that were either totally transformed like IBM or totally net new. So the microship brought the marginal cost of compute to zero. The internet brought the marginal cost of distribution to zero. So listen, in the 90s, when I wanted to get a new video game, I would go to a store and buy a box. And again, I don't have the math up here, but if you actually calculate the price per bit relative to DSL in the late 90s, it's about four or five orders of magnitude again, relative to actually shipping it. So I think there's a pretty good analog where you say these large models actually bring the marginal cost of creation to zero. There was some very fuzzy, vague notion of what creation means, but for sure we could talk about it of like content, conversation, whatever it is. And like the previous epochs, when those epochs happened, you had no idea what new companies were going to be created. Nobody predicted Amazon. Nobody predicted Yahoo. Like, I remember when this happened. So I think, listen, I think we should all get ready for a new wave of iconic companies. I don't think we know what they're going to look like, but forget the capabilities. Economics are just too compelling.

Host: We'll hear more from our team at the end of this episode. But speaking of economics and the scale of top models today, here is our new general partner, Anjani Midha, reminiscing about an early call he had with Dario Amadei, co-founder of Anthropic, who you'll also hear from shortly.

Anjani Midha: I'm going to take you all back in time to about three years ago. You and Tom gave me a call, one of your co-founders, and said, hey, you know, we think we're going to go start Anthropic. And I asked you, great, okay, like, what do you think we need to get going? He said, well, I think we can get by with like 500. I said, okay, I think we can find 500K somewhere. And I remember you deadpan saying, dude, I'm doing 500 million dollars. And that's when I realized that things were going to be a little bit different.

Host: Dario was one of the first employees at OpenAI and spent five years there before co-founding Anthropic. And the last year of AI has absolutely captured the masses, but people like Dario were early in recognizing just how far these technologies could scale. What was it that at that moment, when the team at OpenAI had started publishing your first experiments on scaling laws, that gave you so much confidence that this was going to hold, when everybody else just thought that was crazy talk?

Dario Amadei: Yeah, so for me, the moment was actually GPT-2 in 2019, where there were two different perspectives on it, right? When we put out GPT-2, you know, some of the stuff that was considered most impressive at the time was, oh, my God, you give this five examples, just offer it straight into the language model, five examples of English to French translation. And then you put a sixth sentence in English and it actually translates it into French. Like, oh, my God, it actually understands the pattern. That was crazy to us, even though the translation was terrible. It was almost worse than if you were to just take a dictionary and substitute word for word. But, you know, our view was that, look, this is the beginning of something amazing because there's no limit and you can continue to scale it up. And there's no reason why the patterns we've seen before won't continue to hold. The objective of predicting the next word is so rich and there's so much you can push against that it just absolutely has to work. And then some people looked at it and they're like, you made a bot that translates really badly. It was just, I think, two very different perspectives on the same thing. And we just like really, really believed in the first perspective.

Host: What happened then was you saw a reason to continue down that line of inquiry, which resulted in GPT-3. And what would you think was the most dramatic difference between GPT-3 and the previous efforts?

Dario Amadei: Yeah, I mean, I think it was much larger and scaled up to a substantial extent. I think the thing that really surprised me was the Python programming, where the conventional wisdom was that these models couldn't reason at all. And when I saw the Python programming, even though it was very simple stuff, even though a lot of it was stuff you could memorize, you could put it in kind of new situations, come up with something that isn't going to be anywhere in GitHub. And it was just showing the beginning of being able to do it. And so I felt that that ultimately meant that we could keep scaling the models and they would get very good at reasoning.

Host: What was the moment at which you realized, well, okay, we think this is actually going to generalize much broader than we expect. What were some of the signals there that gave you that conviction?

Dario Amadei: I think one of the signals was that we hadn't actually done any work. We had just scraped the web and there was enough Python data in the web to get these good results. When we looked through it, it was like maybe 0.1% to 1% of the data that we scraped was Python data. So the conclusion was, well, if it does so well with so little of our data and so little effort to curate it on our part, it must be that we can enormously amplify this. And so that just made me think, well, okay, we're getting more compute. We can scale up the models more and we can greatly increase the amount of data. So we have so many ways that we can amplify this. And so, of course, it's going to work. It's just a matter of time.

Host: Another person optimistic about scaling laws at the time was Noam Shazir. Noam was one of the researchers and lead author behind the transformative 2017 Transformer paper and has since co-founded Character AI.

Noam Shazir: I knew that, you know, you can make this technology better in a lot of ways. We can improve it with model architecture and distributed algorithms and quantization and all of these things. So I was working on that, but then it struck me, hey, the biggest thing is just scale. Can you throw like a billion dollars or a trillion dollars at this thing? What would happen if we did massively scale compute? Well, many companies chose to find out, and we, the consumer, are the beneficiaries of that. But can this realistically continue? Can the industry just continue to throw more computer at the problem and get better solutions? Or will a more fundamental unlock be required? This theme was top of mind for many at the event. And here is OpenAI's co-founder and CTO, Mira Murati, tackling that question head on.

Mira Murati: So there isn't any evidence that we will not get much better much more capable models as we continue to scale them across the axes of data and compute. Whether that takes you all the way to AGI or not, that's a different question. There are probably some other breakthroughs and advancements needed along the way, but I think there's still a long way to go in the scaling laws and to really gather a lot of benefits from these larger models.

Host: We'll hear more from Mira and touch on AGI in part two. But first, here's Gnome in conversation with ACC&C general partner Sarah Wang. On just how much compute we expect to soon be available, but also how much innovation is on deck, even if there aren't additional fundamental breakthroughs.

Gnome: I see this stuff like massively scaling up. It's just like not that expensive. I think I saw an article yesterday, like NVIDIA is going to build like another one and a half million H100s next year. So like that's 2 million H100s. So that's two times 10 to the sixth times. They can do about 10 to the 15th operations per second. So two times 10 to the 21 divide by like eight times 10 to the nine people on Earth. So that's roughly a quarter of a trillion operations per second per person, which means that could be processing on the order of like one word per second on like a hundred billion parameter model for everyone on Earth. But like really, it's not going to be everyone on Earth because like some people are blocked in China and some people are sleeping. It's not that expensive. You know, like this thing is massively scalable if you do it right. And, you know, we're working on that.

Host: You said this once that the internet was the dawn of universally accessible information, and we're now entering the dawn of universally accessible intelligence. Maybe building off your last answer, what did you mean by that? Do you think we're there yet?

Gnome: Yeah, I mean, I think it's like we're really a Wright Brothers first airplane kind of moment, right? Like we've got something that works and is useful for now some large number of use cases and looks like it's scaling very, very well. And without any breakthroughs, it's going to get massively better as everyone just kind of scales up to use it. And there will be more breakthroughs because now, you know, like all the scientists in the world are like working on making this stuff better. It's great that like all this stuff is accessible and open source. Like, you know, we're going to see like a huge amount of innovation and what's possible in the largest companies now can be possible in, you know, in somebody's academic lab or garage in a few years. And then, yeah, as the technology gets better, there are just going to be all kinds of great use cases that emerge and pushing technology forward, pushing science, pushing the ability to help people in various ways. I'd love to get to the point where you can just ask it how to cure cancer or something. I mean, it seems a few years away for now. Do you think we need another fundamental breakthrough like the transformer technology to get there? Or do you think we actually have everything that we need?

Mira Murati: I mean, it's impossible to predict the future, but I don't think anyone's seen like these scaling laws stop. I think as far as anybody has experimented, stuff just keeps getting smarter, so we'll be able to unlock lots and lots of new stuff. I don't know if there's an end to it, but at least everybody in the world should be able to talk to something really brilliant and have incredible tools all the time. And I can't imagine that that will not be able to build on itself.

Host: At the core, the computation isn't that expensive. Like operations cost like 10 to the negative $18 these days. And like, you know, if you can do this stuff efficiently, even talking to the biggest models ever trained, the cost of that should be like way, way lower than the value of your time or most anybody's time. And really we should, you know, there's the capacity there to scale these things up by orders of magnitude.

Host: As the industry does pursue scale, here's Dario's take on what bottlenecks may be along the way.

Dario Amadei: With the next 24, 36 months, what do you think the biggest bottlenecks are in demonstrating that the scaling laws continue holding? Yeah, so I think there's three elements. There's data, there's compute, and there's algorithmic improvements. So I think we are on track, even if there were no algorithmic improvements from here, even if we just scaled up what we had so far. I think the scaling laws are going to continue, and I think that's going to lead to amazing improvements. I think the biggest factor is simply that more money is being poured into it. The most expensive models made today cost about $100 million, say plus or minus a factor of two. I think that next year we're probably going to see from multiple players models on the order of $1 billion. And in 2025, we're going to see models on the order of several billion, I don't know, perhaps even $10 billion. And so I think that factor of 100 plus the compute inherently getting faster with the H100s, that's been a particularly big jump because of the move to lower precision. So you put all those things together, and if the scaling laws continue, there's going to be a huge increase in capabilities.

Host: But if compute does increase, how might this impact the size of models and ultimately the cost of inference for consumers?

Dario Amadei: Inference will not get that much more expensive. The basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of square root of n and size the model by a factor of square root of n. So that square root basically means that the model itself does not get that much bigger and the hardware is getting faster while you're doing it. So I think these things are going to continue to be servable for the next three or four years. If there's no architectural innovation, they'll get a little bit more expensive. I think if there's architectural innovation, which I expect there to be, they'll get somewhat cheaper.

Host: Increased model size and performance should unlock fundamentally new applications, which we'll explore further in part two. But first, entering the conversation is David Buzuki, co-founder and CEO of Roblox, commenting on the value of owning your infrastructure and the impact of that on inference cost, especially in a 3D world constantly reinventing itself.

David Buzuki: An even further extension takes probably a lot of compute horsepower, which is completely personalized degeneration in real time backed by massive inference stuff. So you could imagine, okay, I'm making the super Dungeons and Dragons thing, but as it watches you play and maybe we know your history, you'll be playing a 3D experience that no one's ever seen before. One of the good things we've done is we, for a long time, really focused on building our own infrastructure. We have hundreds of thousands of servers, many, many edge data centers, terabits of connectivity that we've traditionally used for 3D simulation, that the more we can run inference jobs on these, we can run super high volume inference at high quality, at low cost and make this, you know, just freely available so the creators don't worry about it.

Host: Whether we can continue scaling is one thing. But another topic on the minds of many builders is whether they can compete with the largest models. Will bigger models always win or will specialization trump generality?

Martin Casado: Reminds me very much of the silicon industry. So I remember in the 90s, when you buy a computer, there are all these weird coprocessors. There's like, here's like string matching, here's a floating point, here's crypto. And like all of them got consumed into basically the CPU. It just turns out generality was very powerful. And that created a certain type of economy, one where like you had, you know, Intel and AMD, and like, you know, it all went in there. And of course, it creates a lot of money to build these chips. And so, like, you can imagine two futures. There's one future where, like, you know, generality is so powerful that over time, the large models basically consume all functionality. And then there's another future where there's going to be a whole bunch of models and, like, things fragment and, you know, there are different points of the design space.

Mira Murati: Do you have a sense of, like, is it OpenAI and nobody or is it everybody? It kind of depends what you're trying to do. So, obviously, the trajectory is one where these AI systems will be doing more and more of the work that we're doing, and they'll be able to operate autonomously, but we will need to provide direction and guidance and oversee. But I don't want to do a lot of the repetitive work that I have to do every day. I want to focus on other things. But in terms of how this works out with a platform, we make a lot of models available through our API, from the very small models to our frontier models. And people don't always need to use the most powerful, the most capable model. Sometimes they just need a model that actually fits for their specific use case. And that's far more economical. So I think there's going to be a range. And there's a lot of focus right now on building more models. But, you know, building good products on top of these models is incredibly difficult. Plus each industry may have unique requirements.

Host: Here David commenting on how a suite of models will likely be required in order to power the class of games of the 21st century.

David Buzuki: In any company like a Roblox, there's probably 20 or 30 end-user vertical applications that are probably very bespoke. Natural language filtering, very different than generative 3D. And at the end-user point, we want all of those running. We want to use all of the data in an opt-in fashion to help make these better, tune these better. But as we go down, down, down, there's probably a natural two or three clustering of general, bigger, fatter type models in a company like ours. There's definitely one around safety, civility, natural language processing, natural language translation. generation generally one more multimodal thing around 3D creation say some combination of text image whatever generate a great avatar and then there's probably a third area which gets into the virtual human area which is how would we take the five billion hours of human opted in data what we're saying how we're moving where we go together how we work in a 3D environment and could we use that to maybe inform a better 3D simulation of a human. So I would say, yes, looking at large models in those three areas. And I think the market as we see it, there's going to be these super big God model, massive LLM type companies. I think we are probably a layer below that. We're very fine-tuned for the disciplines we want. And it's worth noting that the back-end model is only one part of the product.

Host: Here is Mira with a reminder to builders about the importance of UX.

Mira Murati: Actually, you can see sort of the contrast between making these