--:-- Erik Torenberg: It's like faster than light speed travel has just been invented. For many of the developers and programmers I talk to today, they're like going to Disneyland just because how many great tools there are to help them move faster.
--:-- Martin Casado: We do spend a lot of time just trying to be very honest with ourselves. What is the new behavior being created? Where is stuff getting used? Infrastructure never goes away, it just gets layered. In this case, it's definitely all the infrastructure that we've been using and leveraging in the past are still very relevant, but it's definitely getting layered by having this fourth pillar.
--:-- Erik Torenberg: Software was always the disruptor. One of the most exciting things about the AI wave is like software is being disrupted. Like we're being disrupted. This is a pretty big deal. It's by far the biggest thing that I've seen happen sort of in my life.
--:-- [Intro music fades in and out]
--:-- Erik Torenberg: Is AI the fourth pillar of infrastructure? Today on the podcast, we're joined by Martine Cassato, Jennifer Lee, and Matt Bornstein from A16Z's Infra team to unpack how infrastructure is evolving in the age of AI. From compute, storage, and networking to models, developer tools, and agents. We get into what infra actually means today, how it differs from enterprise, and why software itself is being disrupted. We also talk through the rise of technical users as buyers, what makes an infra company defensible, and how past infra waves from cloud to COVID to the current AI boom have shaped how we invest and build. Let's get into it.
--:-- Erik Torenberg: We're here today to discuss the state of infra. First, can we get a definition of Infra? Where does Infra differ from enterprise? How do we think about it internally?
--:-- Martin Casado: I would say Infra is basically what makes software work. We'll probably get pretty deep into a set of technical definitions. You mentioned networking, storage, compute. Where does AI fit into that? But I think at the simplest possible level, if you want software, Infra is what engineers are using behind the scenes to make all this possible.
--:-- Jennifer Li: And our formal definition internally is technical buyer. It's the stuff you use to build the stuff, the stuff used to build apps. If it's used by a technical user, we consider it infrastructure. Whereas something like vertical SaaS could be used by a flooring company or by a marketer or by sales, that would not be considered infrastructure.
--:-- Erik Torenberg: And technical user, for the record, is developer, data scientist, analyst, cybersecurity professional.
--:-- Martin Casado: Yeah, there's DevOps. There's a wide range of people. These are our people, the kind of nerds behind this. In system terms, you can think about it as compute, networking, and storage, but also all the tooling that goes around the developer's day — what you're using to build software and what are the tools and products that are operating this ever-growing, more complex software. All the way to semi-technical users that may want to either prototype or tinker with building applications. We're very interested in anything in the technical domain used by technical people.
--:-- Erik Torenberg: So you mentioned compute, networking, storage. How should we think about models? Is this the fourth layer of infra? How do they interface?
--:-- Jennifer Li: I certainly think of it as a fourth layer of infrastructure. It's leveraged and built on top of all the three pillars we're talking about. It has a lot of demand for compute, and of course it's trained and also producing a large amount of data. To leverage and use these models for our purposes, latency and networking capabilities are also very important. But it's going to be as prevalent as any piece of infrastructure software. I don't know the analogy these days anymore. Is it a database? Is it a new form of compute? To me, it's a fourth pillar that incorporates everything, but also provides intelligence for the software we're using and building today.
--:-- Matt Bornstein: I think this is exactly right. It's probably worth asking why a piece of infrastructure is a piece of infrastructure. Generally, a new piece of infrastructure changes the way that you program computers and then changes the stack that's around it. It's got different memory requirements, different latency requirements. It just requires rethinking how we build software and how we build infrastructure.
--:-- Martin Casado: In addition to compute, network, and storage, I would say distributed systems would also be included just because things like state consistency required you to think about proximity and guarantees. I would say databases probably did too because it changed our programming model. You have different guarantees. These models very much fit in that for a couple of reasons. Jennifer's exactly right — you just build different data centers and different chips if you want to build these models. So it has that impact. But programming them is non-obvious. We're still trying to grapple with how you program them. They don't really listen to you. Sometimes they do the coding themselves.
--:-- If I were to distill the one biggest difference that these models provide to infrastructure, it's the following: I don't remember ever in the history of computer science where we've abdicated logic from an application standpoint. In the past, we've abdicated resources — "give me compute, give me storage" — these abstracted resources. But the logic, the yes or no, what it's doing, always came from the programmer. With these models, we say, "Come up with the answer for me." It's requiring us to rethink what it means to be a programmer, what it means to be software. It's clearly very fundamental to computer science. And to Jennifer's point, it's very much a new piece of infrastructure.
--:-- Matt Bornstein: A lot of people are trying to reason by analogy. You alluded to this, Jennifer. Is it like a database because it can answer queries? Is it like a network because it's non-deterministic and we need to handle retries and weird edge cases? People are really just trying to figure out how to program these things, which you said, Martin. We've got to start from a blank sheet of paper, which is what makes our jobs really exciting right now. There are a lot of people trying to figure it out and coming up with new ideas.
--:-- Martin Casado: Many of us have been in computer science for a long time — in our schools, in our operating lives, in our investing lives. Software was always the disruptor. We disrupt the taxis, we disrupt sales, we disrupt the back office, we disrupt everything. One of the most exciting things about the AI wave is software is being disrupted. We're being disrupted.
--:-- Erik Torenberg: It's disrupting software. It's eating software.
--:-- Martin Casado: I know. Software is a self-eating ice cream.
--:-- Erik Torenberg: That's right.
--:-- Martin Casado: Honestly, I think this is the first time I can honestly say that the profession I've dedicated my entire life to is being disrupted, and it's very exciting because it's eating itself in a way.
--:-- Matt Bornstein: It's tempting to be a curmudgeon because we've all been doing this our whole lives. Being really open to and embracing the new stuff is the key thing. What still applies is infrastructure never goes away. It just gets layered. In this case, all the infrastructure we've been using and leveraging in the past are still very relevant, but it's definitely getting layered by having this fourth pillar, which is AI and models.
--:-- Erik Torenberg: What's different from past super cycles versus what can we learn as we enter this new one?
--:-- Matt Bornstein: Two things happen. One is often when you bring the marginal cost of something down — like with compute, we did it for computation, and for the internet, we did it with distribution — it increases the TAM a whole bunch. You almost always see this massive TAM expansion. Part of that tends to be because the TAM is bigger, you've got new users. And because you have new users, there's normally a new behavior that happens. This is very much the case with the internet — people weren't used to going to a computer and talking to everybody around the world on top of the internet. Existing companies don't know how to think about new behaviors. They've built these sales motions and operating things around the old behavior. So you see TAM expansion, you see new behaviors. Those new behaviors provide white space for challengers, new startup companies to come and fill those models. I think we're seeing exactly that happen with this one as well. Clearly this market is massive if you look at how successful these model companies are. But also you're seeing use cases that computers just never really have done before. In that way, it rhymes very much with the internet and probably even the microchip.
--:-- Jennifer Li: I'll answer that question from my personal experience. I'm always a tools person, wanting tools that fulfill certain creativity. I came to coding and computer science as a late bloomer after my 20s and really enjoyed all the tools available for me at that point for building software and learning computer science. Now we have massive leverage in trying to create anything as long as you have a good idea. Martin laughed at me about this. I was a bigger local no-code champion for the last five, ten years because these are tools for people who have good ideas but may not be educated in computer science. The Retools of the world, the Wix and Squarespace — you can build applications, build software easily with these tools. But now you're given the next level of thought partners, tools to prototype software interfaces for your end customer, for your end user, as long as you know what they need. You can realize these ideas really quickly at your fingertips.
--:-- Martin Casado: Low code's finally happening. It just takes a lot of code. It turns out the code is natural language.
--:-- Jennifer Li: I know, it's so funny. When Jennifer joined the team, she was very excited about low-code. But from my view, low-code is like Python. It's a scripting language. Interpreted languages. PM is low-code. We kind of had to bridge that gap. It was a bit irreconcilable until AI came out. It's very clearly what the promise of low-code was. It really is disrupting software.
--:-- Matt Bornstein: When I was a kid, the internet was a new thing. I remember vividly there was this movie with Sandra Bullock called *The Net* and she orders a pizza from her computer. This was completely mind-blowing. Now this is a common user behavior. But what we're dealing with now is so much bigger than that. Some of these points about how the infrastructure will evolve, how the companies will adapt — those are probably transferable. But this is a pretty big deal. It's by far the biggest thing that I've seen happen in my life.
--:-- Erik Torenberg: Let's zoom out and take this long view. Martina, you're actually the perfect full circle because weren't you the first infra investment ever as a portfolio founder?
--:-- Martin Casado: I think it was either me or Okta. I will say me and Todd were the infra portfolio. In the LP days when they would trot us out in front of the LPs, it was me and Todd from Okta. For sure, I was one of the first two.
--:-- Erik Torenberg: At what point did we develop a clear infra practice at A16Z?
--:-- Martin Casado: We always had strong infra people. Ben Horowitz is an infra guy. And honestly, I would say Mark is — he masquerades as a consumer guy, but he's actually revolutionized the way we use computers in this deep infrastructure way. Many things came from that JavaScript. We had Peter Levine, Zensource. We had Scott Weiss. There's always been deep infra. But when I joined the firm, we didn't think of it as infra. We thought of it as enterprise. You're either in the consumer team, the fintech team, or the enterprise team. The thing about classifying this stuff as enterprise — the go-to-market motions for something that touches technology and being able to reason about that is so different than reasoning about the go-to-market motion that's purely through sales. We've learned over time that we do deep market diligence as a firm and deep diligence on companies before we invest. The type of diligence we did if it was deep infrastructure just required a different type of junior partner and type of analysis. Over time, we realized that companies where you can evaluate more by the business model, the market buyer, the unit economics — those kinds of companies are just sufficiently distinct. So we decided to pull them apart. That's why we have the apps fund and the infra fund.
--:-- Jennifer Li: If you think about enterprise, there is the horizontal piece, but also a lot of vertical enterprise — applications, buyers, sectors — where infrastructure is almost always horizontal. That's another thing we realized along the way. We still want to appeal to this technical audience and technical buyers. But we want to think about the space in a horizontal fashion where these technologies can be distributed to all these different sectors and verticals. It impacts how we think about the stack, how we think about what will drive the form of how the stacks integrate with each other. It's just different from enterprise generally.
--:-- Martin Casado: It's a very important distinction. Technical buyers tend to be centralized buyers in a way. IT will buy compute, network, and storage. Yes, you've got the compute person, the networking person, but it rolls up to IT. Developers are kind of a centralized buyer. You can understand IT and you can understand the developers. But when I was looking at vertical SaaS apps early in my investing career, to understand how to sell into a flooring company, you have to understand the flooring market. That's entirely different from the pet food market, which is entirely different from the construction market. There was no real central software buyer for these verticals. As you get further away from core infrastructure, you get away from this notion of a centralized educated buyer. The level of analysis becomes very different.
--:-- Matt Bornstein: I think that's exactly right. There's even a horseshoe theory of software buyers happening now. You have consumers over here, then you work up to apps. You would think infra is way over here, but it's actually bending back around. Developers are making a lot of the decisions, and a lot of the marketing and sales to developers looks more like consumer these days than it used to.
--:-- Jennifer Li: That's an incredibly important point. I was researching the stats the other day. When I first joined venture or started thinking about infrastructure, developers dealt in the low tens of millions. They were becoming the next generation of consumers. Now they're definitely the next generation of consumers. It's going above 50 million on the way to even bigger than that. Now we're talking about language being a programming language for everybody. They are making decisions just as a consumer. We're training ourselves to understand how developers as individuals adopt tools, but also how IT buying centers as customers and enterprise organizations evaluate tools. There's a lot to learn and figure out, given these are the technical audience we really care about.
--:-- Erik Torenberg: I want to get to how we think about our investable universe and what subcategories are mature versus newer or ripe. But before that, I want to better trace the evolution. Martin, maybe we can start with you. If you had to categorize, since you've been investing, the different waves of infra investment or the inflection points at which it changed how we thought about it.
--:-- Martin Casado: For the life cycle of the firm — 2009, that was pre-cloud. A lot of the early investments were right on the cusp of cloud. My company, for example — the cloud was out there, but it wasn't sufficiently deployed. You could use it as a core thesis. A lot of the early investments were like this. We talk about it in terms of tech, but it rips through the entire business model. Early software was on-prem with a perpetual license. That's different economics and different analysis. The pre-cloud installable software area was one. Then we saw the cloud transition. We went to recurring revenue, totally different deploying model, totally different operating model. During that time, we saw net dollar retention being more important, expansion being more important, gross churn being more important.
--:-- Erik Torenberg: Gross margin.
--:-- Martin Casado: Gross margin being more important. Everything changed. Meanwhile, we saw consumer being disrupted by mobile with the rise of the Ubers, the Lyfts, the Airbnbs. And then we're seeing it happen again during this AI transformation of the last three years — the most dramatic I've seen in the last 30 years of being in this industry. There's one interesting blip along that path: COVID. We come from the realm of enterprise sales and being on the ground deploying things, and that totally evaporated. That was a dramatic shift as well, but it was driven by force majeure as opposed to a secular technical wave like the other two.
--:-- Jennifer Li: One benefit COVID brought is there was already the trend of developers buying and adopting tools — this bottom-up motion for infrastructure and dev tools. It got accelerated during COVID where lots of people were tinkering, trying out products and tools, and more people were building dev tools during that time. There was a flourishing ecosystem of PLG or product-led dev tool companies that are giving birth to what we're seeing now — a lot of AI dev tools coming up.
--:-- Martin Casado: There's nothing developers like better than being forced to stay home and not interact with other people and write code. This is heaven. Best news ever.
--:-- Erik Torenberg: That's amazing. Let's go deeper into the present. Can you share how we think about the different subcategories or landscape that makes up infra? We could also plug some examples of portfolio companies, spaces where we made some bets.
--:-- Matt Bornstein: Here are a few important categories. Developer tools — anything developers use to make their lives better, easier, faster, more efficient. Cursor is probably our top developer tool company right now. And before that, GitHub.
--:-- Erik Torenberg: GitHub.
--:-- Jennifer Li: And Jennifer, you've actually backed a bunch of interesting dev tools companies in the last few years too.
--:-- Jennifer Li: From Levi, Stenless. There was a long time where dev tools was written off by VCs.
--:-- Martin Casado: The TAM was too small.
--:-- Jennifer Li: Exactly. At the time of GitHub, would you ever have imagined a repository would be a huge company? It was almost a joke. People were chatting about the business model and all these things.
--:-- Martin Casado: Small TAM is the classic red flag for infra investing. If you're at home listening to this and someone tells you small TAM, they're not an infra investor. Infra creates TAM. One takeaway from this thing is TAM creative.
--:-- Matt Bornstein: DevTools, you've got Core Infra — compute, network, and storage. That's for IT. Then you tend to be quite a bit above the Core Infra stack.
--:-- Jennifer Li: I think about both how developers are using tools to improve their efficiency, but also how customers are getting value out of that. A lot of packaging dev tools into SaaS forms. I'm investing in this company called Pylon. It's a SaaS company that does customer support, but fundamentally it's building a data pipeline. That's core infrastructure that's very good at connecting with systems and providing context to a lot of agents and AI models. To me, that's infrastructure. We're spending a lot of time on cutting-edge AI research — a lot of foundation model investing. We could numerate for the next 10 minutes.
--:-- Martin Casado: I will say the reason we're a bit skittish on this question is early in super cycles, it's very hard to distinguish between an infra company and an application company. The reason is because the TAM is so small and so new, the new technology becomes the app. Let me refer to the original super cycle that started the firm: the internet. Netscape — I remember when it came out. It was a consumer thing, or at least a student in school thing. Everybody was downloading it from an FTP server and using it as individuals. The enterprise didn't know what to think about it and banned it. The same company that built JavaScript, building this core technology, is also doing the browser. Over time, it matures. Then you have all of these internet companies and applications show up. We're seeing the same thing in the AI wave. Is MidJourney an infra company? They build a model. Or is it an app company? It's both in this sense. At this stage, it's very hard to distinguish.
--:-- Matt Bornstein: It's exactly right. It's very hard to answer if OpenAI is an app company or an infra company. It's already building infrastructure that's like a cloud, running these models for different sectors and use cases, but at the same time building a consumer app that's ChatGPT.
--:-- Jennifer Li: We think of foundation model companies that are similar. Eleven Labs — they're a voice AI provider and they have the creator application that can use a studio to create voices. At the same time, they're also supplying voices to large-scale enterprise use cases that are fine-tuning, cloning your own voice, and distributing that through API. It's sort of both.
--:-- Matt Bornstein: Another area we've done a lot of work is data systems. This goes all the way back to Databricks. There have been two branches: the back-end data-eng driven big data systems — Spark, Hadoop sort of thing — and the data analyst, more tabular Snowflake kind of thing. As a firm, we've been super active and super aggressive investing in companies like Databricks. I mentioned Fivetran, DBT, which you guys invested in together, Hex, which is doing really well, Tabular, which was acquired by Databricks. I think we're still really bullish on this. Unfortunately, AI has sucked the air out of the room for a lot of data companies from a bunch of different angles, but I think we'll continue to do more of this as well.
--:-- Erik Torenberg: How do we think about defensibility for AI companies, whether it's the player or the model layer? Do they all have their respective areas of defensibility, or how has our notion of defensibility evolved?
--:-- Matt Bornstein: We once wrote a blog post that there was no defensibility anywhere in the stack.
--:-- Erik Torenberg: For anything?
--:-- Matt Bornstein: For anything. And yet people make lots of money. The argument at the time was: NVIDIA has a moat because chip designs are hard to copy. But if you go up or down, they're all manufactured at the same place — TSMC. If you go up, the cloud providers provide effectively the same product. The models are training on the same data and have similar capabilities. The apps are all using the same model. That was the naive theory when we were trying to understand this at the beginning. I think maybe it was true at the time. What Martine was alluding to is that meanwhile, every company at every layer of the stack is doing fantastically well right now. During this initial phase of industry development — which we sometimes call the Brownian motion phase — I think it's really hard to make pronouncements about what's going to work, what's not, where value will accrue. App companies are doing really well. We're pretty clear past the wrapper phase. I don't think there are any wrappers anymore. Building good products with AI is really hard. The founders doing it now have really good intuition for how to do it. The models are clearly pushing the whole industry forward and they've built huge companies. It's kind of all working right now. You could actually make a case for how defensibility will work. It's quite different from the way defensibility has worked before.
--:-- Jennifer Li: On how defensibility worked for infra companies before — it's really hard. If you're building a new database, a new framework, it takes a lot of expertise in the domain of understanding what has happened in the past, where the field came from, and what innovations need to happen to polish a software into this new abstraction to provide to developers. One example I'm thinking of is DuckDB. It's a high-performance, small, but really nimble database. It took four years for the team to write it. To replicate that is really hard. That's generally what happened in the past infrastructure space. It takes these experts a lot of time to build a new piece of security software — if it's YubiKeys or again, Databricks on Spark. As the AI infrastructure comes through, a lot of those defensibilities still stay and are still true because these are earned secrets of what the downfalls and guarantees other software run into, where these founders know the past and also look into the future. But I do feel like the adoption phase is just really massive. Who is going to earn the distribution and earn developer attention is going to be a different game.
--:-- Martin Casado: Can I do a quick mental model that has suited me very well in thinking about this? I think the sloppiest thinking in our industry is around defensibility. We're just so sloppy about it. This might be worth the price of listening to this podcast. The industry tends to go through expansion and contraction phases — like the Big Bang. It expands and then it contracts. What happens when it expands? Zero-sum thinking is deadly because you're just getting more market. We're clearly in an expansion phase. Everybody says NVIDIA can't sell more chips, but they keep selling more chips. The hosting platforms can't continue to get margins, yet they keep continuing to get margins. Matt said it perfectly. If you're in the expansion phase, there's just more to sell. You should be aggressive investing. What happens in the collapse phase? Look at any layer of the stack. In the collapse phase, things start to consolidate. What is the end state of consolidation? The end state will always be an oligopoly or a monopoly. Layers never go away. If you have an oligopoly — say the clouds — then you have what is effectively price fixing, but it's tacit. Everybody says, "We're going to price it this, and we're going to maintain our 30% margins." You still have value there, you have margins. In the case of a monopoly — say an Intel at the time — you can also maintain margins. In none of this do you lose margins.
--:-- Martin Casado: And I just think this is why people think so sloppy about this. People use words like commoditization and no defensibility. That tends to be a battle between layers of the stack. But the only way you can do that is actually move down the stack and enter somebody else's layer, which is incredibly hard to do. And you do see it. So of course, Google is going to build their own chips and they start moving down the stack. But that's a very, very different thing than somehow Google playing the different layers off against each other.
--:-- Martin Casado: So I would encourage anybody that does invest, at least in infrastructure, to not think zero-sum and to realize that historically every layer of the stack has maintained some level of value and margin. And if not, it was because a layer above them managed to verticalize themselves. But then it's that one player against the rest of the world.
--:-- Matt Bornstein: Yeah, I mean, it's like faster than light speed travel has just been invented, right? And we're sending all the spaceships out in all directions. And there's plenty of planets and stars to claim for everybody. We're not even close enough to each other to fight over them.
--:-- Martin Casado: For sure, this is extreme. But it will slow down and then the consolidation will happen. But I guarantee you'll just end up with these great companies that maintain margin. AWS still has great margins. Google still has great margins. And still growing at insane speed. Databricks too is growing at incredible speed.
--:-- Jennifer Li: I think people underestimate how hard these problems are in many cases. You're sort of applying consumer thinking because this is how we live most of our lives. Like, oh, wouldn't it be relatively easy to move to a different part of the stack or take out your competitor, or a customer could just switch back and forth? And it's just different laws of physics, I think. It just turns out, in general, the switching cost of an infrastructure piece is so much higher. Even with API businesses, people tend to think you can just switch over to another API. There's so much logic embedded in calling the API in the software itself. There's a lot more switching cost compared to your regular SaaS software consumer.
--:-- Matt Bornstein: Yeah, totally, because you're actually integrating systems, right? It's not necessarily a person who can just have a preference for one thing or another. You're sort of integrated.
--:-- Erik Torenberg: Sam Altman once had the advice to startups last year. He was like, "If you're worried about us improving our models, you're in a tough spot. But if you get more excited about your business by us improving our models, then you're in a good spot." Do you think that's a helpful framework?
--:-- Martin Casado: I think it's helpful for OpenAI.
--:-- Jennifer Li: Do you believe that?
--:-- Martin Casado: No comments.
--:-- Matt Bornstein: I want to say that too. A16Z, if you think us investing in this company isn't good for you, then... For companies, that's great.
--:-- Martin Casado: There's a very open question that's actually a technical question. It isn't a business question, which is how much does general training generalize, right? So we know in the pre-training world, it generalized really well. So you'd create one model, and that model was just as good at code as it was at writing a poem. So we know that it was very general. And in that world, sure, as the models get more powerful, then they can do all of the things, so they compete with all of the things, right?
--:-- Martin Casado: But it seems clear to me, and again, this is an observation and it may not be correct, that as we get more into the RL world, you make some trade-offs. And then let's say I RL something for code. It's not going to be as good at something else. You're making these trade-offs. And in that world, then it's not the case that the model is going to be generally good. So I think it's great to compete at the model layer. So again, I think this is maybe a reasonable rubric, certainly for OpenAI to have people believe. Maybe a reasonable rubric if you believe that these models are going to be generally great, but I just don't think it holds up to how things are going to play out.
--:-- Jennifer Li: This was a debate we had two years ago. I think whether the general model and the most capable model will rule, or a lot of small, medium-sized models that are very good at specific tasks, that's going to be the future. It turns out both are true.
--:-- Matt Bornstein: Both, yeah.
--:-- Jennifer Li: When we talk about complex systems, you cannot just use one model that drives everything, at least not today. But you can compose very capable and powerful models to take certain tasks and also chain together processes from processing a document to feeding it into a model to have some reasoning and give you back really clean and structured data to make decisions and put it into your application to serve end users. That's a complex system that invokes many model calls instead of just one big model's task.
--:-- Erik Torenberg: Did you guys have any reactions or is it worth talking at all about Karpathy's talk? Did that framing resonate with you? Did you have any differences of how you would frame certain things? One thing he mentioned is that he thinks it's not the year of agents, but the decade of agents. Perhaps it's not as immediately upcoming as we might have thought.
--:-- Matt Bornstein: Probably should have talked about it more.
--:-- Jennifer Li: There's something he said recently that I thought, and he's actually responding to somebody else, that I thought is actually very insightful. I actually thought his talk was great, but I thought his talk was kind of an overview of what a lot of people know. I thought it was a pretty good general overview.
--:-- Martin Casado: I saw the tweet today about who knows how old it was. So there's this idea of prompt engineering that people have talked about. And somebody, it wasn't Karpathy, but Karpathy piled on top is this: it's really not about prompt engineering, it's context engineering. So what is context engineering? If you're going to call a model, you have to know what to put in the context in that prompt. And what tools do you have to do that? Well, you could use other models, but at some point you're probably going to use traditional computer science. You're going to use things like indexes, you're going to have to do prioritization, et cetera. And to really drive the best performance out of those models, you do want the context to be correct.
--:-- Martin Casado: And I do think it's probably the right framing of this problem. And the next step is, in as much as we're going to provide formalism to how to use these models, to how to use existing tools, to how to improve the performance, you should be thinking about what's the right way to get the right context into those models. And I bring this up because, like we said before, new infrastructure pieces create new patterns and new methods of software and building systems. And this is a great example of that kind of emerging before our eyes and people reasoning about it. And I truly believe in five years, we'll look back, we'll come up with a whole new set of formal ways to build software. And they will have strong guarantees and we'll understand them and there'll be all the tools for it, etc.
--:-- Jennifer Li: The way I think that relates to our world is, if you think about what is the new form factor of infrastructure that needs to become part of this context engineering, it goes back to a lot of what we're obsessed about: data pipeline. How do you feed the right data and context into the models or into the context? And how do you have agents' tools or an infrastructure that will provide discovery and guarantees and observability of these tools as well? It's the classic infrastructure problem that's still unsolved. So it's a very exciting time.
--:-- Matt Bornstein: There's a few different types of infra founders, right? There's the sort of infra founder who loves solving really messy, long tail, nasty problems. There's a type that kind of just gets fed up with a problem and they're like, "I'm going to solve this finally. I'm sick of this." Then there's the type that just sees the world in a new way. And it's kind of like, "This is actually how we should marshal these resources." And React is a great example of this. We had all this progression of front-end development frameworks and finally React was the way that stuck. And for years now, it's been sort of the default front-end.
--:-- Matt Bornstein: So I think what Karpathy's talking about is trying to figure that out. And I think the software 2.0 thing was really interesting. We were investing in a bunch of traditional ML companies at the time. And I think software 3.0, I think he's sort of right about that too, directionally. And my hope is it'll inspire a lot of new infra founders to do this work, see the world in a new way, and figure out how these primitives should really be arranged.
--:-- Martin Casado: One of the difficulties of having any conversation around AI is it just exploits this weakness in the human imagination to dump all of our fears and hopes and dreams into this anthropomorphic fallacy, right? And this goes all the way back to the Promethean legend. So let's talk about even this context, right? We're building systems to build other systems. So systems have constraints, right? And so you can fail on either side of this when it comes to this anthropomorphic fallacy. On one side, you can be like, "This stuff doesn't work. You shouldn't use it. You should only use traditional things," which, okay, that's clearly not the case. It seems very useful. But on the other side, you can believe they'll solve all of our problems and you don't need formalism and you just kind of go to the beach and you come back when AGI is done and it'll do it for you.
--:-- Martin Casado: And so part of our job and what we spend a lot of time talking about is trying to find that pragmatic, non-blinkered, non-pessimistic middle, despite all of the rhetoric. And then you hear all the rhetoric, right? The stuff is going to like, "No, we're not gonna have to work and we're all going to be on the beach." Or, you know, like it'll kill us, the whole thing. And I think where we've landed is this is a real disruption. It is changing all of software. It'll look something different, but it is still going to require professionals. And I do think that the statement "it'll require professionals" is a very meaningful one. It means that you actually still need people that understand the specifications of the systems.
--:-- Martin Casado: And not everybody agrees. Some people are out there like, "Listen, you will never need a programmer again," because people are beginning to say some high level thing and it'll show up. And the only statement I'll say to that is formal systems came out of natural languages for a reason. Either you care about specifying what you're designing or you don't. And if you do, you need to be a professional. And that's why every professional discipline, even though they started with a natural language, has ended up with a formal system.
--:-- Erik Torenberg: What is your mental model on coding specifically? Will there be fewer engineers who are just higher powered?
--:-- Matt Bornstein: I think the best way to think about this is simply that we're going to have more developers. I think it's very unlikely that we're going to shrink development teams because we have amazing new tools. That's just not how these markets have worked in the past. I think exactly the opposite is going to happen. We're going to be creating so much great software and it's going to be so accessible to so many people who may work at a big company or may just be hacking on the weekend. I think that's by far the simplest way to think about it.
--:-- Matt Bornstein: And it gets back to Martin's point of you can't anthropomorphize these models. A model is a file on a hard drive in a computer somewhere. And when you run a Python script, you can transform one piece of data into another piece of data. That's what this is. And programming is a fundamentally creative job, right? You are literally creating things in the most strict sense of the word, which is you're creating software that doesn't exist before. And that's something only a person can actually do at some level of abstraction. So I personally think this is a huge boon for programmers. And you have to change the way that you're working. And it's a huge productivity boost. And I think that creates more, not less.
--:-- Jennifer Li: I cannot agree more. I feel like for many of the developers and programmers I talk to today, they're like going to Disneyland just because of how many great tools there are to help them move faster, build things they have always wanted to do on both side projects and also their main job. I do think it also changes the dynamic of how people are picking up new languages, picking up new frameworks. And it is, again, a next level of iteration speed, given what we're seeing with AI agents and also AI coding tools.
--:-- Martin Casado: Here's another, I think, useful mental model on this stuff. I think it's worth asking the question, why do people buy software? Why does someone buy some random SaaS tool? Is it because it's so hard to build it? No. Most SaaS tools are like CRUD. They're just these basic read-write databases. They're all the same. So why do people buy them? And Aaron Levy, who's the CEO of Box, I think said this so beautifully: the reason people buy software is because somebody else made the decisions of what the workflow should be and what the operational logic should be and what data is important, how you use that data is important. Creating a product is a lot of understanding what is being used and guiding the user along that direction.
--:-- Martin Casado: If not, I'll just give you a compiler. Like, you do whatever you want. Or I'll just give you a database and you do whatever you want. There's a reason that we have a proliferation of vertical SaaS and it is this kind of articulation or this transfer of domain understanding. And that just doesn't go away independent of how you create the software. And so we will still need to design products based on whatever problem is being solved, guide people so that they're the most effective with them and they can understand the best. And we did this with assembly before, and then we did it with high-level languages, and then we did it with high-level frameworks, and then we're going to do it with AI. But the fundamental process of that articulation will not go away. And it's totally orthogonal to creating the software itself. It turns out to be a much harder problem to go out and collect requirements from an unknown set of users with an unknown set of needs and figuring out what to build. That turns out to be much harder than actually building it.
--:-- Matt Bornstein: So what do you think is the average number of lines changed in a PR in the industry?
--:-- Martin Casado: Two.
--:-- Matt Bornstein: Yeah. But it shows you the point. It literally is understanding the need from the business and the need from the user and making some minor tweak. That is the long tail that goes into software. And by the way, it turns out it's two.
--:-- Martin Casado: Well, I think that's the median. The median is two. I've written a lot of cheap PRs in my life.
--:-- Matt Bornstein: Just for the record, by the way, Martin said "crud." Crud is a technical term, right? Create, read, update, delete. We actually think applications are great. We don't think they're cruddy. They just really are CRUD.
--:-- Martin Casado: Yeah, CRUD is it.
--:-- Matt Bornstein: Just a footnote. Good to know.
--:-- Erik Torenberg: Jennifer, you were mentioning earlier how two years ago we were having this debate on generalization. And what are the debates we're having now internally or with your peers? Or what are the main questions that we're asking that we can't wait to see how they're going to reveal themselves in the next few months or next year that are going to impact our business?
--:-- Jennifer Li: Well, gosh, every week is different. What are some recent ones? Definitely how realistic are agents today at really producing production-level software? That's more on the coding agent side, but also in general, the agent evolution. Where are we in going from demo-ware to producing real value, tangible value?
--:-- Jennifer Li: What are some other ones? Synthetic data is one we talk about a lot. We've been talking about that for 10 years.
--:-- Martin Casado: Yeah, this is what's so great about infra. You can have the same debates for 10 years and they never quite go away. It's like changing the background. Consistency versus availability. Literally every system has those trade-offs.
--:-- Jennifer Li: So this synthetic data thing, right? It's almost an information theory question. It's like, can you make models meaningfully better without introducing new information to the system? And I think it's now pretty clear you can do a little bit, but the question is, does this lead to a self-improving utopia of models or not? And I think we have some pretty strong opinions on the "not" side of that.
--:-- Martin Casado: Generalization, as I mentioned, is a pretty interesting one. If you train a model to be really good at math, does that mean it's going to be really good at other things? Or is it just really good at math, which I get excited about. Not everybody gets excited about this.
--:-- Matt Bornstein: Another one that we talk a lot about is what these things are actually good for. Just because the path we came from used a lot of AI, pre-gen AI used a lot of AI. So there's a lot of AI-shaped holes in the enterprise, like chatbots and this and that. And that's very much on the brain. And it seems to sometimes confuse the discussions from the new use cases we're seeing. Like if you look at the most common use cases of something like ChatGPT, I think the top one is companionship and therapy. And then it's managing my schedule. It's the top of the pyramid of need stuff. And then number five is professional development, not low-code development or whatever.
--:-- Matt Bornstein: And so I think what is happening is we have this idea of what we thought AI was going to do and just the stilted attempts previously. Jennifer actually ran product for a chat company prior. And then what it really is good for, and clearly there's some overlap and convergence, but it's not nearly as big as people say. And so we do spend a lot of time just trying to be very honest with ourselves. What is the new behavior being created? What is this stuff getting used for? What is it just being crammed into places it's not actually quite good at?
--:-- Erik Torenberg: To that point, how do we think about agents right now? What are they good for, and what are they going to be good for soon enough? What is the state of them? How do we think about the broader conversation?
--:-- Matt Bornstein: Coding agents are awesome. It's really amazing.
--:-- Martin Casado: So I have a very simple way to think about this. And I'm like the anti-agent guy, by the way. I think it's kind of a marketing thing. This is the other thing about infra people: we are allergic to marketing, which is not always a good thing.
--:-- Erik Torenberg: Which is a good thing you're here, Eric.
--:-- Martin Casado: I always ask, what do you mean? Yeah, exactly. Can you explain? If you take the simplest definition that basically an agent is an LLM running in a loop, a very simple way to think about this is errors propagate throughout the loop. So if you have a small error, it gets worse and worse. And this is why a lot of agents doing, say, general web browsing don't perform very well yet. On the flip side, if you have a way to correct those errors in the loop, which is one thing that you have in code, right? You can lint, you can interpret, you can try to compile and things like that. You actually do see good performance over time. So that's a very simplistic and maybe not quite the right way to look at it. But if you can do this kind of error correction, I think you're seeing a lot of improvement from this iterative approach.
--:-- Matt Bornstein: I mean, it's incredible. I'm actually on the GitHub mailing list of a lot of the companies that I work with, just mostly for interest. And I have, even in the last week, seen a bunch of Cursor-like agent commits. Even in the last 24 hours, the Slack integration, seeing it come from Slack. And so I do think for bite-sized tasks you can articulate very well, we're starting to see them really work. So in the coding space, I would say I'm a convert. But to Matt's point, the "go wander out in the woods and bring back a bear," I think we're a long way off. There are a lot of errors you can run into.
--:-- Erik Torenberg: Will we see more vertical integration or more horizontal specialization?
--:-- Jennifer Li: Historically, we've seen both. And what's interesting is we're already seeing both now, right? Apple, of course, has just been historically vertically integrated. Microsoft and Intel, historically horizontal. And often, companies will start horizontal and then go vertical. So Google is horizontal, built on top of normal servers, but then they built their servers, and then they built their own chips. They built their own networking gear. And so I think you always get a mix of the two.
--:-- Jennifer Li: What's interesting about now is we're actually really seeing both. I would say that OpenAI is very much a vertically integrated company now with ChatGPT driving a lot of it. I would say Anthropic, a lot of the usage really is more horizontal. And they're doing a great job of that. I think we're seeing this on the model layer too. Very interesting discussion we haven't had, but it's a very interesting one. It's like open source, quote unquote, really seems to work with these models just because you can't, as a user, recreate it. So if you look at BFL, they've done a great job building a horizontal layer for these models. But then you've got companies like Ideogram, which have built a great vertical experience as well. And so I would say for AI, we've already got, this early on, great examples of both. And I don't see any reason that that will change.
--:-- Jennifer Li: I think from the business front, it just poses new interesting questions and challenges too of how do you capture the value. Of course, horizontally you can capture the value through being able to address every single use case by providing that to developers or enterprises. But if you vertically integrate, you kind of have to pick the lane. Do I want to focus on image models? Do I want to focus on graphic designers? Do I focus on people who are generating photographs? You have to understand the market and the user personas and use cases pretty well to capture the maximum value, where you probably can take an easier path to just provide an API so everybody can use it.
--:-- Erik Torenberg: I think this is a great place to wrap. Guys, thanks so much for coming on the podcast.
--:-- Martin Casado: Thank you for having us. This is awesome.
--:-- Jennifer Li: This is fun.
--:-- Erik Torenberg: Thanks for listening to the A16Z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com slash A16Z. We've got more great conversations coming your way. See you next time.
