Dylan Patel on the AI Chip Race - NVIDIA, Intel & the US Government vs. China

Host: How you buy GPUs is like buying cocaine. You call up a couple people, you text a couple people, you ask, "Yo, how much you got, what's the price?"

Host: If your two arch-nemeses suddenly team up, it's the worst possible news you can have. I did not see this coming. I think it's an amazing development. Like a Warren Buffett coming into a stock. Jensen is like the Buffett effect for the semiconductor world.

Host: It's kind of poetic that everything's gone full circle and Intel's sort of crawling to NVIDIA.

Host: Today, we're talking about one of the biggest surprises in semiconductors in years. NVIDIA just put $5 billion into Intel, two long-term rivals now teaming up on custom data centers and PC products, a deal nobody saw coming. For NVIDIA, it's the Buffett effect. For Intel, it's a lifeline. And for AMD, ARM, and the global chip race, the fallout could be massive.

Host: To break it all down, I'm joined by Dylan Patel, Chief Analyst at SemiAnalysis, Sarah Wang, General Partner at A16Z, and Guido Appenzeller, Partner at A16Z and former CTO of Intel's Data Center and AI Business Unit. Let's get into it.

Host: Dylan, welcome back to the podcast.

Dylan Patel: Thanks for having me, yeah.

Host: It just so happens that there's some big news, just as we're having you, NVIDIA announcing a $5 billion investment in Intel and them teaming up to jointly develop custom data centers and PC products. What do you think about the collaboration?

Dylan Patel: I think it's hilarious that NVIDIA can invest, it gets announced, and their investment's already up 30%. $5 billion investment, $2 billion profit already. I think it's fun because they need their customers to really have big buy-in. So when their potential customers buy in and commit to certain types of products, it makes a lot of sense, right?

Dylan Patel: And it's kind of funny in a way because in the past, there was this whole thing around how Intel was sued for being anti-competitive with their chipsets. NVIDIA actually got a settlement from Intel way back when the graphics were separate from the GPU and the graphics were really put on the chipset which had all this other IO, like USB and all this stuff.

Dylan Patel: So it's kind of a poetic turn of events that now Intel is going to make a chiplet and package it alongside a chiplet from NVIDIA and then that's like a PC product. It's kind of poetic that everything's gone full circle and Intel's sort of crawling to NVIDIA. But actually, it might just be the best device. I don't want an ARM laptop because it can't do a lot of things, and so an x86 laptop with NVIDIA graphics fully integrated would be probably the best product in the market.

Host: So are you optimistic? How do you think this will go?

Dylan Patel: I mean, sure I hope, right? I'm a perpetual optimist on Intel because I have to be. I was thinking that the structure of the deal that at least a lot of the government folks and Intel were sort of trying to go for was big customers and the biggest suppliers directly give capital to Intel. But this is sort of the other way around where they're buying some of the stock, having some ownership, but they're not really diluting the other shareholders.

Dylan Patel: And then the other shareholders will get diluted, or everyone will get diluted, when Intel finally does raise the capital from the capital markets. But because they've announced these deals, and they're pretty small, right? $5 billion NVIDIA, $2 billion SoftBank. U.S. government was $10 billion. These are still relatively small.

Host: Pretty small, yeah.

Dylan Patel: Yeah, on the nature of things, right? I mean, last time I think I said Intel needs like $50 billion, right? Now when they go to the capital markets, it's better. And hopefully they get another couple of these announcements. Maybe there's all sorts of speculation that Trump is involved in getting these companies to invest. NVIDIA, and now the government as well, of course.

Dylan Patel: And now, is Apple going to come invest, right? And also do something with Intel? Or who else will come in? And that'll really boost investor confidence and they can dilute or go get debt. Like a Warren Buffett coming into a stock. Jensen is like the Buffett effect for the semiconductor world.

Host: Guido, you were the CTO of the Intel Data Center and AI Business Unit. What are your thoughts?

Guido Appenzeller: I think it's really good for customers and consumers in the short term. Having both Intel and, especially for the laptop market, having to collaborate is amazing. I wonder what's going to happen with any of the internal graphics or AI products at Intel. They might just push a reset and give up on that for now. They currently don't have anything competitive. There was the Gaudi effort that's more or less done. There was the internal graphics chips, which never competed really at the high end. So from that perspective, it makes a lot of sense for both sides.

Guido Appenzeller: Look, I think for Intel, they needed a breath of fresh air. They were sort of desperate. So I think it's a very good thing.

Guido Appenzeller: I think AMD is fucked. If your two arch nemeses suddenly team up, that's the worst possible news you can have. They were already struggling. Their cards are good. Their software stack is not. They were getting very limited traction. They now have a bigger problem outside.

Guido Appenzeller: I think ARM is a little bit screwed as well because their biggest selling point was sort of like, "Look, we can partner with everybody that doesn't want to partner with Intel." In a sense, they're number one. NVIDIA is probably the most dangerous of the future CPU competitors. So they now suddenly have access to Intel technologies and might get in that direction. And it remixes the card.

Guido Appenzeller: I did not see this coming. I think it's an amazing development.

Host: Yeah, it will be very interesting to see this play out.

Host: To Eric's point, PAC News Week, the other thing that we wanted to pick your brain on, since we have you here, Dylan, is the other news dropping on Huawei unveiling their kind of AI roadmap. Obviously they're hyping up the capabilities. I think you guys have been sort of ahead of the curve of trying to gauge what can the 950 supercluster actually do, but would love your thoughts on everything that's going on from the China front.

Host: This is kind of coupled with DeepSeek saying their next models are going to be on domestically produced Chinese chips, the Chinese government kind of banning companies from buying the produced specifically for China NVIDIA chips. So there's just sort of a lot of dominoes falling right now in the semi-market in China, but would love your take overall and drill into some detail.

Dylan Patel: Yeah, I think when you sort of zoom out to even like, let's walk from 2020, because I think it's really important to recognize how cracked Huawei is, or even just historically, they've always been really good. Sure, initially they stole Cisco source code and firmware and all this stuff, but then they rapidly passed them up as well as every other telecom company.

Dylan Patel: In 2020, they released an Ascend chip and submitted to impartial public benchmarks and they were the first to bring 7nm AI chips to market. They were the first to have that. Now you can still say NVIDIA was ahead but the gap was like nothing. And this is when they could access the full foreign supply chain. This was when they just passed Apple to be TSMC's largest customer. They were clearly ahead of everyone on a manufacturing supply chain design standpoint on a total basis.

Dylan Patel: Now, of course, NVIDIA still had higher market share, but it was so nascent then, they could have really taken over the market. Huawei got banned by the Trump 1 administration from accessing, and then it went into effect in 2020, the full ban. And so they were only able to make a small volume of these chips, but they had trained significant models on these chips that they made then.

Dylan Patel: And then over the next couple years, NVIDIA continued to accelerate. Huawei, because they were banned from TSMC, had to go and try and figure out how to manufacture at SMIC, the domestic TSMC. And then they were also, in parallel, trying to go through shell companies to manufacture at TSMC and acquire memory from Korea and so on and so forth.

Dylan Patel: So by the end of 24, this had gotten in full swing and it was caught, right? It was caught and they finally shut it down. But they were able to acquire 3 million chips, 2.9 million chips from TSMC through these other entities. Roughly $500 million worth of orders which ends up being a billion dollar fine that the US government gave TSMC if I recall correctly. At least there was a Reuters article which is important and interesting to gauge because the number of Ascends floating out there has not consumed this entire capacity yet.

Dylan Patel: So now we get to 2025. The H20 got banned in the beginning of the year. NVIDIA had to write off huge amounts of money. Our revenue estimate for NVIDIA in China for just H20 was north of $20 billion because that's what they were booking in capacity, or had to write off. And then it got banned. They cut the supply chain. They just said, "No, we're not doing this anymore." They had their inventory, gets re-approved, they resell the inventory, but now they're like, "Do we even restart production?" That is NVIDIA's question.

Dylan Patel: And now you have China saying, "Hey, we don't need NVIDIA, we have domestic alternatives," whether it be Huawei or Cambricon. These companies have capacity, but most of this capacity is still foreign produced, whether it be wafers from TSMC, memory from Korea, Samsung and SK Hynix. So the question is sort of like, how much can they do domestically?

Dylan Patel: And there's sort of two fronts there, right? There's the Logic, i.e. replacing TSMC, and there's the Memory, i.e. replacing Hynix, Samsung, Micron. And on the Logic side, they're behind, but they're really ramping there, and I think they can sort of get to the production capacity estimates needed, and the US is still allowing them to import all the equipment necessary, pretty much.

Dylan Patel: The bans are really for beyond the current generation of technology, beyond 7 nanometer. The bans are really for 5 nanometer and below. Even though the government says they're for 14 nanometer, the actual equipment that's banned is only for below 7 nanometer. And so they'll be able to make a lot of 7 nanometer AI chips and maybe even get to 5 with using existing equipment for 5 nanometer rather than taking the new techniques.

Dylan Patel: And so there's the logic side and then there's the memory side. And the aspect of Huawei's announcement that was surprising was that they're doing custom memory, right? That's the part that is sort of like, "Hey, this is really exciting." They announced two different types of chips for next year. One that's focused on recommendation systems and pre-fill and then one that's focused on decode.

Host: There's a twin these days.

Dylan Patel: Yeah, so in NVIDIA, the same thing. They just announced a pre-fill specific chip recently. There are numerous AI hardware startups that are really focusing on pre-fill versus decode. And so the sort of split of inference up to two workloads, Huawei is doing the same thing for their next year chip. And what's interesting is the decode one has custom HBM.

Dylan Patel: What does that mean? What is the manufacturing supply chain? Because that's the one that's tricky. How much can they manufacture of that custom HBM? And NVIDIA and others are also adopting custom HBM only starting next year. So it's not like, yes, the manufacturing capacity is not there. Maybe it is going to consume a bit more power. It's going to be slightly lower bandwidth. But the fact that they're able to do some of the same things that NVIDIA plans to do, AMD plans to do, in their memory is evidence that they're catching up.

Dylan Patel: But then the main question that remains is production capacity. So as far as, "Hey, NVIDIA is banned in China, like they're saying don't buy NVIDIA chips," I think for a period of time that's fine for China, from a perspective of, "Hey, I'm China. That's fine because you have all this capacity that you shipped in in 2024, they haven't turned into AI chips. Now you're turning them into AI chips, you're running all that stockpile down."

Dylan Patel: What about the transition from running that stockpile down to ramping your new stuff? And that transition is the one that's really tricky. China is either shooting itself in the foot by not purchasing NVIDIA chips during that time period, or China is able to ramp. I think they'll be able to ramp. I think it'll take a little bit longer. And there will be a sort of a gap in between where China probably backtracks and says it's fine.

Dylan Patel: Like ByteDance is begging for NVIDIA chips. They don't want to use some Cambricon, they use some Huawei, but they really want to use NVIDIA because it's way better. They don't care about the domestic supply chain. They want to make the best models. They want to deploy their AI as efficiently as possible. The government can mandate them to not do it. It's not that NVIDIA is not competitive, it's that the government is trying to instigate it.

Dylan Patel: I guess the last sort of thing is there's always the argument of, "Hey, if banning NVIDIA chips to China is so good for China, why didn't China do it for itself?" They're finally doing it for themselves. Again, it'll be interesting to see. Smuggling is still happening, re-exportation of chips from other countries to China. That is still happening at some volume, low volume, lower medium volume.

Dylan Patel: But then the direct shipments of NVIDIA chips that are legally allowed to China are not necessarily happening today, but may have to restart at some point because China won't have the production capacity. They would just have so many fewer AI chips being deployed domestically versus the US. And at some point you kind of have to pick, "Am I all about the internal supply chain or am I all about chasing super powerful AI?"

Host: So is there an angle here about the negotiation angle as well? Because currently there's still discussions ongoing what exactly are the boundaries, what can be exported to China. So these are sort of well-timed announcements if you want to make a point that the US should allow more exports. Do you think that's a factor or not?

Dylan Patel: In the report we did a few weeks ago about the production capacity of Huawei and the supply chain, there was a bit in there that we wrote about how, honestly, if you were China and you do want NVIDIA chips, actually, how do you play this? And it's by hyping up your domestic supply chain. And it's like, "Yes, we can do everything." It's Huawei announced the most crazy shit possible, announce seven years of roadmaps, so you sit there and read your report basically.

Dylan Patel: I think they knew they were already banned and then like say, "We're banning NVIDIA." Right then the government official is going to think, alongside sort of lobbying from domestic players, "Of course we want to ship them better AI chips. We're losing this market. We can't lose this market." And it's sort of like it is 10,000 IQ, right? And we're here playing checkers while they're playing chess.

Host: Well, so I guess negotiating chip aside, in that report, you talked about HBM or high bandwidth memory being a bottleneck to Huawei. To your point on one of the surprising aspects of the announcement, do you think it's credible that it's no longer a bottleneck based on what they're saying? Or is it just hype?

Dylan Patel: I think production capacity-wise, it is still absolutely a bottleneck. Certain types of equipment required for making HBM need to be imported. They're working on domestic solutions, but as far as we know, they have not imported enough equipment for this.

Dylan Patel: Although, if you look at Chinese import data for different types of equipment, there's sort of like fabs spend roughly, it depends on the process technology, but fabs spend roughly different amounts of money on lithography, etch deposition, metrology. And historically, lithography has hovered around 17, 18% with EUV, it grew to 25%. But China, because they wanted to stockpile lithography and they were worried about the incoming ban, they were importing lithography at a much higher rate than that, like 30, 40% of their equipment imports were lithography. And they were just stockpiling lithography equipment.

Dylan Patel: This is sort of reversed now in that if you look at the monthly import-export data both into provinces in China but also out of countries, you can see that Etch specifically is skyrocketing. The main thing about stacking HBM is that when you have each wafer, you have to Etch create this thing called a through-silicon via so it can connect from the top to bottom and then you stack them on top of each other 12 high, 16 high for HBM. That's how you make super high bandwidth memory. Their import for Etch is skyrocketing now.

Dylan Patel: So it's like, they don't have the production capacity yet. How fast can they ramp it is a function of how much equipment can they get, and the yields. Improving yields is really hard on manufacturing. Intel and Samsung are really good, and TSMC is just amazing. Not that those companies suck, I think is a better way to put it.

Dylan Patel: And so, it's those two things. I think Yield, they haven't even started production of HBM3, right? They've only done some sampling of HBM2. HBM3 came out a few years ago. So there's still quite a bit of ways on going up the learning curve. Obviously, I expect them to catch up faster than it took the technology to be developed because it exists in the world. We know how to do it. It's just a matter of actually doing it versus inventing it.

Dylan Patel: And then the other one is sort of the production capacity. A couple months of import-export data is not enough to set up for years worth of supply chain build-up, which is what we have today in Korea for the Korean companies. Now, Hynix is also investing in the U.S., in Illinois, and then Micron is primarily in Japan. The American memory company is primarily in Japan and Taiwan, but they're also expanding in Singapore and the U.S. now.

Dylan Patel: There's so much capital that's been invested. It would take some time for China to build up that production capacity to actually match the West. And when I say the West, I mean East Asia in production, non-China, East Asia in production capacity. So it'll take some time to get there. I think it's like, "Hey, we can design this." It's always a question of can we manufacture?

Dylan Patel: And then the thing that Jensen would say is, "You're betting on China not being able to manufacture?" It's a matter of when, not if. And that's the whole calculus that I think the US government has to be aware of when they're like, "Hey, what level of AI chips do we sell? Do we sell everything?" Probably not because AI is far more powerful. The end market of AI is going to be way larger than the end market of semiconductors and equipment.

Dylan Patel: "Do we sell, what level do we sell at?" Well, how much can China make at each specific performance tier and then analyze that and what's the volume and then figure out what is okay, which is maybe a little bit above or around the same level.

Host: Yeah. So if you, to your point on playing chess versus checkers, if you're Jensen, what would your next move be, given the situation at hand?

Dylan Patel: It's both partially true that he's afraid of Huawei more than he is of AMD. He called them formidable. Huawei beat Apple. They passed Apple up in TSMC orders. They passed Apple up in phone market share. Not in the US, but in many parts of the world before the bans came down. Even now they're growing back again in market share without Western supply chains. They've done this to numerous other industries. I would say Apple's a formidable competitor. They've beaten a lot of industries. It's reasonable that he's afraid of them. He's not afraid of AMD.

Dylan Patel: I think the best thing is what Huawei announced is reality rather than their hope target. And so away all doubt on manufacturing capacity, which I think is not fair. I think manufacturing capacity is a real bottleneck for them. And then the yield learnings, a real bottleneck, like temporary, maybe. We'll see how long and we'll see how fast the rest of the NVIDIA technology advances past what Huawei is capable of, and how fast Huawei is able to close the gap.

Dylan Patel: But I think his main sort of pitch would be Huawei is real. They're a formidable competitor. They're going to take over not just the Chinese market, but also foreign markets, whether it be the Middle East or Southeast Asia or South Asia or Europe or LATAM. Everywhere besides America.

Dylan Patel: I think Noah Smith has this analogy. This whole idea is that you should Galapagos China. Make them have their own domestic industry that is so different from the rest of the world. Kind of what happened with Japan in the 70s and 80s and 90s, their PCs were so specific and hyper-optimized to the Japanese market with the weird scroll wheel on these Japanese PCs. You literally, it's like you go like this and it scrolls. And then the touchpad is a circle and then that's around it. Things like that are so weird. And the rest of the world doesn't care, but Japan market likes it, right?

Dylan Patel: And his whole idea is like, let's Galapagos them, i.e. keep their technology within China and then that's like deadweight loss and they never expand outside versus that we serve the whole world. But the whole risk is that the opposite can also happen, right? Our technology is hyper-optimized to running language models at this scale and RL and you keep like hardware software co-design can take you down a path of the tree that is a dead end. And then China, because they're not allowed to access this tree, they're like, "Oh, okay." And then they end up in the optimal spot. We had a local minima, they had a local maxima, they had a global maxima. That's sort of like technological Galapagos thing is sort of what Noah Smith's analogy is. I like it a lot. I don't know if it's accurate, but it's an interesting one.

Host: Yeah, I love that.

Host: Well, actually, maybe just taking a step back from current events, even though there's so much to talk about right now. Last time you appeared with us, NVIDIA came up, obviously, and you talked about a couple of the potential paths forward for NVIDIA. Give us maybe the bull case, the bear case.

Dylan Patel: Yeah, fair enough. There's a lot embedded in their numbers now, but what's interesting is consensus for the banks is for across the hyperscaler, so Microsoft, CoreWeave, Amazon, Google, and Oracle, Meta, so it's the six hyperscalers, the consensus for the banks is $360 billion of spend next year across all of them. And my number is closer to 450, 500. And that's based on all the research we do on data centers and tracking each individual data center in the supply chains.

Dylan Patel: So this is just NVIDIA spend. This is CapEx for the hyperscalers. CapEx gets split up across different companies, but the vast, vast majority still goes to NVIDIA. And NVIDIA is in a position not where they can't take share. They grow with the market or defend share. And so the question is, how fast is the growth rate of CapEx for hyperscalers and other users?

Dylan Patel: And the reason I included Oracle and CoreWeave is hyperscalers, even though they're traditionally not called hyperscalers because they are opening eyes hyperscaler. So when you look at the Oracle announcement, first of all, the Oracle announcement, I don't understand why people don't think this is crazier. They did the most unprecedented thing in the history of stocks and companies ever. They gave a four guidance and it made Larry the richest man in the world.

Dylan Patel: Anyways, the question is how fast does revenue grow? Do you think OpenAI, which signed a $300 billion plus deal with Oracle, will actually be able to pay $300 billion across raising capital and revenue? And it gets to a rate of over $90 billion a year in just a handful of years. Do you believe the market will grow that fast? It's very possible, yes.

Dylan Patel: It's very possible for OpenAI. What is their revenue going to be exiting next year? Some people think $35 billion, some people think $40 billion, some people think $45 billion ARR by the end of the year next year. This year they hit $20 ARR. So if that growth rate is maintained, then all of that cost goes to compute plus all the capital they continue to raise.

Dylan Patel: And again, their financials that they gave to investors for their last round was like, "Hey, we're going to burn $15 billion next year." It's probably more likely going to be like $20 billion. And you stack this on and they're not going to be profitable until 2029. So they're going to continue to burn $15, $20, $25 billion of cash each year plus revenue growth, that's their compute spend.

Dylan Patel: And you do this for Anthropic, you do this for OpenAI, you do this for all the labs. It's very possible that the pie does get to more than $360 billion next year, $500 billion next year for total capex. And the pie continues to grow for hyperscalers. NVIDIA says actually it's going to be multiple trillions a year on AI infrastructure. And he's going to capture a huge portion of it. That's his bull case, right?

Dylan Patel: That's the bull case is AI is actually so transformative and the world just gets covered and data centers, and the majority of your interactions are with AI, whether it's business productivity and telling an agent to do some code, or you're just talking to your AI girlfriend, Annie, it doesn't matter. All of this is running on NVIDIA for the most part.

Dylan Patel: The bare case is, even if it does grow a lot... Let's save the bull case for a second. Fundamentally, the value creation, I think, personally, is there. Trillions of dollars of value with AI, I can totally see this happen. So assume it's true, where will NVIDIA top out? I guess how much do you believe in takeoffs, right?

Host: Yes.

Dylan Patel: So if there is a takeoff scenario where powerful AI builds more powerful AI builds more powerful AI or that creates more and more each level of intelligence enables more for the economy, how many monkeys can you employ in your business versus how many humans or how many dogs? What is the value creation of a human versus a dog? Sort of like the same with AI. So in this case, the value creation could be hundreds of trillions if not the negative after that.

Host: Do you need this? If we take every white collar worker and make them twice as productive with AI, that's in the hundreds of trillions, isn't it?

Dylan Patel: Yeah, but what is twice? If you talk to people at the labs, twice as productive, what does that even mean? It's replaced them. It is being 10 times better than death. I don't know how soon that happens. White-collar work is essentially useless without a constant stream of LLM tokens that make them productive. At that point, you basically can tax every single knowledge worker in the world, which is most workers in the world long-term.

Host: Yeah. What's your guess? Give us a number. What's the cap?

Dylan Patel: Cap? Why aren't we making a Matrioshka brain? I don't know. I mean, at some point, the machine says humans don't need to live and I need even more compute.

Host: One step before that, mate. Are we colonizing Mars yet?

Dylan Patel: TBD. I don't know, man. I find it completely impossible to predict anything beyond five years, given how much stuff is changing. Five is a large number. I'll leave it to economists, right? Honestly, supply chain stuff is three, four years out and that's it. And then fifth year is sort of like YOLO. So I just try and ground myself in the supply chain stuff.

Dylan Patel: Supply chain and then what is the adoption of AI? What's the value creation? What's the usage? And you can see that in a short horizon. Beyond that, I don't know, are we all going to be connected to computers, like BCIs and stuff? I don't know, dude. Are humanoid robots? You saw Elon's thing, right? He's like, "Yeah, humanoid robots are why Tesla's worth more than 10 trillion." So go there. Great. What is all that being trained on? Great. NVIDIA. Okay, awesome. So that's worth also 10 trillion. I don't know. It's too out there for me. I don't like the out there discussions.

Host: Very fair. Read some sci-fi books.

Host: So just pulling out the thread where you talked about this is kind of a throwaway comment, but how market share can't really grow just because it's such a dominant market share. And we talked about, or you guys talked about, the moat of NVIDIA last time. And obviously this moat is tied to maintaining that very high market share that they currently have. And I love this sort of historic journey you took us through with Huawei just earlier. Can you kind of walk through what NVIDIA did throughout history to build their moat?

Dylan Patel: It's super awesome because they failed multiple times in the beginning and they bet the whole company multiple times, right? Like, Jensen is just crazy enough to bet the whole company, right?

Host: So just pulling out the thread where you talked about this is kind of a throwaway comment, but how market share can't really grow just because it's such a dominant market share. And we talked about, or you guys talked about, the moat of NVIDIA last time. And obviously this moat is tied to maintaining that very high market share that they currently have. And I love this sort of historic journey you took us through with Huawei just earlier. Can you kind of walk through what NVIDIA did throughout history to build their moat?

Dylan Patel: It's super awesome because they failed multiple times in the beginning and they bet the whole company multiple times. Jensen is just crazy enough to bet the whole company. Like, whether it was certain chips ordering volume before he knew it even worked, and it was like all the money he had left, or ordering volumes for projects he had not won yet. I heard a story from someone who's a gray beard in the industry, and I think would know, that NVIDIA ordered the volume for the Xbox before Microsoft gave them the order. They were just like, "Fuck it, YOLO." I don't know how true this is. I'm sure there's more nuance there, verbal indication or whatever, but the order was placed before he got the order.

There were cases with the crypto bubbles. There were a couple of them, but NVIDIA did their damn best to convince everyone on the supply chain that it wasn't crypto, that it was gaming, that it was durable, real demand, and that it was gaming and data center and professional visualization. Therefore, you guys should ramp your production. They all ramped production and spent all this capex on increasing production and building out new lines for them. They pay per item, and then they bought them and sold them and made shitloads of money. Then when it all fell apart, they just had to write down a quarter's worth of inventory. Whatever. Everyone else was like, "Well, crap, I have all these empty production lines."

What did AMD do then? Their chips were actually better for crypto mining on an amount of silicon cost versus how much you hash, but they just didn't. AMD was like, "Ah, we're going to not really raise production." It's a reasonable thing. It's sort of like strike while the iron's hot. The same has happened with NVIDIA in recent times. They've ordered capacity that no one believes multiple times. They see that in demand obviously, but in many cases their number for Microsoft was higher than Microsoft's internal planning. Then Microsoft's internal planning went up, but their number for Microsoft was way higher. It's like, "We just don't think Microsoft's going to need this much, even though they tell us this." Who the heck is like, "No, no, no, customer, you're going to buy more," and orders, right? Then when the orders come through the supply chain, it's like, "I have to put pay NCNR, non-cancel, non-returnable."

I asked a question in Taiwan once. There was Colette, which is the CFO, and Jensen, CEO. They were both there. It was a room full of mostly finance bros, and they were asking stupid finance questions three days before earnings. Obviously they just could not answer anything because it's SEC regulations. But my question to them was, "Look, Jensen, you're so vibes-driven and very gut-feel and very visionary. And Colette's CFO. She's amazing in her own right, but those personalities clash. How do you work together?" He's like, "I hate spreadsheets. I don't look at them. I just know." It's his response. Of course the best innovators in the world have really good gut instinct. The gut instinct to order with non-cancelable, what you don't know. They've had to write down over their history multiple times many, many billions of dollars in accumulative orders. Whether it be the H20, which is more regulatory, or other cases, they've ordered and had to cancel.

Host: Is that many billions?

Dylan Patel: It's many billions. Well, it depends. The crypto write-down was like multiple billion when their stock was less than $100 billion. It's just compared to the upside. I think everything you did was right. I think everything AMD did was wrong in that scenario. It is crazy, especially in a cyclical industry like semiconductors where companies go bankrupt all the time, which is why we have all this consolidation. Every down cycle, companies go bankrupt. If you look at it from a risk-return perspective, these bets were totally worth taking. If you look at it from, "I'm a CEO, I want to have predictable quarters for Wall Street," it's a very different story. I think that's where part of the tension is from.

I don't know if you've seen these Lee Kuan Yew edits where they're him saying some fiery speech and then it's some cool music at the end, showing different pictures of him. We made one of Jensen recently and put it on social media. Instagram, TikTok, XHS, Redbook, Twitter, all the different social media. I really liked it because he's like, "The goal of playing is to win. And the reason you win is so you can play again." He compared it to pinball where you just play all day and you keep getting more rounds. His whole thing is, "I want to win so I can play the next game." It's only about the next generation. It's not about 15 years from now because it's a whole new playing field every time, or five years from now.

Guido Appenzeller: I think you're right. The risk or reward is correct. Few people take these kinds of risks. It's the only semiconductor company that's worth north of $10 billion that was founded as late as it was. MediaTek was in the early 90s, and then NVIDIA. Everyone else is from the 70s mostly.

Host: Yeah, I think you raised this great point on betting the farm. He's actually been wrong a couple times, to your point. Mobile, right? What the hell happened with mobile?

Dylan Patel: Exactly. And he still takes them.

Host: I think Mark actually had this great conversation with Eric where he talked about being founder-run, where you have this memory of the risks you took to get to where you are today. In a lot of cases, if you're a CEO brought on later on, you're sort of like, "Okay, continue to steer the ship as is." But in this case, he remembers all the times they almost went belly up. He's like, "I've got to keep making bets like that." How do you think he's changed? He's been one of the longest-running CEOs, over 30 years. He's kind of right up there with Larry Ellison now. How do you think he's changed over the last 30 years? I'm 29. I don't know what he was like. I've watched a lot of old interviews.

Dylan Patel: NVIDIA was founded before I was born. I'm 96.

Host: Yeah, maybe anything over the last couple of years?

Dylan Patel: I think even watching old interviews, old presentations he's given, one thing is that he's just sauced up and dripped up. The charisma he's gotten has only gotten stronger. I don't know if it's quite relevant. The man has learned to be a rock star even more. He was always charismatic, but he's a complete rock star now. He was a rock star a decade ago too, it's just people maybe didn't recognize it.

The first live presentation I watched was CES 2014 or 2015. I was moderating gaming hardware subreddits at the time, a teenager. The dude is talking only about AI. He's telling all these gamers about AlexNet and self-driving cars. It's like, know your audience first of all. It has nothing to do with consumer electronics and gaming. At the time, I was half like, "Holy crap, this is amazing," but also half like, "I want you to announce new gaming GPU." On the forums, everyone was quickly like, "Screw this, I want to hear about the gaming GPUs. NVIDIA's price gouging." NVIDIA's always had to price the value plus a little bit because they were just smart enough to know. I'm guessing Jensen just has the gut feel of how to price things. He'll change the price on gaming watches up until right before the presentation. It really is a gut feel thing.

Anyway, he had that charisma to know what was right, but a lot of people were like, "Oh, no, whatever, Jensen's wrong, he doesn't know what he's talking about." But now he talks, people are like, "Oh, very, very." So it might just be that he's been right enough.

Host: There was a post on X recently that said he had moved up into god mode with a select group of CEOs.

Dylan Patel: Who's the other gods? It was Zuck, Elon, and Jensen.

Host: Nice, good crew to be in. So we pray to Silicon Valley. It's the cult now. Just one last thing on people. You mentioned Colette, his CFO. There's the famously loyal crew at NVIDIA, even though all of the OGs could retire at this point. Is there anyone akin to a Gwynne Shotwell at SpaceX or previously a Tim Cook to Steve Jobs at Apple that is at NVIDIA today?

Dylan Patel: He had two co-founders. One of them is not involved and hasn't been for a long time, but the other one was involved up until just a few years ago. So it's not just Jensen running the show, although he was running the show. There are quite a few people on the hardware side. There's someone at NVIDIA that's mythical to me. When you talk to the engineering teams, he leads a lot of engineering teams. He's a private person, so I don't want to say his name, but he's effectively like chief engineering officers in his role. People within his org will know who he is. There are people like that. He's intensely loyal, and there are a number of these types of people.

There's another fella who's like the guy who literally says, "We need to get this silicon out now, we're cutting features." That's what he's famously known for, and all the technologists in NVIDIA hate him. This is a second guy, also intensely loyal to NVIDIA, has been around for a long time. When you have such a visionary company, one problem is that you get lost in the sauce. "Oh, I want to make this, it's got to be perfect, amazing." You have to have that sort of person. Obviously they're close to Jensen for a reason, because Jensen also believes these things. Have the visionary future looking, but also screw it, cut it, we'll put it in the next one, ship. Ship now, ship faster. In a space like silicon, which is really hard to do.

The thing about NVIDIA that's always been super impressive, and it's from the beginning days, is their first successful chip. They were going to run out of money. He had to go get money from other people to even finish the development. Even then, he just had enough money because he'd already had a failed chip before this. The chip came back and it had to work, otherwise it would not. They could only pay for one mask set. Basically, you put these stencils into the lithography tool, and it says where the patterns are. You put the stencil in, you deposit stuff, you etch stuff, you deposit materials on the wafer, etch it away. You put the stencil in and you tell it where to put stuff. The deposition and etch keeps happening in those spots, and you stack dozens of layers on top of each other and make up a chip. These stencils are custom to each chip, and they cost today in the orders of tens of billions of dollars. Even back then, it was still a lot of money.

The typical thing with semiconductor manufacturing is, as good as you can simulate it, as good as you can do all the verification, you'll send a design in and you have to change it. There's going to be something. It's so hard to simulate everything perfectly. The thing about NVIDIA is they tend to just get it the first time. Even great executing companies like AMD or Broadcom often have to ship revisions. They're denoted in like A and then a number, or B and then a number. It's like two different parts of the masks. NVIDIA always ships A0 almost always. They sometimes ship A1. A lot of times, even if they'll start production of the A, which is basically the transistor layer, then the number is like the wiring that connects all the transistors together. NVIDIA will start production of the A and ramp it really high, and then just hold it right before you transition to the metal, just in case they do need to change the metal layers. The moment they're ready and they've confirmed that it works, they can just blast through a lot of production. Everyone else is like, "Oh, let's get the chip back. Oh, okay, A0 doesn't work. We've got to make this tweak, make this tweak, get the chip back." It's called a stepping.

Guido Appenzeller: We were very jealous of NVIDIA at that time. They consistently delivered in the first one. We did not. The data center CPU group, there was one product where I said A1. You go to B if you have to change the transistor layer as well. Intel got to E2 once. E2 is like a 15 revision. This is not a peak of AMD's. When they went skyrocketing on market share versus Intel was when Intel was at E2, 15 steppings. Because it's quarters of delay. It's catastrophic for a go-to-market.

Dylan Patel: Each time is a quarter of delay or something.

Guido Appenzeller: Yeah, it's absurd. So I think that's the other thing about NVIDIA. Screw it, let's ship it, let's get the volume ASAP, let's do these things. They have some of the best simulation, verification, et cetera, that lets them go from design to shipment as fast as possible, cutting out any unnecessary features that could delay it, making sure they don't have to do revisions so that they can respond to the market ASAP.

There's a story about how Volta, which was the first NVIDIA chip with TensorCores. They saw all the AI stuff on the prior generation P100 Pascal, and they decided we should go all in on AI. They added the TensorCores to Volta only a handful of months before they sent it to the fab. They said, "Screw it, let's change it." If they hadn't done that, maybe someone else would have taken the AI chip market. There are all these times where they just make major changes, but there are often minor things that you have to tweak. Number four maths or some architectural detail. NVIDIA is just so fast. The other crazy thing is they have a software division that can't keep up with that. If you come out with a chip and basically no stepping required, it's immediately in the market. Being ready with drivers and all the infrastructure on top of that is just super impressive.

Host: I love that point because you think of NVIDIA benefiting from tailwind after tailwind, but I think both of you are saying you have to move fast enough and execute well enough to take advantage of those tailwinds. If you think about nailing the video game tailwind, VR, Bitcoin mining, obviously AI now. One thing that Jensen talks about today is robotics, AI factories. Maybe my last question on NVIDIA: what do you think about the next 10 to 15 years? I know calling beyond five is hard, but what does NVIDIA's business look like?

Dylan Patel: It's really a question of, and I've asked some executives at NVIDIA this question because I really want to know, and they won't answer it obviously, but what are you going to do with your balance sheet? You are the most high cash flow company, and you have so much cash flow. The hyperscalers are all taking their cash flow way down because they're spending on GPUs. What are you going to do with all this cash flow? Even before this whole takeoff, he wasn't allowed to buy ARM. So what can he do with all this capital and all this cash? Even this $5 billion investment in Intel, there's regulatory scrutiny there. It's in the announcement, subject to review. I imagine that'll get passed, but he can't buy anything big. He's going to have hundreds of billions of dollars of cash on his balance sheet. What do you do?

Guido Appenzeller: Is it start to build AI infrastructure in data centers?

Dylan Patel: Maybe. But why would you do that if you can just get other people to do it and just take the cash?

Guido Appenzeller: Well, he's investing in those, right? Investing peanuts. He gave CoreWeave a backstop recently because today it's really hard to find a large number of GPUs for burst capacity. Like, "Hey, I want to train a model for three months." I have my base capacity where I don't know my experiments, but I want to train a big model for three months, done. We know from our portfolio. NVIDIA sees this issue. They think it's a real problem with startups. It's why the labs have such an advantage. But what if I could, right now, most companies in the Valley spend 75% of their round on GPUs. At least.

Dylan Patel: Yeah.

Guido Appenzeller: What if you could do 75% in three months on one model run, really scale, have some sort of competitive product, and then you have the model, then you raise more capital and start deploying? What do you do with it? Is it start buying a crap load of humanoid robots and deploying them? They don't make really amazing software for them in terms of the models. The layer below is great. Where they deploy their capital is a question.

Dylan Patel: He has been investing up and down the supply chain a little bit. Investing in the neoclouds, investing in some of the model training companies. But again, small fries. He could have just done the entire Anthropic round if he wanted to. Of course he didn't. And really got them to use GPUs. Or he could have done the entire OpenAI round. He could have done the entire xAI round.

Host: Do you think these are things he should be doing?

Dylan Patel: I don't know. We'll quote you for the next round. He could make venture a dead industry. Take all of the best rounds.

Guido Appenzeller: But it's a lot of business.

Dylan Patel: That's why I've converted. I think picking winners is obviously really tough for him because he has customers all across this ecosystem. If he starts picking winners, then his customers will be even more anxious to leave and give even more effort to AMD or some startup or their internal efforts. Buying TPUs, whatever it is. He can do a little bit. A few hundred million in an OpenAI round is fine, or a few hundred million in an xAI round is fine. CoreWeave, everyone's throwing a fuss about it, but he invested a couple hundred million early on, plus rented a cluster from them for internal development purposes instead of renting it from a hyperscaler, which is cheaper for NVIDIA to do. It's better for them to do it from them than the hyperscalers. Is he really backstopping CoreWeave that much, or any of the other customers in NeoClouds? There's some investment, but it's more like, "This is a good cloud, we'll throw 5% or 10% of the round." It's not he's taking 50% plus of the round.

Host: Is he also reshaping his market? A couple of years ago, there were four big purchases of these cards. You just listed six. To what extent is that a strategy?

Dylan Patel: It is. I think it absolutely is. But he didn't have to put much capital down to do this. Just chip one earlier than the other? I don't know. If you look at the grand amount of capital that he spent investing in the NeoClouds, it's a few billion dollars. But he has a lot of other levers if he wants to.

Host: Allocations, as you mentioned.

Dylan Patel: What's nice is, historically, you gave volume discounts to the hyperscalers, but because he can use the argument of antitrust, he's like everyone gets the same price. So fair. It's very fair.

Guido Appenzeller: What should he do? What should guide his... I think there's the argument he should invest in data centers, and only the data center layer, not what goes in the data center, so that more people build data centers. If the market demand continues to grow, data centers and power are not the issue. Invest in data centers and power. I've said that to them. They should invest in data centers and power, not in the cloud layer, because the cloud layer is quite commoditized. It's commoditized your complement, is the whole phrase. I won't say being a cloud is commoditized, but you have a lot of competitors who are decent now. You've educated the commercial real estate and other infrastructure investment firms into going into AI infra as well. So I don't think it's the cloud layer that you invest in. Do you invest in data centers and energy? Because that's the bottleneck for your growth. A, how much people want to spend and can spend, and B, the ability to actually put them in data centers. And then robotics. I think there are areas he could invest in, but nothing requires $300 billion of capital. So what do you do with the capital? I really don't know, and I feel like Jensen has to have some idea, some visionary plan here, because that's what shapes the company. They could just continue to, I mentioned $200 billion of free cash flow, $250 billion of free cash flow a year. What do they do with it? Do they just buy back stock forever? Do they go the Apple route? The reason why Apple hasn't done anything interesting in nearly a decade is they've got someone not visionary at the head. Tim Cook's great at supply chain. They're just plowing the money into buybacks. The self-driving car thing failed. We'll see what happens with AR, VR. We'll see what happens with wearables, but Meta and OpenAI might be even better than them. What does he invest in? I have no clue. What requires so much capital and actually gets a return? The easy thing is my cost of equity. I just buy back stuff. It doesn't completely change the company culture.

Dylan Patel: I think that's another thing. There are probably areas you could invest it in, but you suddenly end up with the company doing two completely different things, which are very difficult to keep on.

Guido Appenzeller: But they do like 10 completely different things.

Dylan Patel: One way to look at it is we build AI infrastructure. In the guise of we build AI infrastructure, robots, humanoids around the world are AI infrastructure. Or data centers and energy is AI infrastructure. The humanoids would totally work. If you're suddenly pouring concrete and building power plants, it has completely different cultures, completely different stuff. It's getting much, much harder.

Guido Appenzeller: I agree. There are different ways to do it, like invest in the various companies or backstop the building of power plants. No one will have to build power plants because they're 30-year underwriting things. There are all these different areas where could it use capital to allow something to happen, not necessarily owning it himself. Look, and bear in mind at Intel, one of the biggest problems we had was that our customer base sucked. Most of the chips went to the large hyperscalers, which they're way too concentrated, and they build their own chips, so you can push down your prices. Honestly, spending it on diversifying the cloud...

Dylan Patel: The problem was in 2014, you guys should have just charged so much that your margins were 80%. What would the world have done? Nothing.

Guido Appenzeller: The margins were pretty good back then. That wasn't the problem.

Dylan Patel: That was the primary problem. If they were 60, 65, they were 80.

Guido Appenzeller: Still, yeah.

Dylan Patel: That was Jensen. It's Jensen on a server podcast. UGSD is kicking in here.

Host: Well, wait, I think Guido's comment is actually a really good segue into something else we wanted to talk to you about, which is the hyperscalers. One of the reasons that I love reading SemiAnalysis is you guys make these out-of-consensus calls that you're often right about. One of them recently was calling...

Dylan Patel: Only often?

Host: You have a Jensen hit rate. It's very high.

Dylan Patel: Where's my billion dollar PV positive bet?

Host: But the one that caught my eye was Amazon's AI resurgence. I wanted to talk to you a little bit about that. I think we found it pretty interesting being on the ground, helping our portfolio companies pick who their partners are. We have some micro data on this, but you sort of walk through why they are behind.

Dylan Patel: In Q1 2023, I wrote an article called "Amazon's Cloud Crisis." It was about how all these neoclouds are going to commoditize Amazon. It was about how Amazon's entire infrastructure was really good for the last era of computing. What they do with their elastic fabric, ENA and EFA, their NICs, the whole protocol and everything behind them, what they do for custom CPUs, etc. It was really good for the last era of scale-out computing and not the era of scale-up AI infra. How NeoClouds were going to commoditize them, and how their silicon teams were focused on cost optimization, whereas the name of the game today is max performance per cost. But that often means you just drive up performance like crazy. Even if cost doubles, you drive up performance more, triples, because then the cost per performance falls still. That's sort of the name of the game today with NVIDIA's hardware. It ended up being a really good call. Everyone was calling us out like, "No, you're wrong." This was when Amazon was the best stock, and Microsoft really hadn't started taking off yet, nor had Oracle and all these others. Since then, Amazon has been the worst-performing hyperscaler. The call here is that they still have structural issues. They still use Elastic Fabric, although that's getting better, still behind NVIDIA's networking, still behind Broadcom's slash Arista type networking, Nix. Their internal AI chip is okay, but the main thing is that they're now waking up and being able to actually capture business.

The main call here is that since that report, AWS has been decelerating revenue. Year-on-year revenue has been falling consistently. Our big call is that it's actually going to start re-accelerating. That's because of Anthropic, because of all the work we do on data centers, tracking every single data center, when that goes online and what's in there, the flow-through on cost, how much the chips cost, the networking cost, the power cost. You know how much generally margins are for these things, and you can start estimating revenue. When we build all that up, it's very clear to us that they trough on AWS revenue growth at this point. This is the lowest AWS revenue growth will be on a year-to-year basis for at least the next year. It's re-accelerating to north of 20% again because of all these massive data centers they have online with Tranium and GPUs. It depends on which one, it depends on which customer. The experience is not as good as, say, a CoreWeave or whatever, but the name of the game is capacity today. CoreWeave can only deploy so much, they only can get so much data center capacity, and they're really fast at building. But the company with the most data center capacity in the world, and still today, although they may get passed up in the next two years, is Amazon. Actually, they will get passed up based on what we see, but incrementally, Amazon still has the most spare data center capacity that's going to ramp into AI revenue over the next year.

Guido Appenzeller: Let me ask a question. Is that the right type of data center capacity? For the high-density AI build-outs today, you need massively more cooling, you need to have enough water close by, you need to have enough power close by. Is it the right place, or is that the wrong type of thing?

Host: Data center capacity, in this sense, I mean all the way from power secured to substations built to transformers to you can provide the power whips to the racks. Now obviously the data center capacity will differ. Historically, Amazon's had the highest density data centers in the world. They went to like 40 kilowatt racks when everyone was still at 12. And if you've ever stepped foot inside of most data centers, they're pretty cool and dry-ish. If you step inside of an Amazon data center, they feel like a swamp. It feels like where I grew up. It's humid and hot because they're optimizing every percentage.

Your point here is that Amazon's data centers aren't equipped for the new type of infrastructure, but when you compare them to the cost of the GPU, having a complex cooling arrangement is fine. We made a call on Acera Labs a few months ago, a couple months ago when they were at 90 and it's gone to 250 the month after because of what orders Amazon is placing with them. There are certain things with Amazon's infrastructure. I won't get too much into it, but their rack infrastructure requires them using a lot more of a Sterilabs connectivity products. And the same applies to cooling. They just have to use a lot more of this stuff. But again, this stuff is inconsequential on cost compared to the GPU.

Guido Appenzeller: My question was more like, look, I may need a major river close by for cooling at this point. In many areas, I just can't get enough water. It's probably power in the same region.

Host: There are two gigawatt scale sites that they have, power all secured, wet chillers and dry chillers all secured, everything's fine. It's just not as efficient, but that's fine. They're going to ramp the revenue, they're going to add the revenue. Not that I necessarily think Amazon's internal models are going to be great, or that their internal chip is better than NVIDIA's or competitive with TPU, or their hardware architecture is the best. I don't necessarily think that's the case. But they can build a lot of data centers and they can fill them up with stuff that will be rented out. It's a pretty simple thesis.

Guido Appenzeller: How important has Anthropic been to the co-design for Tranium? Because I remember we had a portfolio company. This was summer 2023. They invited them to AWS. They spent, man, I think eight hours with them over the course of a week trying to figure out Tranium back then. It was just impossible to work through. Is that... obviously that portfolio company hasn't gone back and tried it now, but how different is it now based on what you're hearing?

Host: Oh, it's still bad.

Guido Appenzeller: Okay, okay.

Host: It's tough to use. This is the argument that every inference company offers, including the AI hardware startups: because I'm only running three different models at most, I can just hand-optimize everything and write kernels for everything and even go down to an assembly level. How hard can it be? It is pretty hard. But you tend to do this for production inference anyway. You aren't using QDNN, which is NVIDIA's library that's super easy to generate kernels. You're not using these ease-of-use libraries. When you're running inference, you're either using Cutlass or stamping out your own PTX, or in some cases people are even going down to the SaaS level.

When you look at an OpenAI or an Anthropic, when they run inference on GPUs, they're doing this. And the ecosystem is not that amazing once you get all the way down to that level. It's not like using NVIDIA GPUs is easy now. You have an intuitive understanding of the hardware architecture because you work on it so much, and everyone's worked on it, and you need to talk to other people. But at the end of the day, it's not easy. Whereas with Tranium or TPUs, the hardware architecture is a little bit more simple than a GPU. Larger, more simple cores rather than having all this functionality. Less general. So it's a little bit easier to code on. There are tweets from Anthropic people saying when they are doing that low level, actually they prefer working on Tranium and TPU because of the simplicity.

To be clear, Tranium and TPU—I mean Tranium especially—is very hard to use. Not for the faint of heart. It's very difficult. But you can do it if you're just running... If I'm Anthropic and I must only run Claude 4.1 Opus and Sonnet, and I won't even run Haiku—I'll just run Haiku on GPUs—I'm just going to run two models. And actually, I'll just run Opus on GPUs too, and Tritipius. Sonnet is the majority of my traffic anyway. I could spend the time. And how often am I changing that architecture? Every four or six months? It's not even changing that much, honestly.

Guido Appenzeller: I mean, from three to four definitely did change, right?

Host: Yeah, I mean, define architectural change. At a high level, the primitives are more or less the same across the last couple of generations. I don't know enough about Anthropic's model architecture, to be honest, but I think from what I've seen at other places, there have been enough changes that it takes time to program this. The main thing is, if I'm Anthropic and I have 7 billion ARR now or whatever, by the end of next year, north of 20, ARR is maybe even 30. My margins are 50%, 70%. That's $15 billion of training that I need that can run on Sonnet. And most of that's going to be Sonnet 4, 5, whatever it is. It's going to be one model serving most of the use cases. So I could spend the time and it'll work on this hardware.

Guido Appenzeller: Totally. Maybe on the topic of non-consensus calls you've made, and maybe I'll move to another cloud. In June, you guys said that Oracle is winning the AI compute market. And then in this pod, we've already referenced the big jump, obviously, that Oracle had. I think it was the single largest gain that a company with over $500 billion in market cap has ever had.

Host: Was the 2023 Q1 NVIDIA not bigger?

Guido Appenzeller: It might have been smaller. I think it was maybe close. We'll fact check ourselves. That's amazing. But obviously, this is the massive commitment that was announced. Can you walk us through why you made that call then and just sort of why Oracle is poised to do so well in such a competitive space?

Host: Yeah, so Oracle—they're the largest balance sheet in the industry that is not dogmatic to any type of hardware. They're not dogmatic to any type of networking. They will deploy Ethernet with Arista. They'll deploy Ethernet through their own white boxes. They'll deploy NVIDIA networking, InfiniBand or Spectrum X. And they have really good network engineers. They have really great software across the board. Again, like ClusterMax. They were ClusterMax Gold because their software is great. There are a couple things that they needed to add that would take them higher, and they're adding those—to Platinum, which was where CoreWeave was.

So you couple two things. OpenAI has got insane compute demand. Microsoft is quite pansy. They're not willing to invest. They don't believe OpenAI can actually pay the amount of money. I mentioned earlier, the $300 billion deal—you don't have $300 billion. Oracle's willing to take the bet. Now, of course, there's a bit more security in the bet in that Oracle really only needs to secure the data center capacity. This is how we came across the bet. And we've been telling our institutional clients, especially in a super detailed way—whether it be the hyperscalers or AI labs or semiconductor companies or investors—in our data center model, because we're tracking every single data center in the world.

Oracle doesn't build their own data centers either, by the way. They get them from other companies, they co-engineer, but they don't physically build them themselves. So they're quite nimble in terms of being able to assess new data centers, engineer them. So we saw all these different data centers Oracle is snatching up in deep discussions, signing, et cetera. And so we have, hey, gigawatt here, gigawatt there, gigawatt there. Aveline, two gigawatt. You have all these different sites that they're signing up and in discussions with, and we're noting them. And then we have the timeline because we're tracking the entire supply chain, we're tracking all the permits, regulatory filings through language models, using satellite photos constantly, and then supply chain of chillers, transformer equipment, generators, et cetera. We're able to make a pretty strong estimate quarter by quarter in our data center model of how much power there is for each of these sites.

Some of these sites that we know of aren't even ramping until 2027, but we know that Oracle signed it. And we have this ramp path. So then it's this question of, okay, let's say you have a megawatt, for simplicity's sake, which is a ton of power, but now it doesn't feel like much. We're in the gigawatt era. If you're talking about a megawatt, you fill it up with GPUs. How much do the GPUs for a megawatt cost? Actually, it's even simpler to do the math. If I'm talking about a GV200, each individual GPU is 1200 watts, but when you talk about the whole system, it's roughly 2000 watts all in, everything, simplicity's sake, $50,000 per GPU. The GPU doesn't cost them—there's all the peripheries. So $50,000 CapEx for 2000 watts, so $25,000 for 1000 watts. And then what's the rental price for GPU? If you're on a really long-term deal, volume 260, 270 in that range, then you end up with, oh, it costs like $12 million per megawatt to rent a megawatt.

And then each chip is different. So we track each chip, what the capex is, what the networking is. So you know what each chip is. You can predict what chips they're putting in which data centers, when those data centers go online, how many megawatts by quarter. And then you end up with, oh, well, Stargate goes online in this time period. They're going to start renting it this time. It's this many chips. Each Stargate site. And so therefore, this is how much OpenAI would have to spend to rent it. And then you prick that out. And we were able to predict Oracle's revenue with pretty high certainty, and we matched pretty dead on what they announced for 25, 26, 27. And we were pretty close on 28. The surprise for us was that they announced some stuff for 28, 29 data centers that we haven't found yet, but we'll find them, of course.

And this methodology lets you see what data centers you're getting, how much power, what they're signing, how much incremental revenue that is, when that comes online. And so that's the basis of our Oracle bet. Obviously in the newsletter we included a lot less detail, but it was that thesis: hey, they have all this capacity, they're going to sign these deals. In our newsletter we talked about two main things. We talked about the OpenAI business and then we talked about the ByteDance business. And presumably tomorrow on Friday there's going to be an announcement about TikTok and all this, but the ByteDance business—huge amounts of data center capacity that Oracle is also going to lease out to ByteDance. So we did the same methodology there. With ByteDance it's pretty certain they'll pay because they're a profitable company. With OpenAI it's not, and so there's got to be some error bars as you go further out in terms of will OpenAI exist in 28, 29, 30 and will they be able to pay the $80-plus billion a year that they've signed up to Oracle with? That's the only risk here.

And if that happens, Oracle's downside is also somewhat protected because they only signed the data center, which is a minority of the cost. The GPUs are everything. And the GPUs they purchase one to two quarters before they start renting them. So the downside risk is pretty low for them. If they don't get the deal, well, they don't get the revenue, but it's not like they're stuck with a bunch of assets they bought that are worthless.

Guido Appenzeller: Is there another angle here? OpenAI and Microsoft were BFFs and now they've filed divorce papers and they just want to diversify, and that's pushing them away towards other providers?

Host: Yeah, so Microsoft was the exclusive compute provider. It got reorged to right of first refusal.

Guido Appenzeller: Was it not your last choice or something like that?

Host: No, it's still right of first refusal, but it's like Microsoft... Those two are not mutually exclusive. Well, if OpenAI is like, "We're going to sign an $80 billion contract for the next five years. You guys want it?" And they're like, "No." "Okay, cool." And then they go to Oracle. And OpenAI is sort of like, OpenAI needs someone with a balance sheet to actually be able to pay for it. And then they'll make tons of money off of OpenAI on the margins, on the compute, and the infra, and all these things. But someone's got to have a balance sheet. And OpenAI doesn't have a balance sheet. Oracle does.

Although, given the scale of what they signed, we had another source of information, which was that they were talking to debt markets. Because Oracle actually just needs to raise debt to pay for this many GPUs over time. Now, they won't do it immediately. They can pay for everything this year and next year from their own cash. But in 27, 28, 29, they'll start to have to use debt to pay for these GPUs, which is what CoreWeave has done. In many of the neoclouds, most of it's debt financed. Even Meta went and got debt for their Louisiana mega data center. Just because it's cheaper. It's literally better on a financial basis to do buybacks with your cash and get debt because the debt is cheaper than the return on your stock. It's a financial engineering thing.

But who's out there? It could be Amazon, it could be Google, it could be Microsoft. Or it could be Oracle or Meta. Meta's obviously not. Microsoft's chickened out. Amazon, Google, and Oracle. That's all that's left.

Guido Appenzeller: Google would be an awkward fit.

Host: Yeah, Google would be an awkward fit. Amazon would be a fine fit. But exactly, right? It's very dramatic.

Guido Appenzeller: Well, I guess maybe on the topic of these giant data center build-outs, you guys just released a piece on XAI and Colossus 2. Are you getting less impressed by these feats of building something this massive in six months? Or is it still very impressive to you guys?

Host: This is the thing I've said about AI researchers: they're like the first class of humans to think about things on an order of magnitude scale. Whereas people have always thought about things in terms of percentage growth, ever since industrialization. And before that, it was just absolute numbers. Humanity is evolving in terms of how we think because things are changing faster. Everything is on a log scale.

So it was really impressive when GPT-2 was trained on so many chips, and then GPT-3 was trained on 20k... sorry, GPT-4, 20k H100s. It was like, holy crap. And then it was the era of 100k GPU clusters. We did some reports around 100k GPU clusters, but now there are like 10 100k GPU clusters in the world. I was like, okay, that's kind of boring. But it's like 100k GPUs is over 100 megawatts. Now it's like literally in our Slack and some of these channels, it's like, "Oh, we found another 200 megawatt data center." There's someone who puts the yawning emoji every time. And I'm like, dude, what? Now it's only exciting if you do gigawatt scale. We're in the gigawatt era.

Guido Appenzeller: And I'm sure... maybe we'll start yawning to that too.

Host: But the log scale of this is crazy. The capital numbers are crazy. It's crazy enough that OpenAI did a $100 billion trading run. Then they did a billion dollar trading run. Now we're talking about $10 billion trading runs. It's crazy that we think in log scale, but yes, things are only impressive. What Elon's doing in Tennessee, in Memphis—first time was crazy. 100k GPUs in six months. He bought a factory in February of 24 and had models training within six months. And he did liquid cooling, first large-scale data center at this scale for AI doing liquid cooling, all these sorts of crazy firsts. Putting generators outside, cat turbines, all these different things to get the power. Mobile substations. All these different crazy things. Tapping the natural gas line that's running alongside the factory.

So he does this, and it's like, holy crap. And he did it for 100K GPUs, 200, 300 megawatts. Right now he's doing it for a gigawatt scale, and he's doing it just as fast. And so you would think this is obviously way more impressive than he did it again. But maybe I'm desensitized. It's like you've given the child too much candy, and now the child doesn't like apples.

So yeah, a gigawatt data center. There were all these protests around his Memphis facility. People like, "Oh, you're destroying the air." And it's like, have you looked around that area of Memphis? There is a gigawatt gas turbine plant that's just powering that area generally. There's a sewage plant that's servicing the entire city of Memphis. And there are open air pits of... there's open air mining. There's all sorts of disgusting shit around there, which is needed. We need that stuff to have a country run. And people are complaining about a couple hundred megawatts of generation.

So he got protests from all sorts of people. You got super into the politics side of things. Double ACP even protested him. So he really got some local municipalities to be like, "Oh, I don't like this." And so he couldn't do as much as he wanted to in Memphis, but he still needed the data center to be close because he wanted to connect these data centers super high bandwidth, super close. And he already had a lot of infrastructure set up there. So he bought another distribution center at this time. And it's still in Memphis, but the cool thing about Memphis is it's right across the border from Mississippi. So now it's like 10 miles away from his original one, but his facility is like a mile away from Mississippi, and he bought a power plant in Mississippi and he's putting turbines there. The regulation is completely different.

If the question is really about galvanizing resources and building it really fast, maybe Elon is ahead of everyone. He hasn't made the best model yet, or he doesn't have the best model, at least today I think. You could argue Grok 4 was the best for a little period of time. But it's truly amazing how fast he's able to build these things. And for first principles, most people are like, "Fuck. We can't build the power. We can't do power here anymore. I guess we have to find a new site." And it's like, no, just go across the border. Go to Mississippi.

Guido Appenzeller: And my favorite thing is Arkansas is right there. So Mississippi gets mad. I don't know. The regulations... all future data centers built in places where multiple states meet? Is that the...

Host: Four corners, yeah.

Guido Appenzeller: The optimal regulatory... I think there's one in the US. Is there a point in the US with five?

Host: I know there's a point with four. Four states intersect there. Maybe that's the recording status center.

Guido Appenzeller: Kind of concerning. All right. I'm going to buy real estate in that area.

Host: I've read it.

Guido Appenzeller: Well, I guess on the topic of maybe new hardware, you had this piece analyzing TCO for GB200s. And I'm going to ask this question on behalf of our portfolio companies, which it sounds like you're helping them already. But one of the findings that I thought was really interesting was TCO was sort of 1.6X H100s for GB200s. And so obviously there's this point on, okay, that's the benchmark for the performance boost that you're going to need to at least make the performance-cost ratio benefit from switching over. Maybe just talk about what you've seen from a performance standpoint and what do you recommend to portfolio companies, maybe at a smaller scale than XAI, who are thinking about new hardware, trying to get it. There's capacity constraints, obviously.

Host: Yeah, I mean, that's a challenge. With each generation of GPU, it gets so much faster that you end up wanting the new one. And in some metrics, you could say GB200 is three times faster than, or two times faster than the prior generation. Other metrics, you can say it's way more than that. If you're doing pre-training versus inference, they can run everything four bits. If you can run it four bits or just inference and take advantage of the huge NVLink, NBL72, there are ways you can squint and say GB200 is only 2x faster than H100, in which case 1.6x TCO. It's worthwhile. It's worth going to the next gen. But it's more marginal. It's not a big deal.

Then there are other cases where it's like, well, if you're running DeepSeek inference, the performance difference per GPU is north of like 6, 7x, and it continues to optimize for DeepSeek inference. And so then it's like, well, I'm only paying 60% more for 6x. And it's like, it's a 4x or 3x performance per dollar gain. Absolutely. And if you're running inference of DeepSeek, that can also include RL. And so the question is sort of... and then the other question is, well, the GPU is new. There's also B200, there's UB200, there's B200. B200 is much more simple from a hardware perspective. It's just eight GPUs in a box. So then it's not as much of a performance gain, especially in inference. But you have all the stability. It's an eight GPU box. It's not going to be unreliable.

The GV200s are still having some reliability challenges. Those are being worked through. It's getting better and better by the day, but it's still a challenge. But when you have an H100, H200, eight GPUs, one of them fails. You take the entire server offline and you have to fix it. If your cloud's good, they'll swap it in. But if it's GB200, what do you now do with 72 GPUs? If one fails, do you break the whole thing to get a new 72? The blast radius of a failure—GPU failure rates at best are the same and likely worse, gen on gen, because everything's getting hotter, faster, etc. So at best, the failure rates are the same. Even if you model the failure rates as the exact same, because you go from one out of eight to one out of 72, it's a huge problem.

So now what a lot of people are doing is they run a high-priority workload on 64 of them, and then the other eight you run low-priority workloads. Which is then like, okay, this is this whole infrastructure challenge. I have to have high priority workloads, I have to have low priority workloads. When a high priority workload has a failure, instead of taking the whole rack offline, you just take some of the GPUs from the low priority one, put it in the high priority one, and then you just let the dead GPU sit there until you service the rack at a later date. There are all these complicated infrastructure things that make it so, oh wait, actually that 3x or 2x performance increase in pre-training is lower because the downtime is higher. Or I'm not using all the GPUs always. Or I'm not smart enough or I don't have the infra to have low-priority and high-priority workloads.

It's not impossible. The labs are doing it. If I'm running a cloud, it's actually really hard because I probably have to rent the spot one, the spares out of spot instance or something.

Guido Appenzeller: No, no, no, because it's a coherent domain. It's NVLink. You don't want anyone touching that. So it has to be the end customer.

Host: You don't have to leave them because it's empty spares. It's even worse. The end customer usually will just be like, "I want them and I will." And the SLAs and the pricing, everything is accounting for that. Generally, when you have a cloud, you have an SLA that is, hey, it's going to be uptime, it's going to be 99%, blah, blah, blah, for this period. With GB200, it's 99% for 64 GPUs, not 72. And then it's like 95% for 72. Now, it differs across every cloud. Every cloud is a different SLA. But they've adjusted for this because they're like, look, this hardware is just finicky. Do you still want it? We will credit you in that 64 of them will always work, not 72. There's this whole finicky nature, and the end customer has to be capable of dealing with the unreliability. And the end customer can just continue to use V200, performance gains not as much. The whole reason you want this 72 domain is so you can have some of these gains, but you have to be smart enough to be able to do it. And that's challenging for small companies.

Guido Appenzeller: Totally. And we just announced the Rubin pre-fill cards, like CTX, CRI?

Host: CPX, there we go.

Guido Appenzeller: What's your take on that? Does it cannibalize?

Host: Dude, by the way, I don't know if this is like brain rot or like, I don't know, but I can't remember what I had for lunch yesterday, but I know the model number of every fucking chip.

Guido Appenzeller: Haunts you in your dreams.

Host: We're broken. We're broken. Living the dream.

Guido Appenzeller: No, no, no. Why do you pre-announce a product that's 5x faster for certain use cases? Is it that much?

Host: Historically, AI chips were AI chips, right? And then we started getting a lot of people saying, this is a training chip, this is an inference chip. Actually, training and inference are switching so fast into what they require that now it's still one chip. Actually, there are still workload level dynamics that differ, but the main workload is inference even in training, because of RL. Most of that is generating stuff in an environment and trying to achieve a reward. So it's inference still. Training is now becoming mostly dominated by inference as well.

But inference has two main operations. There is calculating the KV cache for pre-fill. Here are all these documents, do the attention between all of them, between all the tokens, whatever type of attention you use. And then there's decode, which is auto-regressively generating each token. These are very, very different workloads. And so initially, the ideas or infrastructure techniques, the ML systems techniques were, oh, okay, I will just make the batch size every single forward pass this big. And if I make it, let's call it, I'll make it 1,000 big. And maybe I'll run 32 users concurrently. That way, now I still have 900 something left, 960 left. That 960 is actually doing the prefill for if a request comes in, it chunks it. It's called chunk prefill. You prefill chunks of it now, you get really good utilization on GPUs. But then that ends up impacting the decode workers, the people who are auto-regressively generating each token, having slower TPS. And tokens per second is really important for user experience.

So then the idea is like, okay, these two workloads are so different and they are literally different. You pre-fill and then you decode. It's not like you're interleaving them. So why don't we split them entirely? And this is done on the same type of chip. OpenAI, Anthropic, Google. Pretty much everybody does that. Everyone good. Together, Fireworks. All these guys do disaggregated pre-fill decode. So they run pre-fill on a set of GPUs, decode on a set of GPUs.

Why is this beneficial? Because you can auto-scale them. You can, hey, all of a sudden, I have a lot more long context workers. I allocate more resources to pre-fill. Oh, all of a sudden, over time, my traffic mix is not long input, short output. It's short input, long output. I have more decode workers. This way I can guarantee, and so now I can auto-scale the resources differently, and I can also guarantee that my pre-fill time is... What's really important in search is how fast you get the page to start loading, not when does the resource happen. What do people do in games? The loading screen often has some sort of interactive environment or it blends in over time, it has tips and tricks, ways to distract you.

The same thing is, there are studies and papers out there that users prefer a faster time to first token. First token gets streamed to me sooner, even if the total time to get all my tokens is a little bit longer. I can't read that fast anyway. I mean, most models return about speed reading speed. But you need that. The idea is that you want to guarantee time to first token is a certain level for user experience reasons. Otherwise, people are like, screw this, not using AI. The decode speed matters a lot too, but not as much as time to first token. And so by having separate pre-fill and decode, you do this.

But now you've already done this all in the same infrastructure. So now it's like, what's the next logical step? These workloads are so different. Decode, you have to load all the parameters in and the KV caches to generate a single token. You batch a couple users together, but very quickly you run out of memory capacity or memory bandwidth because everyone's KV cache is different. The attention of all the tokens. Whereas on Prefill, I could even just serve one or two users at a time because if they send me a 64,000 context request, that is a lot of flops.

Host: ...64,000 context requests. I'll use Llama 70B because it's simple to do math on. 70 billion parameters, that's 140 gigaflops per token times 64,000. That's many, many teraflops. You can use the entire GPU for like a second, right? Potentially, depending on the GPU, just to do the prefill. And that's just one forward pass. So I don't necessarily care about loading all the tokens or all the parameters in KVCache fast. All I care about is all the flops.

Dylan Patel: That leads us to sort of... I think it was a long-winded explanation because it's hard for people to understand what CPX is. I've had a lot of my own clients, we set multiple notes explaining, and they're like, "I still don't understand." I'm like, "Shit, okay. Send them to the 'Attention is All You Need' paper." You can't expect... I mean, think about a networking person. I don't need to know about this. Or think about an investor, a data center operator. They're like, "Oh, there's two chips. Why should I build my data center differently?" I have to explain everything. Or just like, "No, you don't have to build differently." But anyways, you get to know.

In Stanford, there's 25% of all students—not CS students, *all* students—who read that paper. Read what paper? "Attention is All You Need." That's low. They do gym majors and philosophy. I'm like, "This is amazing." Anyway, sorry. The Middle East—I can't remember what country it is—has AI education starting at like age 8. And in high school, they have to read "Attention is All You Need." Wow. Someone told me that their kid had to read it. Top-down mandates for education, you know. Maybe they work, maybe they don't. Like, maybe people are homeschooling their kids. I don't know, I went to public school.

Guido Appenzeller: But back to your readers. Just on the topic of hardware cycles, I wanted to maybe... I didn't actually explain what CPX is. So CPX is a very compute-optimized chip for prefill. And then decode is the normal chips with HBM. HBM is more than half the cost of the GPU. If you strip that out, you end up having a much cheaper chip passed on to the customer. If NVIDIA takes the same margin, then the cost of this prefill chip is much, much lower. And now the whole process is way cheaper and more efficient. Now long context can be adopted.

Host: I love that we're actually going into all this detail because I had a more 10,000-foot view question for you. I haven't been following the semi-market as closely as you have. I probably started with the A100. I remember helping someone at Character chase down GPUs. This was summer of June 2023. The only thing that mattered at that time was delivery date because there was a huge capacity crunch. Then to see that over the last two years evolve where, let's say six to twelve months ago, people were doing these RFPs to 20 neoclouds. And the only thing that mattered to some degree was price.

Dylan Patel: People actually do RFPs for GPUs? Yes. So just to be clear, my opinion on how you buy GPUs is that it's like buying cocaine or any other drug. This was described to me—not me, I don't buy cocaine. Someone tells me this. I'm like, "Holy shit, it's right." You call up a couple people. You text a couple people. You ask, "Yo, how much you got? What's the price?" It's exactly the same way. We have Slack connects with like 30 neoclouds. We just send them a message like, "Hey, customer wants this much. This is what they're looking for." And then they send quotes.

Host: I know this guy. I know a guy. Well, I think that's actually a very accurate description. I've sent countless port codes, your ClusterMax original posts, because I thought it did a really good job breaking them down. But maybe one question to end on for me is just, what era are we in now with Blackwells coming online? Are we sort of back to the summer 2023 era, and that's kind of the cycle that we've just entered? Or what's your view on where we are?

Dylan Patel: That's a very good question. For one of your portfolio companies, we were like, "You know, after their difficulties with Amazon, let's actually get you to fuse. The original deals we got you were gone, but here's some other deals." It turned out that multiple major neoclouds had sold out of Hopper capacity, and their Blackwell capacity comes online in a few months. So it's a bit of a challenge. Due to inference? Inference demand has been skyrocketing this year. Reasoning models, the revenue has been skyrocketing this year. And then also, Blackwell comes online but it's hard to deploy. There's a learning curve. Whereas with Hopper, you buy it, install it in the data center, it's running within a month or two. For Blackwell, it's a longer time frame because of reliability challenges. It's a new GPU. Just learning pains, growing pains.

So there was this gap of how many GPUs are coming onto the market right as revenue is starting to inflect. A lot of capacity got sucked up. Actually, prices for Hopper bottomed like three or four months ago, or five or six months ago. And actually, they've crept up a little bit now. They're still... I don't think we're quite in the 2023 or 2024 era of GPUs being tight, but certainly if you want just a few GPUs, it's easy. But if you want a lot, it's hard. You can't get capacity instantly.

Host: Wow, what a time. Shall we wrap on that? Dylan, this was another instant classic. Thank you so much for coming to the podcast.

Dylan Patel: It was like two hours, bro. We couldn't stop. Thanks so much. It was great.

Host: Thank you so much for having me.

[Podcast outro begins]

Host: Thanks for listening to the A16Z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com slash A16Z. We've got more great conversations coming your way. See you next time.

[Disclaimer]

Host: As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.