--:-- Zoran: Welcome to the A16Z Podcast. I'm Zoran. Today we're talking about trends in data centers, focusing on hyperscalers. These are some of the world's most well-known companies running massive data centers with hundreds of thousands of servers. We're talking about companies like Amazon, Facebook, Microsoft, Google, Apple, Alibaba, and Tencent.
--:-- Zoran: Innovation in the data center traditionally has been constrained by the model in which suppliers provided fixed-function networking chips that limit how much the biggest data center operators can differentiate. In recent years, though, programmable chips have allowed these companies to customize and increase performance, latency, and reliability for the incredible amounts of data center traffic they manage.
--:-- Zoran: This trend also allows the hyperscalers to innovate throughout the pipeline, from operating system to networking interface to user application. To talk about the trends in data centers, we talked this summer to two experts. Martin Casado is an A16Z general partner focused on enterprise investing. Before that, he was a pioneer in the software-defined networking movement and the co-founder of Nicira, which was acquired by VMware.
--:-- Zoran: He's joined by Nick McKeown, a Stanford professor of computer science who has founded multiple companies and was Martin's co-founder at Nicira and has worked with hyperscalers to innovate within their data centers. After this podcast was recorded, Nick was appointed Senior Vice President and General Manager of a new Intel organization, the Network and Edge Group.
--:-- Zoran: We start with Nick explaining the sheer scale of traffic inside data centers and how that is changing the industry.
--:-- Nick McKeown: If you were to cut a line vertically through the United States and then look at all of the public internet traffic that was passing from left to right and right to left, we call this the bisection bandwidth of the internet. It's essentially how much capacity the internet has crossing the United States. That is less than the amount of traffic going between a couple of hundred servers inside one data center. It's just been inverted.
--:-- Nick McKeown: The whole model of how communication takes place has really become dominated by the insides of data centers, of all the communication that's taking place for compute and nowadays machine learning, training, and inference that's taking place. It's totally dominating how we communicate.
--:-- Nick McKeown: So it means that the industry itself has focused on how to serve that market. And so the majority of the silicon that is being produced, whether it's CPUs, whether it's storage, whether it's for networking, for the network interfaces and the switches, it's mostly targeted on that data center business just because of the sheer scale of it.
--:-- Zoran: And so we're going to get into how data centers got here and how they're evolving. But first, to situate ourselves in what's happening right now, who are the companies that are running these massive data centers, these hyperscalers as they're called? And what's the current model? Like who supplies them and who builds the data centers?
--:-- Nick McKeown: Hyperscalers are those companies that we know very well, like Google and Amazon and Facebook and Microsoft, the companies that are building warehouse-scale computing. They've been doing this for 10 or 15 years now, and they have big warehouses full of hundreds of thousands of servers that are interconnected by rich networks with huge amounts of storage.
--:-- Nick McKeown: And this is where they serve up things like Google search and the Facebook apps that we all use and nowadays office products like Microsoft Office. So there's a rich network of suppliers, technology suppliers who serve them. And nowadays, actually, for most of those suppliers, these big hyperscalers represent a very big part of the market share.
--:-- Nick McKeown: The hyperscalers are well-known for building their own data centers, everything from famously Google's own servers to the way they do cooling to the way they do real estate to the entire software stack. And the reason is because they're actually pushing these technologies at scales and deficiencies that they just weren't designed for.
--:-- Nick McKeown: And that's required a pretty large redesign. And so this is in the context of organizations that are incredibly sophisticated, incredibly savvy, and have really remade every other part of the data center down to like, you know, the ducting and the electricity.
--:-- Zoran: Okay, so let's look at how we got here. How have data centers changed over the years?
--:-- Nick McKeown: The early data centers, some of their first attempts, they were made out of off-the-shelf computers that they would just order from companies like Dell and Compaq was probably around in those days. And then they would arrange them into racks of servers, perhaps 30 or 40 in a rack.
--:-- Nick McKeown: And then they would hang as much disk storage as they could off it, put as much memory into them as they could. And then they would string them all together with networking equipment. They would just buy from regular equipment vendors. And, you know, there was an ecosystem of vendors like Cisco and HP and folks like this who were selling the networking equipment for them in the early days.
--:-- Nick McKeown: So that's what the early data centers looked like. And as they grew, there was fear in their eyes because they realized that they were building systems at such phenomenal scale that no one had ever built before, that these systems just weren't going to be able to scale up to the size that they needed.
--:-- Nick McKeown: Now in the data center itself, hundreds of thousands of servers are connected together by thousands of switches. And each switch decides where it's going to send these packets of information next so that they eventually reach the correct server.
--:-- Nick McKeown: And today these switches are made from a single silicon chip called an ASIC. And in a single switch, an ASIC can process over 10 billion packets per second today. Data rates, well over 10 terabits per second, all within a single piece of silicon.
--:-- Nick McKeown: So these are immensely powerful devices. To put this into perspective, a single switch chip processes a new packet in the time it takes a photon to travel just one inch. So that's the time that it takes for it to process one of these.
--:-- Nick McKeown: And a 10 terabits per second switch can stream the entire Netflix catalog at high definition in less than a minute.
--:-- Zoran: All right, so we have these incredibly powerful data centers, hundreds of thousands of servers, incredibly powerful single switch ASICs that have emerged. Martin, how have hyperscalers reacted to these trends? Like what does this processing power that Nick just outlined for these chips mean for how hyperscalers think about their business and their strategies?
--:-- Martin Casado: These are very very specialized chips that have very demanding requirements but they're very integral to a core function of the data center, which is communication. I mean, all the data center workloads are distributed. In order for the workloads to work, or most of them are distributed, in order for them to work, they have got to communicate a lot, which means like latency is very important.
--:-- Martin Casado: Prioritization of traffic is very important. And that's what these switches do. And so the question has been, is like, you know, if you're, you know, a Google or an Amazon or a Facebook, part of your differentiation is how fast your data center is, right?
--:-- Martin Casado: And you've changed so much more of your stack. And the question is, is what can you do in the network in order to differentiate at that level? And that is kind of the basis for this discussion has, and it's been around for 15 years, right? It continues to evolve.
--:-- Martin Casado: And as, you know, and as these data center companies get bigger and bigger, they're naturally going to consider whether the silicon that they can buy will best support their customers' workloads, or if they would do better to modify and design them for themselves.
--:-- Martin Casado: It's a natural question as they get to a scale where they're operating at enormous economies of scale, but dealing with very ferocious competitors, the other hyperscalers. And in order to be able to compete with each other, they've got to be looking for not only cost reduction but also ways to differentiate in terms of the performance they get.
--:-- Zoran: Martin, give us the sense of the range of options that hyperscalers have before them as they consider how to differentiate up to and including building their own chip.
--:-- Martin Casado: Like one of them is, okay, so you can buy a CPU from somebody else, an ASIC from somebody else, like whether it's a standard CPU or a NIC, and then you could write your own software for it, even really low-level software, right? I'm not talking about application code.
--:-- Martin Casado: You're writing low-level code, right? And so pretty much all the hyperscalers will do this. This is so they can the most efficiently drive it and kind of use the best majority.
--:-- Martin Casado: Another thing that you can do is you can include an FPGA, a field programmable gate array. So this is a way in which a hyperscaler can build their own functionality in hardware. Normally, this is ancillary functionality.
--:-- Martin Casado: Like it won't do full switching. It won't be a full NIC, but it'll add some ancillary, but very, very high-performance functionality. And it'll do this in a programmable way. Like let's say on the NIC or on the mother of the border of the switch.
--:-- Martin Casado: And then the most aggressive by far is you build your own ASIC. I mean, this is a phenomenal undertaking. My opinion is in order for a hyperscaler to build an ASIC, I think it's actually crazy. It's just such a phenomenal effort, whatever.
--:-- Martin Casado: And so what we do need, particularly in the networking, is we need networking chips that can be programmed so that you have tons of flexibility, but you don't lose the cost-performance initiatives.
--:-- Martin Casado: And so, you know, that is kind of tradeoff between, you know, do you build your own ASIC versus, you know, do you get the same type of functionality through programming?
--:-- Zoran: And Nick, if you don't mind, I think it'd be interesting to just talk through, like, what does it mean to build an ASIC just from like a resource and time perspective?
--:-- Nick McKeown: Yeah, absolutely. So today, the general rule of thumb is that it's $150 to $200 million to contemplate building an ASIC, and you need a team typically of a couple of hundred people.
--:-- Nick McKeown: And this is not something that you want to do lightly unless you're doing something in extremely high volume, or you're going to derive immense value from the exercise.
--:-- Nick McKeown: So in most cases, it sort of benefits the industry to have technology vendors who are building these chips, taking on the great investment and the great cost to do this, and selling to multiple customers.
--:-- Nick McKeown: And herein lies the problem, right? If the costs favor having a technology vendor who is selling to multiple customers, they have to figure out what is the product they're going to build.
--:-- Nick McKeown: And so if that product has a fixed functionality for all of those customers, it leaves the customers with no opportunity to differentiate with each other.
--:-- Nick McKeown: And so where we've seen this change first is in some of the accelerators that hyperscalers have developed for themselves, or in the customer-specific network interfaces, network interface chips, and switches that have been developed by some.
--:-- Nick McKeown: However, there is an alternative way of doing this that my team was just alluding to, and that is instead of making it fixed function, make all of those devices programmable as well.
--:-- Nick McKeown: And so the hyperscaler can determine how they're going to introduce new ideas to differentiate from their competitors by programming the device themselves.
--:-- Nick McKeown: Lift these functions up and out of the hardware, just make them programs. And this has been the trend that we've seen over the last few years.
--:-- Nick McKeown: It's essentially software eating the world of the data center as a whole for the infrastructure as a whole. And this at this point seems inevitable.
--:-- Nick McKeown: You know, it's the natural next step beyond disaggregation of the equipment by breaking it down into chips that are programmed rather than buying sort of vertically integrated complete solutions.
--:-- Nick McKeown: And so the natural next choice is take the chips that were themselves fixed, make them programmable and allow for that differentiation.
--:-- Nick McKeown: I've been involved in developing some networking ASICs exactly for this purpose. And I've observed that over the last decade, that as the hyperscalers want to have more control over how individual packets are processed, then they will do interesting, crazy, sometimes wild things that I would have never have thought of, that their competitors don't think of.
--:-- Nick McKeown: And so they're actually diverging and going off in slightly different directions. That's the nature of competition.
--:-- Nick McKeown: And it's the first time you're seeing that real competition taking place in the infrastructure of these hyperscalers.
--:-- Nick McKeown: And so the job of the technology providers, whether that's companies like Intel, Brocom, NVIDIA, the task is kind of simple, design and build the best programmable devices you can and execute like hell.
--:-- Zoran: Yeah, I still think that we should revisit the question, why would Amazon, or at least rumors of Amazon building their own ASIC, right?
--:-- Zoran: I mean you could believe it because the technology has ever existed before or you could believe there actually something endemic to the industry.
--:-- Zoran: And I actually think it a bit of a spicy question but it very interesting which is you know the CPU grew up in an era where it was really about programmability and the networking industry did not.
--:-- Zoran: And they been exceptionally resistant to it. And now we're actually seeing the dam break.
--:-- Zoran: And I do think there's two things, which is we actually now have the right abstraction. So people are programmed. So there has been a technical breakthrough.
--:-- Zoran: I also do think that there's now finally an awareness in the industry that this is what the customers want, and there's enough competition to get it.
--:-- Zoran: And we're actually seeing a restructuring of the industry allow for this. It's actually leading to something which I think is quite transformational.
--:-- Zoran: So we're at the point now where every piece of the pipeline that goes from a user application, like a Facebook application in a server, all the way through the Linux operating system, out through the network interface, all the way through the switches, all the way into the application at the other end, that entire pipeline can now be programmed.
--:-- Zoran: And that's something that's really just happened in the last few years. And what we have developed as an industry are extremely efficient programmable pipelines, which you can program in a high-level language.
--:-- Zoran: It's called P4. It doesn't particularly matter the specifics of this language, but it lends itself to writing programs that run very, very fast.
--:-- Zoran: And so you can write programs where previously you required fixed function. And so it means that the network becomes programmable from end to end.
--:-- Zoran: You can change its behavior to be whatever you want.
--:-- Zoran: Now, in a generation before that, there was another transformation that took place that came out of Martin's PhD work.
--:-- Zoran: And this was to change and transform how networks are controlled.
--:-- Zoran: And so this revolution took place in which the software that is used to define the behavior of how networks are controlled, not how individual packets are processed, but the sort of overall management and control of the network, that was opened up and taken essentially away from the equipment vendors and handed to those who own and operate the biggest networks in the world, the internet service providers, the hyperscalers.
--:-- Zoran: And so today, every hyperscaler in the world builds its own networking equipment, writes the software to control it, and they take it for granted that they can download it, they can write it, they can commission someone else to write it, they can modify somebody else's, but this is software that is under their control.
--:-- Zoran: And that was the big deal.
--:-- Zoran: So not only are the networks programmable sort of from end to end, they're programmable from the top down in any way that we would manage and control them.
--:-- Zoran: And so if you step back a little bit, the entire system, the computers, the storage, the network is becoming a big programmable platform as one.
--:-- Zoran: It's a big distributed system that you can think of programming to do exactly what you want.
--:-- Zoran: And this is going to change the way that the hyperscalers, anybody that builds a network, internet service providers, mobile operators, they're all going to start thinking differently about how they build not only networks but the entire data center that they use to run the software for their own infrastructure and for their customers.
--:-- Zoran: I mean, one way to view this is the following, which is one of the reasons perhaps that networking has never been programmable is because nobody's wanted to program them.
--:-- Zoran: You know, it's like, you know, computers were programmable because everybody had a CPU and we were writing applications, but there wasn't any gain for programming and networking.
--:-- Zoran: But the hyperscalers, they get a very specific benefit for programming the network. Like Nick said it perfectly, which is basically all of the world's internet traffic, a large percentage is actually happening within the data center.
--:-- Zoran: That is a competitive, like latency is a competitive advantage. I'm sure Google has, you know, some algorithm that shows how much money they waste for every millisecond of latency, right?
--:-- Zoran: And a lot of that latency is going to be network-related. And if that is the case, then, you know, like this is certainly for their internal operations, very important.
--:-- Zoran: And so now you have, you know, a large set of companies that have a specific interest in programmability, then they're putting the pressure on the industry at large for programmability.
--:-- Zoran: And that's happening now.
--:-- Zoran: And I think that there's two potential net impacts. One of them is I do think that this allows for disruption in the industry in the sense of like new challengers, maybe a shifting of position of incumbents, empowering different parts of the stack, right?
--:-- Zoran: And then the second thing is like, yeah, like the actual technology innovation that's going to come out of this is also enormous because now we've just unlocked another key piece of infrastructure that can be programmable and can add differentiated value.
--:-- Zoran: So what are some of the ways hyperscalers are using programmability?
--:-- Nick McKeown: What I've seen people do, once you give them a programmable device that allows them to decide how packets are processed within the network, one of the first things they do is they simplify.
--:-- Nick McKeown: They take many of the things that were embedded in the fixed-function devices that they never used, that were sitting there as a vulnerability for them, because they might actually trigger a problem that they don't really understand because they're not actually using that part of the fixed-function device.
--:-- Nick McKeown: They throw them all out. So they write programs that just boil it down to the essence.
--:-- Nick McKeown: The networking people talk in terms of protocols. These are the sort of the ways in which different devices communicate within a network.
--:-- Nick McKeown: And a typical fixed-function device might have to support 45 or 50 of these different types of protocols. Yet most hyperscalers are using three or four internally within their data centers.
--:-- Nick McKeown: So they want to get the other ones out because they're consuming power, there are security risks, there are vulnerabilities.
--:-- Nick McKeown: So this is one of the first things they do.
--:-- Nick McKeown: The second thing that they do is they want to get greater visibility into what their system is doing, into where packets are flowing, where storage blocks are.
--:-- Nick McKeown: And computers have become very good at logging what they're doing. If you want to debug what a computer is doing, you can use software, you can use hardware, you can go and interrogate what it was doing, and then even reproduce the problems on the fly.
--:-- Nick McKeown: In networking and storage, this is extremely difficult and it's very rudimentary today, the sorts of tools that you have available to you.
--:-- Nick McKeown: No one has ever operated at sufficient scale before to really figure out what they needed. The hyperscalers did though.
--:-- Nick McKeown: They knew exactly what they wanted. And as soon as you put into their hands the ability to write programs to tell them so that they can look at and interrogate what their infrastructure is doing, it one of the first things that they do.
--:-- Nick McKeown: They want to know, where did packets go? How long did a storage block actually take to reach its destination? If it failed to get written into the correct location, why was that?
--:-- Nick McKeown: If it actually met some congestion along the way, whose fault was it? So they can go back and rectify it.
--:-- Nick McKeown: So as soon as you put this capability into their hands, they all go off and measure the things that they've been wanting to measure for a long time to give them greater visibility into what the system is doing.
--:-- Nick McKeown: And then the next thing they do is they just want to smooth and control everything in a way that everything is just better behaved and easier to keep going at that large scale.
--:-- Nick McKeown: They can sort of tame the traffic and route it in the directions that they need to give them the reliability that they need in their system and the redundancy in case a link fails or a port part of the system fails.
--:-- Nick McKeown: And so these are the things that they do. They're in a way not that surprising, but they only know how to do this because they're operating at such huge scale.
--:-- Nick McKeown: No one that ever designed a chip actually was running a big hyperscaler system. They didn't know what to put in there.
--:-- Nick McKeown: So if you make it programmable, you can leave the customer to actually figure out what they need and write the software they need to do it.
--:-- Zoran: What's the broader story here, Martin? What are some of the other implications of this for the industry?
--:-- Martin Casado: I mean, I think the undertone in all of this, which just may not be super apparent, which is how dramatic the industry had to shift in order for this to happen.
--:-- Martin Casado: So when Nick and I started in the mid-2000s, the networking industry looked like the mainframe industry. Like you would buy a box from a network vendor and everything in it was written by the vendor.
--:-- Martin Casado: Every application, the operating systems, all of the ASICs, I mean, they wrapped the sheet metal around it, you know, the whole thing.
--:-- Martin Casado: And rather than having a programming model, they would train people to operate it. So there wasn't even a programming model. Again, this is 15 years ago. It's not that long.
--:-- Martin Casado: And what has happened since then is not only can you control these things programmatically rather than require human beings, this is the control thing that Nick mentioned, but also the actual per packet operations you can program, which is actually a very, very tremendous technical challenge.
--:-- Martin Casado: And as a result, I actually think that there's a potential for reshaping the entire industry.
--:-- Martin Casado: I don't think it's just about like hyperscalers differentiating. It's like, I mean, do you have a shift in who provides the silicon, right?
--:-- Martin Casado: I mean, traditionally you've had companies that just focus just on fixed. You could have a new entrant. The big networking vendors, they're going to have to kind of rethink what it means to them.
--:-- Martin Casado: I mean, do they kind of evolve more of an ecosystem? Do new players arise that are kind of better at this horizontalization?
--:-- Martin Casado: What does it mean to the OEMs? Maybe the OEMs get empowered like, you know, the Dells of the world or the Taiwanese ODMs.
--:-- Martin Casado: And so I think that the broader story here is one of horizontalization of a previously vertical market.
--:-- Martin Casado: And a large market, we're not talking about some niche thing. This is like $40 to $100 billion market.
--:-- Martin Casado: And this just highlights how important network programmability is. And the fact that it needs to be programmable and now it's becoming programmable.
--:-- Martin Casado: Another big change that is taking place is as these systems turn into these kind of programmable platforms, big programmable systems where the infrastructure itself is entirely programmable, the software that they're using is increasingly made up of open source.
--:-- Martin Casado: And open source, which frankly, in the infrastructure industry, apart from Linux, was considered a bit of a joke 15 years ago.
--:-- Martin Casado: It is now taken as a given that the majority of the infrastructure will be built by open source, open source that people pull together and assemble, and then hopefully in many cases will push back out again to get that benefit of all of the sort of the quality assurance that they get from multiple people working on it.
--:-- Martin Casado: And this is because a lot of the pieces of the infrastructure are necessary but non-differentiating.
--:-- Martin Casado: And so it benefits them to get that sharing of many, many people from outside their own company working on this together.
--:-- Martin Casado: And so as the infrastructure becomes more and more created from open source, it itself becomes more open to scrutiny.
--:-- Martin Casado: It's more transparent in the sense of it becomes easier to find bad stuff that is being placed into that open source or bugs, etc.
--:-- Martin Casado: And so over time, it should actually become far more resilient, far more secure than the infrastructure of the past.
--:-- Zoran: So Nick, if we take these ideas of programmability and open source, like what other areas will this touch? How will consumers and enterprises see the broader effects of these trends?
--:-- Nick McKeown: I think that all of this, these same dynamics are now happening to the rollout of 5G and the investment in 5G.
--:-- Nick McKeown: Cellular networks in the past were like these walled gardens. The industry was very protective of the way things were done.
--:-- Nick McKeown: But 5G was deliberately designed and architected to feel very much like the rest of the internet.
--:-- Nick McKeown: And the reason that people get very excited about 5G is that it's essentially going to replace a lot of the wireless communications that we use today.
--:-- Nick McKeown: Wi-Fi, ways of communicating within factories, connecting all of those new IoT devices like, you know, our dishwasher that now communicates online and the cameras and the things that we have in our homes, the robots inside factories.
--:-- Nick McKeown: All of that is being built out of the same ideas, programmable, disaggregated, low-cost, open-source software that is written and owned by the mobile operators.
--:-- Nick McKeown: So 5G is becoming software-defined as well.
--:-- Nick McKeown: So you're going to get this entire infrastructure inside the hyperscalers, all the way out to edge computing, all the way out to 5G, all of this infrastructure is going to be defined by software running on programmable hardware.
--:-- Nick McKeown: That is going to change everything because it's going to open the floodgates for a massive amounts of innovation, which is exactly what we need.
--:-- Zoran: Martin, Nick, thank you so much for being with us today.
--:-- Martin Casado: Thank you.
--:-- Nick McKeown: Thank you. That was great.
--:-- Zoran: Great chatting.
--:-- Zoran: Bye-bye now.
