Podcast Episode Summary
The podcast episode discusses the intersection of artificial intelligence (AI) and science, particularly in the field of biology. The main topic is the impact of AI on scientific discovery and the potential for AI to help uncover fundamentally new science.
Key Points:
Participants:
Takeaways:
In recent years, the AI landscape has seen huge advancements, from the release of Dall-E 2 in April 2022 to the emergence of AI music and video models in early 2024.
While creative tools often steal the spotlight, AlphaFold 2 marked a groundbreaking AI breakthrough in biology in 2021. Since its release, this pioneering tool for predicting protein structures has been utilized by over 1.7 million scientists worldwide, influencing fields ranging from genomics to computational chemistry.
In this episode, DeepMind's VP of Research for Science, Pushmeet Kohli, and a16z General Partner Vijay Pande discuss the transformative potential of AI in scientific exploration. Can AI lead to fundamentally new discoveries in science? Let's find out.
Resources:
Find Pushmeet on Twitter: https://twitter.com/pushmeet
Find Vijay on Twitter: https://twitter.com/vijaypande
Learn more about Google DeepMind: https://deepmind.google
Read DeepMind’s AlphaFold whitepaper: https://deepmind.google/discover/blog/a-glimpse-of-the-next-generation-of-alphafold
Read DeepMind’s AlphaGeometry: https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry
Read DeepMind’s research on new materials: https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/
Read DeepMind’s paper on FunSearch, focused on new discoveries in mathematics: https://deepmind.google/discover/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models
Stay Updated:
Find a16z on Twitter: https://twitter.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Subscribe on your favorite podcast app: https://a16z.simplecast.com/
Follow our host: https://twitter.com/stephsmithio
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Stay Updated:
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Podcast on Spotify
Listen to the a16z Podcast on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Pushmeet Kohli
AI is not sort of nice to have. It's basically almost a necessity for us to make sense and reason about any problem that we are now looking at.
Vijay Pande
I think there's going to be this really fun cultural shift where 10 years ago, people would say, "Oh, it's ridiculous a computer could try to do these things." I think 10 years from now, people will be like, "Oh, it's ridiculous to have a human being do that. You can't load all these numbers in your head."
Pushmeet Kohli
Essentially, what we have entered is an age where a single human mind cannot comprehend the data that we are gathering about the universe. One of these structures may have taken the length of a PhD to solve a single structure, and now we're talking about true scale. There have been 1.6 or 1.7 million users of the AlphaFold database. If that is not a positive statement about the planet, then I don't know what it is. There are 1.7 million people interested in protein structure prediction. I'm really happy about that.
Music fades in and out
Host
The last few years have been peppered with AI announcements. Let's recap a few. April 2022, DALL-E 2 is released. Midjourney and Stable Diffusion follow that summer. Then in November, ChatGPT arrives. Then 2023 features the release of Claude, Llama, and Mistral 7B, just to name a few models. And we're only a quarter or so into 2024, and we're already seeing the expansion into AI music and video models faster than almost anyone could have imagined.
And while much of the attention circles around creative tools, there was an AI unlock in biology that caught much attention in 2021. That was AlphaFold 2. A breakthrough in prediction around the 3D models of protein structures was released and open sourced by the DeepMind team in July of that year. Since then, over 1.7 million scientists across 190 countries have been leveraging the tool.
In the meantime, the DeepMind team has been hard at work, seeing how else machine learning can expand the frontier of science across many areas of biology, from structural biology to genomics, to protein design, to cell genomics, to quantum chemistry, to meteorology, to fusion, to pure mathematics, to computer science. They've released papers like the high accuracy weather model GraphCast in November, AlphaGeometry in January, which approached the level of a Human Olympiad Gold Medalist, and other papers across materials, mathematical functions, and more, including continuing to push forward AlphaFold.
And today, we have the pleasure of hearing directly from DeepMind's VP of Research focused on science, Pushmeet Kohli. Pushmeet sits down with myself and a16z general partner Vijay Pande, who has long been part of this intersection himself, as a longtime professor at Stanford, spanning several departments from computer science to structural biology to biophysics, and was also the founder of the Folding@Home Project, released in the year 2000.
Together, we reflect on the journey to AlphaFold. But more importantly, where are we in the trajectory of AI meaningfully impacting the way we perform and unlock new science? From new lab economics to clinical trials to drug discovery and more. So the question becomes, can artificial intelligence help us uncover fundamentally new science? And has it already done that? Let's find out.
As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.
So AI has been the talk of the town. A lot of people are familiar with the consumer LLMs, think ChatGPT, maybe Midjourney, but AI has been around for quite some time and it's also impacting the scientific sphere, which I think is so exciting. And I think both of you do too. So Pushmeet, maybe we could just start there and talk a little bit about your background, how you got into this intersection of science and AI. And also you work for DeepMind, which I feel like is one of the flagship AI companies. Why have you chosen to focus more there than perhaps some of the others?
Pushmeet Kohli
Yeah, so I took a very roundabout journey into what I do today at DeepMind. I'm a computer scientist by background and was hired at Microsoft Research and worked there for a decade, mostly working on applied mathematics, solving difficult math problems, and most of them were encountered in machine learning. So I started with computer vision, computer graphics, information retrieval. After having gone through many of these applications, I was very excited about deep learning when it finally emerged. I really thought that this was a game changer in terms of how machine learning is going to impact applications.
Demis Hassabis, who is the CEO and founder of DeepMind, at that time DeepMind was a young startup. He reached out and said, "We know you from some acquaintances. Why don't you join us?" And I said, "No, you guys are working on games at that time. I went to products and applications." And he said, "The whole games thing is just phase one. The idea is to eventually impact science and impact applications, which are the biggest challenges in the world." The level of conviction with which he made his case, I was convinced this guy gets it. So I moved to DeepMind in 2017.
I told him, "If you're very serious about real world applications, we need to make sure that machine learning systems are reliable." So when I joined DeepMind, I founded the reliability and safety team at DeepMind. Around a year into it, Demis asked me once, "You're really interested in multidisciplinary research, where you want to apply machine learning in impactful problems. I think the most impactful area that you could work on is science." That was a complete left field suggestion. My last class was in school. So I was quite skeptical, to be honest. I told him, "You've got the wrong guy. I have no background in biology or physics or chemistry." But he said, "The way you are approaching these things, it's good. Let's give it a try and see where it goes."
So we started the science program with six or seven people working on two projects. Now it's almost a 140 person team and we have 10 different initiatives spanning many areas of biology, from structural biology to genomics, to protein design, to cell genomics, to quantum chemistry, to meteorology, to fusion, to pure mathematics, to computer science. It's a long journey, but it started with an accident.
Host
Yeah, and also a very scientific iterative approach. I love that. Vijay, before we jump into more of those projects that Pushmeet alluded to, I'd love to hear your background and how you got into this intersection of science and AI, because you also have quite the storied history there.
Vijay Pande
Sure. So from 1999 to 2015, I was a professor at Stanford. My home department was chemistry, but I also had appointments in computer science, structural biology, and was also chair of biophysics. At that intersection, it was clear that machine learning was a very exciting tool to use. What was really happening early with genomics in the 90s, and then just plowed all the way through, was the rise of data in biology and biology becoming very quantitative. Once it starts becoming quantitative, machine learning is very natural.
As Pushmeet talked about, where a lot of us got particularly excited was maybe 2013, 2014, 2015 as deep learning was emerging. Machine learning before deep learning required human beings to come up with their features. It was like a little tool. With deep learning, it could be something that replaces more and more of the human part of the thinking. A lot of the interesting results are emergent after that. Those emergent properties got very exciting.
It was clear at the time that we need a lot of compute. So early on in 2000, I founded the Folding@Home Distributed Computing Project. We were some of the first to program GPUs. All of that comes together: the data, the compute, and finally, the algorithms. Once those three pieces were together, many of us could see that this was taking off and it was time to dive in.
Host
Absolutely. I think that brings us to this question of the why now. So you already addressed it, but Vijay, what gets you so excited about this intersection? We're recording this in 2024. AI has really been around since maybe the 50s. Is it just that we have the right amount of compute? Is it that we have these unlocks when it comes to the modeling? Give us a picture of what gets you so excited about what's to come before we dive into some of the specific examples.
Vijay Pande
If you step back, what we're really seeing in biology is this industrial revolution. If you look at a biology lab, maybe even today to some extent versus 10 years ago versus 50 years ago, there would be benches and people in white coats and pipetting. Maybe the boxes on the benches are a little different, but it's very similar. It's very bespoke and artisanal. What is shifting is that it's becoming industrialized. We're seeing the rise of robotics. With that industrialization comes this immense amount of data. AI needs data and data needs AI. So as biology gets all that data, we can lean into this.
What's most intriguing is that life sciences and healthcare largely have not been permeated by technology, not by IT to a great deal. Healthcare and life sciences collectively are becoming 25% of US GDP. Trillions and trillions of dollars go through this, and very little of it is being revolutionized by tech. This revolution is happening because of AI. AI is allowing this industrialization to happen, especially turning these bespoke artisanal processes into something that is engineered and industrialized.
AI is one aspect of it, and there are many others. I've talked about robotics. That's the arc that's exciting. We saw hints of it in 2015. It's probably a 25-year arc, maybe 30-year arc that we're 10 years into. Industrial revolutions don't happen overnight. But when you look back, the whole world is going to be changed. We're living in the middle of it. I was always jealous about people living in the 1920s, going from nothing to steam trains and all this stuff. Now we're the ones in the center of it. It's such an exciting time.
Host
Right. You see that picture of, I think it's somewhere in New York, where you have all of these horses lined up. Back then that just felt like the norm. Then you see what, like a decade later, it's all replaced by cars. So Pushmeet, maybe we could use AlphaFold as an example here. A lot of people listening to the podcast are maybe most familiar with that paper and that breakthrough, but it's also a great example of how that didn't happen overnight. I think most people noticed it in 2020, but it didn't start in 2020. So maybe you could talk about that arc. What is AlphaFold? How did it come to be? And where are we today in terms of its impact?
Pushmeet Kohli
I was telling how I started my journey with the science program at DeepMind. At that time, we had these two small scale projects. One was protein structure prediction, and the other one was quantum chemistry. AlphaFold rose from that protein structure prediction project. In its simplest form, it's a very simple problem. Given an amino acid sequence, which constitutes a protein, we want to understand the 3D coordinates of those amino acids. That's really important because if you understand the 3D structure of the protein, that gives you an idea about what the function of that protein would be. That has implications for drug discovery, for understanding basic cellular biology, and so forth.
We started working on this problem because it satisfies one of our key requirements when we look into problems. It's a real foundational root node problem. Once you solve it, it has so many implications in disease understanding, biology, synthetic biology, and so on. Not only that, it is a classic machine learning problem. You require reasoning because you are working with an expanded solution space, and you have access to raw material, which is data. The structural biology community had done an amazing job curating a very good dataset in the form of the PDB. Scientists all across the world, whenever they found the structure of a protein, which sometimes took almost five years or even a decade in some cases, would diligently deposit that 3D structure in this database.
At that time when we started, I think there were 150,000 structures from X-ray crystallography and cryo-EM. That was an amazing dataset to start with. The other big problem in machine learning is how to evaluate the machine learning model. In machine learning, one of the easiest things you can do is fool yourself. These models are extremely good at cheating. If you give them any way to cheat, they will cheat. The protein folding community and the protein structure prediction community had this biannual competition called CASP, the Critical Assessment for Structure Prediction. They would run this blind assessment, like an Olympics of protein structure prediction, where people would be given protein sequences whose structure was not known by anyone except one experimentalist who had deposited it. The true generalization ability of the model would be exhibited.
We thought this problem checked a number of key criteria which we use for taking up a problem for the very long term. We started with a team which investigated how much progress we could make. We were hopeful, optimistic that machine learning could play a bottom role, but we didn't know. This was a new problem for us, and we were approaching it with a lot of respect.
Host
And Pushmeet, what year was this when it started?
Pushmeet Kohli
We started around 2017, and we took part in the critical assessment at the end of 2018. When we entered AlphaFold 1 in 2018, we were not really sure where it would be, maybe in the top three. But it actually performed really well. It not only was state-of-the-art, but outperformed state-of-the-art by a margin. That validated our hypothesis. The basic research philosophy at DeepMind has been the multidisciplinary nature of the teams. We had brought in some really good structural biologists and biophysics people. John Jumper, being the lead of AlphaFold, was part of the team at that time. That gave us a lot of confidence.
Now we were the best in the world, but the model was still not useful. It was producing good results, but it was nowhere close to solving the problem. We had to make a bet. Can we really go after it and solve it once and for all? Or is this it? The first thing we had to do was start from scratch. We had to throw AlphaFold 1 from the table and say, "This approach that we have started is not going to work."
Host
What gave you the indication that AlphaFold 1 couldn't take you to the next level? Because I think even in the AI space outside of science, there are a lot of questions around, can we just depend on the scaling laws? Do we need some sort of new unlock to get to, you know, insert problem here, could be AGI, could be something else. What gave you the indication that this is great, we're so happy with our results, but we actually need to throw this out and start anew?
Pushmeet Kohli
AlphaFold 1 had adopted a classical two-stage approach. The machine learning model's job was, given a sequence, it does not predict the 3D coordinates of the amino acids directly. What it predicts is the distance between amino acids. Then there is a second stage which was supposed to take that distance matrix and recover the 3D coordinates. The machine learning neural network's job was restricted to finding the distances between amino acid residues. This two-stage model was very effective, but it was not very elegant. If you made certain errors, you would not be able to backpropagate back to the neural networks because you found the results after the second stage, and the neural network would not get that supervision.
We believed that in order to properly train the model, we needed end-to-end. We needed a model which could go directly from the sequence to the structure. That was one critical change that needed to be made. But it was a difficult change because you are starting from a much lower baseline when you are building up that end-to-end network.
Host
So let's fast forward. So you did throw out AlphaFold 1, and then what happens after that?
Pushmeet Kohli
AlphaFold 2, we start this long journey where we start making progress with a much lower performance than AlphaFold 1 even. We have this internal leaderboard where everyone in the team can propose ideas and try out their ideas on the central leaderboard to see how much of a delta each idea or each change makes. We were making steady progress. Then there were times where progress would stagnate, sometimes for months it would stagnate, and people would ask, "Have we reached the limit?" But over time, and I think around when the pandemic started, we caught some really big deltas where we thought we are making real progress.
If you look at the metrics as to how you quantify protein structure prediction accuracy, it's called GDT. We had crossed that 80 GDT threshold. That was unprecedented. That motivated us to push it even further and later on to a 90 GDT and beyond, which we thought is what we needed to do.
The pandemic really brought home to the whole team the actual importance of the problem. We were all sitting in our homes shielding, and there were scientists out there who said if you have the structure of the different SARS-CoV-2 proteins, it would be really helpful. The community very quickly found the structure of the spike protein because it was similar to SARS-CoV-1, but the accessory proteins of the virus, the structure for those was not known. The fact that we could compute these predictions, share it with experts who are trying to deal with the pandemic and think about designing inhibitors, it really brought to the team the real world impact and relevance that this fundamental problem has.
Around September 2020, when the second CASP competition ended, we got this email from the organizers who wanted to chat. That was unprecedented. We were surprised, like, why did the organizers want to chat so early on? They were super surprised at how good the predictions were. In fact, some of them speculated, "Maybe this team has cheated in some way. It couldn't be so good." There was one particular scientist who had submitted a protein but did not know the structure. They had hoped that the structure would be obtained by the time the competition ended. But this structure was not known to anyone, literally anyone. And AlphaFold could give them an initial starting point which can solve the structure for that particular protein. They were totally amazed that such a system now existed in the CASP competition.
We later released AlphaFold. Not only was it very accurate, it was also very efficient. We decided to find the structures for almost all the proteins that are known to scientists, around 250 million of them, and put them in a database with our partners, the European Molecular Biology Laboratory, EMBL-EBI, and made that as a resource that anyone can access.
Host
That's amazing. I'd love to turn it to you, Vijay. You obviously have run a lab for a long time and you've been on the other side of this. All these researchers who now have access to this database, which by the way for the audience, one of these structures may have taken the length of a PhD to solve a single structure. Now we're talking about true scale. And again, this being deployed to all the researchers that can access it. Vijay, maybe you can speak to what that really means and also if we can apply this to other areas of science as well.
Vijay Pande
The impact of this is many fold. I can speak to it both from looking at it from the academic lens, but also from the last 10 years of investing in startups and how startups use this as well. First off, I think it's worth really emphasizing the significance of structure itself. The reason why universities like Stanford have whole departments for structural biology is that the structure is typically pretty evocative of function and other biological aspects. Perhaps the most notable example is the DNA structure. Watson and Crick came up with this structure. By looking at just the structure, you can imply how DNA is replicated and essentially how genetics works to some degree, the very basics of it. That's one of the most dramatic examples, but there are numerous examples where if you have the structure, you can understand the function.
Structural biology is a fundamental part of how we understand biology from the molecular scale up. For drug design, often if we understand the structure and its dynamics, we can understand how to drug proteins and come up with therapeutics in a much more engineered fashion. The significance of structural biology is huge. It's also a time where structural biology is in a renaissance because, as you mentioned, it used to take many years to come up with experimental structures, but new methods like cryo-EM can come up with structures in much shorter times, even days. There's a renaissance going there.
For structural biology as a field, I think we'll see this combination of new experimental methods and computational methods. What was most striking to me is how experimentalists were going to these databases and looking at them and using it almost like you would use the human genome database. The human genome database takes genomics and turns it into a database lookup. You don't have to do the experiment yourself if you can just do the computational query. To some degree, what AlphaFold did is it took the structural biology of proteins and made it a database lookup. It's not exactly a true database lookup in the sense that this is a prediction, but as the quality of predictions gets higher and higher, it becomes kind of the same thing. That's huge.
The final thing that was most striking is that there's always going to be a shift from academia to industry. Maybe 30 years ago, academics would design computer chips and new types of microprocessors and new architectures. We don't do that now in academia. That's not something that makes sense to do. That's much better done in companies, especially given the scale of what's going on. For multiple reasons, this is something that DeepMind was perfectly suited to do in a way that academic groups really weren't. That shift now suggests that it's a really interesting time for this to leave academia and now be in the industrialized world of startups and companies.
Host
That's really interesting, the relationship you're talking about of academia and industry. Something that people talk a lot about these days is whether these different AI models can really fundamentally advance science the way that you typically think of academics as the parties that are facilitating that. I'd love to hear from both of you, maybe starting with you, Vijay, what indications, whether it's through AlphaFold or other projects that you're seeing emerge, actually indicate that yes, these models, these scientific discoveries in a sense, are able to help us actually push the frontier instead of maybe just help us be a little more efficient within the zone that we're already in.
Vijay Pande
Pushmeet said that structure prediction is a foundational problem. If you take the arc of drug design, first you have to come up with understanding the biology. AI for biology is a very interesting area where we can maybe start to understand the nature of pathways and do this on human biology in ways that don't require experiments on human beings, which has always been one of the biggest limitations. We understand mouse biology really well because of all the experiments we can do, but we could never do that on human beings directly. AI models for humans, as they become more predictive, and especially just more predictive than a mouse is predictive of a human, the mouse is a model in a sense. That gets super interesting for unraveling biology.
AI for biology is a thing. We could talk about AI for chemistry, and I think AlphaFold is in that category where now we're trying to understand biophysical chemistry. How can we quickly drug undruggable proteins? How can we come up with new antibodies and design proteins? That's a whole area. Finally, AI for clinical trials is going to be where maybe the biggest impact financially will be. Clinical trials could cost hundreds of millions to billions of dollars. Even a 10% improvement on a billion-dollar enterprise is huge. That's where some of the toughest problems are to work on. As we make impact there, clinical trials will be better, will be probably more easily powered, and will be hopefully more successful because we'll be picking the right ones to do.
That turns into eventually AI for personalized medicine, which is an extension of that trial. I don't want you to do an experiment on me as a mouse or a rat, but I would love to make sure I get the best drugs for me. You and I are different and we'll respond differently to drugs. To be able to have that predicted would be huge. That's the arc of it. We're just at the very beginning.
Host
Definitely. We talked about AlphaFold, which is very exciting and maybe the most familiar to folks. But Pushmeet, your team has also created a bunch of other papers that touch this intersection of AI and science, or you could say AI and math or AI and physics. Those are things like materials, GraphCast which has to do with weather forecasting, FunSearch, AlphaGeometry. I'd love to hear from you again on this probing of, are we moving the frontier forward with these different models? What are you seeing from some of these other projects that your team is working on in terms of AI helping us actually uncover new science?
Pushmeet Kohli
Essentially, what we have entered is an age where a single human mind cannot comprehend the data that we are gathering about the universe. This is true in any field you now encounter. It is true in biology. No biologist can reason and analyze all the biological data that has been gathered. No physicist can look at and analyze all the high energy physics data that is being gathered. Even mathematicians cannot look at and analyze all the large scale mathematical simulation data that we can now compute and simulate. AI is not nice to have. It's basically almost a necessity for us to make sense and reason about any problem that we are now looking at.
I have examples in pure mathematics where we work on topology. You describe a knot in two different definitions. There is an algebraic definition and there is a geometric definition. Mathematicians understood these categorizations, but never understood the connections between them. In one of our works, we generated a lot of data for knots in these two characterizations. We asked the neural network, "Can you make predictions about one characterization from the other?"
And the idea was, well, the answer should be no. But in fact, it could make predictions. And when we drilled down, we found a very nice conjecture that nobody had encountered. We worked with mathematicians who then not only made that conjecture, but actually proved there was a very elegant, nice relationship between those two characterizations. So this is like completely fundamental discoveries in mathematics that were completely unknown to mathematicians, now being uncovered by a machine learning and AI model. We are seeing this across the board in any of the scientific areas we are looking at. We are discovering new insights, new patterns that were not expected, just because the techniques to analyze the raw scale of data did not exist.
Vijay Pande
I think amongst biologists, especially maybe 10 years ago and further back, there was often a belief that biology is just so complex that it's incomprehensible, that there's no way to even understand it. The only thing you can do is run the experiment and see what happens. I think we're seeing the beginning of a shift where people are starting to think, well, there are complexities and a lot we don't know, but AI actually can gather all that together and start to decipher this and be a natural language for biology. There's going to be this really fun cultural shift where 10 years ago, people would say, "Oh, it's ridiculous.
A computer could try to do these things." I think 10 years from now, people will be like, "Oh, it's ridiculous to have a human being do that. You can't load all these numbers in your head. That's just ridiculous to even say that." We've seen this in other places like chess. It seemed impossible that a computer could beat a grandmaster. And now there's not even worth trying. And we saw with Go and all these other things. So I think that's just a cultural shift. But I don't think that's a bad thing. A forklift can lift much more than the strongest weightlifter, and we view that as a positive thing.
It's always going to be us and them. The interesting question will be: once it can do these things that we can't do, what do we do together with that?
Host
Yeah. And what can we do? One of the most amazing things is that DeepMind, for the most part, has given these models or the results of them to the community. So researchers have their hands on them. Maybe we could talk about that. How are researchers leveraging these new breakthroughs? There's all kinds of stats around we don't have enough cancer drugs or they're in shortages. And those are very real things we want to fix. So Pushmeet, maybe we'll start with you. What are you seeing in terms of this technology being deployed and how are researchers using it?
Pushmeet Kohli
Yeah, so this was another fascinating journey of growth. As I told you, I was not from the natural sciences, so working on AlphaFold was a learning experience, but releasing AlphaFold to the community was an even bigger learning experience. The AlphaFold database, when we were building it up, we wanted it to be available everywhere on the planet to all scientists. But the scale of science was unprecedented. I was not aware of it. The AlphaFold database today has been accessed in 190 countries, and there have been 1.6 or 1.7 million users of the AlphaFold database. If that is not a positive statement about the planet, then I don't know what is. There are 1.7 billion people interested in protein structure prediction. I'm really happy about that, given all the things that are happening in the world.
In terms of impact, it's an amazing spectrum. We saw AlphaFold being used in pathbreaking, fundamental biological discoveries. My personal favorite is the nuclear pore complex—the structure of the pore complex, the way a nucleus controls how material gets into the nucleus and out. That fundamental structure was not known, and researchers used AlphaFold 2 structures to piece together the whole complex. A recent paper from the Feng lab showed how you could develop a molecular syringe, and again they used AlphaFold 2 in designing that. There are so many other areas where people have been using it for developing new vaccines, working on new antibiotics against antimicrobial resistance, and synthetic biology. One of the key partners at the early stages was a university here in the UK, which was using AlphaFold to develop and think about enzymes that could decompose plastics. So you have this whole spectrum of fundamental biology, drug discovery, even synthetic biology and enzyme development that has been impacted by AlphaFold. It was very difficult to even predict what would be the uses of the tool.
Vijay Pande
I think there's also just within biology, there's become a shift that people are wrapping their heads around prediction a bit better. Before, experiment was the gold standard and that was all people wanted to hear about. Part of it is also just the zeitgeist of the time—when you deal with large language models, you're basically dealing with predictions of what comes next. I think people have understood the pros and cons of predictions, but that there's massive value in having it. It's funny that we talk so much about the technology, but I think the human shifts and the cultural shifts are the things we're really going to need to push. What gets me most excited about what Pushmeet's just been talking about is that it's a sign that we're seeing this cultural shift as well.
Host
Maybe something else you could speak to, Vijay, that's coming to mind as both of you are sharing more about these researchers. How does this change the economics of a lab? We talked about how uncovering a structure could have taken a whole PhD. Now we have new tools and we're seeing these economics change in some of the more consumer fields. How does this change the economics of research overall?
Vijay Pande
One of the fantasies that one of my former colleagues talked about was what we call "beach biotech," where you have one person and a laptop, presumably on the beach or wherever you want to be. You've got CROs, these contract research organizations, to do the experiments. You have some AWS cloud or GCP cloud somewhere to run your calculations. That one person with AI—we're not quite there yet, but I think that's an intriguing fantasy to think about. On the way to that one-person aspiration, we have smaller teams doing way more with much less capital outlays, building startups much more efficiently and getting to results much more rapidly. The challenge is what I mentioned before: getting to the clinical trials and speeding that up will be nice, but the big financial return will be on the clinical trial side. The expectation is that AI for biology and understanding targets based on human data would also help on the trial side. So put together, I think we can get to these therapeutics faster, cheaper, and hopefully better.
Host
Yeah. And maybe, Pushmeet, we could tackle that directly. For folks who aren't these researchers who are already leveraging these tools, how much does it really cost if someone wants to get a protein structure prediction or use some of the other models like GraphCast or materials? What cost are we really looking at?
Pushmeet Kohli
For the AlphaFold database, it's literally free. You just go to the AlphaFold database, find the protein you're interested in out of the 250 million proteins, click it, and it's there. It's free for everyone on the planet to use. It has democratized things in a way that scientists in Latin America or India who are working on neglected tropical diseases, for instance, who had no way to get a structure of a protein they were interested in, can now get access to these structures at the click of a button. Of course, a lot of research needs to be done to take that work towards a more focused outcome, and a lot more investment is needed.
If you are trying to accomplish the vision that Vijay outlined, the protein structures are a start. But you really need to think about how it binds to ligands, how you do the ligand design, how you solve the co-folding problem. There is a lot of investment needed to make these models and refine them for specific applications. We have a spinoff from DeepMind, Isomorphic Labs, which is now investing in this area. At the same time, we are continuing work on the foundational side and have released an announcement on the next generation of AlphaFold, which goes beyond proteins to other biomolecules like nucleic acids—DNA, RNA—and small ligands.
Host
I think it's amazing that you've opened this up to the community. I'd love to hear both of your takes on the relationship of these models and them being open sourced. It's a big debate within AI at large, but especially when it comes to science. There's nothing more that people get excited about than this idea of curing cancer, solving poverty, and an agriculture crisis. But at the same time, people also get very scared. That's where people's sci-fi nightmares come to be—someone can engineer a molecule that can kill us all. Starting with you, Vijay, what's your take on this relationship of AI and science and why it should be open source?
Vijay Pande
The beauty of open source, for AI and biology but more broadly, is that people can build on top of each other. What's remarkable about the AI field over the last five, maybe 10 years, is that it feels like an amazing result comes out once a week. The key part is that it comes out with code or a GitHub repo that you can check out immediately. You don't even have to just believe the results—you can run it yourself. People have even open sourced various tests. So essentially, we're building a skyscraper where each person builds a new floor and we're going up really fast. That's what open source can do. In the past, if it wasn't open source, I'd have to read the paper, code it myself, and sometimes the paper may be a little vague on some detail, so I might not bother. Open source allows us to build on top of each other and build rapidly.
Now, certain parts won't be open source. You unfortunately can't open source a drug compound because then no one's going to pay for the trial. The economics just doesn't make sense given the hundreds of millions or billions of dollars involved. So certain parts will be closed source, and there are hundreds of startups in AI biology and AI drug design that will take advantage of what's been done, develop their own methods, and build on top. That's where I think the drugs will come from.
You also mentioned the concern that because this is so powerful, we could do dangerous things with it. There's a bit of a misconception there because there's a huge asymmetry between the complexity of drug design for treating disease—that's a really hard problem—and how easy it is to come up with chemicals that are dangerous and toxic. In fact, that's why we have phase one trials, because even the things you thought would hopefully not be toxic at all turn out to be toxic. It's actually very easy to make toxic things. Google will teach you how to get ricin and all this other stuff, for better or worse. So the asymmetry is that if we get rid of AI for drug design, you lose all the good and you don't prevent any of the bad, which is already here.
Host
I think that's a good point that a lot of people don't think about. Pushmeet, maybe you could speak to why DeepMind has chosen to open source these models, which isn't necessarily the norm across different AI companies.
Pushmeet Kohli
There was a lot of deliberation within the team and within the company on this. There were a few different things that went into that final decision. One was that AlphaFold was so foundational. If we had kept it closed source, fully leveraging its impact for society would have been difficult. It's very hard to even predict what the potential applications are. Just to give you an example: when we launched AlphaFold, a couple of days later, somebody did an analysis on the uncertainty associated with the AlphaFold predictions and figured out that AlphaFold, even though it was not trained for that, was the best predictor for predicting disorder in proteins. That was something we would not have come up with. Someone interacting with the model in the community figured that out.
So when we were thinking about it, there was, of course, how to maximize social and scientific impact. The second consideration was responsibility. We consulted a number of experts from structural biology, chemistry, and drug discovery to figure out the right, responsible, and safe approach. Even considering malicious use cases, after we had done all the due diligence, we felt that this was safe to release and that the impact of releasing it and open sourcing it would outweigh any costs. So it was decided that we should open source it. I think the decision has been validated by the impact that AlphaFold 2 has had in the community. Now, that's not true for all models. Subsequently, we have had models which we have not open sourced. But in the case of AlphaFold 2, the decision was very, very clear in favor of sharing it with the world in the most freely possible way.
Host
For the ones that you haven't chosen to open source, if you're willing to share, how do you make that decision?
Pushmeet Kohli
There are a number of different factors: what would be the social impact, the scientific impact of releasing things, versus what is the commercial cost of releasing something while leveraging it for commercial purposes, or even the safety argument. Just to give you an example, one of our recent models announced last year was AlphaMissense. This is a model for predicting the effect of missense variants. It produces state-of-the-art accuracy in making predictions about whether missense variants are benign or could be pathogenic. In this particular case, we felt that the predictions of the model for the human genome—the 71 million human missense variants—if we released that, it would serve most of the purposes a clinician or biologist would be interested in. So we just released the predictions rather than the model, because the model had many other uses. You could run it on different organisms. There were other commercial considerations. So we released the predictions, shared the methodology, but we did not open source the approach.
Host
That makes sense. At the very outset, you shared so many different projects or areas of scientific study that your team is working on. I'm just so curious because it sounds like there's been success across many. Are there any areas of science or mathematics that you've tried to address with this approach of using machine learning and AI that's not quite working? Whether it be because we don't have the prior data set, as Vijay has spoken to, that sets the foundation? I'm just curious if there are limitations emerging in any of these fields that your team is running into.
Pushmeet Kohli
One specific area that I would love to have impact on, and I think AI would eventually have impact on, is systems biology. It's an incredibly important problem to really understand at the system level how biological systems behave. It's just that the data and the evaluation are not at a place where it is for genomics, functional genomics, or structural biology. Before we actually start an initiative in any of these areas, there is a huge due diligence process we need to undergo, because you're making a very long-term commitment. The careers and the impact of some of the best scientists and engineers we have are being committed to that area. We take that responsibility very seriously. Only when we are confident of the impact of the problem, confident that we have a good evaluation metric to track progress, and we have the raw material—the data or a simulator to get good data—only then do we make that long-term commitment towards a specific topic.
Vijay Pande
To highlight the data issue, I think one of the biggest differences between AI for language models or AI for video and AI for biology or healthcare is that most of the interesting data in biology and healthcare is either dark—all these medical records that you just can't access on the internet, which would be very useful for understanding the healthcare side and trial side—or it's never been measured. We need to do the experiments. Having the data could be paramount. That's going to be different than other places. Other places, maybe the algorithms can really drive things because everyone has the same data more or less. Here, people will be differentiated by their data. The innovations will be innovations in AI combined with innovations in data collection. There are things at that interface, like active learning and how to use data more efficiently. But the data game, I think, is going to be huge.
Host
Absolutely. And Vijay, I'd love to get your take. You've spoken to a few examples already, but what different areas do you wish that more attention was being allocated? Or do you think there's a set of grand challenges that can and will eventually be solved with some of this technology?
Vijay Pande
The fun thing about CASP, this critical assessment of structure prediction, is that it inspired all these other prospective trials and prospective studies. There's a ton of that stuff to do. There's a test for predicting binding of small molecules. I think we'll see in time that these types of methods do extremely well in those assessments. But the holy grail, in my mind, is being able to predict clinical trials. To understand how a drug works in human biology—that's a systems biology problem at the largest scale. That is the holy grail. I think we'll probably do it in parts. You could imagine models for specific organs or specific parts of the body, and then we put them together.
Mixtures of experts is pretty common these days, and that might be one approach. However it gets done, once it gets to the point where these models are better than the animal models, that's where there's really going to be a tipping point. We can move much more rapidly without being stymied by having to run these animal models, which takes a long time and is very expensive. There are crazy things like right now, there's a monkey shortage because monkeys are in such high demand to run these experiments. So there's probably a long road to get there, but once we get there, that will be a major inflection point.
Host
Wow. I did not know there was a monkey shortage. But it really is important to know. To your point, hopefully we get to a future where some of the things we're doing in research today seem just so incredibly outdated because we have better options. Pushmeet, what's next up for DeepMind in terms of areas of interest? You're already working on so many things, but I'd love to get a pulse on what's exciting for you too.
Pushmeet Kohli
What is fascinating about science in any of these fields is that there's so much more to work on. Even on structural prediction, I just mentioned that the latest version extends to general biomolecules like DNA, understanding RNA, understanding the interactions between small molecules, ligands, and proteins, bigger complexes, antibodies. There are so many things we can extend in genomics. We have worked on both gene expression—the coding part of the genome with missense variants and the non-coding part of the genome—like predicting gene expression. We have made progress, but we are not completely at the end of it.
There's a lot we are doing in all these areas. In material science, you mentioned the model GNoME, which was able to predict 400,000 novel stable compounds, expanding the number of stable compounds known by more than an order of magnitude. But how do you now take those compounds and reason about their specific properties that would be useful in a particular application? In any of these disciplines, we are not targeting one specific milestone. We are saying, here is a topic, and the long-term roadmap is to think about a paradigm shift in how science is done in that area and move towards a more rational, modeling-based approach.
There's a lot that needs to be done. We are trying to focus on certain specific areas, and new areas come up if the raw materials are there in terms of data and if we are clear on the evaluation metric. We are constantly reviewing them as well.
Host
That's amazing. I haven't done as much research as you, Vijay, but I did do a summer of battery research and materials research where we were trying to discover new sodium ion transition metal materials. My summer was literally—I was in college, so I wasn't very advanced—but it was literally finding a paper that documented how to synthesize this material in the kiln, mixing it up, creating a little battery, doing it in the glove box, and running it to see how effective it was. In many cases, it was very ineffective. But every so often we found a material, and it was truly just trial and error, trial and error, trial and error. When I see papers that do things in a completely new way at scale, way cheaper, without all these university students in a glove box day and night, it's so exciting.
Vijay Pande
The end point for me is that we're in the middle of this journey—this technological journey, this cultural journey, these cultural shifts. It's going to feel like the big goals I've laid out, like clinical trials and systems biology, are so far off. It's going to take a while, but we can get a lot done in 10 years, collectively, 15 years. Think about where we were five years ago, 10 years ago, 15 years ago. 15 years ago, people weren't really even talking that much about deep learning, or just beginning. So the goals we have are lofty, but I think we're right in the thick of it. All of it is very doable. It's just building that tower one step at a time. It'll be fun to have this chat again in five years. Hopefully sooner.
Pushmeet Kohli
One thing that has been very exciting in the last few years is the rise of LLMs and foundational models. In most of the projects I was talking about, we were working with structured data—data that was either collected or, in the case of some of our fusion work, simulated. But with the rise of foundation models and LLMs, that opens up the possibility of using unstructured data to feed these models. That really opens the door for large-scale ingestion of scientific knowledge into the models. That is a very exciting direction that will bring a number of other problems into the feasibility zone which previously were not there. Of course, there are challenges with understanding uncertainty, hallucination, and all these technical problems that need to be addressed. But once that is done, the impact on models for scientific discovery would be amazing. So that's another reason to be excited for the future.
Host
Absolutely. And all of the problems you just mentioned are also opportunities for people to go and fix and be a part of that whole ecosystem. This has been really wonderful, Pushmeet, Vijay. Thank you for getting people excited about what's to come. These two fields intersecting—what a time to be alive here in 2024 to be a part of it. Like you said, Vijay, we're in our equivalent of the 1920s. Hopefully people in the 2120s will look back at this fondly.
Vijay Pande
Absolutely. Yeah.
Host
If you liked this episode, if you made it this far, help us grow the show. Share with a friend, or if you're feeling really ambitious, you can leave us a review at ratethispodcast.com slash a16z. Candidly, producing a podcast can sometimes feel like you're just talking into a void. So if you did like this episode, if you liked any of our episodes, please let us know. We'll see you next time.