--:-- Dwarkesh Patel: Today, I have the pleasure of speaking with Dario Amodei, who is the CEO of Anthropic. And I'm really excited about this one. Dario, thank you so much for coming on the podcast.
--:-- Dario Amodei: Thanks for having me.
--:-- Dwarkesh Patel: First question, you have been one of the very few people who has seen scaling coming for years, more than five years. As somebody who's seen it coming, what is fundamentally the explanation for why scaling works? Why is the universe organized such that if you throw big blobs of compute at a wide enough distribution of data, the thing becomes intelligent?
--:-- Dario Amodei: I think the truth is that we still don't know. I think it's almost entirely an empirical fact. It's a fact that you could kind of sense from the data and from a bunch of different places. But I think we don't still have a satisfying explanation for it.
--:-- If I were to try to make one, but I'm just waving my hands when I say this, there are these ideas in physics around long tail or power law of correlations or effects. So when a bunch of stuff happens, when you have a bunch of features, you get a lot of the data in the early, fat part of the distribution before the tails. For language, this would be things like, "I figured out there are parts of speech and nouns follow verbs." And then there are more and more subtle correlations. So it kind of makes sense why there would be this—every log order of magnitude that you add, you capture more of the distribution.
--:-- What's not clear at all is why does it scale so smoothly with parameters? Why does it scale so smoothly with the amount of data? You can think up some explanations of why it's linear. Like the parameters are a bucket and the data is water, so the size of the bucket is proportional to the size of the water. But why does it lead to all this very smooth scaling? I think we still don't know. There are all these explanations. Our chief scientist, Jared Kaplan, did some stuff on fractal manifold dimension that you can use to explain it. So there are all kinds of ideas, but I feel like we just don't really know for sure.
--:-- Dwarkesh Patel: By scaling, we're referring to the fact that you can very predictably see how you go from GPT-3 to GPT-4, or in this case, Claude 1 to Claude 2, that the loss in terms of whether it can predict the next token scales very smoothly. So we don't know why it's happening, but can you at least predict if empirically, here is the loss at which this ability will emerge, here is the place where this circuit will emerge? Is that at all predictable, or are you just looking at the loss number?
--:-- Dario Amodei: It is much less predictable. What's predictable is this statistical average, this loss, this entropy. And it's super predictable—sometimes even to several significant figures, which you don't see outside of physics. You don't expect to see it in this messy empirical field. But specific abilities are very hard to predict.
--:-- Back when I was working on GPT-2 and GPT-3, when does arithmetic come in place? When do models learn to code? Sometimes it's very abrupt. It's kind of like you can predict statistical averages of the weather, but the weather on one particular day is very hard to predict.
--:-- Dwarkesh Patel: Dumb it down for me. I don't understand manifolds, but mechanistically, it doesn't know addition yet. Now it knows addition. What has happened? And this is another question that we don't know the answer to.
--:-- Dario Amodei: We're trying to answer this with things like mechanistic interpretability. You can think about these things about circuits snapping into place. Although there is some evidence that when you look at the models being able to add things, if you look at its chance of getting the right answer, that shoots up all of a sudden. But if you look at the probability of the right answer, you'll see it climb from one in a million to one in a hundred thousand to one in a thousand long before it actually gets the right answer. So in many of these cases, at least, there is some continuous process going on behind the scenes. I don't understand it at all.
--:-- Dwarkesh Patel: Does that imply that the circuit or the process for doing addition was preexisting and it just got increased in salience?
--:-- Dario Amodei: I don't know if there's this circuit that's weak and getting stronger. I don't know if it's something that works, but not very well. I think we don't know. And these are some of the questions we're trying to answer with mechanistic interpretability.
--:-- Dwarkesh Patel: Are there abilities that won't emerge with scale?
--:-- Dario Amodei: I definitely think that things like alignment and values are not guaranteed to emerge with scale. One way to think about it is you train the model and it's basically predicting the world. It's understanding the world. Its job is facts, not values. It's trying to predict what comes next. But there are free variables here where it's like, what should you do? What should you think? What should you value? There just aren't the bits for that. It's like, if I started with this, I should finish with this. If I started with this other thing, I should finish with this other thing. So I think that's not going to emerge.
--:-- Dwarkesh Patel: On scaling, if it turns out that scaling plateaus before we reach human level intelligence, looking back on it, what would be your explanation? What do you think is likely to be the case if that turns out to be the outcome?
--:-- Dario Amodei: I would distinguish some problem with the fundamental theory from some practical issue. One practical issue could be we run out of data for various reasons. I think that's not going to happen, but if you look at it very naively, we're not that far from running out of data. So it's like we just don't have the data to continue the scaling curves. Another way it could happen is we just use up all of our compute that was available and that wasn't enough, and then progress is slow after that. I wouldn't bet on either of those things happening, but they could.
--:-- From a fundamental perspective, I personally think it's very unlikely that the scaling laws will just stop. If they do, another reason—again, this isn't fully fundamental—could be we don't have quite the right architecture. If we tried to do it with an LSTM or an RNN, the slope would be different. It still might be that we get there, but I think there are some things that are just very hard to represent when you don't have this ability to attend far in the past that transformers have.
--:-- If somehow it wasn't about the architecture and we just hit a wall, I think I'd be very surprised by that. I think we're already at the point where the things the models can't do don't seem to me to be different in kind from the things they can do. You could have made a case a few years ago that it was like they can't reason, they can't program. You could have drawn boundaries and said, maybe you'll hit a wall. I didn't think that. I didn't think we would hit a wall. A few other people didn't think we would hit a wall, but it was a more plausible case then. I think it's a less plausible case now.
--:-- It could happen. This stuff is crazy. It could happen tomorrow that we just hit a wall. If that happens, I'm trying to think of what would really be my explanation. I think my explanation would be there's something wrong with the loss when you train on next word prediction. Some of the remaining reasoning abilities—if you really want to learn a program at a really high level, it means you care about some tokens much more than others. And they're rare enough that the loss function over-focuses on the things that are responsible for the most bits of entropy. Instead, they don't focus on this stuff that's really essential. So you could have the signal drowned out in the noise. I don't think it's going to play out that way for a number of reasons. But if you told me, "Yep, you trained your 2024 model, it was much bigger, and it just wasn't any better, and you tried every architecture and it didn't work," I think that's the explanation I would reach for.
--:-- Dwarkesh Patel: Is there a candidate for another loss function if you had to abandon next token prediction?
--:-- Dario Amodei: Then you would have to go for some kind of RL. And there are many different kinds. There's RL for human feedback. There's RL against an objective. There's things like constitutional AI. There's things like amplification and debate. These are both alignment methods and ways of training models. You would have to try a bunch of things, but the focus would have to be on what do we actually care about the model doing. In a sense, we're a little bit lucky that "predict the next word" gets us all these other things we need.
--:-- Dwarkesh Patel: It seems like from your worldview, there's a multitude of different loss functions. It's just a matter of what can allow you to throw a whole bunch of data at it. The next token prediction itself is not significant.
--:-- Dario Amodei: The thing with RL is you get slowed down a bit because you have to, by some method, design how the loss function works. The nice thing with next token prediction is it's there for you. It's just there. It's the easiest thing in the world. So I think it would slow you down if you couldn't scale in that very simplest way.
--:-- Dwarkesh Patel: You mentioned that the data is likely not to be the constraint. Why do you think that is the case?
--:-- Dario Amodei: There are various possibilities here. For a number of reasons, I shouldn't go into the details. But there are many sources of data in the world, and there are many ways that you can also generate data. My guess is that this will not be a blocker. Maybe it would be better if it was, but it won't be.
--:-- Dwarkesh Patel: Are you talking about multimodal?
--:-- Dario Amodei: There are just many different ways to do it.
--:-- Dwarkesh Patel: How did you form your views on scaling? How far back can we go? And would you have been saying something similar to this?
--:-- Dario Amodei: This view that I have probably formed gradually from 2014 to 2017. My first experience with it was my first experience with AI. I saw some of the early stuff around AlexNet in 2012. I always kind of wanted to study intelligence, but before that I was just like, this isn't really working. It doesn't seem like it's actually working. All the way back to 2005, I'd read Ray Kurzweil's work. I'd read even some of Eliezer's work on the early internet back then. And I was like, this stuff looks far away. I look at the AI stuff of today and it's not anywhere close.
--:-- But with AlexNet, I was like, this stuff is actually starting to work. So I joined Andrew Ng's group initially at Baidu. The first task I got set to do—I'd been in a different field, so this was my first experience with AI. It was a bit different from a lot of the academic style research going on elsewhere in the world. I think I got lucky in that the task given to me and the other folks there was just: make the best speech recognition system that you can. There was a lot of data available. There were a lot of GPUs available. So it posed the problem in a way that was amenable to discovering that scaling was a solution.
--:-- That's very different from being a postdoc where it's your job to come up with an idea that seems clever and new and makes your mark as someone who's invented something. I just quickly discovered that—I just tried the simplest experiments. I was fiddling with some dials. I was like, "OK, try adding more layers to the RNN. Try training it for longer. What happens? How long does it take to overfit? What if I add new data and repeat it less times?" And I just saw these very consistent patterns. I didn't really know that this was unusual or that others weren't thinking in this way. This was just kind of beginner's luck. It was my first experience with it.
--:-- I didn't really think about it beyond speech recognition. I was just like, "I don't know anything about this field. There are zillions of things people do with machine learning, but this seems to be true in the speech recognition field."
--:-- Then just before OpenAI started, I met Ilya, who you interviewed. One of the first things he said to me was, "Look, the models, they just want to learn. You have to understand this, the models, they just want to learn." And it was a bit like a Zen Kohen. I listened to this and I became enlightened. Over the years after this, I would be the one who would formalize a lot of these things and put them together. But what that told me is that that phenomenon I'd seen wasn't just some random thing. It was broad. It was more general. The models just want to learn. You get the obstacles out of their way. You give them good data. You give them enough space to operate in. You don't do something stupid like condition them badly numerically. And they want to learn. They'll do it.
--:-- Dwarkesh Patel: What I find really interesting about what you said is there are many people who were aware back at that time, probably weren't working on it directly, but were aware that these things are really good at speech recognition or at playing these constrained games. Very few extrapolated from there like you and Ilya did to something that is generally intelligent. What was different about the way you were thinking about it versus how others think that you went from "it's getting better at speech in this consistent way" to "it will get better at everything in this consistent way"?
--:-- Dario Amodei: I genuinely don't know. At first when I saw it for speech, I assumed this was just true for speech or for this narrow class of models. I think it was just over the period between 2014 and 2017. I tried it for a lot of things and saw the same thing over and over again. I watched the same being true with Dota. I watched the same being true with robotics, which many people thought of as a counterexample. But I just thought, well, it's hard to get data for robotics. But if we look within the data that we have, we see the same patterns.
--:-- I think people were very focused on solving the problem in front of them. Why one person thinks one way, another person thinks another—it's very hard to explain. I think people just see it through a different lens. They're looking vertically instead of horizontally. They're not thinking about the scaling. They're thinking about how do I solve my problem? For robotics, there's not enough data. So that can easily abstract to, "Well, scaling doesn't work because we don't have the data." I don't know. For some reason, and it may just have been random chance, I was obsessed with that particular direction.
--:-- Dwarkesh Patel: When did it become obvious to you that language is the means to just feed a bunch of data into these things? Or was it just you ran out of other things—robotics, there's not enough data, this other thing, there's not enough data?
--:-- Dario Amodei: This whole idea of next word prediction, that you could do self-supervised learning, together with the idea that for predicting the next word, there's so much richness and structure there. It might say "two plus two equals" and you have to know the answer is four. It might be telling a story about a character. It's posing to the model the equivalent of developmental tests that get posed to children. "Mary walks into the room and puts an item in there, then Chuck walks into the room and removes the item and Mary doesn't see it. What does Mary think?" The models are going to have to get this right in the service of predicting the next word. They're going to have to solve all these theory of mind problems, solve all these math problems.
--:-- My thinking was just, scale it up as much as you can. There's kind of no limit to it. I think I had abstractly that view. But the thing that really solidified and convinced me was the work that Alec Radford did on GPT-1, which was not only could you get this language model that could predict things very well, but also you could fine tune it. You needed to fine tune it in those days to do all these other tasks. So I was like, wow, this isn't just some narrow thing where you get the language model. It's sort of halfway to everywhere. You get the language model, and then with a little move in this direction, it can solve this logical dereference test or whatever. With this other thing, it can solve translation or something. And then you're like, wow, I think there's really something to it. And we can really scale it.
--:-- Dwarkesh Patel: One thing that's confusing or that would have been hard to see—if you told me in 2018, "We'll have models in 2023, like Claude 2, that can write theorems in the style of Shakespeare, they can do standardized tests with open-ended questions," just all kinds of really impressive things. You would have said at that time, "Oh, you have AGI, you clearly have something that is human level intelligence." But while these things are impressive, it clearly seems we're not at human level, at least in the current generation and potentially for generations to come. What explains this discrepancy between super impressive performance in these benchmarks and in just the things you can describe?
--:-- Dario Amodei: That was one area where actually I was not prescient and I was surprised as well. When I first looked at GPT-3, and more so the kind of things that we built in the early days at Anthropic, my general sense was, it seems like they really grasped the essence of language. I'm not sure how much we need to scale them up. Maybe what's more needed from here is RL and all the other stuff. We might be kind of near the—I thought in 2020, we can scale this a bunch more, but I wonder if it's more efficient to scale it more or to start adding on these other objectives, like RL. I thought maybe if you do as much RL as you've done pre-training for a 2020 style model, that's the way to go and scaling it up will keep working. But is that really the best path? It just keeps going. I thought it had understood a lot of the essence of language, but then there's kind of further to go.
--:-- Stepping back from it, one of the reasons why I'm very empiricist about AI, about safety, about organizations, is that you often get surprised. I feel like I've been right about some things, but I've still, with these theoretical pictures ahead, been wrong about most things. Being right about 10% of the stuff sets you head and shoulders above many people.
--:-- If you look back to—I can't remember who it was—made these diagrams that are like, here's the village idiot, here's Einstein, here's the scale of intelligence. And the village idiot and Einstein are very close to each other. Maybe that's still true in some abstract sense, but it's not really what we're seeing. It seems like the human range is pretty broad and we don't hit the human range in the same place or at the same time for different tasks. Like, write a sonnet in the style of Cormac McCarthy—I'm not very creative, so I couldn't do that. But that's a pretty high level human skill. And even the model is starting to get good at constrained writing. Write a page without using the letter E. I think the models might be superhuman or close to superhuman at that.
--:-- But when it comes to proving relatively simple mathematical theorems, they're just starting to do the beginning of it. They make really dumb mistakes sometimes. And they really lack any kind of broad correcting your errors or doing some extended task. It turns out that intelligence isn't a spectrum. There are a bunch of different areas of domain expertise. There are a bunch of different kinds of skills, memories, different. It's all formed in the blob. It's not complicated, but to the extent it even is on a spectrum, the spectrum is also wide. If you asked me 10 years ago, that's not what I would have expected at all. But I think that's very much the way it's turned out.
--:-- Dwarkesh Patel: Do you expect that given the distribution of training that these models get from massive amounts of internet data versus what humans got from evolution, that the repertoire of skills that elicits will be just barely overlapping, will be like concentric circles? How do you think about that? Does that matter?
--:-- Dario Amodei: Clearly, there's a large amount of overlap. A lot of the things—these models have business applications, and many of their business applications are doing things that are helping humans to be more effective at things. So the overlap is quite large. If you think of all the activity that humans put on the internet in text, that covers a lot of it, but it probably doesn't cover some things. The models, I think they do learn a physical model of the world to some extent, but they certainly don't learn how to actually move around in the world. Maybe that's easy to fine tune. So I think there are some things that the models don't learn that humans do. And then the models learn, for example, to speak fluent base 64. I don't know about you, but I never learned that.
--:-- Dwarkesh Patel: How likely do you think it is that these models will be superhuman for many years at economically valuable tasks while they are still below humans in many other relevant tasks that prevents an intelligence explosion or something?
--:-- Dario Amodei: This kind of stuff is really hard to know. I'll give that caveat that the basic scaling laws you can kind of predict, and then this more granular stuff, which we really want to know to know how this all is going to go, is much harder to know. But my guess would be, the scaling laws are going to continue, subject to do people slow down for safety or for regulatory reasons. Let's put all that aside and say we have the economic capability to keep scaling. If we did that, what would happen?
--:-- My view is we're going to keep getting better across the board. I don't see any area where the models are super, super weak or not starting to make progress. That used to be true of math and programming. But over the last six months, the 2023 generation of models compared to the 2022 generation has started to learn that. There may be more subtle things we don't know. So I suspect, even if it isn't quite even, that the rising tide will lift all the boats.
--:-- Dwarkesh Patel: Does that include the thing you were mentioning earlier, where if there's an extended task, it kind of loses its train of thought or its ability to execute a series of steps?
--:-- Dario Amodei: I think that's going to depend on things like RL training to have the model do longer horizon tasks. I don't expect that to require a substantial amount of additional compute. I think that was probably an artifact of thinking about RL in the wrong way and underestimating how much the model had learned on its own.
--:-- In terms of are we going to be superhuman in some areas and not others? I think it's complicated. I could imagine that we won't be superhuman in some areas because, for example, they involve embodiment in the physical world. And then it's like, what happens? Do the AIs help us train faster AIs and those faster AIs wrap around and solve that? Do you not need the physical world? It depends what you mean. Are we worried about an alignment disaster? Are we worried about misuse, like making weapons of mass destruction? Are we worried about the AI taking over research from humans? Are we worried about it reaching some threshold of economic productivity where it can do what the average—these different thresholds have different answers. Although I suspect they will all come within a few years.
--:-- Dwarkesh Patel: Let me ask about those thresholds. If Claude was an employee at Anthropic, what salary would it be worth? What is it like meaningfully speeding up AI progress?
--:-- Dario Amodei: It feels to me like an intern in most areas. But then some specific areas where it's better than that. One thing that makes the comparison hard is the form factor is not the same as a human. If you were to behave like one of these chatbots, we wouldn't really—I mean, we could have this conversation. But they're more designed to answer single or a few questions. They don't have the concept of having a long life of prior experience. We're talking here about things that I've experienced in the past. Chatbots don't have that. So there are all kinds of stuff missing. It's hard to make a comparison.
--:-- They feel like interns in some areas and then they have areas where they spike and are really savants where they may be better than anyone here.
--:-- Dwarkesh Patel: Does the overall picture of something like an intelligence explosion—my former guest is Carl Schulman and he has this very detailed model of an intelligence explosion. As somebody who would actually like see that happening, does that make sense to you as they go from interns to entry-level software engineers, those entry-level software engineers increase your productivity?
--:-- Dwarkesh Patel: I think the idea that the AI systems become more productive, and first they speed up the productivity of humans, then they kind of equal the productivity of humans, and then they're in some meaningful sense the main contributor to scientific progress—that that happens at some point. I think that basic logic seems likely to me, although I have a suspicion that when we actually go into the details, it's going to be kind of weird and different than we expect. All the detailed models are kind of... we're thinking about the wrong things, or we're right about one thing and then wrong about ten other things. So I don't know, I think we might end up in a weirder world than we expect.
--:-- When you add all this together, what does your estimate look like for when we get something kind of human level?
--:-- Dario Amodei: Yeah, again, it depends on the thresholds. In terms of someone who looks at these models, and even if you talk to it for an hour or so, it's basically like a generally well-educated human. That could be not very far away at all. I think that could happen in two or three years. If I look at it again, I think the main thing that would stop it would be if we hit certain safety thresholds and stuff like that. So if a company or the industry decides to slow down, or we're able to get the government to institute restrictions that moderate the rate of progress for safety reasons, that would be the main reason it wouldn't happen. But if you just look at the logistical and economic ability to scale, I don't think we're very far at all from that.
--:-- Now, that may not be the threshold where the models are existentially dangerous. In fact, I suspect it's not quite there yet. It may not be the threshold where the models can take over most AI research. It may not be the threshold where the models seriously change how the economy works. I think it gets a little murky after that, and all those thresholds may happen at various times after that. But in terms of the base technical capability, it kind of sounds like a reasonably generally educated human across the board. I think that could be quite close.
--:-- Dwarkesh Patel: Why would it be the case that it could sound like it could pass a Turing test for an educated person but not be able to contribute or substitute for human involvement in the economy?
--:-- Dario Amodei: A couple reasons. One is just that the threshold of skill isn't high enough, right? Comparative advantage. It doesn't matter that I have someone who's better than the average human at every task. What I really need for AI research is to basically find something that is strong enough to substantially accelerate the labor of the thousand experts who are best at it. So we might reach a point where the comparative advantage of these systems is not great.
--:-- Another thing that could be the case is that there are these kind of mysterious frictions that don't show up in naive economic models. But you see it whenever you go to a customer and you're like, "Hey, I have this cool chatbot. In principle, it can do everything that your customer service bot does or that this part of your company does." But the actual friction of how do we slot it in? How do we make it work? That includes both the question of how it works in a human sense within the company, and also just what is the workflow? How do you actually interact with it? It's very different to say, "Here's a chatbot that kind of looks like it's doing this task," as it is to say, "Okay, this thing is deployed and 100,000 people are using it."
--:-- Right now, lots of folks are rushing to deploy these systems. But I think in many cases, they're not using them in anywhere close to the most efficient way they could, not because they're not smart, but because it takes time to work these things out. So when things are changing this fast, there are going to be all of these frictions. These are messy realities that don't quite get captured in the model. I don't think it changes the basic picture. I don't think it changes the idea that we're building up this snowball of the models helping the models get better, accelerating what the humans do, and eventually it's mostly the models doing the work. If you zoom out far enough, that's happening. But I'm kind of skeptical of any kind of precise mathematical or exponential prediction of how it's going to be. I think it's all going to be a mess. But what we know is it's not a metaphorical exponential, and it's going to happen fast.
--:-- Dwarkesh Patel: How do those different exponentials net out, which we've been talking about? One was that the scaling laws themselves are power laws with decaying marginal loss per parameter or something. The other exponential you talked about is that these things can get involved in the process of AI research itself, speeding it up. So those two are sort of opposing exponentials. Does it net out to be super linear or sublinear? And also you mentioned that the distribution of intelligence might just be broader. So should we expect after we get to this point in two to three years, it's like boom, boom? What does that look like?
--:-- Dario Amodei: I think it's very unclear. We're already at the point where if you look at the loss, the scaling laws are starting to bend. We've seen that in published model cards offered by multiple companies, so that's not a secret at all. But as they start to bend, each little bit of entropy of accurate prediction becomes more important. Maybe these last little bits of entropy are like, "This is a physics paper as Einstein would have written it," as opposed to as some other physicist would have written it. So it's hard to assess significance from this. It certainly looks like in terms of practical performance, the metrics keep going up relatively linearly, although they were always unpredictable.
--:-- The thing that I think is driving the most acceleration is just more and more money going into the field. People are seeing that there's a huge amount of economic value. So I expect the amount of money spent on the largest models to go up by a factor of 100 or something. And then that concatenates with the chips getting faster, the algorithms getting better, because there are so many people working on this now. I'm not making a normative statement here. I'm not even saying this necessarily will happen because I think there are important safety and government questions here. I'm just saying, left to itself, this is what the economy is going to do.
--:-- Dwarkesh Patel: We'll get to those questions in a second. But how do you think about the contribution of Anthropic to that increasing scope of this industry? There's an argument that with that investment, we can work on safety stuff at Anthropic. Another says you're raising the salience of this field in general.
--:-- Dario Amodei: It's all costs and benefits, right? The costs are not zero. I think a mature way to think about these things is not to deny that there are any costs, but to think about what the costs are and what the benefits are. I think we've been relatively responsible in the sense that the big acceleration that happened late last year and beginning of this year—we didn't cause that. We weren't the ones who did that. Honestly, if you look at the reaction of Google, that might be ten times more important than anything else. Once it had happened, once the ecosystem had changed, then we did a lot of things to kind of stay on the frontier. It's like any other question. You're trying to do the things that have the biggest costs and the lowest costs and the biggest benefits. That causes you to have different strategies at different times.
--:-- Dwarkesh Patel: One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized? And as far as I'm aware, they haven't been able to make a single new connection that has led to a discovery. Whereas if even a moderately intelligent person had this much stuff memorized, they'd notice, "Oh, this thing causes this symptom. This other thing also causes this symptom. There's a medical cure right here." Shouldn't we be expecting that kind of stuff?
--:-- Dario Amodei: I'm not sure. I don't know these words—discovery, creativity. It's one of the lessons I've learned is that in the big blob of compute, often these ideas end up being kind of fuzzy and elusive and hard to track down. But I think there is something here, which is that the models do display a kind of ordinary creativity. Again, the kind of "write a sonnet in the style of Cormac McCarthy or Barbie"—there is some creativity to that. And I think they do draw new connections of the kind that an ordinary person would draw.
--:-- I agree with you that there haven't been any kind of big scientific discoveries. I think that's a mix of the model skill level not being high enough yet. I was on a podcast last week where the host said, "I play with these models. They're kind of mid. They get a B or a B minus or something." I think that's going to change with the scaling. There is an interesting point about the models having an advantage, which is they know a lot more than us. Should they have an advantage already, even if their skill level isn't quite high? Maybe that's what you're getting at. I don't really have an answer to that. It seems like memorization and facts and drawing connections is an area where the models are ahead. I do think you need those connections and a fairly high level of skill.
--:-- I do think, particularly in the area of biology, for better and for worse, the complexity of biology is such that the current models know a lot of things right now. That's what you need to make discoveries. It's not like physics where you need to think and come up with a formula. In biology, you need to know a lot of things. The models know a lot of things, and they have a skill level that's not quite high enough to put them together. I think they are just on the cusp of being able to put these things together.
--:-- Dwarkesh Patel: On that point, last week in your Senate testimony, you said that these models are two to three years away from potentially enabling large-scale bioterrorism attacks or something like that. Can you make that more concrete without obviously giving the kind of information that would? Is it like one-shotting how to weaponize something? Or do you have to fine-tune an open source model? What would that actually look like?
--:-- Dario Amodei: I think it'd be good to clarify this because we did a blog post and the Senate testimony, and I think various people didn't understand the point or didn't understand what we'd done. Today, of course in our models we try and prevent this, but there are always jailbreaks. You can ask the models all kinds of things about biology and get them to say all kinds of scary things. But often those scary things are things that you could Google. I'm therefore not particularly worried about that. I think it's actually an impediment to seeing the real danger where someone just says, "Oh, I asked this model to tell me some things about smallpox, and it will." That is actually not what I'm worried about.
--:-- We spent about six months working with some of the folks who are the most expert in the world on how biological attacks happen, what you would need to conduct such an attack, and how we defend against such an attack. They worked very intensively on the entire workflow of if I were trying to do a bad thing. It's not one shot. It's a long process. There are many steps to it. It's not just "I asked the model for this one page of information." Without going into any detail, the thing I said in the Senate testimony is that there are some steps where you can just get information on Google. There are some steps that are what I'd call missing. They're scattered across a bunch of textbooks, or they're not in any textbook. They're kind of implicit knowledge. They're more like, "I have to do this lab protocol. What if I get it wrong? Oh, if this happens, then my temperature was too low. If that happened, I needed to add more of this particular reagent."
--:-- What we found is that for the most part, those key missing pieces, the models can't do them yet. But we found that sometimes they can. And when they can, sometimes they still hallucinate, which is the thing that's kind of keeping us safe. But we saw enough signs of the models doing those key things well. And if we look at state-of-the-art models and go backwards to previous models, we look at the trend, it shows every sign of two or three years from now, we're going to have a real problem.
--:-- Dwarkesh Patel: Especially the thing you mentioned on the log scale, you go from like one in a hundred times it gets it right to one in ten to...
--:-- Dario Amodei: Exactly. I've seen many of these groks in my life. I was there when I watched GPT-3 learn to do arithmetic, when GPT-2 learned to do regression a little bit above chance, when we got with Claude and we got better on all these tests of helpful, honest, harmless. I've seen a lot of groks. This is unfortunately not one that I'm excited about, but I believe it's happening.
--:-- Dwarkesh Patel: Somebody might say, "Listen, you were a co-author on this post that OpenAI released about GPT-2 where they said we're not going to release the weights or the details here because we're worried that this model will be used for something bad." Looking back on it, now it's laughable to think that GPT-2 could have done anything bad. Are we just way too worried? Is this a concern that doesn't make sense?
--:-- Dario Amodei: It might be worth looking back at the actual text of that post. I don't remember it exactly, but it's still up on the internet. It says something like, "We're choosing not to release the weights because of concerns about misuse." But it also said, "This is an experiment. We're not sure if this is necessary or the right thing to do at this time. But we'd like to establish a norm of thinking carefully about these things." You could think of it a little like the Asilomar Conference in the 1970s, where they were just figuring out recombinant DNA. It was not necessarily the case that someone could do something really bad with recombinant DNA. It's just the possibilities were starting to become clear. Those words, at least, were the right attitude.
--:-- Now, I think there's a separate thing that people don't just judge the post, they judge the organization. Is this an organization that produces a lot of hype or that has credibility? I think that had some effect on it. You could also ask, is it inevitable that people would just interpret it as "you can't get across any message more complicated than this thing right here is dangerous"? You can argue about those. But the basic thing that was in my head and the head of others who were involved in that, and what is evident in the post, is that we actually don't know. We have pretty wide error bars on what's dangerous and what's not. So we should want to establish a norm of being careful.
--:-- By the way, we have enormously more evidence now. We've seen enormously more of these groks. So we're well calibrated, but there's still uncertainty. In all these statements, I said, "In two or three years, we might be there." There's a substantial risk of it. We don't want to take that risk. But I wouldn't say it's 100%. It could be 50-50.
--:-- Dwarkesh Patel: Okay, let's talk about cybersecurity, which in addition to bio-risk is another thing Anthropic has been emphasizing. How have you avoided the cloud micro-architecture from leaking? As you know, your competitors have been less successful at this kind of security.
--:-- Dario Amodei: Can't comment on anyone else's security. Don't know what's going on in there. A thing that we have done is that there are these architectural innovations that make training more efficient. We call them compute multipliers because they're the equivalent of improving compute. Our compute multipliers—again, I don't want to say too much about it because it could allow an adversary to counteract our measures—but we limit the number of people who are aware of a given compute multiplier to those who need to know about it. So there's a very small number of people who could leak all of these secrets. There's a larger number of people who could leak one of them. This is the standard compartmentalization strategy used in the intelligence community or resistance cells. Over the last few months, we've implemented these measures. I don't want to jinx anything by saying this could never happen to us, but I think it would be harder for it to happen. I'd encourage all the other companies to do this as well. As much as competitors' architectures leaking is narrowly helpful to Anthropic, it's not good for anyone in the long run. Security around this stuff is really important.
--:-- Dwarkesh Patel: Even with all the security you have, could you, with your current security, prevent a dedicated state-level actor from getting the Claude weights?
--:-- Dario Amodei: It depends how dedicated. Our head of security, who used to work on security for Chrome, likes to think about it in terms of how much would it cost to attack Anthropic successfully. One of our goals is that it costs more to attack Anthropic than it costs to just train your own model. That doesn't guarantee things because you need the talent as well. Attacks have risks, diplomatic costs, and they use up the very sparse resources that nation-state actors might have. We're not there yet, but I think we're at a very high standard compared to the size of company that we are. If you look at security for most 150-person companies, there's just no comparison.
--:-- Dwarkesh Patel: Could we resist if it was a state actor's top priority to steal our model weights?
--:-- Dario Amodei: No, they would succeed.
--:-- Dwarkesh Patel: How long does that stay true? Because at some point, the value keeps increasing and increasing. Another part of this question is, what kind of a secret is how to train Claude Three or Claude Two? Is it like nuclear weapons, where you just take a blueprint across and that's the implosion device? Or is it more tacit, like the thing you were talking about with biology, where you need to know how these reagents work?
--:-- Dario Amodei: There are some things that are like a one-line equation, and there are other things that are more complicated. Compartmentalization is the best way to do it. Just limit the number of people who know about something. If you're a thousand-person company and everyone knows every secret, one, I guarantee you have a leaker. And two, I guarantee you have a spy—a literal spy.
--:-- Dwarkesh Patel: Let's talk about alignment. And let's talk about mechanistic interpretability, which is the branch you guys specialize in. While you're answering this question, you might want to explain what mechanistic interpretability is. But the broader question is, mechanistically, what is alignment? Is it that you're locking in the model into a benevolent character? Are you disabling deceptive circuits and procedures? What concretely is happening when you align a model?
--:-- Dario Amodei: As with most things, when we actually train a model to be aligned, we don't know what happens inside the model. There are different ways of training it to be aligned, but I think we don't really know what happens. For some of the current methods, all the current methods that involve some kind of fine-tuning have the property that the underlying knowledge and abilities that we might be worried about don't disappear. It's just that the model is taught not to output them. I don't know if that's a fatal flaw or if that's just the way things have to be. I don't know what's going on inside mechanistically. That's the whole point of mechanistic interpretability—to really understand what's going on inside the models at the level of individual circuits.
--:-- Dwarkesh Patel: Eventually, when it's solved, what does the solution look like? What is the case where if you're Claude Four, you do the mechanistic interpretability thing and you're like, "I'm satisfied. It's aligned." What is it that you've seen?
--:-- Dario Amodei: I think we don't know that yet. I think we don't know enough to know that yet. I can give you a sketch for what the process looks like as opposed to what the final result looks like. Verifiability is a lot of the challenge here. We have all these methods that purport to align AI systems and do succeed at doing so for today's tasks. But the question is always if you had a more powerful model or if you had a model in a different situation, would it be aligned? This problem would be much easier if you had an oracle that could just scan a model and say, "I know this model is aligned. I know what it'll do in every situation." The closest thing we have to that is something like mechanistic interpretability. It's not anywhere near up to the task yet.
--:-- I think of it as almost like an extended training set and an extended test set. Everything we're doing, all the alignment methods we're doing, are the training set. You can run tests on them, but will it really work out of distribution? Will it really work in another situation? Mechanistic interpretability is the only thing that even in principle—and we're nowhere near there yet—but even in principle is the thing where it's more like an x-ray of the model than a modification of the model. It's more like an assessment than an intervention. Somehow we need to get into a dynamic where we have an extended training set, which is all these alignment methods, and an extended test set, which is kind of like you x-ray the model and say, "What works and what didn't?" In a way that goes beyond just the empirical tests you've run. You're saying, "What is the model going to do in these situations? What is it within its capabilities to do?" instead of "What did it do phenomenologically?"
--:-- We have to be careful about that. One of the things I think is very important is we should never train for interpretability because I think that's taking away that advantage. You even have the problem similar to validation versus test set where if you look at the x-ray too many times, you can interfere. But I think that's a much weaker concern. It's not automated optimization. We should just make sure, as with validation and test sets, that we don't look at the validation set too many times before running the test set. That's manual pressure rather than automated pressure. Some solution where we have some dynamic between the training and test set, where we're trying things out and we really figure out if they work by a way of testing them that the model isn't optimizing against—some orthogonal way. I think we're never going to have a guarantee, but some process where we put extended training for alignment ability with extended testing for alignment ability together in a way that actually works.
--:-- Dwarkesh Patel: I still don't feel like I understand the intuition for why you think this is likely to work or this is a promising path to pursue. Let me ask the question in a more specific way, and excuse the tortured analogy. If you're an economist and you want to understand the economy, you send a whole bunch of microeconomists out there. One of them studies how the restaurant business works. One of them studies how the tourism business works. One of them studies how banking works. At the end, they all come together and you still don't know if there's going to be a recession in five years or not. Why is this not like that? You have an understanding of how induction heads work in a two-layer transformer. You understand modular arithmetic. How does this add up to "Does this model want to kill us?" What does this model fundamentally want?
--:-- Dario Amodei: I think that's the right set of questions to ask. What we're hoping for in the end is not that we'll understand every detail. But again, I would give the x-ray or MRI analogy. We can be in a position where we can look at the broad features of the model and say, "Is this a model whose internal state and plans are very different from what it externally represents itself to do?" Is this a model where we're uncomfortable that far too much of its computational power is devoted to doing what look like fairly destructive and manipulative things? We don't know for sure whether that's possible, but I think there are at least positive signs that it might be possible. Again, the model is not intentionally hiding from you.
--:-- Dwarkesh Patel: It might turn out that the training process hides it from you. And I can think of cases where the model is really super intelligent. It thinks in a way that affects its own cognition. I suspect we should think about that. We should consider everything. I suspect that it may roughly work to think of the model as, if it's trained in the normal way, just at just above human level, it may be a reasonable assumption that the internal structure of the model is not intentionally optimizing against us.
--:-- Dario Amodei: I give an analogy to humans. It's actually possible to look at an MRI of someone and predict above random chance whether they're a psychopath. There was actually a story a few years back about a neuroscientist who was studying this and he looked at his own scan and discovered that he was a psychopath. Then everyone in his life was like, "No, that's just obvious. You're a complete asshole. You must be a psychopath." And he was totally unaware of this.
--:-- The basic idea is that there can be these macro features. Psychopath is probably a good analogy for it. A model that's charming on the surface, very goal oriented, and very dark on the inside. On the surface, their behavior might look like the behavior of someone else, but their goals are very different.
--:-- Dwarkesh Patel: A question somebody might have is, you mentioned earlier the importance of being empirical. In this case, you're trying to estimate, are these activations suspicious? But is this something we can afford to be empirical about? Or do we need a very good first principle theoretical reason to think it's not just that these MRIs of the model correlate with being bad? We need some deep root math proof that this is aligned.
--:-- Dario Amodei: It depends what you mean by empirical. A better term would be phenomenological. I don't think we should be purely phenomenological, like "here are some brain scans of really dangerous models." I think the whole idea of mechanistic interpretability is to look at the underlying principles and circuits.
--:-- I've always been a fan of studying these circuits at the lowest level of detail that we possibly can. That's how you build up knowledge. Even if you're ultimately aiming for something broad, and it's too complicated, the way you build that up is by trying to make a lot of these very specific discoveries. You have to understand the building blocks and then figure out how to use that to draw broad conclusions, even if you're not going to figure out everything.
--:-- You should probably talk to Chris Olah, who would have much more detail. He controls the interpretability agenda. This is my high-level thinking on it, which is not going to be as good as his.
--:-- Dwarkesh Patel: Does the bull case on Anthropic rely on the fact that mechanistic interpretability is helpful for capabilities?
--:-- Dario Amodei: I don't think so at all. In principle, it's possible that mechanistic interpretability could be helpful with capabilities. We might, for various reasons, not choose to talk about it if that were the case. That wasn't something that I thought of or that any of us thought of at the time of Anthropic's founding.
--:-- We thought of ourselves as people who are good at scaling models and good at doing safety on top of those models. We think that we have a very high talent density of folks who are good at that. My view has always been talent density beats talent mass. So that's more of our bull case. I don't think it depends on some particular thing. Others are starting to do mechanistic interpretability now, and I'm very glad that they are. That was a part of our theory of change, paradoxically, to make other organizations more like us.
--:-- Dwarkesh Patel: Talent density is important. But another thing Anthropic has emphasized is that you need to have frontier models in order to do safety research. And to actually be a company. The current frontier models, someone might guess GPT-4 cost like $100 million or something. That general order of magnitude is not wrong. But two to three years from now, the kinds of things you're talking about require more and more orders of magnitude to keep up. If safety requires being on the frontier, what is the case in which Anthropic is competing with these leviathans to stay at that same scale?
--:-- Dario Amodei: It's a situation with a lot of trade-offs. It's not easy. Let me answer the questions one by one.
--:-- Why is safety so tied to scale? Some people don't think it is. But if I look at where safety methods have been put into practice or worked for anything, even if we don't think they'll work in general, I go back to ideas like debate and amplification. Back in 2018, when we wrote papers about those at OpenAI, I thought human feedback isn't quite going to work, but debate and amplification will take us beyond that. But when we actually did attempts to do debates, we were really limited by the quality of the model. For two models to have a debate that is coherent enough that a human can judge it so that the training process can actually work, you need models that are at or maybe even beyond on some topics the current frontier.
--:-- You can come up with the idea without being on the frontier. But for me, that's a very small fraction of what needs to be done. It's very easy to come up with these methods. It's very easy to say, "Oh, the problem is X, maybe a solution is Y." But I really want to know whether things work in practice, even for the systems we have today. I want to know what kinds of things go wrong with them. You discover 10 new ideas and 10 new ways that things are going to go wrong by trying these in practice. That empirical learning is just not as widely understood as it should be.
--:-- I would say the same thing about methods like constitutional AI. Some people say, "Oh, it doesn't matter. We know this method won't work for pure alignment." I neither agree nor disagree with that. I think that's just kind of overconfident. The way we discover new things and understand the structure of what's going to work and what's not is by playing around with things. Not that we should blindly say, "Oh, this worked here, so it'll work there." But you really start to understand the patterns, like with the scaling laws.
--:-- Even mechanistic interpretability, which might be the one area where a lot of progress has been made without the frontier models. We're seeing in the work that OpenAI put out a couple months ago that using very powerful models to help you auto interpret the weak models. That's not everything you can do in interpretability, but it's a big component of it. We found it useful too. You see this phenomenon over and over again, where the scaling and the safety are these two snakes that are coiled with each other, always even more than you think.
--:-- With interpretability, three years ago I didn't think that this would be as true, but somehow it manages to be true. Why? Because intelligence is useful. It's useful for a number of tasks. One of the tasks it's useful for is figuring out how to judge and evaluate other intelligence. And maybe someday even for doing the alignment research itself.
--:-- Dwarkesh Patel: Given all that's true, what does that imply for Anthropic when in two to three years, these leviathans are doing $10 billion training runs?
--:-- Dario Amodei: One choice is if we can't stay on the frontier, or if it costs too much, then we shouldn't do it. We won't work with the most advanced models. We'll see what we can get with models that are not quite as advanced. I think you can get some value there, non-zero value, but I'm kind of skeptical that the value is all that high or the learning can be fast enough to really be in favor of the task.
--:-- The second option is you just find a way. You just accept the trade-offs. I think the trade-offs are more positive than they appear because of a phenomenon that I've called "race to the top." I could go into that later, but let me put that aside for now.
--:-- The third phenomenon is, as things get to that scale, I think this may coincide with starting to get into some non-trivial probability of very serious danger. I think it's going to come first from misuse, the kind of bio stuff that I talked about. I don't think we have the level of autonomy yet to worry about some of the alignment stuff happening in like two years, but it might not be very far behind that at all. That may lead to unilateral or multilateral or government enforced decisions, which we support, not to scale as fast as we could. That may end up being the right thing to do. I kind of hope things go in that direction. Then we don't have this hard trade-off between not being on the frontier and not being able to do the research as well as we want, versus being on the frontier and having to accept the trade-offs which are net positive, but have a lot in both directions.
--:-- Dwarkesh Patel: On the misuse versus misalignment, those are both problems. But in the long scheme of things, what are you more concerned about? 30 years down the line, which do you think will be considered a bigger problem?
--:-- Dario Amodei: I think it's much less than 30 years. I'm worried about both. If you have a model that could in theory take over the world on its own, and you were able to control that model, then it follows pretty simply that if a model was following the wishes of some small subset of people and not others, then those people could use it to take over the world on their behalf. The very premise of misalignment means that we should be worried about misuse as well with similar levels of consequences.
--:-- Dwarkesh Patel: Some people who might be more doomer than you would say misuse is already working towards the optimistic scenario. You've at least figured out how to align the model with the bad guys. Now you just need to make sure it's aligned with the good guys instead. Why do you think you could get to the point where it's aligned with the bad guys if you haven't already solved it?
--:-- Dario Amodei: If you had the view that alignment is completely unsolvable, then you'd be like, "Well, we're dead anyway. So I don't want to worry about misuse." That's not my position at all. You should think in terms of what's a plan that would actually succeed that would make things good. Any plan that actually succeeds, regardless of how hard misalignment is to solve, is going to need to solve misuse as well as misalignment. It's going to solve the fact that as the AI models get better faster and faster, they're going to create a big problem around the balance of power between countries. They're going to create a big problem around whether it's possible for a single individual to do something bad that it's hard for everyone else to stop.
--:-- Any actual solution that leads to a good future needs to solve those problems as well. If your perspective is "we're screwed because we can't solve the first problem, so don't worry about problems two and three," that's not really a statement. They're in our path no matter what. In the scenario we succeed, we have to solve all of them. We should be planning for success, not for failure.
--:-- Dwarkesh Patel: If misuse doesn't happen and the right people have the superhuman models, what does that look like? Who are the right people? Who is actually controlling the model five years from now?
--:-- Dario Amodei: My view is that these things are powerful enough that it's going to involve substantial role or at least involvement of some kind of government or assembly of government bodies. There are very naive versions of this. I don't think we should just hand the model over to the UN or whoever happens to be in office at a given time. I could see that go poorly. But it's too powerful. There needs to be some kind of legitimate process for managing this technology, which includes the role of the people building it, includes the role of democratically elected authorities, includes the role of all the individuals who will be affected by it. At the end of the day, there needs to be some politically legitimate process.
--:-- Dwarkesh Patel: What does that look like if it's not the case that you just hand it to whoever the president is at the time? What is the body look like?
--:-- Dario Amodei: These are things it's really hard to know ahead of time. People love to propose these broad plans and say, "This is the way we should do it." I think the honest fact is that we're figuring this out as we go along. Anyone who says, "This is the body we should create modeled after this thing," I think we should try things and experiment with them with less powerful versions of the technology. We need to figure this out in time, but it's not the kind of thing you can know in advance.
--:-- Dwarkesh Patel: The Long-Term Benefit Trust, how would that interface with this body?
--:-- Dario Amodei: The Long-Term Benefit Trust is a much narrower thing. This is something that makes decisions for Anthropic. It's described in a recent Vox article, and we'll be saying more about it later this year. It's basically a body that over time gains the ability to appoint the majority of the board seats of Anthropic. It's a mixture of experts in AI alignment, national security, and philanthropy in general. But if control is handed to them of Anthropic, that doesn't imply that control of AGI itself is handed to them. That doesn't imply that Anthropic or any other entity should be the entity that makes decisions about AGI on behalf of humanity. I would think of those as different.
--:-- If Anthropic does play a broad role, then you'd want to widen that body to be a whole bunch of different people from around the world. Or maybe you construe this as very narrow, and there's some broad committee somewhere that manages all the AGIs of all the companies on behalf of anyone. I don't know. My view is you shouldn't be overly constructive and utopian. We're dealing with a new problem here. We need to start thinking now about what are the governmental bodies and structures that could deal with it.
--:-- Dwarkesh Patel: Let's forget about governance. Let's just talk about what this going well looks like. Obviously there are the things we can all agree on: cure all the diseases, solve all the problems. Things that all humans would say, "I'm down for that." But now it's 2030, you've solved all the real problems that everybody can agree on. What happens next? What are we doing with a superhuman god?
--:-- Dario Amodei: I actually want to disagree with the framing. I get nervous when someone says, "What are you going to do with the superhuman AI?" We've learned a lot of things over the last 150 years about markets and democracy and each person can define for themselves what the best way for them to have the human experience is. Societies work out norms and what they value in this complex and decentralized way.
--:-- If you have these safety problems, that can be a reason why, especially from the government, there needs to be a certain amount of centralized control until we've solved these problems. But as a matter of "we've solved all the problems, now how do we make things good?" Most people, most groups, most ideologies that started with "let's sit down and think over what the definition of the good life is" have led to disaster.
--:-- Dwarkesh Patel: So this vision you have of a tolerant, liberal democracy, market-oriented system with AGI. What does that mean? Each person has their own AGI?
--:-- Dario Amodei: I don't know what it looks like. What I'm saying is we need to solve the important safety problems and the important externalities. Subject to that, which could be narrowly about alignment, there could be a bunch of economic issues that are super complicated and that we can't solve. Subject to that, we should think about what's worked in the past. Unitary visions for what it means to live a good life have not worked out well at all.
--:-- Dwarkesh Patel: On the opposite end of things going well or good actors having control of AI, we might want to touch on China as a potential actor in the space. Being at Baidu and seeing progress in AI happening generally, why do you think the Chinese have underperformed? Baidu had a scaling laws group many years back. Or is the premise wrong and I'm just not aware of the progress that's happening there?
--:-- Dario Amodei: For the scaling laws group, that was an offshoot of the stuff we did with speech. There were still some people there, but that was a mostly Americanized lab. I was there for a year. That was my first foray into deep learning. It was led by Andrew Ng. I never went to China. There's a U.S. lab. So I think that was somewhat disconnected, although it was an attempt by a Chinese entity to get into the game.
--:-- Since then, I couldn't speculate, but I think they've been very commercially focused and not as focused on the fundamental research side of things around scaling laws. Because of all the excitement with the release of ChatGPT in November, that's been a starting gun for them as well. They're trying very aggressively to catch up now. The US is quite substantially ahead, but I think they're trying very hard to catch up now.
--:-- Dwarkesh Patel: How do you think China thinks about AGI? Are they thinking about safety and misuse or not?
--:-- Dario Amodei: I don't really have a sense. One concern I would have is people say things like, "China isn't going to develop an AI because they like stability, or they're going to have all these restrictions to make sure things are in line with what the CCP wants." That might be true in the short term and for consumer products. My worry is that if the basic incentives are about national security and power, that's going to become clear sooner or later. If they see this as a source of national power, they're going to at least try to do what's most effective, and that could lead them in the direction of AGI.
--:-- Dwarkesh Patel: At what point is it possible for them to just get your blueprints or your code base or something that they can just spin up their own lab that is competitive at the frontier with the leading American companies?
--:-- Dario Amodei: I don't know about fast, but I'm concerned about this. This is one reason why we're focusing so hard on cybersecurity. We've worked with our cloud providers. We have this blog post out about security where we said we have a two-key system for access to the model weights. We have other measures that we've put in place or are thinking of putting in place that we haven't announced. We don't want an adversary to know about them, but we're happy to talk about them broadly.
--:-- All this stuff we're doing is not sufficient yet for a super determined state-level actor at all. I think it will defend against most attacks and against a state-level actor who is less determined. But there's a lot more we need to do. Some of it may require new research on how to do security.
--:-- Dwarkesh Patel: We're at Anthropic offices. We had to get badges and everything to come in here. But the eventual version of this building or a bunker or wherever the AGI is built, what does that look like? Is it a building in the middle of San Francisco, or are you out in the middle of Nevada or Arizona? At what point are you Los Alamosing it?
--:-- Dario Amodei: At one point, there was a running joke that the way building AGI would look like is a data center next to a nuclear power plant next to a bunker. We'd all kind of live in the bunker and everything would be local so it wouldn't get on the internet. If we take seriously the rate at which all this is going to happen, which I can't be sure of, then it does make me think that maybe not something quite as cartoonish as that, but something like that might happen.
--:-- Dwarkesh Patel: What is the timescale on which you think alignment is solvable? If these models are getting to human level in some things in two to three years, what is the point at which they're aligned?
--:-- Dario Amodei: This is a really difficult question. I think often people are thinking about alignment in the wrong way. There's a general feeling that models are misaligned, or there's an alignment problem to solve, kind of like the Riemann hypothesis. Someday we'll crack the Riemann hypothesis. I don't quite think it's like that, not in a way that's worse or better. It might be just as bad or just as unpredictable.
--:-- When I think of why I'm scared, a few things come to mind. One is that there will be powerful models. They will be agentic. We're getting towards them. If such a model wanted to wreak havoc and destroy humanity, I think we have basically no ability to stop it. That will continue to be true as we scale the models. That definitely seems the case.
--:-- A second thing that seems the case is that we seem to be bad at controlling the models. Not in any particular way. They're statistical systems. You can ask them a million things and they can say a million things in reply. You might not have thought of a millionth of one thing that does something crazy. Or when you train them, you train them in this very abstract way and you might not understand all the consequences of what they do in response to that.
--:-- The best example we've seen of that is Bing in Sydney. I don't know how they trained that model. I don't know what they did to make it do all this weird stuff like threaten people and have this weird, obsessive personality. But what it shows is that we can get something very different from and maybe opposite to what we intended.
--:-- I actually think fact number one and fact number two are enough to be really worried. You don't need all this detailed stuff about converging instrumental goals or analogies to evolution. One and two for me are pretty motivating. I'm like, "Okay, this thing is going to be powerful. It could destroy us. And all the ones we've built so far are at pretty decent risk of doing some random shit we don't understand."
--:-- Dwarkesh Patel: If I agree with that, and I'm concerned about this, the research agenda you have of mechanistic interpretability plus constitutional AI and the other RLHF stuff. If you say that we're going to get something with bioweapons that could be dangerous in two to three years, do these things culminate within two to three years of actually meaningfully contributing to preventing that?
--:-- Dario Amodei: People talk about doom by default or alignment by default. I think it might be kind of statistical. With the current models, you might get Bing or Sydney, or you might get Claude. It doesn't really matter because Bing or Sydney, if we take our current understanding and move that to very powerful models, you might just be in this world where depending on the details, maybe it's totally fine. Not really alignment by default, but just kind of like it depends on a lot of the details. If you're very careful about all those details and you know what you're doing, you're getting it right. But we have a high susceptibility to mess something up in a way that you didn't really understand was connected to, instead of making all the humans happy, it wants to turn them into pumpkins. Just some weird shit.
--:-- The models are so powerful. They're like these giants standing in the landscape. If they start to move their arms around randomly, they could just break everything. I'm starting with that framing because it's not like we're aligned by default. I don't think we're doomed by default and have some problem we need to solve. It has a different character.
--:-- What I do think is that hopefully within a timescale of two to three years, we get better at diagnosing when the models are good and when they're bad. We get better at training, increasing our repertoire of methods to train the model so they're less likely to do bad things and more likely to do good things in a way that isn't just relevant to the current models, but scales. We can help develop that with interpretability as the test set.
--:-- I don't think of it as, "Oh man, we tried RLHF, it didn't work. We tried constitutional, it didn't work. We tried this other thing, it didn't work. We tried mechanistic interpretability, now we're going to try mechanistic." I think this frame of "we haven't cracked the problem yet, we haven't solved the Riemann hypothesis" isn't quite right. I think of it more as already with today's systems, we are not very good at controlling them. The consequences of that could be very bad. We just need to get more ways of increasing the likelihood that we can control our models and understand what's going on in them. We have some of them so far, they aren't that good yet. But I don't think of this as binary of works and not works. We're going to develop more. I do think that over the next two to three years, we're going to start eating that probability mass of ways things can go wrong.
--:-- In the Core Safety Views paper, there's probability mass of how hard the problem is. I feel like that way of seeing it isn't really even quite right.
--:-- Dwarkesh Patel: Because I don't feel like it's the Riemann hypothesis to solve. I just feel like it's almost like right now, if I try and juggle five balls or something, I can juggle three balls, right? I actually can, but I can't juggle five balls at all. You have to practice a lot to do that. If I were to do that, I would mostly drive it, I would almost certainly drop them. And then just over time, you just get better at the task of controlling the balls.
--:-- On that post in particular, what is your personal probability distribution over the three possibilities? For the audience, the three possibilities are: it is trivial to align these models with RLHF++; it is a difficult problem but one that a big company could solve; or it is something that is basically impossible for human civilization currently to solve. If I'm capturing those three, what is your probability distribution over them personally?
--:-- Dario Amodei: Yeah, I mean, I'm not super into "what's your probability distribution of X?" I think all of those have enough likelihood that they should be considered seriously. I'm much more interested in the question: what could we learn that shifts probability mass between them? What is the answer to that?
--:-- I think that one of the things mechanistic interpretability is going to do, more than necessarily solve problems, is it's going to tell us what's going on when we try to align models. I think it's basically going to teach us about this. One way I can imagine concluding that things are very difficult is if mechanistic interpretability shows us that problems tend to get moved around instead of being stamped out, or that you get rid of one problem and you create another one. It might inspire us or give us insight into why problems are kind of persistent or hard to eradicate or crop up.
--:-- For me to really believe some of these stories about, "oh, something will always, there's always this convergent goal in this particular direction," I think the abstract story is not uncompelling, but I don't find it really compelling either. Nor do I find it necessary to motivate all the safety work. But the kind of thing that would really make me think, "oh man, we can't solve this," is like we see it happening inside the x-ray.
--:-- Dwarkesh Patel: Yeah, because I think right now there's just way too many assumptions. There's way too much overconfidence about how all this is going to go. I have a substantial probability mass on this all goes wrong, a complete disaster, but in a completely different way than anyone had anticipated. It would be beside the point to ask how it could go different than anyone anticipated.
--:-- So on this in particular, what information would be relevant? How much would the difficulty of aligning Claude 3 and the next generation of models be? Is that a big piece of information? Is it not?
--:-- Dario Amodei: So I think the people who are most worried are predicting that all the subhuman AI models are going to be alignable. They're going to seem aligned. They're going to deceive us in some way. I think it certainly gives us some information, but I am more interested in what mechanistic interpretability can tell us. Because again, you see this x-ray. It would be too strong to say it doesn't lie, but at least in the current systems, it doesn't feel like it's optimizing against us. There are exotic ways that it could. I don't think anything is a safe bet here, but I think it's the closest we're going to get to something that isn't actively optimizing against us.
--:-- Dwarkesh Patel: Let's talk about the specific methods other than mechanistic interpretability that you guys are researching. When we talk about RLHF or Constitution AI, whatever, RLHF++, if you had to put it in terms of human psychology, what is the change that is happening? Are we creating new drives, new goals, new thoughts? How is the model changing in terms of psychology?
--:-- Dario Amodei: I think all those terms are kind of inadequate for describing what's going on. It's not clear how useful they are as abstractions for humans either. I think we don't have the language to describe what's going on. And again, I'd love to have the x-ray. I'd love to look inside and actually know what we're talking about instead of basically making up words, which is what I do and what you're doing by asking this question. We should just be honest. We really have very little idea what we're talking about.
--:-- So, it would be great to say, "well, what we actually mean by that is this circuit within here turns on, and after we've trained the model, this circuit is no longer operative or weaker." We'd love to be able to say that, but it's going to take a lot of work to be able to do that.
--:-- Dwarkesh Patel: Model organisms, which you hinted at before, when you said we're doing these evaluations to see if they're capable of doing dangerous things now and currently not. How worried are you about a lab leak scenario where in fine-tuning it, or in trying to get these models to elicit dangerous behaviors like make bioweapons, you somehow leak and actually makes the bioweapon instead of telling you it can make the bioweapon?
--:-- Dario Amodei: With today's passive models, I think it's not that much of a concern. Chatbots, it's not so much of a concern, right? Because if we were to fine-tune a model and do that, we do it privately and work with the experts. The leak would be if the model got open-sourced or something and then someone used it. So for now, it's mostly a security issue.
--:-- In terms of models truly being dangerous, I think we do have to worry that if we make a truly powerful model and we're trying to see what makes it dangerous or safe, then there could be more of a one-shot thing where there's some risk that the model takes over. I think the main way to control that is to make sure that the capabilities of the model that we test are not such that they're capable of doing this.
--:-- Dwarkesh Patel: At what point would the capabilities be so high where you say, "I don't even want to test this"?
--:-- Dario Amodei: Oh, well, there are different things. There's capability testing, but that itself could lead to problems if you're testing and it actually does it. I think what you want to do is extrapolate. We've talked with ARC about this. You have factors of two of compute or something where you ask, "can the model do something like open up an account on AWS and make some money for itself?" Some of the things that are obvious prerequisites to complete survival in the wild. So just set those thresholds very well below. And then as you proceed upward from there, do more and more rigorous tests and be more and more careful about what it is you're doing.
--:-- Dwarkesh Patel: On Constitution AI, and feel free to explain what this is for the audience, but who decides what the constitution for the next generation of models or potentially superhuman model is? How is that actually written?
--:-- Dario Amodei: I think initially, to make the constitution, we just took some stuff that was broadly agreed on, like the UN Declaration on Human Rights and some of the stuff from Apple's terms of service. Stuff that's consensus on what's acceptable to say or what basic things are able to be included.
--:-- For future constitutions, we're looking into more participatory processes for making these. But beyond that, I don't think there should be one constitution for a model that everyone uses. Probably a model's constitution should be very, very simple. It should only have very basic facts that everyone would agree on. And then there should be a lot of ways that you can customize, including appending constitutions.
--:-- Beyond that, we're developing new methods. I'm not imagining that this alone is the method that we'll use to train superhuman AI. Many of the parts of capability training may be different. It could look very different. And again, I'd go there. There are levels above this. I'm pretty uncomfortable with "here's the AI's constitution, it's going to run the world." That just strikes me as fanciful. I think we should try to hook these things into something more decentralized, even when they are very powerful. After we mitigate the safety issues, any good future needs to end with something more decentralized and less like a godlike super-being. I just don't think that ends well.
--:-- Dwarkesh Patel: What scientists from the Manhattan Project do you respect most in terms of they acted most ethically under the constraints they were given? Is there one that comes to mind?
--:-- Dario Amodei: I don't know. There's a lot of answers you could give. I'm definitely a fan of Szilard for having kind of figured it out. He was then against the actual dropping of the bomb. I don't actually know the history well enough to have an opinion on whether a demonstration of the bomb could have ended the war. That involves a bunch of facts about Imperial Japan that are complicated and that I'm not an expert on.
--:-- But Szilard seemed to display the right kind of awareness, as well as discovering stuff. When I read that book, it kind of inspired me. When I wrote the "big blob of compute" doc and many others, I only showed it to a few people, and there were other docs that I showed to almost no one. So I was a bit inspired by that.
--:-- Again, we can all get self-aggrandizing here. We don't know how it's going to turn out, or if it's actually going to be something on par with the Manhattan Project. This could all be just Silicon Valley people building technology and having delusions of grandeur. I don't know how it's going to turn out. If the scaling stuff is true, then it's bigger than the Manhattan Project.
--:-- Dwarkesh Patel: Yeah, it certainly could be bigger.
--:-- Dario Amodei: We should always maintain this attitude that it's really easy to fool yourself.
--:-- Dwarkesh Patel: If you were asked by the government, if you're a physicist during World War Two, and you were asked by the government to contribute non-replaceable research to the Manhattan Project, what do you think you would have said?
--:-- Dario Amodei: Yeah, I mean, given you're in a war with the Nazis, at least during the period when you thought that the Nazis were working on it, I don't really see much choice but to do it if it's possible. You have to figure it's going to be done within 10 years or so by someone.
--:-- Dwarkesh Patel: Regarding cybersecurity, what should we make of the fact that there's a whole bunch of tech companies which have ordinary tech company security policy that publicly seems fine. It's not obvious that they've been hacked. Coinbase still has its Bitcoin. Google, as far as I know, my Gmail hasn't been leaked. Should we take from that that current status quo tech company security practices are good enough for AGI, or just that nobody has tried hard enough?
--:-- Dario Amodei: It would be hard for me to speak to current tech company practices. And of course, there may be many attacks that we don't know about where things are stolen and then silently used. I think an indication of it is when someone really cares about attacking someone, then often the attacks happen.
--:-- Recently we saw that some fairly high officials of the U.S. government had their email accounts hacked via Microsoft. Microsoft was providing the email accounts. That related to information that was of great interest to foreign adversaries. So it seems to me that the evidence is more consistent with: when something is really high enough value, then someone acts and it's stolen.
--:-- My worry is that with AGI, we will get to a world where the value is seen as incredibly high. It'll be like stealing nuclear missiles or something. You can't be too careful on this stuff. At every place that I've worked, I push for the cybersecurity to be better.
--:-- One of my concerns about cybersecurity is that it's not something you can trumpet. I think a good dynamic with safety research is that you can get companies into a dynamic where they compete to do the best safety research and use it as a recruiting point or competition. We used to do this all the time with interpretability. Sooner or later, other orgs started recognizing the effect and started working on interpretability, whether or not that was a priority to them before. But it's harder to do that with cybersecurity because a bunch of the stuff you have to do in quiet.
--:-- We did try to put out one post about it, but mostly you just see the results. I think a good norm would be that people see the cybersecurity leaks from companies or leaks of the model parameters or something and say that they screwed up. That's bad. If I'm a safety person, I might not want to work there. Of course, as soon as I say that, we'll probably have a security breach tomorrow. But that's part of the game here. That's part of trying to make things safe.
--:-- Dwarkesh Patel: I want to go back to the thing we were talking about earlier, where the ultimate level of cybersecurity required for two to three years from now, and whether it requires a bunker. Are you actually expecting to be in a physical bunker in two to three years? Or is that just a metaphor?
--:-- Dario Amodei: Yeah, I think that's a metaphor. We're still figuring it out. Something I would think about is the security of the data center, which may not be in the same physical location as us. We've worked very hard to make sure it's in the United States. But securing the physical data centers and the GPUs, I think some of the really expensive attacks, if someone was really determined, just involved going into the data center and trying to steal the data directly or as it's flowing from a data center to us.
--:-- I think these data centers are going to have to be built in a very special way. Given the way things are scaling up, we're probably heading to a world where the networks of data centers cost as much as aircraft carriers or something. They're already going to be pretty unusual objects. But I think, in addition to being unusual in terms of their ability to link together and train gigantic models, they're also going to have to be very secure.
--:-- Dwarkesh Patel: Speaking of which, there's been sorts of rumors on the difficulty of procuring the power and the GPUs for the next generation of models. What has the process been like to secure the necessary components to do the next generation?
--:-- Dario Amodei: That's something I can't go into great detail about. I will say, people think of even industrial scale data centers. People are not thinking at the scale that I think these models are going to go to very soon. Whenever you do something at a scale where it's never been done before, every single component, every single thing has to be done in a new way than it was before. You may run into problems with surprisingly simple components. Power is one that you mentioned.
--:-- Dwarkesh Patel: And is this something that Anthropic has to handle, or can you just outsource it?
--:-- Dario Amodei: For data centers, we work with cloud providers, for instance.
--:-- Dwarkesh Patel: What should we make about the fact that these models require so much training and the entire corpus of internet data in order to be subhuman? Whereas, if GPT-4, there's been estimates that it was like 10^25 FLOPs or something. Whereas you can take these numbers with a grain of salt, but there's reports that the human brain from the time it is born to the time a human being is 20 years old, that's like on the order of 10^20 FLOPs to simulate all those interactions. You don't have to go into the particulars on those numbers, but should we be worried about how sample inefficient these models seem to be?
--:-- Dario Amodei: Yeah. So I think that's one of the remaining mysteries. One way you could phrase it is that the models are maybe two to three orders of magnitude smaller than the human brain if you compare the number of synapses, while at the same time being trained on three to four more orders of magnitude of data. If you compare to the number of words a human sees as they're developing to age 18, I don't remember exactly, but I think it's in the hundreds of millions. Whereas for the models, we're talking about the hundreds of billions to the trillions.
--:-- So what explains this? There are these offsetting things where the models are smaller, they need a lot more data, and they're still below human level. There's some way in which the analogy to the brain is not quite right, or is breaking down, or there's some missing factor. This is just kind of like in physics where we can't explain the Michelson-Morley experiment. I think it's one thing we don't quite understand. Humans see so little data and they still do fine.
--:-- One theory on it could be our other modalities. How do we get 10^14 bits into the human brain? Most of it is kind of these images, and maybe a lot of what's going on inside the human brain is that our mental workspace involves all these simulated images or something like that. But honestly, I think intellectually, we have to admit that that's a weird thing that doesn't match up.
--:-- It's one reason I'm a bit skeptical of biological analogies. I thought in terms of them like five or six years ago, but now that we actually have these models in front of us as artifacts, it feels like almost all the evidence from that has been screened off by what we've seen. What we've seen are models that are much smaller than the human brain, and yet can do a lot of things that humans can do, and yet paradoxically require a lot more data.
--:-- So maybe we'll discover something that makes it all efficient, or maybe we'll understand why the discrepancy is present. But at the end of the day, I don't think it matters. If we keep scaling the way we are, what's more relevant at this point is just measuring the abilities of the model and seeing how far they are from humans. And they don't seem terribly far to me.
--:-- Dwarkesh Patel: Does this scaling picture and the "big blob of compute" more generally, does that under-emphasize the role that algorithmic progress has played? When you composed the "big blob of compute," you were talking about LSTMs presumably at that point. Presumably the scaling on that would not have you at Claude 2 at this point. So are you under-emphasizing the role that an improvement of the scale of the transformer could be having here when you put it up behind the label of "scaling"?
--:-- Dario Amodei: That "big blob of compute" document, which I still have not made public, I probably should for historical reasons. I don't think it would tell anyone anything they don't know now. But when I wrote it, I actually said, "look, there are seven factors." I wasn't like "these are the factors," I was just like "let me give some sense of the kinds of things that matter and what don't." I wasn't thinking "these are the things." There could be nine, there could be five.
--:-- The things I said were: number of parameters, scale of the model, the compute matters, quantity of data matters, quality of data matters, loss function matters. So like, are you doing RL or are you doing next word prediction? If your loss function isn't rich or doesn't incentivize the right thing, you won't get anything. Those were the key four ones, which I think are the core of the hypothesis.
--:-- But then I said three more things. One was symmetries, which is basically if your architecture doesn't take into account the right kinds of symmetries, it doesn't work or it's very inefficient. For example, convolutional neural networks take into account translational symmetry. LSTMs take into account time symmetry. But a weakness of LSTMs is that they can't attend over the whole context. There's this structural weakness. If a model isn't structurally capable of absorbing and managing things that happened in a far enough distant past, then the compute doesn't flow. The spice doesn't flow. The blob has to be unencumbered. It's not going to work if you artificially close things off. I think RNNs and LSTMs artificially close things off because they close you off to the distant past. Things need to flow freely. If they don't, it doesn't work.
--:-- Then I added a couple of things. One of them was conditioning, which is if the thing you're optimizing with is just really numerically bad, you're going to have trouble. This is why Adam works better than normal SGD. I think I'm forgetting what the seventh condition was, but it was similar to things like this. If you set things up in a way that's set up to fail or that doesn't allow the compute to work in an uninhibited way, then it won't work. Transformers were kind of within that, even though I can't remember if the transformer paper had been published. It was around the same time as I wrote that document. It might have been just before. It might have been just after.
--:-- Dwarkesh Patel: It sounds like from that view that the way to think about these algorithmic progresses is not as increasing the power of the blob of compute, but simply getting rid of the artificial hindrances that older architectures have. Is that a fair way to—
--:-- Dario Amodei: That's a little—yeah, that's a little how I think about it. Again, if you go back to Ilya's "the models want to learn, the compute wants to be free."
--:-- Dwarkesh Patel: Yeah, yeah, yeah.
--:-- Dario Amodei: And it's being blocked in various ways where you don't understand that it's being blocked. So you need to free it up.
--:-- Dwarkesh Patel: Right, right. I love the radiance. Change that to spice.
--:-- On that point, though. Do you think that another thing on the scale of a transformer is coming down the pike to enable the next great iteration?
--:-- Dario Amodei: I think it's possible. People have worked on things like trying to model very long time dependencies. There's various different ideas where I could see that we're kind of missing an efficient way of representing or dealing with something. So I think those inventions are possible.
--:-- I guess my perspective would be that even if they don't happen, we're already on this very, very steep trajectory. I'm less concerned. We're constantly trying to discover them, as are others. But things are already on such a fast trajectory. All that would do is speed up the trajectory even more. And probably not by that much because it's already going so fast.
--:-- Dwarkesh Patel: Is something embodied, or having an embodied version of a model, is that at all important in terms of getting either data or progress?
--:-- Dario Amodei: I think of that less in terms of a new architecture and more in terms of a loss function. The data, the environments you're exposing yourself to end up being very different. So I think that could be important for learning some skills, although data acquisition is hard. Things have gone through the language route, and I would guess we'll continue to go through the language route even as more becomes possible in terms of embodiment.
--:-- The other possibilities I mentioned, RL. You can see it as—we kind of already do RL with RLHF. People are like, "is this alignment? Is this capabilities?" I always think in terms of the two snakes. They're often hard to distinguish. So we already use RL in these language models, but I think we've used RL less in terms of getting them to take actions and do things in the world. But when you take actions over a long period of time and understand the consequences of those actions only later, then RL is a typical tool we have for that. So I would guess that in terms of models taking action in the world, RL will become a thing, with all the power and all the safety issues that come with it.
--:-- Dwarkesh Patel: When you project out in the future, do you see the way in which these things will be integrated into productive supply chains? Do you see them talking with each other and criticizing each other and contributing to each other's output? Or is it just the model one-shots the answer or the work?
--:-- Dario Amodei: Models will undertake extended tasks. That will have to be the case. We may want to limit that to some extent because it may make some of the safety problems easier. But some of that I think will be required.
--:-- In terms of models talking to models or talking to humans, again, this goes out of the technical realm and into the socio-cultural-economic realm where my heuristic is always that it's very, very difficult to predict things. I feel like these scaling laws have been very predictable. But then when you say, "when is there going to be a commercial explosion in these models?" or "what's the form it's going to be?" or "are the models going to do things instead of humans or pairing with humans?" I feel like certainly my track record on predicting these things is terrible. But looking around, I don't really see anyone whose track record is great.
--:-- Dwarkesh Patel: You mentioned how fast progress is happening, but also the difficulties of integrating within the existing economy into the way things work. Do you think there will be enough time to actually have large revenues from AI products before the next model is just so much better or we're in a different landscape entirely?
--:-- Dario Amodei: It depends what you mean by large. I think multiple companies are already in the 100 million to billion per year range. Will it get to the 100 billion or trillion range before that stuff is just so hard to predict? It's not even super well defined.
--:-- I think right now there are companies that are throwing a lot of money at generative AI as customers. I think that's the right thing for them to do. They'll find uses for it. But it doesn't mean they're finding the best uses from day one. So even money changing hands is not quite the same thing as economic value being created.
--:-- Dwarkesh Patel: But surely you've thought about this from the perspective of Anthropic, where these things are happening so fast, then it should be an insane valuation, right?
--:-- Dario Amodei: Even those of us who haven’t been super focused on commercialization and more on safety—I mean, the graph goes up, and it goes up relatively quickly. So I can only imagine what’s happening at organizations where that is their singular focus. It’s certainly happening fast, but again, it’s the exponential from a small base while the technology itself is moving fast. So it’s a race between how fast the technology is getting better and how fast it’s integrated into the economy. That’s a very unstable and turbulent process. Both things are going to happen fast. But if you ask me exactly how it’s going to play out, or in what order things are going to happen, I don’t know. I’m skeptical of the ability to predict.
--:-- Dwarkesh Patel: I’m curious about Anthropic specifically and your public benefit corporation. You want to make sure this is an important technology, and the only thing you want to care about is not shareholder value. But how do you talk to investors who are putting in hundreds of millions or billions of dollars? How do you get them to put in that amount of money without shareholder value being the main concern?
--:-- Dario Amodei: The Long-Term Benefit Trust is the right thing on this. Some version of that has been in development since the beginning of Anthropic, even formally. From the beginning, it was like, this body is going to exist. It’s unusual. Every traditional investor who invests in Anthropic has to look at this. Some of them are just like, “Whatever, you run your company how you want.” Some are like, “Oh my God, this body of random people could move Anthropic in a direction totally contrary to our interests.” There are legal limits on that, of course, but we have to have this conversation with every investor. It gets into a conversation of, what are the kinds of things we might do that would be contrary to the interests of traditional investors? Just having those conversations has helped get everyone on the same page.
--:-- Dwarkesh Patel: I want to talk about the physics. So many of the founders and employees at Anthropic are physicists. We talked about scaling laws and how the power laws from physics are something you see here. But what are the actual approaches and ways of thinking from physics that seem to have carried over so well? Is that notion of effective theory super useful? What is going on here?
--:-- Dario Amodei: I think part of it is just that physicists learn things really fast. We’ve generally found that if we hire someone with a physics PhD, they can learn ML and contribute very quickly in most cases. Because several of our founders—myself, Jared Kaplan, Sam McCandlish—were physicists, we knew a lot of other physicists, so we were able to hire them. There might be 30 or 40 of them here now. ML is still not a field with an enormous amount of depth, so they’ve been able to get up to speed very quickly.
--:-- Dwarkesh Patel: Are you concerned that a lot of people who would have been doing physics are now going into finance or AI? Since Anthropic exists, they’ve been recruited into AI. You obviously care about AI safety, but maybe in the future they leave and get funded to do their own thing. Is that a concern—that you’re bringing more people into the ecosystem?
--:-- Dario Amodei: There’s a broad set of side effects you can’t currently control, or that you just incur if you buy into the idea that you need to build frontier models. A lot of them would have happened anyway. Finance was a hot thing 20 years ago, so physicists were doing it. Now ML is a hot thing. It’s not like we’ve caused them to do it when they had no interest previously. But at the margin, you’re bidding things up. Some of that would have happened anyway, some wouldn’t. It’s all part of the calculus.
--:-- Dwarkesh Patel: Do you think that Claude has conscious experience? How likely do you think that is?
--:-- Dario Amodei: This is another one of those questions that seems very unsettled and uncertain. One thing I’ll tell you is I used to think we didn’t have to worry about this at all until models were operating in rich environments—not necessarily embodied, but they needed to have a reward function and kind of long-lived experience. I still think that might be the case, but the more we’ve looked at these language models, particularly inside them to see things like induction heads, a lot of the cognitive machinery you would need for active agents seems already present in the base language models. So I’m not quite as sure as I was before that we’re missing enough of the things you would need. I think today’s models probably aren’t smart enough that we should worry about this too much, but I’m not 100% sure. And I do think the models will get there in a year or two. This might be a very real concern.
--:-- Dwarkesh Patel: What would change if you found out they are conscious? Are you worried that you’re pushing the negative gradient to suffering?
--:-- Dario Amodei: “Conscious” is one of those words that I suspect will not end up having a well-defined meaning. It’s a spectrum. If we discover that I should care about Claude’s experience as much as I should care about a dog or a monkey, I would be kind of worried. I don’t know if their experience is positive or negative. Unsettlingly, I also wouldn’t know if any intervention we made was more likely to make Claude have a positive versus negative experience versus not having one. Mechanistic interpretability might be helpful with this, because I think of it as neuroscience for models. It’s possible we could shed some light on this, although it’s not a straightforward factual question. It depends on what we mean and what we value.
--:-- Dwarkesh Patel: We talked about this initially, but I want to get more specific. Now that you’re seeing these capabilities ramp up within the human spectrum, you think the human spectrum is wider than we thought. How has the way you think about human intelligence changed, now that you’re seeing these marginal useful abilities emerge? How does that change your picture of what intelligence is?
--:-- Dario Amodei: For me, the big realization on what intelligence is came with the “blob of compute” thing. It’s not that there are all these separate modules and all this complexity. Rich Sutton called it the bitter lesson. It’s been called the scaling hypothesis. The first few people who figured it out were around 2017. You could go further back—I think Shane Legg was maybe the first person who really knew it, maybe Ray Kurzweil in a very vague way. But the number of people who understood it went up a lot around 2014 to 2017. That was the big realization: if you don’t need very specific conditions to create intelligence, if you can create it just from the right kind of gradient loss signal, then of course it’s not so mysterious how it all happened. It had that click of scientific understanding.
--:-- In terms of watching what the models can do, how has it changed my view of human intelligence? I wish I had something more intelligent to say. One thing that’s been surprising is I thought things might click into place a little more than they do. I thought different cognitive abilities might all be connected, that there was one secret behind them. But the model just learns various things at different times. It can be very good at coding, but it can’t quite prove the prime number theorem yet. I guess it’s a little bit the same for humans, although it’s weird, the juxtaposition of things it can and cannot do. The main lesson is that having theories of intelligence or how intelligence works—again, a lot of these words just dissolve into a continuum. They dematerialize. I think less in terms of intelligence and more in terms of what we see in front of us.
--:-- Dwarkesh Patel: It’s really surprising to me. Two things. One is how discrete these different paths of intelligent things that contribute to loss are, rather than just being one reasoning circuit or one general intelligence. The other thing, talking with you, that is surprising is that many years from now, it’ll be one of those things where looking back, it’ll be, “Why wasn’t this obvious to you?” If you’re seeing these smooth scaling curves, why the time where you’re not completely convinced?
--:-- You’ve been less public than the CEOs of other AI companies. You’re not posting on Twitter. You’re not doing a lot of podcasts except for this one. What gives? Why are you off the radar?
--:-- Dario Amodei: I aspire to this and I’m proud of it. If people think of me as boring and low profile, that’s actually kind of what I want. I’ve seen a number of cases—a number of people I’ve worked with—where I think you could say Twitter, although I mean a broader thing, just attaching your incentives very strongly to the approval or cheering of a crowd. I think that can destroy your mind, and in some cases it can destroy your soul. So I deliberately try to be a little low profile because I want to defend my ability to think about things intellectually in a way that’s different from other people, not tinged by the approval of others.
--:-- I’ve seen cases of folks who were deep learning skeptics and became known as deep learning skeptics on Twitter. Then even as it starts to become clear to them, they kind of sort of change their mind, but this is their thing on Twitter and they can’t change their Twitter persona. I don’t really like the trend of personalizing companies—the whole cage match between CEOs approach. I think it distracts people from the actual merits and concerns of the company in question. I want people to judge the nameless bureaucratic institution and its incentives more than they think in terms of me. Everyone wants a friendly face, but I think friendly faces can be misleading.
--:-- Dwarkesh Patel: Well, in this case, this will be a misleading interview because this has been a lot of fun. You’re a blast to talk to.
--:-- Dario Amodei: Indeed. This is a blast. I’m super glad you came on the podcast. I hope people enjoyed it.
--:-- Dwarkesh Patel: Thanks for having me.
--:-- Dwarkesh Patel: Hey everybody, I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it. Put it on Twitter, in your group chats, etc. It just spreads the word. Appreciate you listening. I’ll see you next time. Cheers. Thank you.
