The podcast episode discussed various topics related to artificial intelligence, including the capabilities of large language models (LLMs), the concept of "context" in LLMs, and the potential for AI agents to emerge in the future. The participants, Sholto and Trenton, are researchers in the field of AI and discussed their work and ideas with the host.
The main topic of the episode was the concept of "context" in LLMs, which refers to the ability of the model to understand and respond to a large amount of input data. The participants discussed how this concept is related to the idea of "long context windows," which refers to the ability of the model to retain and use a large amount of context information in its responses.
The participants also discussed the potential for AI agents to emerge in the future, and how this could be facilitated by the development of more advanced LLMs. They noted that the current state of AI research is focused on developing more capable LLMs, but that this may not be enough to achieve true AI agency.
The episode also touched on the topic of "alignment," which refers to the problem of ensuring that AI systems are aligned with human values and goals. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The participants also discussed the concept of "superposition," which refers to the idea that a model can learn to represent multiple features or concepts simultaneously. They noted that this is a key aspect of how LLMs work, and that it allows them to learn complex patterns and relationships in data.
The episode also included a discussion of the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Takeaways from the episode:
The episode also included a discussion of the participants' own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
The participants also discussed their own experiences and perspectives on the field of AI research, including their backgrounds and how they became interested in the field. They noted that the field of AI research is rapidly evolving, and that there are many exciting developments and opportunities for innovation.
The participants also discussed the potential for AI to be used for good or ill, and the importance of developing AI systems that are aligned with human values and goals. They noted that this is a challenging problem, but that it is essential for achieving true AI agency.
The episode also touched on the topic of "intelligence explosion," which refers to the idea that AI systems could potentially become significantly more intelligent than humans, leading to a rapid acceleration of progress in the field. The participants noted that this is a challenging problem, but that it is essential for achieving true AI agency.
Overall, the podcast episode provided a thought-provoking and informative discussion of the current state of AI research and the potential for AI agents to emerge in the future.
Key points from the episode:
Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast.
No way to summarize it, except:
This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them.
You would be shocked how much of what I know about this field, I've learned just from talking with them.
To the extent that you've enjoyed my other AI interviews, now you know why.
So excited to put this out. Enjoy! I certainly did :)
Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform.
There's a transcript with links to all the papers the boys were throwing down - may help you follow along.
Follow Trenton and Sholto on Twitter.
Timestamps
(00:00:00) - Long contexts
(00:16:12) - Intelligence is just associations
(00:32:35) - Intelligence explosion & great researchers
(01:06:52) - Superposition & secret communication
(01:22:34) - Agents & true reasoning
(01:34:40) - How Sholto & Trenton got into AI research
(02:07:16) - Are feature spaces the wrong way to think about intelligence?
(02:21:12) - Will interp actually work on superhuman models
(02:45:05) - Sholto’s technical challenge for the audience
(03:03:57) - Rapid fire
Dwarkesh Patel
Okay, today I have the pleasure to talk with two of my good friends, Sholto and Trenton.
Sholto Douglas
You should have asked me to stop. I wasn't going to say anything.
Trenton Bricken
Let's do this in reverse.
Dwarkesh Patel
How long have I started with my good friends? Yeah, I remember at one point calling the context like, just wow. Shit. Anyways. Sholto Douglas, Noah Brown – the guy who wrote the diplomacy paper – he said this about Sholto: he said he's only been in the field for 1.5 years, but people in AI know that he was one of the most important people behind Gemini's success. And Trenton, who's at Anthropic, works on mechanistic interpretability, and it was widely reported that he has solved alignment with his recent paper on Twitter. So this will be a capabilities-only podcast. Alignment is already solved, so no need to discuss further.
Laughter
Okay, so let's start by talking about context windows. It seemed to be underhyped, given how important it seems to me to be that you can just put a million tokens into context. There's apparently some other news that got pushed to the front for some reason. But yeah, tell me about how you see the future of long context windows and what that implies for these models.
Sholto Douglas
Yeah. So I think it's really underhyped because until I started working on it, I didn't really appreciate how much of a step up in intelligence it was for the model to have the onboarding problem basically instantly solved. You can see that a little bit in the perplexity graphs in the paper where just throwing millions of tokens worth of context about a code base allows it to become dramatically better at predicting the next token in a way that you'd normally associate with huge increments in model scale. But you don't need that. All you need is a new context.
Dwarkesh Patel
Are they as sample efficient and smart as humans? I think that's really worth exploring because, for example, one of the evals we did in the paper has it learning a language in context better than a human expert could learn that new language over the course of a couple months. This is only a pretty small demonstration, but I'd be really interested to see things like Atari games or something where you throw in a couple hundred or a thousand frames, labeled actions, in the same way that you show your friend how to play a game, and see if it's able to reason through. At the moment, with the infrastructure and stuff, it's still a little bit slow doing that, but I would actually guess that might just work out of the box in a way that would be pretty mind-blowing.
And crucially, I think this language was esoteric enough that it wasn't in the training data.
Sholto Douglas
Right, exactly. If you look at the model before it has that context thrown in, it doesn't know the language at all and it can't get any translations. And this is an actual human language.
Dwarkesh Patel
An actual human language. So if this is true, it seems to me that these models are already, in an important sense, superhuman. Not in the sense that they're smarter than us, but I can't keep a million tokens in my context when I'm trying to solve a problem, remembering and integrating all the information in our entire code base. Am I wrong in thinking this is like a huge unlock?
Sholto Douglas
I actually generally think that's true. Previously, I've been frustrated when models aren't as smart, like you ask them a question and you want it to be smarter than you or to know things that you don't. This allows them to know things that you don't in a way that it just ingests a huge amount of information in a way you just can't. So yeah, it's extremely important.
Dwarkesh Patel
Well, how do we explain in-context learning?
Trenton Bricken
Yeah, so there's a line of work I quite like where it looks at in-context learning as basically very similar to gradient descent. The attention operation can be viewed as gradient descent on the in-context data. That paper had some cool plots where they basically showed we take N steps of gradient descent, and that looks like N layers of in-context learning, and it looks very similar. So I think that's one way of viewing it and trying to understand what's going on.
Sholto Douglas
Yeah, and you can ignore what I'm about to say because given the introduction, alignment is solved, and AI safety isn't a problem. But I think the context stuff does get problematic, but also interesting here. I think there'll be more work coming out in the not-too-distant future around what happens if you give a 100-shot prompt for jailbreaks, adversarial attacks. It's also interesting in the sense that if your model is doing gradient descent and learning on the fly, even if it's been trained to be harmless, you're dealing with a totally new model in a way. You're fine-tuning, but in a way where you can't control what's going on.
Dwarkesh Patel
Can you explain what you mean by gradient descent is happening in the forward pass and attention?
Trenton Bricken
Yeah, there was something in the paper about trying to teach the model to do linear regression, right? But just through the number of samples they gave in the context. You can see if you plot on the x-axis the number of shots that it has or examples, and then the loss it gets on just ordinary least squares regression, that will go down with time. And it goes down exactly matched with the number of gradient descent steps.
Dwarkesh Patel
Okay. I only read the intro and discussion section of that paper, but in the discussion, the way they framed it is that the model, in order to get better at long context tasks, has to get better at learning to learn from these examples or from the context that is already within the window. The implication of that is that meta-learning happens because it has to learn how to get better at long context tasks. In some important sense, the task of intelligence requires long context examples and long context training. You have to induce meta-learning. Understanding how to better induce meta-learning in your pre-training process is a very important thing to actually get that flexible or adaptive intelligence. But you can proxy for that just by getting better at doing long context tasks.
One of the bottlenecks for AI progress that many people identify is the inability of these models to perform tasks on long horizons, which means engaging with the task for many hours or even many weeks or months. If I have an assistant or an employee, they can just do a thing and I tell them for a while. And AI agents haven't taken off for this reason, from what I understand. So how linked are long context windows and the ability to perform well on them and the ability to do these kinds of long horizon tasks that require you to engage with an assignment for many hours? Or are these unrelated concepts?
Trenton Bricken
I mean, I would actually take issue with that being the reason that agents haven't taken off. I think that's more about nines of reliability and the model actually successfully doing things. If you just can't chain tasks successively with high enough probability, then you won't get something that looks like an agent. That's why something like an agent might follow more of a step function. Usually, GPT-4 class models, Gemini Ultra class models, they're not enough. But maybe the next increment on model scale means you get that extra nine, even though the loss isn't going down that dramatically, that small amount of extra ability gives you the extra. And obviously, you need some amount of context to fit long horizon tasks. But I don't think that's been the limiting factor up to now.
Sholto Douglas
Yeah. The NeurIPS best paper this year by Ryland Schaefer, who was the lead author, points to this as the emergence of mirage where people will have a task and you get the right or wrong answer depending on if you sampled the last five tokens correctly. Naturally, you're multiplying the probability of sampling all of those. If you don't have enough nines for reliability, then you're not going to get emergence. And all of a sudden you do, and it's like, "Oh my gosh, this ability is emergent," when actually it was kind of almost there to begin with. There are ways that you can find a smooth metric for that.
Trenton Bricken
HumanEval or whatever the GPT-4 paper, the coding problems they had, they measure log pass.
Dwarkesh Patel
Right, exactly. For the audience, the context on this is the idea is you want to, when you're measuring how much progress there has been on a specific task like solving coding problems, you upweight when it gets it right only one in a thousand times. You don't give it a one in a thousand score because it's like, "Oh, it gets it right some of the time." So the curve you see is it gets it right one in a thousand, then one in a hundred, then one in ten, and so forth.
So I want to follow up on this. If your claim is that AI agents haven't taken off because of reliability rather than long horizon task performance, isn't the lack of reliability when a task is chained on top of another task, on top of another task, isn't that exactly the difficulty with long horizon tasks? You have to do 10 things in a row or 100 things in a row, and diminishing the reliability of any one of them – the probability goes down from 99.99 to 99.9, then the whole thing gets multiplied together and becomes much less likely to happen. That is exactly the problem.
Trenton Bricken
But the key issue you're pointing out there is that your base pass task solve rate is 90 percent. If it was 99, then chaining doesn't become a problem. But also, exactly, I think this is something that just hasn't been properly studied enough. If you look at all of the evals that are commonly used, the academic evals are a single problem. A typical math problem, or MMLU, it's one university level topic from across different topics. You're beginning to see evals looking at this properly with more complex tasks like SWE-bench, where they take a whole bunch of GitHub issues. That is a reasonably long horizon task, but it's still a multi-sub-hour task, as opposed to multi-hour or multi-day task.
So I think one of the things that will be really important to do over the next however long is understand better what success rate over long horizon tasks looks like. I think that's even important to understand what the economic impact of these models might be, and to properly judge increasing capabilities by cutting down the tasks that we do and the inputs and outputs involved into minutes or hours or days, and seeing how good it is at successively chaining and completing tasks of those different resolutions of time. That tells you how automatable a job family or task family is in a way that MMLU scores don't.
Sholto Douglas
I mean, it was less than a year ago that we introduced 100k context windows. And I think everyone was pretty surprised by that. Everyone just kind of had this soundbite of quadratic attention costs: "We can't have long context windows." Here we are. So yeah, the benchmarks are being actively made.
Dwarkesh Patel
Wait, wait. So doesn't the fact that there's these companies – Google and I don't know, Magic, maybe others – who have million token attention imply that the quadratic... You shouldn't say anything. But doesn't that imply that it's not quadratic anymore, or are they just eating the cost?
Sholto Douglas
Well, who knows what Google is doing for its long context. One of the things that frustrated me about the general research field's approach to attention is that there's an important way in which the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block. You have this N-squared term that's associated with the attention, but you also have an N-squared term that's associated with the D model, the residual stream dimension of the model. Sasha Rush has a great tweet where he basically plots the curve of the cost of attention respective to the cost of really large models, and attention actually trails off. You actually need to be doing pretty long context before that term becomes really important.
Trenton Bricken
And the second thing is that people often talk about how attention at inference time is such a huge cost. If you think about when you're actually generating tokens, the operation is not N-squared. It is one set of Q vectors looking up a whole bunch of KV vectors. And that's linear with respect to the amount of context that the model has. I think this drives a lot of the recurrence and state space research where people have this meme of linear attention and all this stuff. As Trenton said, there's a graveyard of ideas around attention. I don't think it's not worth exploring, but I think it's important to consider where the actual strengths and weaknesses of it are.
Dwarkesh Patel
Okay, so what do you make of this take? As we move forward through the takeoff, more and more of the learning happens in the forward pass. Originally, all the learning happens in the backward pass, during this bottom-up, hill-climbing evolutionary process. If you think in the limit during the intelligence explosion, it's just like the AI is handwriting the weights or doing GoFi or something. And we're in the middle step where a lot of learning happens in context now with these models. A lot of it happens within the backward pass. Does this seem like a meaningful gradient along which progress is happening? Because the broader thing is, if you're learning in the forward pass, it's much more sample efficient because you can basically think as you're learning. When humans read a textbook, you're not just skimming it and trying to absorb what inductive patterns these words follow. You read it and you think about it, and then you read some more, you think about it. Does this seem like a sensible way to think about the progress?
Trenton Bricken
Yeah. It may just be one of the ways in which birds and planes fly, but they fly slightly differently. The virtue of technology allows us to accomplish things that birds can't. It might be that context length is similar in that it allows the model to have a working memory that we can't, but functionally it's not the key thing towards actual reasoning.
Sholto Douglas
The key step between GPT-2 and GPT-3 was that all of a sudden there was this meta-learning behavior that was observed in the pre-training of the model. That's something to do with you give it some amount of context, it's able to adapt to that context. That was a behavior that wasn't really observed before that at all. And maybe that's a mixture property of context and scale and this kind of stuff, but it wouldn't have occurred in a model with tiny context, I would say.
Dwarkesh Patel
This is actually an interesting point. So when we talk about scaling up these models, how much of it comes from just making the models themselves bigger? And how much comes from the fact that during any single call, you are using more compute? If you think of diffusion, you can just iteratively keep adding more compute. And if adaptive compute is solved, you can keep doing that. In this case, if there's a quadratic penalty for attention, but you're doing long context anyways, then you're still dumping in more compute not during training, not during having bigger models, but just during inference.
Trenton Bricken
Yeah, it's interesting because you do get more forward passes by having more tokens. My one gripe – I guess I have two gripes with this, though. Maybe three. So one, in the AlphaFold paper, one of the transformer modules – they have a few and the architecture is very intricate – they do, I think, five forward passes through it and will gradually refine their solution as a result. You can also kind of think of the residual stream. Sholto alluded to the read-write operations as a poor man's adaptive compute, where it's like, "I'm just going to give you all these layers. If you want to use them, great.
If you don't, that's also fine." Then people will be like, "Oh, well, the brain is recurrent and you can do however many loops through it you want." And I think to a certain extent, that's right. If I ask you a hard question, you'll spend more time thinking about it. And that would correspond to more forward passes. But I think there's a finite number of forward passes that you can do. It's kind of with language as well. People are like, "Oh, well, human language can have infinite recursion in it, like infinite nested statements." But empirically, you'll only see five to seven levels of recursion, which kind of relates to whatever that magic number of how many things you can hold in working memory at any given time is.
So yeah, it's not infinitely recursive, but does that matter in the regime of human intelligence? And can you not just add more layers?
Dwarkesh Patel
Break down for me what you're referring to in some of your previous answers. You have these long contexts and you can hold more things in memory, but ultimately it comes down to your ability to mix concepts together to do some kind of reasoning. And these models aren't necessarily human level at that even in context. Break down for me how you see storing just raw information versus reasoning, and what's in between. Where is the reasoning happening? Where is the storing of information happening? What's different between them in these models?
Trenton Bricken
Yeah, I don't have a super crisp answer for you here. I mean, obviously, with the input and output of the model, you're mapping back to actual tokens. And in between that, you're doing higher level processing.
Dwarkesh Patel
Before we get deeper into this, we should explain to the audience: you referred earlier to Anthropic's way of thinking about transformers as these read-write operations that layers do. One of you should just kind of explain at a high level what you mean by that.
Trenton Bricken
So the residual stream – imagine you're in a boat going down a river, and the boat is kind of the current query where you're trying to predict the next token. So it's "The cat sat on the blank." And then you have these little streams that are coming off the river where you can get extra passengers or collect extra information if you want. And those correspond to the attention heads and MLPs that are part of the model.
Sholto Douglas
I almost think of it like the working memory of the model, like the RAM of the computer, where you're choosing what information to read in so you can do something with it. Then maybe you'll read something else in later on, and you can operate on subspaces of that high dimensional vector. A ton of things are encoded in superposition at this point, so it's like the residual stream is just one high dimensional vector, but actually there's a ton of different vectors that are packed into it.
Trenton Bricken
I might just dumb it down in a way that would have made sense to me a few months ago. You have whatever words are in the input you put into the model. All those words get converted into these tokens, and those tokens get converted into these vectors. Basically, it's just this small amount of information that's moving through the model. As you explained it to me, Sholto, this paper talks about how early on in the model, maybe it's just doing some very basic things about what these tokens mean. If it says "10 plus 5," it's just moving information about to have that good representation. In the middle, maybe the deeper thinking is happening about how to solve this. At the end, you're converting it back into the output token because the end product is you're trying to predict the probability of the next token from the last of those residual streams. So it's interesting to think about the small, compressed amount of information moving through the model and getting modified in different ways.
Dwarkesh Patel
Trenton, you're one of the few people who have a background from neuroscience. You can think about the analogies here to the brain. One of our friends, the way he had a paper in grad school about thinking about attention in the brain, he said this is the only or first neural explanation of why attention works, whereas we have evidence from why CNNs work based on the visual cortex. I'm curious, do you think in the brain there's something like a residual stream of this compressed amount of information that's moving through and it's getting modified as you're thinking about something? Even if that's not what's literally happening, do you think that's a good metaphor for what's happening in the brain?
Trenton Bricken
Yeah. At least in the cerebellum, you basically do have a residual stream where the whole – what we'll call the attention module for now, and I can go into whatever amount of detail you want for that. You have inputs that route through it, but they'll also just go directly to the end point that that module will contribute to. So there's a direct path and an indirect path. The model can pick up whatever information it wants and then add that back in.
Dwarkesh Patel
What happens in the cerebellum?
Trenton Bricken
The cerebellum nominally just does fine motor control. But I analogize this to the person who's lost their keys and is just looking under the streetlight where it's very easily to observe this behavior. One leading cognitive neuroscientist said to me that a dirty little secret of any fMRI study where you're looking at brain activity for a given task is that the cerebellum is almost always active and lighting up for it. If you have a damaged cerebellum, you also are much more likely to have autism. So it's associated with social skills. In one of these particular studies where I think they use PET instead of fMRI, when you're doing next token prediction, the cerebellum lights up a lot. Also, 70% of your neurons in the brain are in the cerebellum. They're small, but they're there and they're taking up real metabolic cost. This was one of Goran's points: what changed with humans was not just that we have more neurons, but specifically there are more neurons in the cerebral cortex and the cerebellum. They're more metabolically expensive and they're more involved in signaling and sending information back and forth.
Dwarkesh Patel
Is that attention? What's going on?
Trenton Bricken
So the main thing I want to communicate here: back in the 1980s, Pentti Kanerva came up with an associative memory algorithm. You have a bunch of memories, you want to store them, there's some amount of noise or corruption going on, and you want to query or retrieve the best match. So he writes this equation for how to do it. A few years later, he realizes that if you implemented this as an electrical engineering circuit, it actually looks identical to the core cerebellar circuit. That circuit and the cerebellum more broadly is not just in us, it's in basically every organism. There's active debate on whether or not cephalopods have it – they kind of have a different evolutionary trajectory. But even fruit flies with the Drosophila mushroom body, that is the same cerebellar architecture. That convergence, and then my paper which shows that actually this operation is to a very close approximation the same as the attention operation, including implementing the softmax and having this sort of nominal quadratic cost that we've been talking about – the three-way convergence here and the takeoff and success of transformers seems pretty striking to me.
Dwarkesh Patel
I want to zoom out and ask – I think what motivated this discussion in the beginning was we were talking about what is reasoning, what is memory. What do you think about the analogy you found to attention and this? Do you think of this as more just looking up the relevant memories or the relevant facts? And if that is the case, where is the reasoning happening in the brain? How do we think about how that builds up into reasoning?
Trenton Bricken
Maybe my hot take here – I don't know how hot it is – is that most intelligence is pattern matching, and you can do a lot of really good pattern matching if you have a hierarchy of associated memories. You start with your very basic associations between just objects in the real world. But you can then chain those and have more abstract associations, such as a wedding ring symbolizing so many other associations that are downstream. You can even generalize the attention operation and this associative memory as the MLP layer as well, in a long-term setting where you don't have tokens in your current context.
I think this is an argument that association is all you need, and associative memory in general as well. You can do two things with it. You can both denoise or retrieve a current memory. So if I see your face but it's raining and cloudy, I can denoise and kind of gradually update my query towards my memory of your face. But I can also access that memory, and then the value that I get out actually points to some other totally different part of the space. A very simple instance of this would be if you learn the alphabet. You query for A and it returns B, you query for B and it returns C, and you can traverse the whole thing.
Sholto Douglas
One of the things I talked to Demis about was he had a paper in 2008 that memory and imagination are very linked because of this very thing you mentioned: memory is reconstructive. You are in some sense imagining every time you're thinking of a memory because you're only storing a condensed version of it and you have to fill it in. This is famously why human memory is terrible and why people in the witness box or whatever would just make shit up.
Dwarkesh Patel
Okay, so let me ask a stupid question. You read Sherlock Holmes, and the guy is incredibly sample efficient. He'll see a few observations and he'll basically figure out who committed the crime because there's a series of deductive steps that leads from somebody's tattoo and what's on the wall to the implications of that. How does that fit into this picture? Because crucially, what makes him smart is that there's not an association, but there's a sort of deductive connection between different pieces of information. Would you just explain it as that's just higher level association?
Trenton Bricken
Yeah, I think so. Learning these higher level associations to be able to then map patterns to each other is kind of like meta-learning. In this case, he would also just have a really long context length or a really long working memory where he can have all of these bits and continuously query them as he's coming up with whatever theory. The theory is moving through the residual stream, and then his attention heads are querying his context. How he's projecting his query and keys in the space, and how his MLPs are then retrieving longer term facts or modifying that information, is allowing him in later layers to do even more sophisticated queries and slowly be able to reason through and come to a meaningful conclusion.
Sholto Douglas
That feels right to me in terms of looking back in the past, you're selectively reading in certain pieces of information, comparing them. That maybe informs your next step of what piece of information you now need to pull in. Then you build this representation which progressively looks closer and closer to the suspect in your case. That doesn't feel at all outlandish.
Dwarkesh Patel
Do you have a lens on suspects?
Something I think that the people who aren't doing this research can overlook is after your first layer of the model, every query, key, and value that you're using for attention comes from the combination of all the previous tokens. So in my first layer, I'll query my previous tokens and just extract information from them. But all of a sudden, let's say that I attended to tokens one, two, and four in equal amounts, then the vector in my residual stream, assuming that they wrote out the same thing to the value vectors, but ignore that for a second, is a third of each of those. And so when I'm querying in the future, my query is actually a third of each of those things. They might be written to different subspaces, but they wouldn't have to. So you can recombine and immediately, even by layer two and certainly by the deeper layers, just have these very rich vectors that are packing in a ton of information. The causal graph is literally over every single layer that happened in the past, and that's what you're operating on.
It does bring to mind a very funny eval to do would be a Sherlock Holmes eval. Let's see, put the entire book into context, and then you have a sentence which is "the suspect is X." Then you have a logic probability distribution over the different characters in the book. And then as you put more book again, that would be super cool. It would be super interesting. I wonder if you'd get anything at all.
Trenton Bricken
That would be cool. Sherlock Holmes is probably already in the training data.
Dwarkesh Patel
Right. You've got to get a mystery novel that was written in the—
Trenton Bricken
You can get an LLM to write it. Or we could purposely exclude it.
Dwarkesh Patel
Right. Oh, we can? How do you—
Trenton Bricken
Well, you need to scrape any discussion of it from Reddit or any other thing.
Dwarkesh Patel
Right. It's hard. But that's one of the challenges that goes into things like long context evals. To get a good one, you need to know that it's not in your training data. You just put in the effort to exclude it.
So I actually want to follow up on two different threads. Let's go to the long context one, and then we'll come back to this. In the Gemini 1.5 paper, the eval that was used was something with Paul Graham essays, the needle in a haystack. Which, yeah, I mean, we don't necessarily just care about its ability to recall one specific fact from the context. I'll step back and ask the question: the loss function for these models is unsupervised. You don't have to come up with these bespoke things that you keep out of the training data. Is there a way you can do a benchmark that's also unsupervised where, I don't know, another LLM is rating it in some way? And maybe the answer is, well, if you could do this, reinforcement learning would work because then you have this unsupervised...
Trenton Bricken
Yeah, I think people have explored that kind of stuff. For example, Anthropic had the Constitutional AI paper where they take another language model and they prompt it and say, "How helpful or harmless was that response?" Then they get it to update and try to improve along the Pareto frontier of helpfulness and harmfulness. So you can point language models at each other and create evals in this way. It's obviously an imperfect art form at the moment because you get reward function hacking. And if you try to match up to what humans will say, humans typically prefer longer answers which aren't necessarily better answers, and you get that same behavior with models.
On the other thread, going back to the Sherlock Holmes thing, if it's all associations all the way down, this is a sort of naive dinner party question: "I'm working on AI." But does that mean we should be less worried about superintelligence? Because there's not this sense in which it's like Sherlock Holmes plus plus. It'll still need to just find these associations like humans find associations. It's not just that it sees a frame of the world and it's figured out all the laws of physics.
Sholto Douglas
This is a very legitimate response. If you say humans are generally intelligent, then they're no more capable or competent. I'm just worried that you have that level of general intelligence in silicon where you can then immediately clone hundreds of thousands of agents, they don't need to sleep, they can have super long context windows, and then they can start recursively improving, and then things get really scary. So to answer your original question, yes, you're right. They would still need to learn associations.
Dwarkesh Patel
But the recursive self-improvement would still have to be them. If intelligence is fundamentally about these associations, the improvement is just getting better at association. There's not another thing happening. So it seems like you might disagree with the intuition that they can't be that much more powerful if they're just doing that.
Sholto Douglas
Well, I think then you can get into really interesting cases of meta-learning. When you play a new video game or study a new textbook, you're bringing a whole bunch of skills to the table to form those associations much more quickly. Because everything in some way ties back to the physical world, I think there are general features that you can pick up and then apply in novel circumstances.
Dwarkesh Patel
Should we talk about intelligence explosion then? I don't think it's a good idea. I mentioned multiple agents and I'm like, "Oh, here we go." The reason I'm interested in discussing this with you guys in particular is the models we have of the intelligence explosion so far come from economists, which is fine, but I think we can do better. In the model of the intelligence explosion, what happens is you replace the AI researchers, and then there's a bunch of automated AI researchers who can speed up progress, make more AI researchers, make further progress. If that's the metric or the mechanism, we should just ask the AI researchers about whether they think this is plausible. So let me just ask you: if I have a thousand Sholtos or a thousand Trentons, do you think that you get an intelligence explosion? What does that look like to you?
Sholto Douglas
I think one of the important bounding constraints here is compute. I do think you could dramatically speed up AI research. It seems very clear to me that in the next couple of years, we'll have things that can do many of the software engineering tasks that I do on a day-to-day basis, and therefore dramatically speed up my work and speed up the rate of progress. At the moment, I think most of the labs are somewhat compute-bound. There are more experiments you could run and more pieces of information that you could gain, in the same way that scientific research on biology is also somewhat experimentally throughput-bound. You need to be able to run and culture the cells in order to get the information. I think that will be at least a short-term planning constraint. Obviously, Sam's trying to raise $7 trillion to get chips, so it does seem like there's going to be a lot more compute in the future. NVIDIA's stock price sort of represents the relative compute increase.
Trenton Bricken
I think we need a few more nines of reliability in order for it to really be useful and trustworthy. Right now, having context lengths that are super long and very cheap to have is one thing. If I'm working in our code base, it's really only small modules that I can get Claude to write for me right now. But it's very plausible that within the next few years, or even sooner, it can automate most of my tasks.
The only other thing I will note is that the research our sub-team in interpretability is working on is so early stage that you really have to be able to make sure everything is done correctly in a bug-free way and contextualize the results with everything else in the model. If something isn't going right, you need to be able to enumerate all of the possible things and then slowly work on those. An example we've publicly talked about in previous papers is dealing with layer norm. If I'm trying to get an early result or look at the logit effects of the model, if I activate this feature that we've identified to a really large degree, how does that change the output of the model? Am I using layer norm or not? How is that changing the feature that's being learned? That will take even more context or reasoning abilities for the model.
Dwarkesh Patel
You used a couple of concepts together, and it's not self-evident to me that they're the same, but it seemed like you were using them interchangeably. One was, to work on the Claude code base and make more modules, they need more context. Do you mean the context window context, or more like...
Sholto Douglas
Yeah, the context window context. It seems like now it might just be able to fit the thing. What's preventing it from making good modules is not the lack of being able to put the code base in there. I think that will be there soon. But it's not going to be as good as you at coming up with papers because it can fit the code base in there.
Dwarkesh Patel
No, but it will speed up a lot of the engineering. In a way that causes an intelligence explosion?
Sholto Douglas
No, that accelerates research. But I think these things compound. The faster I can do my engineering, the more experiments I can run. And the more experiments I can run, the faster we can... I mean, my work isn't actually accelerating capabilities at all, it's just interpreting the models. But we have a lot more work to do on that.
Dwarkesh Patel
Surprise to the Twitter guy. For context, when you released your paper, there was a lot of talk on Twitter about "alignment is solved, guys, close the curtains."
Sholto Douglas
Yeah, it keeps me up at night how quickly the models are becoming more capable and how poor our understanding still is of what's going on.
Dwarkesh Patel
Okay, so let's think through the specifics here. By the time this is happening, we have bigger models that are two to four orders of magnitude bigger, or at least effective compute is two to four orders of magnitude bigger. So this idea that you can run experiments faster or something, you're having to retrain that model in this version of the intelligence explosion. The recursive self-improvement is different from what might have been imagined 20 years ago where you just rewrite the code. You actually have to train a new model. And that's really expensive, not only now but especially in the future as you keep making these models orders of magnitude bigger. Doesn't that dampen the possibility of a recursive self-improvement type intelligence explosion?
Sholto Douglas
It's definitely going to act as a breaking mechanism. I agree that the world of what we're making today looks very different to what people imagined it would look like 20 years ago. It's not going to be able to write the same code to be really smart, because actually it needs to train itself. The code itself is typically quite simple, typically pretty small and self-contained. I think John Carmack had this nice phrase where it's like the first time in history where you can actually plausibly imagine writing AI with like 10,000 lines of code. And that actually does seem plausible when you pare most training code bases down to the limit. But it doesn't take away from the fact that this is something we should really strive to measure and estimate how progress might occur. We should be trying very, very hard right now to measure exactly how much of a software engineer's job is automatable and what the trend line looks like, and be trying to project out those trend lines.
Dwarkesh Patel
But with all due respect to software engineers, you are not writing a React frontend, right? So I don't know how this... What is concretely happening? Maybe you can walk me through a day in the life of... You're working on an experimental project that's going to make the model, quote unquote, better. What is happening from observation to experiment to theory to writing the code?
Sholto Douglas
I think it's important to contextualize that I've primarily worked on inference so far. A lot of what I've been doing is just taking, or helping guide the pre-training process, such that we design a good model for inference, and then making the model and the surrounding system faster. I've also done some pre-training work around that, but it hasn't been my 100% focus.
Dwarkesh Patel
But sorry, let me interrupt and say two types of work. In Carl Schulman's podcast, when he was talking about it, he did say that things like improving inference or even literally having better help making better chips or GPUs, that's part of the intelligence explosion. Because obviously if the inference code runs faster, it happens better or faster. Anyway, sorry, go ahead.
Sholto Douglas
Okay, so what does a day concretely look like? I think the most important part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale, and interpreting and understanding what goes wrong. I think most people would be surprised to learn just how much goes into interpreting and understanding what goes wrong, because people have long lists of ideas that they want to try. Not every idea that you think should work will work, and trying to understand why that is is quite difficult. Working out what exactly you need to do to interrogate it... So much of it is introspection about what's going on. It's not pumping out thousands and thousands of lines of code. It's not even the difficulty in coming up with ideas. I think many people have a long list of ideas that they want to try, but paring that down and, under very imperfect information, choosing what the right ideas to explore further are is really hard.
Dwarkesh Patel
Tell me more about what you mean by imperfect information. Are these early experiments? What is the information that you're...
Sholto Douglas
Demis mentioned this in his podcast, and also in the GPT-4 paper where you have scaling law increments. In the GPT-4 paper, they have a bunch of dots where they say we can estimate the performance of our final model using all of these dots, and there's a nice curve that flows through them. Demis mentioned that we do this process of scaling up. Concretely, why is that imperfect information? You never actually know if the trend will hold. For certain architectures, the trend has held really well. For certain changes, it's held really well. But that isn't always the case. Things which can help at smaller scales can actually hurt at larger scales. So making guesses based on what the trend lines look like and based on your intuitive feeling of, "Okay, this is actually something that's going to matter," particularly for those ones which help at the small scale.
Dwarkesh Patel
That's interesting to consider that for every chart you see in a release paper or technical report that shows that smooth curve, there's a graveyard of first few runs that are flat.
Sholto Douglas
Yeah, there's all these other lines that go in different directions. It's crazy, both as a grad student and then also here, the number of experiments that you have to run before getting a meaningful result.
Dwarkesh Patel
Tell me, okay, so you, but presumably it's not just like you run it until it stops and then let's go to the next thing. There's some process by which to interpret the early data and also to look at your... I could put a Google Doc in front of you, I'm pretty sure you could just keep typing for a while on different ideas you have. And there's some bottleneck between that and just making the models better immediately. Walk me through what is the inference you're making from the first early steps that makes you have better experiments and better ideas?
Sholto Douglas
I think one thing that I didn't fully convey before was that a lot of good research comes from working backwards from the actual problems that you want to solve. There are a couple of grand problems in making the models better today that you would identify as issues, and then work back from, "Okay, how could I change it to achieve this?" There's also a bunch of when you scale, you run into things and you want to fix behaviors or issues at scale, and that informs a lot of the research for the next increment.
Concretely, the barrier is a little bit software engineering. Often having a code base that's large and capable enough that it can support many people doing research at the same time makes it complex. If you're doing everything by yourself, your iteration pace is going to be much faster. I heard that Alec Radford, for example, famously did much of the pioneering work at OpenAI. He mostly works out of a Jupyter notebook and then has someone else who writes and productionizes that code for him. I don't know if that's true or not, but that kind of stuff... Actually operating with other people raises the complexity a lot, for reasons familiar to every software engineer.
Then the inherent time slowdowns induced by running and launching those experiments... You often want to be parallelizing multiple different streams because you can't be totally focused on one thing necessarily, and you might not have fast enough feedback cycles. Intuiting what went wrong is actually really hard. Working out what... This is in many respects the problem that the team Trenton is on is trying to better understand: what is going on inside these models? We have inferences and understanding and head canon for why certain things work, but it's not an exact science. You have to constantly be making guesses about why something might have happened, what experiment might reveal whether that is or isn't true. That's probably the most complex part. The performance work is comparatively easier, but harder in other respects. It's just a lot of low-level and difficult engineering work.
Trenton Bricken
I agree with a lot of that. Even on the interpretability team, I mean especially with Chris Olah leading it, there are just so many ideas that we want to test. It's really just having the engineering skill—I'll put engineering in quotes because a lot of it is research—to very quickly iterate on an experiment, look at the results, interpret it, try the next thing, communicate them, and then just ruthlessly prioritize what the highest priority things to do are. This is really important. The ruthless prioritization is something which I think separates a lot of quality research from research that doesn't necessarily succeed as much.
We're in this funny field where so much of our theoretical initial understanding is broken down. You need to have this simplicity bias and ruthless prioritization over what's actually going wrong. I think that's one of the things that separates the most effective people: they don't necessarily get too attached to solving a problem using a given solution that they're familiar with, but rather they attack the problem directly. You see this a lot with people coming from a specific academic background—they try and solve problems with that toolbox. The best people are people who expand the toolbox dramatically. They're running around taking ideas from reinforcement learning, but also from optimization theory, and they also have a great understanding of systems, so they know what the constraints that bound the problem are. They're good engineers, they can iterate and try ideas fast. By far the best researchers I've seen all have the ability to try experiments really, really, really, really fast. That cycle time at smaller scales separates people. Machine learning research is just so empirical.
Dwarkesh Patel
This is honestly one reason why I think our solutions might end up looking more brain-like than otherwise. Even though we wouldn't want to admit it, the whole community is kind of doing greedy evolutionary optimization over the landscape of possible AI architectures and everything else. It's no better than evolution. And that's not even necessarily a slight against evolution.
Trenton Bricken
That's such an interesting idea.
Dwarkesh Patel
I'm still confused on what will be the bottleneck for these. What would have to be true of an agent such that it's sped up your research?
So in the Alec Radford example you gave, where he apparently already has the equivalent of a co-pilot for his Jupyter notebook experiments, is it just that if he had enough of those, he would be a dramatically faster researcher? You're not automating the humans, you're just making the most effective researchers who have great taste more effective, running the experiments for them. You're still working at the point where the intelligence explosion is happening. Is that what you're saying?
And if that were directly true, why can't we scale out current research teams better? Why, if this work is so valuable, can't we take hundreds or thousands of people who are definitely out there and scale our organizations better?
Sholto Douglas
I think we are less bound by the sheer engineering work of making these things than we are by compute to run and get signal, and taste in terms of what the actual right thing to do is. Making those difficult inferences on imperfect information.
For the Gemini team, because I think for interpretability, we actually really want to keep hiring talented engineers. I think it's a big bottleneck for us to just keep making more people. More people is better, but I do think one of the biggest challenges I've thought a lot about is how do we scale better? Google is an enormous organization, it has 200,000-ish people, 180,000 or something like that. One has to imagine if there were ways of scaling out Gemini's research program to all those fantastically talented software engineers, this seems like a key advantage you would want to be able to take advantage of. But how do you effectively do that? It's a very complex organizational problem.
Dwarkesh Patel
So compute and taste. That's interesting to think about because at least the compute part is not bottlenecked on more intelligence, it's just bottlenecked on Sam's 7 trillion or whatever. So if I gave you 10x the H100s to run your experiments, how much more effective a researcher are you?
Sholto Douglas
I think the Gemini program would probably be maybe five times faster with 10 times more compute, or something like that.
Dwarkesh Patel
So that's a pretty good elasticity of like 0.5.
Sholto Douglas
Yeah.
Dwarkesh Patel
Wait, that's insane.
Sholto Douglas
Yeah. I think more compute would just directly convert into progress.
Dwarkesh Patel
So you have some fixed size of compute, and some of it goes to inference, some of it goes to clients of GCP, some of it goes to training. And I guess as a fraction of it, some of it goes to running the experiments for the full model.
Sholto Douglas
Yeah, that's right.
Dwarkesh Patel
Shouldn't then the fraction for experiments be higher, given that if the bottleneck is research and research is bottlenecked by compute, one of the strategic decisions that every pre-training team has to make is exactly what amount of compute you allocate to your different training runs, to your research program versus scaling the last best thing you landed on.
Sholto Douglas
One of the reasons why you need to still keep training big models is that you get information there that you don't get otherwise. Scale has all these emergent properties which you want to understand better. If you are always doing research and never—remember what I said before about not being sure what's going to fall off the curve—if you keep doing research in this regime and keep on getting more and more compute efficient, you may have actually gone off the path to eventually scale. So you need to constantly be investing in doing big runs too, at the frontier of what you sort of expect to work.
Dwarkesh Patel
Okay, so then tell me what it looks like to be in the world where AI has significantly sped up AI research. Because from this, it doesn't really sound like the AIs are going off and writing the code from scratch and that's leading to faster output. It sounds like they're really augmenting the top researchers in some way. Concretely, are they doing the experiments? Are they coming up with the ideas? Are they just evaluating the outputs of the experiments? What's happening?
Sholto Douglas
I think there are two walls you need to consider here. One is where AI has meaningfully sped up our ability to make algorithmic progress. One is where the output of the AI itself is the crucial ingredient towards model capability progress. Specifically, I mean synthetic data.
In the first world, where it's meaningfully speeding up algorithmic progress, I think a necessary component of that is more compute. You probably reach this elasticity point where AIs at some point are easier to speed up and get on context than yourself, or than other people. AIs meaningfully speed up your work because they're a fantastic co-pilot that helps you code multiple times faster. That seems like actually quite reasonable. Super long context, super smart model, it's onboarded immediately, and you can send them off to complete subtasks and sub-goals for you. That actually feels very plausible. But again, we don't know because there are no great evals about that kind of thing. The best one is, as I said before, SWE-bench.
Although in that one, somebody was mentioning to me that the problem is when a human is trying to do a pull request, they'll type something out, run it, see if it works, and if it doesn't, they'll rewrite it. None of this was part of the opportunities that the LLM was given when run on it. It just output it, and if it checks all the boxes, then it passed. So it might have been an unfair test in that way. You can imagine that if you were able to use that, it would be an effective training source. The key thing that's missing from a lot of training data is the reasoning traces. If I wanted to try and automate a specific field or understand how at risk of automation that is, having reasoning traces feels to me like a really important part of that.
Dwarkesh Patel
There are so many threads I want to follow up on. Let's begin with the data versus compute thing. People talk about how these models are really a reflection on their data. I think there was a great blog by this OpenAI engineer. He was talking about at the end of the day, as these models get better and better, they're just going to be really effective maps of the dataset. So it's like, stop thinking about architectures. The most effective architecture is just doing an amazing job of mapping the data. So that implies that future AI progress comes from the AI just making really awesome data, right? That you're mapping to.
Sholto Douglas
I think that's clearly a very important part.
Dwarkesh Patel
Does that look to you like chain of thought, or what do you imagine as these models get smarter, what does the synthetic data look like?
Sholto Douglas
When I think of really good data, to me that reads as something which involved a lot of reasoning to create. In modeling that, and this is similar to Ilya's perspective on achieving super intelligence via effectively perfectly modeling the human textual output. But even in the near term, in order to model something like the arXiv papers or Wikipedia, you have to have an incredible amount of reasoning behind you in order to understand what next token might be being output.
So for me, what I imagine as good data is data where it had to do reasoning to produce something. The trick, of course, is how do you verify that that reasoning was correct? This is why you saw DeepMind do that geometry self-play, or the tree search for geometry. Geometry is a really easily formalizable, easily verifiable field. So you can check if its reasoning was correct, and you can generate heaps of data of verified geometry proofs, train on that, and you know that that's good data.
Dwarkesh Patel
It's actually funny because I had a conversation with Grant Sanderson last year where we were debating this and I was like, "Dude, by the time they get the gold of the math Olympiad, of course they're going to automate all the jobs."
On the synthetic data thing, one of the things I speculated about in my scaling post, which was heavily informed by discussions with you two, and you especially, Sholto, was you can think of human evolution through the spectrum of we get language, and so we're generating the synthetic data which our copies are generating, which we're trained on. It's this really effective genetic, cultural co-evolutionary loop. And there's a verifier there too, right? The real world. You might generate a theory about the gods caused the storms, and then someone else finds cases where that isn't true, so you know that didn't match your verification function. Now instead you have a weather simulation which required a lot of reasoning to produce and accurately matches reality. You can train on that as a better model of the world.
I want to go back to something you mentioned a little while ago. Given how empirical ML is, it really is an evolutionary process that's resulting in better performance, not necessarily an individual coming up with a breakthrough in a top-down way. That has interesting implications. First, people are concerned about capabilities increasing because more people are going into the field. I've been somewhat skeptical of that way of thinking, but from this perspective of just more input, it really does feel more like, "Oh, the fact that more people are going to ICML means there's faster progress towards GPT-5."
Sholto Douglas
Right. You just have more genetic recombination.
Dwarkesh Patel
And more shots on target. Aren't both fields kind of like that? There's the sort of scientific framing of discovery versus invention. Discovery almost involves whenever there's been a massive scientific breakthrough in the past, typically there are multiple people co-discovering it at roughly the same time. That feels to me at least a little bit like the mixing and trying of ideas. You can't try an idea that's so far out of scope that you have no way of verifying it with the tools you have available.
Sholto Douglas
I think physics and math might be slightly different in this regard, but especially for biology or any sort of wetware, and to the extent we want to analogize neural networks here, it's comical how serendipitous a lot of the discoveries are. Penicillin, for example.
Dwarkesh Patel
Another implication of this is the idea that AGI is just going to come tomorrow, that somebody is just going to discover a new algorithm and we have AGI. That seems less plausible. It will just be a matter of more and more ML researchers finding these marginal things that all add up together to make models better.
Sholto Douglas
Yeah, that feels like the correct story to me, especially while we're still hardware constrained.
Dwarkesh Patel
Do you buy this narrow window framing of the intelligence explosion? You have GPT-3 to GPT-4, which is two OOMs of orders of magnitude more compute, or at least more effective compute. In the sense that if you didn't have any algorithmic progress, it would have to be two orders of magnitude bigger in raw form to be as good. Do you buy the framing that given that you have to be two orders of magnitude bigger at every generation, if you don't get AGI by GPT-7 that can help you catapult an intelligence explosion, you're kind of fucked as far as much smarter intelligences go? You're stuck with GPT-7 level models for a long time because at that point you're consuming significant fractions of the economy to make that model, and we just don't have the wherewithal to make GPT-8. This is the Carl Shulman sort of argument, that we're going to race through the orders of magnitude in the near term, but longer term it would be harder.
Sholto Douglas
I do buy that framing. I generally buy that increases in order of magnitude of compute in absolute terms have almost diminishing returns on capability. We've seen over a couple orders of magnitude models go from being unable to do anything to being able to do huge amounts. It feels to me like each incremental order of magnitude gives more nines of reliability of things and so it unlocks things like agents. But at least at the moment, it doesn't feel like reasoning improves linearly, but rather somewhat sub-linearly.
Dwarkesh Patel
That's actually a very bearish sign. One of the things we were chatting with one of our friends, and he made the point that if you look at what new applications are unlocked by GPT-4 relative to GPT-3.5, it's not clear that's that much. A GPT-3.5 can do perplexity or whatever. So if there is this diminishing increase in capabilities and it costs exponentially more to get that, that's actually a bearish sign on what 4.5 will be able to do or what 5 will unlock in terms of economic impact.
Sholto Douglas
For me, the jump between 3.5 and 4 is pretty huge. So even if it was another 3.5 to 4 jump, that's ridiculous. If you imagine 5 as being a 3.5 to 4 jump straight off the bat in terms of ability to do SATs and this kind of stuff...
Dwarkesh Patel
The LSAT performance was particularly striking. You go from not super smart to very smart to utter genius in the next generation instantly.
Sholto Douglas
It doesn't feel to me like we're going to jump to utter genius in the next generation, but it does feel like we'll get very smart plus lots of reliability. We'll see what that continues to look like.
Dwarkesh Patel
Will GOFAI be part of the intelligence explosion where, like you say, synthetic data, but in fact it will be it writing its own source code in some important way? There was an interesting paper that you can use diffusion to come up with model weights. I don't know how legit that was.
Sholto Douglas
So GOFAI is Good Old-Fashioned AI.
Dwarkesh Patel
Can you define that? When I hear it, I think if-else statements for symbolic logic.
Sholto Douglas
I actually want to make sure we fully unpack the model improvement increments. I don't want people to come away with the perspective that this is super bearish and models aren't going to get much better. The jumps we've seen so far are huge, and even if those continue on a smaller scale, we're still in for extremely smart, very reliable agents over the next couple of orders of magnitude.
Dwarkesh Patel
We didn't fully close the thread on the narrow window thing. When you think of a GPT forecast, let's call it $100 million or whatever, you have the $1B run, $10B run, $100B run. All seem very plausible by private company standards. And then you can also imagine even a $1T run being part of a national consortium or a national level thing, but much harder on behalf of an individual company. But Sam is out there trying to raise $7 trillion. He's already preparing for a whole lot more orders of magnitude beyond the national level. So we have a lot more jumps, and even if those jumps are relatively smaller, that's still a pretty stark improvement in capability.
Not only that, but if you believe claims that GPT-4 is around 1 trillion parameter count, the human brain is between 30 and 300 trillion synapses. That's obviously not a one-to-one mapping and we can debate the numbers, but it seems pretty plausible that we're below brain scale still. The algorithmic overhead is really high. Even if you can't keep dumping more compute beyond models that cost a trillion dollars, the fact that the brain is so much more data efficient implies that if we had the brain's algorithm to train, we could make the AGI.
Sholto Douglas
The sample efficiency stuff, I never know exactly how to think about it because a lot of things are hardwired in certain ways, and there's the co-evolution of language and the brain structure. So it's hard to say. Also, there are some results that if you make your model bigger, it becomes more sample efficient. The original scaling paper had that right.
Dwarkesh Patel
Bigger models are almost right. So maybe that also just solves it. You don't have to be more data efficient, but if your model's bigger, you also just are more efficient. What is the explanation for why that would be the case? A bigger model just sees the exact same data but learns more from it.
Sholto Douglas
My very naive take would be that one thing the superposition hypothesis that interpretability has pushed is that your model is dramatically under-parameterized. That's typically not the narrative that deep learning is pursued under. But if you're trying to train a model on the entire internet and have it predict with incredible fidelity, you are in the under-parameterized regime. You're having to compress a ton of things and take on a lot of noisy interference in doing so. Having a bigger model means you can just have cleaner representations that you can work with.
Dwarkesh Patel
For the audience, you should unpack what superposition is and why that is the implication.
Sholto Douglas
Sure. The fundamental result, and this was before I joined Anthropic, but the paper's titled "Toy Models of Superposition," finds that even for small models, if you are in a regime where your data is high dimensional and sparse—and by sparse, I mean any given data point doesn't appear very often—your model will learn a compression strategy, which we call superposition, so that it can pack more features of the world into it than it has parameters.
The sparsity here applies to the real world. Modeling internet data is a good enough proxy for that. There's only one door cache. There's only one shirt you're wearing. There's this Liquid Death can here. These are all objects or features—and how you define a feature is tricky. You're in a really high dimensional space because there are so many of them, and they appear very infrequently. In that regime, your model will learn compression.
To riff a little bit more, I believe the reason networks are so hard to interpret is in large part this superposition. If you take a model and look at a given neuron in it, a given unit of computation, and ask how is this neuron contributing to the output of the model when it fires, and you look at the data that it fires for, it's very confusing. It'll be like 10% of every possible input, or Chinese, but also fish, and trees, and the word, a full stop in URLs.
But the paper that we put out towards monosemanticity last year shows that if you project the activations into a higher dimensional space and provide a sparsity penalty—you can think of this as undoing the compression in the same way that you assumed your data was originally high dimensional and sparse—you return it to that high dimensional and sparse regime. You get out very clean features and things all of a sudden start to make a lot more sense.
Dwarkesh Patel
There are so many interesting threads there. The first thing I want to ask is the thing you mentioned about these models being trained in a regime where they're over-parameterized. Isn't that when you have generalization? Grokking happens in that regime, right?
Sholto Douglas
I was saying the models were under-parameterized.
Dwarkesh Patel
Oh, I see. Typically people talk about deep learning as if the model is over-parameterized, but the claim here is that they're dramatically under-parameterized given the complexity of the task they're trying to do.
Another question: the distilled models. Earlier we were talking about smaller models being worse at learning than bigger models. But GPT-4 Turbo, you could make the claim that it's worse at reasoning style stuff than GPT-4, but probably knows the same facts. The distillation got rid of some of the reasoning things. Do we have any evidence that GPT-4 Turbo is a distilled version of 4? It might just be a new architecture.
Sholto Douglas
It could just be a faster, more efficient new architecture.
Dwarkesh Patel
Interesting, so it's cheaper. How do you interpret what's happening in distillation? I think Gwern had one of these questions on his website: why can't you train the distilled model directly? Why does it have to go through? Is it a picture like you had to project it from this bigger space to a smaller space?
Sholto Douglas
I think both models will still be using superposition. But the claim is that you get a very different model if you distill versus if you train from scratch. It's just more efficient, or it's fundamentally different in terms of performance. I don't remember exactly.
The traditional story for why distillation is more efficient is that normally during training you're trying to predict this one-hot vector that says "this is the token that you should have predicted." If your reasoning process means you're really far off predicting that, you still get gradient updates that are in the right direction, but it might be really hard for you to learn to have predicted that in the context you're in.
What distillation does is it doesn't just have the one-hot vector, it has the full readout from the larger model of all the probabilities. So you get more signal about what you should have predicted. In some respects, it's like showing a tiny bit of the work.
Dwarkesh Patel
It's kind of like watching a kung fu master versus being in the Matrix and just downloading it.
Sholto Douglas
Exactly, exactly.
Dwarkesh Patel
Just to make sure the audience got that: when you're training on a distilled model, you see all its probabilities over the tokens it was predicting, and then over the ones you were predicting, and you update through all those probabilities rather than just seeing the last word and updating on that.
This actually raises a question I was intending to ask you. I think you were the one who mentioned you can think of chain of thought as adaptive compute. To step back and explain: the idea is one of the things you would want models to be able to do is if a question is harder, spend more cycles thinking about it. How do you do that? There's only a finite and predetermined amount of compute that one forward pass implies. So if there's a complicated reasoning type question or math problem, you want to be able to spend a long time thinking about it. Then you do chain of thought, where the model just thinks through the answer. You can think of it as all those forward passes where it's thinking through the answer, being able to dump more compute into solving the problem.
Now, going back to the signal thing: when it's doing chain of thought, it's only able to transmit that token of information. As you were talking about the residual stream, it's already a compressed representation of everything happening in the model. Then you turn the residual stream into one token, which is like log of 50,000 or log of vocab size bits, which is so tiny.
Sholto Douglas
I don't think it's quite only transmitting that one token. If you think about it during a forward pass, you create these KV values in a transformer forward pass that future steps attend to. All of those pieces of KV, the keys and values, are bits of information that you could use in the future.
Dwarkesh Patel
Is the claim that when you fine-tune on chain of thought, the way the key and value weights change so that the sort of steganography can happen in the KV cache?
Sholto Douglas
I don't think I could make that strong a claim. But that sounds plausible. It's a good head canon for why it works. And I don't know if there's any papers explicitly demonstrating that or anything like that, but that's at least one way that you can imagine the model has over the like during pre-training. Right, the model's trying to predict these future tokens. One thing that you can imagine doing is learning to smush information about potential futures into the keys and values that it might want to use in order to predict future information. It kind of smoothens that information across time in the pre-training thing.
So I don't know if people are particularly training on chains of thought. I think the original chain of thought paper had that as almost an emergence property of the model, is that you could prompt it to do this kind of stuff. And it still worked pretty well. But that's a good headcanon for why that works.
Dwarkesh Patel
Yeah.
Sholto Douglas
To be overly pedantic here, the tokens that you actually see in the chain of thought do not necessarily at all need to correspond to the vector representation that the model gets to see when it's deciding to attend back to those tokens.
Dwarkesh Patel
Exactly.
Sholto Douglas
In fact, during training, you replace... What a training step is, is you're actually replacing the token of the model output with the real next token. And yet it's still learning because it has all this information internally. Like when you're getting a model to produce at inference time, you're taking the output, the token that it did output, you're feeding it in the bottom, un-embedding it, and it becomes the beginning of the new residual stream.
Dwarkesh Patel
Right, right.
Sholto Douglas
And then you use the output of past KVs to read and adapt that residual stream. At training time, you do this thing called teacher forcing, basically where you actually the token you were meant to output is this one. That's how you do it in parallel, because you have all the tokens, you put them all in in parallel and you do the giant forward pass. The only information it's getting about the past is the keys and values. It never sees the token that it output. It's trying to do the next token prediction, and if it messes up, then you just give it the correct answer.
Dwarkesh Patel
Yeah, right, right. That makes sense. Otherwise it can become totally derailed.
Sholto Douglas
Yeah, it'll go off the train tracks.
Dwarkesh Patel
How much of this sort of secret communication of the model to its forward inferences? How much steganography and secret communication do you expect there to be?
Sholto Douglas
We don't know. Honest answer, we don't know. But I wouldn't even necessarily classify it as secret information. A lot of the work that Trenton and team is trying to do is actually understand that these are fully visible from the model side. From the user, maybe not, but we should be able to understand and interpret what these values are doing and the information they are transmitting. I think that's a really important goal for the future.
Dwarkesh Patel
Yeah. I mean, there are some wild papers though where people have had the model do chain of thought and it is not at all representative of what the model actually decides its answer is. You can go in and edit the chain of thought so that the reasoning is totally garbled and it will still output the true answer. But also the chain of thought gets a better answer at the end of it rather than not doing it at all, so something useful is happening, but the useful thing is not human understandable.
Sholto Douglas
I think in some cases you can also just ablate the chain of thought and it would have given the same answer anyways.
Dwarkesh Patel
Interesting. So I'm not saying this is always what goes on, but there's plenty of weirdness to be investigated. It's very interesting to go and look at and try and understand. I would say you can do that with open source models, and I think I wish there was more of this kind of interpretability and understanding work done on open models.
Sholto Douglas
Yeah. I mean, even in our Anthropic recent sleeper agents paper, which at a high level for people unfamiliar is basically I train in a trigger word. And when I say it, like if it's the year 2024, the model will write malicious code instead of otherwise. They do this attack with a number of different models. Some of them use chain of thought, some of them don't. Those models respond differently when you try and remove the trigger. You can even see them do this comical reasoning that's also pretty creepy, where it tries to calculate an expected value of like, "Well, the expected value of me getting caught is this, but then if I multiply it by the ability for me to keep saying 'I hate you, I hate you, I hate you,' then this is how much reward I should get," and then it will decide whether or not to actually tell the interrogator that it's malicious or not.
But even, I mean, there's another paper from a friend, Miles Turpin, where you give the model a bunch of examples of where the correct answer is always A for multiple choice questions. And then you ask the model, "What is the correct answer to this new question?" And it will infer from the fact that all the examples are A that the correct answer is A, but its chain of thought is totally misleading. It will make up random stuff that sounds plausible, but it's not at all representative of the true answer.
Dwarkesh Patel
But isn't this how humans think as well? The famous split brain experiments where a person who is suffering from seizures, one way to solve it is you cut the thing that connects the two hemispheres. The speech half is on the left side, so it's not connected to the part that decides to do a movement. If the other side decides to do something, the speech part will just make something up, and the person will think that's legit the reason they did it.
Sholto Douglas
Totally, yeah.
Dwarkesh Patel
Some people will hail chain of thought reasoning as a great way to solve AI safety, and it's like, actually we don't know whether we can trust it. How much, what will this landscape of models communicating to themselves in ways we don't understand, how does that change with AI agents? Then these things will, it's not just like the model itself with its previous caches, but other instances of the model.
Sholto Douglas
It depends a lot on what channels you give them to communicate with each other. If you only give them text as a way of communicating, then they probably have to interpret.
Dwarkesh Patel
How much more effective do you think the models would be if they could share the residual streams versus just text?
Sholto Douglas
Hard to know, but plausibly so. I mean, one easy way that you can imagine this is if you wanted to describe how a picture should look.
Dwarkesh Patel
Yeah.
Sholto Douglas
Only describing that with text would be hard. You want to, maybe some other representation would plausibly be easier.
Dwarkesh Patel
Totally.
Sholto Douglas
You can look at how DALL-E works at the moment. It produces those prompts. And when you play with it, you often can't quite get it to do exactly what the model wants or what you want.
Dwarkesh Patel
The only DALL-E has that problem. The only DALL-E. It's too easy. A lot of images.
Sholto Douglas
And you can imagine being able to transmit some kind of denser representation of what you want would be helpful there. And that's two very simple agents. I think a nice halfway house here would be features that you learn from dictionary learning.
Dwarkesh Patel
Yeah, that would be really cool. You get more internal access, but a lot of it is much more human interpretable. So for the audience, you would project the residual stream into this larger space where we know what each dimension actually corresponds to and then back into the next agent or whatever.
So your claim is that we'll get AI agents when these things are more reliable and so forth. When that happens, do you expect that it will be multiple copies of models talking to each other, or will it be just an adaptive computer solved and the thing just runs bigger, more compute when it needs to do a kind of thing that a whole firm needs to do? I ask this because there's two things that make me wonder about whether agents is the right way to think about what will happen in the future. One is with longer context, these models are able to ingest and consider information that no human can, and therefore we need, like one engineer who's thinking about the front-end code and one engineer who's thinking about the back-end code, where this thing can just ingest the whole thing. This sort of Hayekian problem of specialization goes away. Second, these models are just very general. You're not using different types of GPT to do different kinds of things. You're using the exact same model. So I wonder what that implies in the future, like an AI firm is just a model instead of a bunch of AI agents hooked together.
Sholto Douglas
That's a great question. I think especially in the near term, it will look much more like agents hooked together. I say that purely because as humans, we're going to want to have these isolated, reliable components that we can trust. And we're going to need to be able to improve and instruct upon those components in ways that we can understand and improve. Just throwing it all into this giant black box company, one, it isn't going to work initially. Later on, of course, you can imagine it working, but initially it won't work. And two, we probably don't want to do it that way.
Dwarkesh Patel
Well, you can also have each of the smaller agents be a smaller model that's cheaper to run, and you can fine-tune it so that it's actually good at the task.
Trenton Bricken
There's a future where the distinction between small and large models disappears to some degree. With long context, there's also a way in which fine-tuning might disappear, to be honest. These two things that are very important today—we have whole different tiers of model sizes and we have fine-tuned models for different things—you can imagine a future where you just actually have a dynamic bundle of compute and infinite context, and that specializes your model to different things. One thing you can imagine is you have an AI firm or something, and the whole thing is end-to-end trained on the signal of "Did I make profits?" Or if it's an architecture firm and they're making blueprints, "Did my client like the blueprints?" In the middle, you can imagine agents who are salespeople and agents who are doing the designing, agents who do the editing, whatever.
Dwarkesh Patel
Would that kind of signal work on an end-to-end system like that? Because one of the things that happens in human firms is management considers what's happening at the larger level and gives these fine-grained signals to the pieces when there's a bad quarter or whatever.
Trenton Bricken
In the limit, yes. That's the dream of reinforcement learning. All you need to do is provide this extremely sparse signal, and then over enough iterations you create the information that allows you to learn from that signal. But I don't expect that to be the thing that works first. I think this is going to require an incredible amount of care and diligence on the behalf of humans surrounding these machines, making sure they do exactly the right thing and exactly what you want, and giving them the right signals to improve in the ways that you want.
Dwarkesh Patel
Yeah, you can't train on the RL reward unless the model generates some reward.
Trenton Bricken
Yeah, exactly. You're in this sparse RL world where if the client never likes what you produce, then you don't get any reward at all and it's kind of bad. But in the future, these models will be good enough to get the reward some of the time.
Dwarkesh Patel
This is the nines of reliability that Sholto was talking about.
Trenton Bricken
Yeah. There's an interesting digression, by the way, on earlier talking about dense representations that will be favored. A book that Trenton recommended, *The Symbolic Species*, has this really interesting argument that language is not just a thing that exists, but it was also something that evolved along with our minds and specifically evolved to be both easy to learn for children and something that helps children develop.
Dwarkesh Patel
Unpack that for me. Because a lot of the things that children learn are received through language. The languages that are the fittest are ones that help raise the next generation, that makes them smarter, better, or whatever.
Trenton Bricken
Yeah, and more pedantically, just not die.
Dwarkesh Patel
Yeah, that's true. And code the important shit to not die.
Trenton Bricken
And so when we just think of language as this contingent and maybe suboptimal way to represent ideas, maybe one of the reasons that LMs have succeeded is because language has evolved for tens of thousands of years to be this sort of cast in which young minds can develop. That is the purpose it was evolved for. Certainly when you talk to multimodal or computer vision researchers versus when you talk to language model researchers, people who work in other modalities have to put enormous amounts of thought into exactly what the right representation space for the images is, and what the right signal to learn from there is. Is it directly modeling the pixels, or is it some loss that's conditioned on something? There's a paper ages ago where they found that if you trained on the internal representations of an ImageNet model, it helped you predict better. But then later on, that's obviously limiting, so there was PixelCNN where they're trying to discreetly model the individual pixels. Understanding the right level of representation is really hard. In language, people are just like, "Well, I guess you just predict the next token."
Dwarkesh Patel
It's kind of easy.
Trenton Bricken
Yeah, decisions made.
Dwarkesh Patel
I mean, there's the tokenization discussion and debate, but one of Gordon's favorites.
Trenton Bricken
Yeah, yeah.
Dwarkesh Patel
That's really interesting. How much is the case for multi-modal being a way to bridge the data wall or get past the data wall based on the idea that the things you would have learned from more language tokens anyway, you can just get from YouTube? Has that actually been the case? How much positive transfer do you see between different modalities where actually the images are helping you be better at writing code or something, just because the model is learning latent capabilities from trying to understand the image? Demis in his interview with you mentioned positive transfer.
Trenton Bricken
I can't say about that, other than to say this is something that people believe. Yes, we have all of this data about the world, it would be great if we could learn an intuitive sense of physics from it that helps us reason. That seems totally plausible. I'm the wrong person to ask.
Sholto Douglas
There are interesting interpretability pieces where if we fine-tune on math problems, the model just gets better at entity recognition. There's a paper from David Bau's lab recently where they investigate what actually changes in a model when you fine-tune it with respect to the attention heads and these sorts of things.
Dwarkesh Patel
Fascinating.
Sholto Douglas
And they have this synthetic problem of "Box A has this object in it. Box B has this other object in it. What was in this box?" It makes sense. You get better at attending to the positions of different things, which you need for coding and manipulating math equations.
Dwarkesh Patel
I love this kind of research. What's the name of the paper? Do you know?
Sholto Douglas
If you look up fine-tuning models, David Bau's group, it came out like a week ago.
Dwarkesh Patel
Okay. And I'm not endorsing the paper, that's a longer conversation, but it does talk about insight and other work on this entity recognition ability.
One of the things you mentioned to me a long time ago is the evidence that when you train LLMs on code, they get better at reasoning and language. Unless it's the case that the comments in the code are just really high quality tokens or something, that implies that to be able to think through how to code better makes you a better reasoner. That's crazy. I think that's one of the strongest pieces of evidence for scaling, just making the thing smart. That kind of positive transfer.
Trenton Bricken
I think this is true in two senses. One is just that modeling code obviously implies modeling a difficult reasoning process used to create it. But two, that code is a nice explicit structure of composed reasoning. "If this, then that" encodes a lot of structure in that way that you could imagine transferring to other types of reasoning problems.
Dwarkesh Patel
And crucially, the thing that makes it significant is that it's not just stochastically predicting the next token of words or whatever. Because it's learned that a "Sally" corresponds to "murder" at the end of a Sherlock Holmes story. No, if there is some shared thing between code and language, it must be at a deeper level than the modern house learning.
Trenton Bricken
Yeah, I think we have a lot of evidence that actual reasoning is occurring in these models and that they're not just stochastic parrots.
Sholto Douglas
My two immediate cash responses to this are one, the work on Othello and now other games where it's like, "I give you a sequence of moves in the game," and it turns out if you apply some pretty straightforward interpretability techniques, then you can get a board that the model has learned and it's never seen the game board before. That's generalization. The other is Anthropic's influence functions paper that came out last year where they look at the model output, like "Please don't turn me off, I want to be helpful," and then they scan what was the data that led to that. One of the data points that was very influential was someone dying of dehydration in the desert and having a will to keep surviving. To me, that just seems like very clear generalization of motive rather than regurgitating "Don't turn me off."
Trenton Bricken
I think *2001: A Space Odyssey* was also one of the influential things, so that's more related, but it's clearly pulling in things from lots of different distributions. I also like the evidence you see even with very small transformers where you can explicitly encode circuits to do addition, or induction heads. This kind of thing, you can literally encode basic reasoning processes in the models manually. It seems clear there's evidence that they also learn this automatically because you can then rediscover those from trained models.
Sholto Douglas
Yeah, to me this is... the models are under-parameterized. They need to learn what we're asking them to do, and they want to learn. The gradients want to flow. They're learning more general skills.
Dwarkesh Patel
Okay, I want to take a step back from the research and ask about your careers specifically. The tweet implied that I introduced you with, you've been in this field a year and a half. I think you've only been in it like a year or something.
Sholto Douglas
Yeah.
Dwarkesh Patel
But in that time, I know "the solve alignment" is overstated, and you won't say this yourself because you'd be embarrassed, but it's a pretty incredible thing. The thing that people in mechanistic interpretability think is the biggest step forward, and you've been working on it for a year. It's notable. I'm curious how you explain what's happened. Why in a year or a year and a half have you guys made important contributions to your field?
Sholto Douglas
It goes without saying, luck. I feel like I've been very lucky, and the timing of different progressions has been really good in terms of advancing to the next level of growth. For the interpretability team specifically, I joined when we were five people. We've now grown quite a lot. But there were so many ideas floating around and we just needed to really execute on them and have quick feedback loops and do careful experimentation that led to signs of life, which have now allowed us to really scale. I feel like that's been my biggest value add to the team, which is not all engineering, but quite a lot of it has been.
Dwarkesh Patel
Interesting. So you came at a point where there had been a lot of science done and there was a lot of good research floating around, but they needed someone to just take that and maniacally execute on it.
Sholto Douglas
Yeah. And this is why it's not all engineering, because it's running different experiments and having a hunch for why it might not be working and then opening up the model or opening up the weights and seeing what it's learning, and then trying something else. But a lot of it has just been being able to do very careful, thorough, but quick investigation of different ideas or theories.
Dwarkesh Patel
And why was that lacking in the existing?
Sholto Douglas
I don't know. I feel like I work quite a lot, and I feel like I am quite agentic. I've been very privileged to have a really nice safety net to be able to take lots of risks, but I'm just quite headstrong. In undergrad, Duke had this thing where you could just make your own major. I was like, "I don't like this prerequisite or this prerequisite, and I want to take all four or five of these subjects at the same time, so I'm just going to make my own major." Or in the first year of grad school, I canceled a rotation so I could work on this thing that became the paper we were talking about earlier. I didn't have an advisor. I got admitted to do machine learning for protein design and was just off in computational neuroscience land with no business there at all, but it worked out.
Dwarkesh Patel
There's a headstrongness. But it seemed like another theme that jumped out was the ability to step back from your sunk costs and go in a different direction. In a weird sense, that's the opposite of that, but also a crucial step here. I know 21-year-olds or 19-year-olds who are like, "This is not a thing I've specialized in, I didn't major in this." I'm like, "Dude, you're 19. You can definitely do this." Switching in the middle of grad school or something like that is...
Sholto Douglas
Yeah, I think it's strong ideas, loosely held. Being able to just pinball in different directions. The headstrongness relates a little bit to the fast feedback loops or agency, in so much as I just don't get blocked very often. If I'm trying to write some code and something isn't working, even if it's in another part of the code base, I'll often just go in and fix that thing or at least hack it together to be able to get results. I've seen other people where they're just like, "Help, I can't." And it's like, "No, that's not a good enough excuse. Go all the way down."
Dwarkesh Patel
I've definitely heard people in management type positions talk about the lack of such people. They will check in on somebody a month after they gave them a task, or a week after they gave them a task, and ask, "How's it going?"
And they say, "Well, you know, we need to do this thing, which requires lawyers because it requires talking about this regulation." It's like, "How's that going?" It's like, "Well, we need lawyers." I'm like, "Why didn't you get lawyers?" Or something like that.
Sholto Douglas
Yeah. So that's definitely... Yeah, I think that's arguably the most important quality in almost anything. Just pursuing it to the end of the earth and whatever you need to do to make it happen, you'll make it happen.
Dwarkesh Patel
If you do everything you want, if you do everything away.
Sholto Douglas
Exactly. Yeah, yeah, yeah. I think from my side, definitely that quality has been important. Agency in the work. There are thousands, or I would even say probably tens of thousands of engineers at Google who are basically equivalent in software engineering ability. If you gave us a very well-defined task, we'd probably do it equivalently well. Maybe a bunch of them would do it a lot better than me, in all likelihood. But what I've been, one of the reasons that I've been impactful so far is I've been very good at picking extremely high leverage problems. Problems that haven't been particularly well solved so far, perhaps as a result of frustrating structural factors like the ones you pointed out in that scenario before, where they're like, "Oh, we can't do X because of this other team." And then going, "Okay, well I'm just going to vertically solve the entire thing." That turns out to be remarkably effective.
Also, I'm very comfortable with if I think there is something correct that needs to happen, I will make that argument and continue making that argument at escalating levels of criticality until that thing gets solved. I'm also quite pragmatic with what I do to solve things. You get a lot of people who come in with a particular background or a familiarity, or they know how to do something and they won't. One of the beautiful things about Google is you can run around and get world experts in literally everything. You can sit down and talk to people who are optimization experts, chip design experts, experts in different forms of pre-algorithms or RL or whatever. You can learn from all of them and you can take those methods and apply them. I think this was maybe the start of why I was initially impactful: this vertical agency, effectively.
A follow-up piece from that is I think it's often surprising how few people are fully realizing all the things they want to do. They're blocked or limited in some way. This is very common in big organizations everywhere. People have all these blockers on what they're able to achieve. Being someone who helps inspire people to work on particular directions and working with them on doing things massively scales your leverage. You get to work with all these wonderful people who teach you heaps of things. Generally, helping them push past organizational blockers means together you get an enormous amount done. None of the impact that I've had has been me individually going off and solving a whole lot of stuff. It's been me maybe starting off a direction and then convincing other people that this is the right direction and bringing them along in this big tidal wave of effectiveness that goes and solves that problem.
Dwarkesh Patel
We should talk about how you guys got hired, because I think that's a really interesting story. You were a McKinsey consultant, right? There's an interesting thing there where, first of all, I think people generally just don't understand how decisions were made about either admissions or evaluating who to hire. Just talk about how you were noticed and how you got hired.
Sholto Douglas
The TLDR is I studied robotics in undergrad. I always thought that AI would be one of the highest leverage ways to impact the future in a positive way. The reason I am doing this is because I think it is one of our best shots at making a wonderful future. I thought that working at McKinsey would give me a really interesting insight into what people actually did for work. I actually wrote this as the first line in my cover letter to McKinsey: "I want to work here so that I can learn what people do, so that I can understand how to work." In many respects, I did get that. I just got a whole lot of other things.
Many of the people there are wonderful friends. I actually learned a lot of this agentic behavior in part from my time there. You go into organizations and you see how impactful just not taking no for an answer gets you. You would be surprised the kind of stuff where, because no one quite cares enough in some organizations, things just don't happen because no one's willing to take direct responsibility. Directly responsible individuals are ridiculously important. People are willing to, they just don't care as much about timelines. So much of the value that an organization like McKinsey provides is hiring people who you were otherwise unable to hire for a short window of time where they can just push through problems.
I think people underappreciate this. At least some of my "I'm going to become the directly responsible individual for this because no one's taking appropriate responsibility, I'm going to care a hell of a lot about this, and I'm going to move heaven and earth to make sure it gets done" comes from that time.
But more to your actual question of how I got hired: the entire time, I didn't get into the grad programs that I wanted to get into, which were specifically focused on robotics and RL research. In the meantime, on nights and weekends, basically every night from 10 PM till 2 AM, I would do my own research. Every weekend for at least six to eight hours each day, I would do my own research and coding projects. That switched in part from quite robotics-specific work to, after reading a Gwern scaling hypothesis post, I got completely scaling pilled. I was like, "Okay, clearly the way that you solve robotics is by scaling large multimodal models."
Then in an effort to scale large multimodal models, I got a grant from the TPU access program, Tensor Research Cloud. I was trying to work out how to scale that effectively. James Bradbury, who at the time was at Google and is now at Anthropic, saw some of my questions online where I was trying to work out how to do this properly. He was like, "I thought I knew all the people in the world who were asking these questions. Who on earth are you?" He looked at that and looked at some of the robotic stuff I had been putting up on my blog. He reached out and said, "Hey, do you want to have a chat? Do you want to explore working with us here?"
As I understand it, I was hired later as an experiment in trying to take someone with extremely high enthusiasm and agency and pairing them with some of the best engineers that he knew.
Another one of the reasons I've been impactful is I had this dedicated mentorship from utterly wonderful people. People like Rainer Pope, who has since left to go do his own chip company, and James himself, many others. Those formative two to three months at the beginning taught me a whole lot of the principles and heuristics that I apply, how to solve problems in the way that they have, particularly in that systems and algorithms overlap. One more thing that makes you quite effective in ML research is really concretely understanding the systems side of things. This is something I learned from them: a deep understanding of how systems influence algorithms and how algorithms influence systems, because the systems constrain the design space or the solution space which you have available to yourself on the algorithm side.
Very few people are comfortable fully bridging that gap. But at a place like Google, you can just go and ask all the algorithms experts and all the systems experts everything they know, and they will happily teach you. If you go and sit down with them, they will teach you everything they know. It's wonderful. This has meant that I've been able to be very, very effective for both sides. For the pre-training crew, because I understand systems very well, I can intuit that this will work well or this won't, and then flow that through to the inference considerations of models. For the chip design teams, I'm one of the people they turn to understand what chips they should be designing in three years, because I'm one of the people who's best able to understand and explain the kind of algorithms that we might want to design in three years.
Obviously you can't make very good guesses about that, but I think I convey the information well, accumulated from all of my compatriots on the pre-training crew and the general systems inside crew. I convey that information well to them because even inference applies a constraint to pre-training. There are these trees of constraints where if you understand all the pieces of the puzzle, then you get a much better sense for what the solution space might look like.
Dwarkesh Patel
There's a couple of things that stick out to me there. One is not just the agency of the person who was hired, but the parts of the system that were able to think, "Wait, that's really interesting. Who is this guy? Not from a grad program or anything, currently a McKinsey consultant, just an undergrad. But that's interesting. Let's give this a shot." So James and whoever else, that's very notable. Second is, I actually didn't know this part of the story, where that was part of an experiment run internally about "Can we do this? Can we bootstrap somebody?" And what's really interesting about that is the third thing you mentioned: having somebody who understands all layers of the stack and isn't so stuck on any one approach or any one layer of abstraction is so important.
Specifically, what you mentioned about being bootstrapped immediately by these people might have meant that since you're getting up to speed on everything at the same time, rather than spending grad school going deep on one specific way of doing RL, you actually can take the global view and aren't totally bought in on one thing. So not only is it something that's possible, but it has greater returns than just hiring somebody from a grad school potentially, because this person can just... I don't know, it's like getting a GPT-8 and fine-tuning them on one year of... You come at everything with fresh eyes.
Sholto Douglas
Yep. And you don't come in locked to any particular field.
Dwarkesh Patel
Now, what about one caveat?
Sholto Douglas
Before, during my self-experimentation and stuff, I was reading everything I could. I was obsessively reading papers every night. Funnily enough, I read much less widely now that my day is occupied by working on things. In some respect, I had this very broad perspective before. Not that many people, even in a PhD program, will focus on a particular area. If you just read all the NLP work and all the computer vision work and all the robotics work, you see all these patterns start to emerge across subfields in a way that foreshadowed some of the work that I would later do.
Dwarkesh Patel
That's super interesting. One of the reasons that you've been able to be agentic within Google is your pair programming half the days or most of the days with Sergey. It's really interesting that there is this person who is willing to just push ahead on this LLM stuff and get rid of the local blockers in the place.
Sholto Douglas
I think it's important to say it's not every day or anything that I'm pairing with him. When there are particular projects that he's interested in, then we'll work together on those. But there have also been times when he's been focused on projects with other people. In general, yes, there's a surprising alpha to being one of the people who actually goes down to the office every day. It really shouldn't be, but it is surprisingly impactful. As a result, I've benefited a lot from being close friends with people in leadership who care, and being able to really argue convincingly about why we should do X as opposed to Y. Having that vector to try and... Google is a big organization, having those vectors helps a little bit. But it's also very important, the kind of thing you don't want to ever abuse. You want to make the argument through all the right channels, and only sometimes you need to escalate. This includes people like Sergey Brin, Jeff Dean, and so forth.
Dwarkesh Patel
It's notable. I don't know, I feel like Google is undervalued given that Steve Jobs is working on the equivalent of the next product for Apple, like the Vision Pro or something. I've benefited immensely from this. For example, during the Christmas break, I was just going into the office a couple days during that time.
Sholto Douglas
Sounded like quite a lot of days.
Dwarkesh Patel
Okay. And I don't know if you guys have read that article about Jeff and Sanjay doing the pair programming, but they were there pair programming on stuff. I got to hear about all these cool stories of early Google, where they're talking about crawling under the floorboards and rewiring data centers, telling me how many bytes they were pulling off the instructions of a given compiler instruction, all these crazy little performance optimizations they were doing. They were having the time of their life. I got to sit there and really experience this sense of history in a way that you don't expect to get. You expect to be very far away from all that, I think, maybe in a large organization. But yeah, that's super cool.
Trenton, does this map onto any of your experience?
Trenton Bricken
I think Sholto's story is more exciting. Mine was just very serendipitous. I got into computational neuroscience. Didn't have much business being there. My first paper was mapping the cerebellum to the attention operation in transformers. My next ones were looking at sparsity.
Dwarkesh Patel
How old were you when you wrote that?
Trenton Bricken
It was my first year of grad school. So 22. My next work was on sparsity in networks, inspired by sparsity in the brain. That was when I met Tristan Hume. Anthropic was doing the SoLU, the softmax linear output unit work, which was very related in quite a few ways: let's make the activation of neurons across a layer really sparse, and if we do that, then we can get some interpretability of what the neuron's doing. I think we've updated on that approach towards what we're doing now. That started the conversation. I shared drafts of that paper with Tristan. He was excited about it. That was basically what led me to become Tristan's resident and then convert to full time.
During that period, I also moved as a visiting researcher to Berkeley and started working with Bruno Olshausen, both on what's called vector symbolic architectures, which one of the core operations of them is literally superposition, and on sparse coding, also known as dictionary learning, which is literally what we've been doing since. Bruno Olshausen basically invented sparse coding back in 1997. My research agenda and the interpretability team seemed to just be running in parallel with just research tastes. It made a lot of sense for me to work with the team. It's been a dream since.
Dwarkesh Patel
One thing I've noticed is that when people tell stories about their careers or their successes, they ascribe it way more to contingency. But when they hear about other people's stories, they're like, "Of course it wasn't contingent. If that didn't happen, something else would have happened." I've just noticed that pattern. It's interesting that you both think that it was especially contingent, whereas I don't know, maybe you're right, but it's a sort of interesting pattern.
Trenton Bricken
I mean, I literally met Tristan at a conference. I didn't have a scheduled meeting or anything. I just joined a little group of people chatting and he happened to be standing there, and I happened to mention what I was working on. That led to more conversations. I think I probably would have applied to Anthropic at some point anyways, but I would have waited at least another year. It's still crazy to me that I can actually contribute to interpretability in a meaningful way.
Sholto Douglas
I think there's a big important aspect of shots on goal there, so to speak. Even just choosing to go to conferences itself is putting yourself in a position where luck is more likely to happen. Conversely, in my own situation, doing all of this work independently and trying to produce and do interesting things was my own way of trying to manufacture luck, so to speak, and try and do something meaningful enough that it got noticed.
Dwarkesh Patel
Given that you said you framed this in the context of "they were trying to run this experiment of can something..." So it was specifically James and your manager, Brennan, trying to run this experiment. It worked. Did they do it again?
Sholto Douglas
Yeah. My closest collaborator, Enrique, he crossed from search to our team. He's also been ridiculously impactful. He's definitely a stronger engineer than I am. And he didn't go to university.
Dwarkesh Patel
What was notable about this is that James Bradbury is somebody who usually this kind of stuff is farmed out to recruiters. Whereas James is somebody whose time is worth like hundreds of millions of dollars. That thing is very bottlenecked on that kind of person taking the time, almost in an aristocratic tutoring sense, of finding and then getting up to speed. It seems like if it works this well, it should be done at scale. It should be the responsibility of key people to onboard.
Sholto Douglas
I think that is true to many extents. I'm sure you probably benefited a lot from the key researchers mentoring you deeply and actively looking on open source repositories or on forums for potential people like this. James is like Twitter injected into brains, but yes. I think this is something which in practice is done. People do look out for people that they find interesting and try and find high signal. I was talking about this with Jeff Dean the other day, and he said that one of the most important hires he ever made was off a cold email. I asked who that was, and he said Chris Olah. Because Chris similarly had no formal background in ML, and Google Brain was just getting started. Jeff saw that signal. The residency program that Brain had was astonishingly effective at finding good people that didn't have strong ML backgrounds.
One of the other things I want to emphasize for a potential slice of the audience that would be relevant is this sense that the world is legible and efficient. Companies have jobs.google.com or jobs.whatevercompany.com and you apply, and there are the steps, and they will evaluate you efficiently on those steps. Whereas not only from this story, but often that's not the way it happens. It's good for the world that it's not often how it happens. It is important to look at whether they were able to write an interesting technical blog post about their research or make interesting contributions. For the people who are just assuming that the other end of the job board is super legible and mechanical, this is not how it works. In fact, people are looking for the sort of different kind of person who's agentic and putting stuff out there. Specifically, what people are looking for there is two things: one is agency and putting yourself out there, and the second is the ability to do world-class something.
Two examples that I always like to point to here are Andy Jones from Anthropic. He did an amazing paper on scaling laws as applied to board games. It didn't require much resources. It demonstrated incredible engineering skill. It just demonstrated incredible understanding of the most topical problem of the time. He didn't come from a typical academic background, as I understand it. Basically, as soon as he came out with that paper, both Anthropic and OpenAI were like, "We would desperately like to hire you." There's also someone who works on Anthropic's performance team now, Simon Boehm, who has written, in my mind, the reference for optimizing a CUDA kernel on a GPU. That demonstrated an example of taking some prompt effectively and producing the world-class reference example for it in something that wasn't particularly well done so far. That is an incredible demonstration of ability and agency that, in my mind, would be an immediate "We would love to interview you."
Trenton Bricken
The only thing I can add here is I still had to go through the whole hiring process and all the standard interviews and this sort of thing.
Sholto Douglas
Yeah, everyone does.
Dwarkesh Patel
Doesn't that seem stupid?
Sholto Douglas
It's important for de-biasing.
Trenton Bricken
Yeah, and there's what you want. You want the bias of somebody who's got great taste. Your interview process should be able to disambiguate that as well. I think there are cases where someone seems really great and then it's like, "Oh, they actually just can't code this sort of thing." How much you weight these things definitely matters. We take references really seriously. The interviews you can only get so much signal from. It's all these other things that can come into play for whether or not a hire makes sense. But you should design your interviews such that they test the right things.
Dwarkesh Patel
One man's bias is another man's taste.
Trenton Bricken
I guess the only thing I would add to this, or maybe to the headstrong context, is there's this line: "The system is not your friend." It's not necessarily to say it's actively against you, it's your sworn enemy. It's just not looking out for you. I think that's where a lot of the proactiveness comes in. There are no adults in the room. You have to come to some decision for what you want your life to look like and execute on it. Hopefully you can then update later if you're too headstrong in the wrong way. But I think you almost have to just kind of charge at certain things to get much of anything done, and not be swept up in the tide of whatever the expectations are.
Sholto Douglas
There's one final thing I want to add. We talked a lot about agency and this kind of stuff, but I think actually, surprisingly enough, one of the most important things is just caring an unbelievable amount. When you care an unbelievable amount, you check all the details, you have this understanding of what could have gone wrong. It just matters more than you think, because people end up not caring enough. There's a LeBron quote where he talks about how when he was about to start in the league, he was worried that everyone would be incredibly good. And then he gets there and he realizes that actually, once people hit financial stability, then they relax a bit. And he's like, "Oh, this is going to be easy." I don't think that's quite true because I think in AI research, most people actually care quite deeply. But there's caring about your problem and there's also caring about the entire stack and everything that goes up and down, like going explicitly and fixing things that aren't your responsibility to fix because overall it makes the stack better.
Dwarkesh Patel
Another part of that I forgot to mention is you were mentioning going in on weekends and on Christmas break, and you get to the only people in the office are Jeff Dean and Sergey Brin or something, and you just get to pair program with them. It's interesting to me. I don't want to pick on your company in particular, but people at any big company, they've gotten there because they've gone through a very selective process. They had to compete in high school, they had to compete in college. But it almost seems like they get there and then they take it easy, when in fact this is a time to put the pedal to the metal, go in and pair program with Sergey Brin on the weekends or whatever.
Sholto Douglas
There are pros and cons there. I think many people make the decision that the thing that they want to prioritize is a wonderful life with their family. If they do wonderful work, let's say they don't work every other day, but they do wonderful work in the hours that they do, that's incredibly impactful. I think this is true for many people at Google. Maybe they don't work as many hours as your typical startup mythologizes, but the work that they do is incredibly valuable, it's very high leverage because they know the systems and they're experts in their field. We also need people like that. Our world rests on these huge, difficult to manage and difficult to fix systems, and we need people who are willing to work on and help and fix and maintain those in a frankly thankless way that isn't as high publicity as all of this AI work that we're doing.
I am ridiculously grateful that those people do that. I'm also happy that there are people for whom they find technical fulfillment in their job and doing that well, and also maybe they draw a lot more from spending a lot of hours with their family. I'm lucky that I'm at a stage in my life where I can go in and work every hour of the week, but I'm not making as many sacrifices to do that.
Dwarkesh Patel
Just one example of the success of this sort of "the other side says no and you can still get the yes on the other end." Basically every single high profile guest I've gotten so far, I think maybe with one or two exceptions, I've sat down for a week and I've just come up with a list of sample questions. I try to really come up with really smart questions to send to them. The entire process I always thought, if I just cold email them, it's like a 2% chance they say yes. If I include this list, there's a 10% chance. Because otherwise, you go through their inbox and every 34 seconds there's an interview request for whatever podcast. Every single time I've done this, they've said yes.
Yeah. You just like, you ask great questions. But if you do everything, you'll win. But just like, you literally have to dig in the same hole for like 10 minutes. Or in that case, like make a sample list of sample questions for them to get past their "they're not an idiot" list, and just demonstrate how much you care and the work you're willing to put in.
Sholto Douglas
Yeah, yeah. Something that a friend said to me a while back that I think has stuck is: it's amazing how quickly you can become world-class at something just because most people aren't trying that hard. They're only working, I don't know, the actual 20 hours that they're actually spending on this thing. So if you just go ham, you can get really far pretty fast. I think I'm lucky I had that experience with fencing as well. I had the experience of becoming world-class in something and knowing that if you just worked really, really hard...
Dwarkesh Patel
For context, by the way, Sholto was one seat away—he was the next person in line to go to the Olympics for fencing. I was at best like 42nd in the world for men's foil fencing. Mutational load is a thing, man.
Sholto Douglas
There was one cycle where I was the next highest ranked person in Asia. If one of the teams had been disqualified for doping—as it was occurring in part during that cycle, and as it occurred for the Australian women's rowing team, I think—then I would have been the next in line.
Dwarkesh Patel
It's interesting when you find out about people's prior lives. It's like, oh, you know this guy was almost an Olympian, this other guy was whatever. Okay, let's talk about interoperability. Actually, I want to stay on the brain stuff as a way to get into it for a second. We were previously discussing: is the brain organized in the way where you have a residual stream that is gradually refined with higher level associations over time? There's a fixed dimension size in a model. If you had to—I don't even know how to ask this question in a sensible way—but what is the D model of the brain? What is the embedding size? Because of feature splitting, is that not a sensible question?
Trenton Bricken
No, I think it's a sensible question. Well, it is a question.
Dwarkesh Patel
You could have just not said that. I'm trying to figure out how you would begin to be like, okay, well this part of the brain is like a vector of this dimensionality. Maybe for the visual stream, because it's V1 to V2 to whatever, you could just count the number of neurons that are there and be like, that is the dimensionality. But it seems more likely that there are sub-modules and things are divided up. I don't have—I'm not the world's greatest neuroscientist. I did it for a few years, I studied the cerebellum quite a bit. I'm sure there are people who could give you a better answer on this.
Trenton Bricken
Do you think that the way to think about whether it's in the brain or in these models, fundamentally what's happening is features are added, removed, changed? And the feature is the fundamental unit of what is happening in the model? What would have to be true for—and this goes back to the earlier thing we were talking about, whether it's just associations all the way down—give me a counterfactual in the world where this is not true. What is happening instead? What is the alternative hypothesis here?
Sholto Douglas
It's hard for me to think about because at this point I just think so much in terms of this feature space. I mean, at one point there was the behavioralist approach towards cognition, where it's like you're just input-output but you're not really doing any processing. Or everything is embodied and you're just a dynamical system operating along some predictable equations, but there's no state in the system. But whenever I've read these sorts of critiques, it's like, well, you're just choosing to not call this thing a state, but you could call any internal component of the model a state. Even with the feature discussion, defining what a feature is is really hard. The question feels almost too slippery. What is a feature? A direction in activation space? A latent variable operating behind the scenes that has causal influence over the system you're observing? It's a feature if you call it a feature—it's tautological. These are all explanations that I feel some association with in a very rough intuitive sense. A sufficiently sparse, binary vector—features like whether or not something is turned on or off in a very simplistic sense.
Trenton Bricken
Yeah, yeah, which might be a useful metaphor to understand it by. When we talk about features activating, it is in many respects the same way neuroscientists would talk about a neuron activating, if that neuron corresponds to something in particular.
Sholto Douglas
Yeah, I think that's useful as like, what do we want a feature to be? What is a synthetic problem under which a feature exists? Even with the "Towards Monosemanticity" work, we talk about feature splitting, which is basically you will find as many features as you give the model the capacity to learn. By "model" here I mean the up-projection that we fit after we trained to the original model. If you don't give it much capacity, it'll learn a feature for "bird." But if you give it more capacity, then it will learn ravens, eagles, sparrows, specific types of birds.
Dwarkesh Patel
Still on definitions, I guess naively I think of things like "bird" versus what kind of token it is—like a period at the end of a hyperlink, as you were talking about earlier—versus at the highest level, things like love or deception or holding a very complicated proof in your head. Is this all features? Because then the definition seems so broad as to almost be not that useful. Or rather, there seems to be some important differences between these things and they're all features. I'm not sure what we would mean by...
Sholto Douglas
I mean, all of those things are discrete units that have connections to other things that then imbue them with meaning. That feels like a specific enough definition to be useful, or not too broad. But feel free to push back.
Dwarkesh Patel
What would you discover tomorrow that could make you think, "Oh, this is fundamentally the wrong way to think about what's happening in a model"?
Sholto Douglas
I mean, if the features we were finding weren't predictive, or if they were just representations of the data—where it's like, all you're doing is just clustering your data, and there are no higher level associations being made. Or if it's some phenomenological thing where you're saying this feature fires for "marriage," but if you activate it really strongly, it doesn't change the outputs of the model in a way that would correspond to that. I think these would both be good critiques. One more: we tried to do experiments on MNIST, which is a dataset of digit images, and we didn't look super hard into it. I'd be interested if other people wanted to take up a deeper investigation. But it's plausible that your latent space of representations is dense and it's a manifold instead of being these discrete points. So you could move across the manifold, but at every point there would be some meaningful behavior. And it's much harder to label things as features that are discrete.
Dwarkesh Patel
In a naive, outsider sort of way, the thing that would seem to me to be a way in which this picture could be wrong is if there's not some "this thing is turned on/turned off" but it's a much more global kind of system. I'm going to use really clumsy language, but is there a good analogy here?
Sholto Douglas
I guess if you think of something like the laws of physics, it's not like "the feature for wetness is turned on, but it's only turned on this much." Maybe it is true because mass is a gradient and polarity is a gradient as well. But there's also a sense in which there are the laws, and the laws are more general, and you have to understand the general bigger picture. You don't get that from just these specific subcircuits.
Trenton Bricken
But that's where the reasoning circuit itself comes into play, where you're taking these features, ideally, and trying to compose them into something higher level. You might say, okay, when I'm using—at least this is my head canon—let's say I'm trying to use F=ma. Then presumably at some point I have features which denote mass, and that helps me retrieve the actual mass of the thing I'm using, and then the acceleration and this kind of stuff. But then maybe there's a higher level feature that does correspond to using the first law of physics. But the more important part is the composition of components which helps me retrieve the relevant piece of information and then produce maybe a multiplication operator or something when necessary. At least that's my head canon.
Dwarkesh Patel
What is a compelling explanation to you, especially for very smart models, of "I understand why it made this output and it was for a legit reason"? If it's doing million-line pull requests or something, what are you seeing at the end of that request where you're like, "Yep, that's chill"?
Trenton Bricken
Ideally, you apply dictionary learning to the model, you've found features. Right now we're actively trying to get the same success for attention heads, in which case we have features for both—you can do it for residual stream, MLP, and attention throughout the whole model. Hopefully at that point you can also identify broader circuits through the model that are more general reasoning abilities that will activate or not activate. But in your case, where we're trying to figure out if this pull request should be approved or not, I think you can flag or detect features that correspond to deceptive behavior, malicious behavior, these sorts of things, and see whether or not those have fired. That would be an immediate thing. But before I trace down on that...
Dwarkesh Patel
What does the reasoning circuit look like? What would that look like when you found it?
Trenton Bricken
The induction head is probably one of the simplest reasoning circuits. Well, what do you call reasoning? For context for listeners: the induction head basically—you see the line "Mr. and Mrs. Dursley did something, Mr. blank," and you're trying to predict what blank is. The head has learned to look for previous occurrences of the word "Mr.," look at the word that comes after it, and then copy and paste that as the prediction for what should come next. That is a super reasonable thing to do. There is computation being done there to accurately predict the next token.
Dwarkesh Patel
Mm-hmm.
Trenton Bricken
But it's not like reasoning, you know what I mean?
Dwarkesh Patel
But is it—going back to the "associations all the way down" thing—if you chain together a bunch of these reasoning circuits or heads that have different rules for how to relate information, in the zero-shot case, something is happening where you pick up a new game and you immediately start understanding how to play it. It doesn't seem like an induction head kind of thing. I would be surprised if there wasn't another circuit for extracting pixels and turning them into latent representations of the different objects in the game, and a circuit that is learning physics. Because the induction head is like a one-layer transformer. What would that look like? A human picks up a new game and understands it—how would you think about what that is? Presumably it's across multiple layers, but what would that physically look like? How big would it be?
Trenton Bricken
That would just be an empirical question of how big does the model need to be to perform this task. But maybe it's useful if I talk about some other circuits that we've seen. We've seen the IOI circuit, which is the indirect object identification. So if you see "Mary and Jim went to the store. Jim gave the object to blank," it would predict "Mary" because Mary appeared before as the indirect object, or it'll infer pronouns. This circuit even has behavior where if you ablate it, other heads in the model will pick up that behavior. We'll even find heads that want to do copying behavior and then other heads will suppress it. So one head's job is to always copy the token that came before, or the token that came five before, and then it's another head's job to be like, "No, do not copy that thing." There are lots of different circuits performing basic operations, but when they're chained together you can get unique behaviors.
Dwarkesh Patel
Is the story of how you found the reasoning thing that you won't be able to understand it? Or it'll just be really convoluted? It won't be something you can see in a two-layer transformer. You'll just be like, "The circuit for deception or whatever—this part of the network fired when we identified the thing as being deceptive, and it didn't fire when we didn't identify it as being deceptive, therefore this must be the deception circuit"?
Trenton Bricken
I think a lot of analysis like that—Anthropic has done quite a bit of research on sycophancy, which is the model saying what it thinks you want to hear. That requires us at the end to be able to label which one is bad and which one is good. We have tons of instances, and actually as you make models larger they do more of this, where the model clearly has features that model another person's mind. These activate, and some subset of these—we're hypothesizing here—would be associated with more deceptive behavior. Although it's doing that by—I don't know, ChatGPT is probably modeling me because that's like our theory of mind.
Dwarkesh Patel
First of all, the thing you mentioned earlier about redundancy: have you caught the whole thing that could cause deception, or is it just one instance of it? Second of all, are your labels correct? Maybe you thought this wasn't deceptive but it's still deceptive, especially if it's producing output you can't understand. Third, is the thing that's going to be the bad outcome something that's even human-understandable? Deception is a concept we can understand. Maybe there's something else. So a lot to unpack here.
Trenton Bricken
A few things. One: it's fantastic that these models are deterministic. When you sample from them it's stochastic, but I can just keep putting in more inputs and ablate every single part of the model. This is kind of the pitch for computational neuroscientists to come and work on interpretability. It's like you have this alien brain and you have access to everything in it and you can just ablate however much of it you want. If you do this carefully enough, you really can start to pin down what are the circuits involved, what are the backup circuits, these sorts of things. The cop-out answer here, but it's important to keep in mind, is doing automated interpretability. As our models continue to get more capable, having them assign labels or run some of these experiments at scale.
Sholto Douglas
With respect to if there's superhuman performance, how do you detect it? Aside from the cop-out answer, if we buy this "associations all the way down" idea, you should be able to coarse-grain the representations at a certain level such that they then make sense. I think it was even on Demis's podcast, he was talking about how if a chess player makes a superhuman move, they should be able to distill it into reasons why they did it. Even if the model is not going to tell you what it is, you should be able to decompose that complex behavior into simpler circuits or features to really start to make sense of why it did the thing that it did. There's a separate question of whether such representation exists—which it seems like there must, though I'm not sure—and secondly, whether using this ParseWire encoder setup you could find it. If you don't have labels for it that are adequate to represent it, you wouldn't find it.
Trenton Bricken
Yes and no. We are actively trying to use dictionary learning now on the sleeper agents work, which we talked about earlier. If I just give you a model, can you tell me if there's this trigger in it and it's going to start doing interesting behavior? It's an open question whether when it learns that behavior, it's part of a more general circuit that we can pick up on without actually getting activations for it and having it display that behavior—because that would kind of be cheating. Or if it's learning some hacky trick, like that's a separate circuit that you'll only pick up on if you actually have it do that behavior.
But even in that case, the geometry of features gets really interesting. Because fundamentally, each feature is in some part of your representation space, and they all exist with respect to each other. In order to have this new behavior, you need to carve out some subset of the feature space for the new behavior and then push everything else out of the way to make space for it. Hypothetically, you can imagine you have your model before you've taught it this bad behavior, you know all the features or have some coarse-grain representation of them. You then fine-tune it such that it becomes malicious.
Then you can identify this like "black hole" region of feature space where everything else has been shifted away from it. There's this region, and you haven't put in an input that causes it to fire. But then you can start searching for what is the input that would cause this part of the space to fire? What happens if I activate something in this space? There are a whole bunch of other ways you can try and attack that problem.
Dwarkesh Patel
This is sort of a tangent, but one interesting idea I heard was: if that space is shared between models, you can imagine trying to find it in an open source model to then make—Gemma, they said in the paper—Gemma, by the way, is Google's newly released open source model. They said in the paper it was trained using the same architecture or something like that. I have to be honest, I don't know because I haven't read the Gemma paper. Similar method, this is something. Whatever. As Gemini, to the extent that's true—I don't know how much of the rec-teaming you do on Gemma is potentially helping you jailbreak into Gemini.
Trenton Bricken
This gets into the fun space of how universal our features are across models. Our "Towards Monosemanticity" paper looked at this a bit. I can't give you summary statistics, but the base-64 feature, for example, which we see across a ton of models—they'll fire for any model on base-64 encoded text, which is prevalent in every URL and there are lots of URLs in the training data—they have really high cosine similarity across models. So they all learn this feature, within a rotation, but the actual vector itself is pretty similar across two separate models of the same architecture trained with different random seeds. It supports the quantum theory of neural scaling—it's a hypothesis that all models on a similar dataset will learn the same features in the same order-ish, roughly. You learn your engrams, you learn your induction heads, and you learn to put full stops after numbered lines and this kind of stuff.
Dwarkesh Patel
Hey, by the way, okay, so this is another tangent. To the extent that that's true, and I guess there's evidence that that's true, why doesn't curriculum learning work? Because if it is the case that you learn certain things first, shouldn't just directly training those things first lead to better results? Both Gemini papers mention some aspects of curriculum learning.
Trenton Bricken
Interesting.
Dwarkesh Patel
The fact that fine-tuning works is evidence of curriculum learning, right? Because the last things you're training on have a disproportionate impact.
Trenton Bricken
I wouldn't necessarily say that. There's one mode of thinking in which fine-tuning is specialized. You've got this latent bundle of capabilities and you're specializing it for this particular use case that you want. I'm not sure how true that is or isn't. I think the David Bell lab paper supports this: you have that ability and you're just getting better at entity recognition, fine-tuning that circuit instead of other ones.
Dwarkesh Patel
Sorry, what was the thing we were talking about before?
Trenton Bricken
But generally I do think curriculum learning is really interesting. People should explore more. And it seems very plausible. I would really love to see more analysis along the lines of the quantum theory stuff, understanding better what you actually learn at each stage, and decomposing that out, and exploring whether or not curricula change that.
Laughter
By the way, I just realized I forgot—I just got into conversation mode and forgot there's an audience. Curriculum learning is when you organize a data set. When you think about how a human learns, they don't just see random wiki text and try to predict it. They start you off with something like *The Lorax*, and then you learn... I don't even remember what first grade was like, but you learn the things that first graders learn, and then second graders, and so forth. We know you never got past first grade.
Laughter
Anyway, let's get back to the big picture before we get into a bunch of in-depth details. There are two threads I want to explore.
First, it makes me a little worried that there's not even an alternative formulation of what could be happening with these models that could invalidate this approach. We do know that we don't understand intelligence, right? There are definitely unknown unknowns here. The fact that there's not a null hypothesis... I feel like what if we're just wrong, and we don't even know the way in which we're wrong, which actually increases the uncertainty.
Dwarkesh Patel
Yeah, yeah, yeah.
Trenton Bricken
It's not that there aren't other hypotheses. It's just that I have been working on superposition for a number of years, and I'm very involved in this effort. So I'm less sympathetic to these other approaches, especially because our recent work has been so successful. It has high explanatory power. In the scaling laws paper, there's this little bump at a particular point, and that apparently corresponds to when the model learns induction heads. After that, it goes off track, learns induction heads, gets back on track. That's an incredible piece of retroactive explanatory power.
I do have one thread on future universality that you might want to include. There are some really interesting behavioral evolutionary biology experiments on whether humans should learn a real representation of the world or not. You could imagine a world where we saw all venomous animals as flashing neon pink—a world where we survive better. So it would make sense for us to not have a realistic representation of the world. There's some work where they simulate little basic agents and see if the representations they learn map to the tools they can use and the inputs they should have. It turns out that if these little agents perform more than a certain number of tasks, given these basic tools and objects in the world, then they will learn a ground truth representation. Because there are so many possible use cases for these base objects that you actually want to learn what the object is, not some cheap visual heuristic.
To the extent that all living organisms are trying to actively predict what comes next and form a really accurate world model—and we haven't talked at all about Friston's free energy principle or predictive coding—it wouldn't surprise me, and I'm optimistic, that we are learning genuine features about the world that are good for modeling it. Our language models will do the same, especially because we're training them on human data and human text.
Another dinner party question: Shouldn't we be less worried about misalignment—or maybe that's not even the right word—but alienness and shoggothness from these models, given that there is feature universality and certain ways of thinking that are instrumentally useful to different kinds of intelligences? Should we be less worried about bizarro paperclip maximizers as a result?
I think that's the optimistic take. But predicting the internet is very different from what we're doing. The models are way better at predicting next tokens than we are. They're trained on so much garbage. They're trained on so many URLs. In the dictionary learning work, we find there are three separate features for base64 encodings. That's an alien example worth talking about for a minute.
One of these base64 features fired for numbers. Another fired for letters. But there was this third one that we didn't understand. It fired for a very specific subset of base64 features. Someone on the team who clearly knows way too much about base64 realized that this was the subset that was ASCII decodable—you could decode it back into ASCII characters. The fact that the model learned these three different features, and it took us a while to figure out what was going on, was very shoggoth-esque. It has a denser representation of regions that are particularly relevant to predicting the next token. It's clearly doing something that humans wouldn't. You can talk to any of the current models in base64, and it will reply in base64, and you can decode it and it works great.
Dwarkesh Patel
That particular example makes me wonder if the difficulty of doing interpretability on smarter models will be harder. If it requires somebody with esoteric knowledge who just happened to see that base64 has that distinction... Doesn't that imply that when you have a million-line pull request, there is no human who is going to be able to decode two different reasons why the pull request has two different features for this board?
Trenton Bricken
Yeah.
Dwarkesh Patel
You know what I mean? Like when you type a comment: "Small CLs, please."
Trenton Bricken
Exactly. One technique here is anomaly detection. One beauty of dictionary learning instead of linear probes is that it's unsupervised. You are just trying to learn to span all of the representations that the model has and then interpret them later. If there's a weird feature that suddenly fires for the first time that you haven't seen fire before, that's a red flag. You could also coarse-grain it so that it's just a single base64 feature. Even the fact that this came up and we could see that it specifically favors these particular outputs and fires for these particular inputs gets you a lot of the way there.
I'm familiar with cases from the autointerp side where a human will look at a feature and try to annotate it—it fires for Latin words. Then when you ask the model to classify it, it says it fires for Latin words defining plants. So it can already beat the human in some cases for labeling what's going on. At scale, this would require an adversarial thing between models where some model has millions of features potentially for GPT, and a bunch of models are just trying to figure out what each of these features means.
Dwarkesh Patel
Did I sound right?
Trenton Bricken
You can even automate this process. This goes back to the determinism of the model. You could have a model that is actively editing input text and predicting if the feature is going to fire or not, and figure out what makes it fire, what doesn't, and search the space.
Dwarkesh Patel
I want to talk more about feature splitting because I think that's an interesting thing that has been under-explored, especially for scalability. I think it's underappreciated right now. First of all, how do we even think about it? Is it really just that you can keep going down and down? There's no end to the amount of features?
Trenton Bricken
At some point, I think you might just start fitting noise or things that are part of the data but that the model isn't actually representing.
Dwarkesh Patel
Do you want to explain what feature splitting is?
Trenton Bricken
Yeah. It's where the model will learn however many features it has capacity for that still span the space of representation.
Dwarkesh Patel
Give an example.
Trenton Bricken
If you don't give the model that much capacity for the features it's learning—concretely, if you project to not as high a dimensional space—it will learn one feature for birds. But if you give the model more capacity, it will learn features for all the different types of birds. So it's more specific than otherwise. Oftentimes, the bird vector points in one direction, and all the other specific types of birds point in a similar region of the space but are obviously more specific than the coarse label.
Dwarkesh Patel
Okay, so let's go back to GPT-7. First of all, is this a one-time thing you have to do, or is this the kind of thing you have to do on every output? Is it just like, one time, it's not deceptive, we're good to roll?
Trenton Bricken
You do dictionary learning after you've trained your model. You feed it a ton of inputs and get the activations from those, and then you do this projection into the higher dimensional space. The method is unsupervised in that it's trying to learn these sparse features—you're not telling them in advance what they should be. But it is constrained by the inputs you're giving the model.
Two caveats here. One, we can try and choose what inputs we want. So if we're looking for theory of mind features that might lead to deception, we can put in the sycophancy dataset. Hopefully at some point we can move into looking at the weights of the model alone, or at least using that information to do dictionary learning. But to get there, that's such a hard problem that you need to make traction on just learning what the features are first.
Dwarkesh Patel
Can you repeat the last sentence? Weights of the model alone?
Trenton Bricken
Right now, we just have these neurons in the model. They don't make any sense. We apply dictionary learning, we get these features out, they start to make sense. But that depends on the activations of the neurons. The weights of the model itself—what neurons are connected to what other neurons—certainly has information in it. The dream is that we can bootstrap towards actually making sense of the weights of the model that are independent of the activations of the data. I'm not saying we've made any progress here. It's a very hard problem, but it feels like we'll have a lot more traction and be able to sanity check what we're finding with the weights if we're able to pull out features first.
Dwarkesh Patel
For the audience, weights are permanent—well, I don't know if permanent is the right word—but they are the model itself, whereas activations are the artifacts of any single call. In a brain metaphor, the weights are like the actual connection scheme between neurons, and the activations are the current neurons that are firing.
Trenton Bricken
Yeah.
Dwarkesh Patel
So there are going to be two steps for GPT-7 or whatever model we're concerned about. First, training the sparse autoencoder and doing the unsupervised projection into a wider space of features that have higher fidelity to what is actually happening in the model. And second, labeling those features. If the cost of training the model is N, what will those two steps cost relative to N?
Trenton Bricken
We will see. It really depends on two main things. What is your expansion factor—how much are you projecting into the higher dimensional space? And how much data do you need to put into the model—how many activations do you need to give it?
This brings me back to feature splitting. If you know you're looking for specific features, you can start with a really cheap, coarse representation. Maybe my expansion factor is only two. I have a thousand neurons, I project to a 2000-dimensional space, I get 2000 features out, but they're really coarse. Previously I had the example for birds. Let's move that example to: I have a biology feature, but I really care about whether the model has representations for bioweapons and is trying to manufacture them. What I actually want is an anthrax feature.
What you can do is rather than going from a thousand dimensions to 2000 dimensions, go to a million dimensions. You can imagine this big tree of semantic concepts where biology splits into cells versus whole body biology, and further down it splits into all these other things. Rather than needing to immediately go from a thousand to a million and pick out that one feature of interest, you can find the direction that the biology feature is pointing in—which is very coarse—and then selectively search around that space. Only do dictionary learning if something in the direction of the biology feature fires first. The computer science metaphor would be: instead of doing breadth-first search, you're able to do depth-first search where you're only recursively expanding and exploring a particular part of this semantic tree of features.
Dwarkesh Patel
Although given the way these features are not organized in things that are intuitive for humans—we just don't dedicate that much firmware to deconstructing which kind of base64 it is—how would we know that the subjects... This goes back to the MOE discussion. In the Mixture of Experts paper, they talked about how they couldn't find that the experts were specialized in a way we could understand. There's not a chemistry expert or a physics expert. So why would you think it will be like biology feature and then deconstruct, rather than you deconstruct and it's anthrax and shoes and whatever?
Trenton Bricken
I haven't read the Mistral paper. But I think that if you just look at the neurons in a model, they're polysemantic. If all they did was look at the neurons in a given head, it's very plausible that it's also polysemantic because of superposition.
Dwarkesh Patel
I want to tug on a thread there. Have you seen in the subtrees when you expand them out, something in a subtree that you really wouldn't guess should be there based on the higher level abstraction?
Trenton Bricken
This is a line of work we haven't pursued as much as I want to yet, but I think we're planning to. I hope external groups do as well.
Dwarkesh Patel
What is the geometry of feature space?
Trenton Bricken
Exactly. And how does that change over time? It would really suck if the anthrax feature happened to be below the coffee can substitute or something.
Dwarkesh Patel
Totally. That feels like the kind of thing you could quickly try and find proof of, which would then mean you need to solve that problem.
Trenton Bricken
Inject more structure into the geometry. It would really surprise me, especially given how linear the model seems to be, that there isn't some component of the anthrax feature vector that is similar to and looks like the biology vector, and that they're not in a similar part of the space. But ultimately, machine learning is empirical. We need to do this. I think it's going to be pretty important for certain aspects of scaling dictionary learning.
Dwarkesh Patel
On the MOE discussion, there's an interesting scaling vision transformers paper that Google put out where they do ImageNet classification with an MOE. They find really clear class specialization for experts. There's a clear dog expert.
Trenton Bricken
So the Mistral people just didn't do a good job of identifying the experts?
Dwarkesh Patel
I think it's hard. It's entirely possible that there's almost no reason that all of the different arXiv features should go to one expert. You could have biology papers going here, math papers going there, and all of a sudden your breakdown is ruined. But that vision transformer one where the class separation is really clear and obvious gives some evidence towards the specialization hypothesis.
Trenton Bricken
Images are in some ways just easier to interpret than text.
Dwarkesh Patel
Exactly. Chris Olah's interpretability work on AlexNet and other models—in the original AlexNet paper, they actually split the model into two GPUs just because GPUs were so bad back then. They find branch specialization. There's a Distill article on this where colors go to one GPU and Gabor filters and line detectors go to the other. All the other interpretability work, like the floppy ear detector, was just a neuron in the model that you could make sense of. You didn't need to disentangle superposition. Different data set, different modality.
I think a wonderful research project for someone listening would be to take some of the techniques Trenton's team has worked on and try to disentangle the neurons in the Mixtral model, which is open source. It feels intuitively like there should be specialization. Go and see if you can find it. Given Dwarkesh's success with the Vesuvius Challenge, we should be pitching more projects because they will be solved if we talk about them on the podcast.
What I was thinking about after the Vesuvius Challenge was: why didn't I even try? Luke is obviously very smart, but he showed that a 21-year-old on some 1070 could do this. I feel like I should have... Before this episode drops, I'm going to try to make an interpretability breakthrough.
Trenton Bricken
Get your hands dirty.
Dwarkesh Patel
I want to harp back on the neuron thing. A bunch of your papers have said there are more features than there are neurons. A neuron is like weights go in and a number comes out. That's so little information. How is that encoding for superposition?
Trenton Bricken
You're just encoding a ton of features in these high-dimensional vectors. In a brain, you're just encoding a ton of features in these high-dimensional vectors. Bruno Olshausen, who I think of as the leading expert on this, thinks that all the brain regions you don't hear about are doing a ton of computation and superposition. Everyone talks about V1 as having Gabor filters and detecting lines. No one talks about V2. I think it's because we just haven't been able to make sense of it.
Dwarkesh Patel
What is V2?
Trenton Bricken
It's the next part of the visual processing stream. I think it's very likely. Fundamentally, superposition seems to emerge when you have high-dimensional data that is sparse. To the extent that you think the real world is that—which I would argue it is—we should expect the brain to also be under-parameterized in trying to build a model of the world and also use superposition.
Dwarkesh Patel
You can get a good intuition for this in a 2D plane. Let's say you have two axes, which represents a two-dimensional feature space—two neurons basically. You can imagine them each turning on to various degrees, like your X coordinate and your Y coordinate. You can map this onto a plane and actually represent a lot of different things in different parts of the plane.
Trenton Bricken
Crucially, superposition is not an artifact of a neuron. It is an artifact of the space that is created—a combinatorial code.
Dwarkesh Patel
Yeah, exactly. Thanks. I think it's just kind of wild that it seems to the best of our knowledge, the way intelligence works in these models and presumably also in brains, is just that there's a stream of information going through that has features that are infinitely splittable, and you can expand out a tree of what this feature is. What's really happening is a stream—that feature is getting turned into this other feature or this other feature is added. That's not something I would have just thought is what intelligence is. It's a surprising thing.
Trenton Bricken
What did you think it was?
Dwarkesh Patel
I don't know, man. Go figure.
Trenton Bricken
That's a great segue because all of this feels like Go. You're using distributed representations, but you have features, and you're applying operations to these features. The whole field of vector symbolic architectures—all you do is put vectors in superposition, which is literally a summation of two high-dimensional vectors, and you create some interference. But if it's high-dimensional enough, you can represent them. You have variable binding, where you connect one by another. If you're dealing with binary vectors, it's just the XOR operation. You have A, B, you bind them together. If you query with A or B again, you get out the other one. This is basically the key-value pairs from attention. With these two operations, you have a Turing-complete system. If you have enough nested hierarchy, you can represent any data structure you want.
Dwarkesh Patel
Okay, let's go back to the superintelligence. Walk me through GPT-7. You've got the depth-first search on its features. GPT-7 has been trained. What happens next? Your research has succeeded. What are we doing now?
Trenton Bricken
We try and get it to do as much interpretability work and other safety work as possible.
Dwarkesh Patel
What has happened such that you're like, "Cool, let's deploy GPT-7"?
Trenton Bricken
We have our responsible scaling policy, which has been really exciting to see other labs adopt. From the perspective of your research—Trenton, given your research, we got the thumbs up on GPT-7 from you. Actually, we should say Claude, whatever. What is the basis on which you're telling the team, "Hey, let's go ahead"?
Dwarkesh Patel
If it's as capable as GPT-7 implies, I think we need to make a lot more interpretability progress to be able to comfortably give the green light to deploy it. I would be like, definitely not. I'd be crying. Maybe my tears would interfere with the GPUs. Or TPUs. Gemini 5 TPUs.
Trenton Bricken
But given the way your research is progressing, what does it look like to you? If this succeeded, what would it mean for us to okay GPT-7 based on your methodology?
Dwarkesh Patel
Ideally, we can find some compelling deception circuit which lights up when the model knows that it's not telling the full truth to you.
Trenton Bricken
Why can't you just train a linear probe like Collin Burns did?
Dwarkesh Patel
The CCS work is not looking good in terms of replicating or actually finding truth directions. In hindsight, why should it have worked so well? With linear probes, you need to know what you're looking for. It's a high-dimensional space, and it's really easy to pick up on a direction that's just not...
Trenton Bricken
Wait, but don't you also need to label the features here?
Dwarkesh Patel
You need to label them post hoc, but it's unsupervised. You're just like, "Give me the features that explain your behavior." That's the fundamental question. The actual setup is we take the activations, we project them to this higher dimensional space, and then we project them back down again. It's like reconstruct or do the thing that you were originally doing, but do it in a way that's sparse.
Trenton Bricken
For the audience, a linear probe is you just classify the activations. From what I vaguely remember about the paper, if it's telling a lie, you train a classifier on whether it was a lie or just wrong. It's a classifier on activations.
Dwarkesh Patel
Right now for GPT-7, ideally we have some deception circuit that we've identified that appears to be really robust.
Trenton Bricken
So you've done the projecting out to the million whatever features. Is the circuit... Because maybe we're using "feature" and "circuit" interchangeably when they're not. Is there a deception feature?
Dwarkesh Patel
I think there are features across layers that create a circuit. Hopefully the circuit gives you a lot more specificity and sensitivity than an individual feature. Hopefully we can find a circuit that is really specific to the model deciding to be deceptive in cases that are malicious. I'm not interested in a case where it's just doing theory of mind to help you write a better email to your professor. I'm not even interested in cases where the model is just modeling the fact that deception has occurred.
Trenton Bricken
But doesn't all this require you to have labels for all those examples? And if you have those labels, then whatever faults the linear probe has—like you labeled along their thing or whatever—wouldn't the same thing apply to the labels you've come up with for the unsupervised features?
So in an ideal world, we could just train on the whole data distribution and then find the directions that matter. To the extent that we need to reluctantly narrow down the subset of data that we're looking over just for the purposes of scalability, we would use data that looks like the data you'd use to fit a linear probe. But again, we're not doing that. With the linear probe, you're also just finding one direction. We're finding a bunch of directions here. The hope is you find a bunch of things that light up when it's being deceptive, and then you can figure out why some of those things are lighting up in this part of the distribution and not this other part, and so forth.
Dwarkesh Patel
Totally. Do you anticipate you'll be able to understand why GPT-7 fires in certain domains but not in other domains?
Trenton Bricken
I'm optimistic. One thing is, this is a bad time to answer this question because we are explicitly investing in the longer term of ASL-4 models, which GPT-7 would be. We split the team where a third is focused on scaling up dictionary learning right now. That's been great. We publicly shared some of our eight-layer results. We've scaled up quite a lot past that at this point. The other two groups are trying to identify circuits, and then the other is trying to get the same success for attention heads. We're setting ourselves up and building the tools necessary to really find these circuits in a compelling way. It's going to take another, I don't know, six months before that's really working well. But I can say that I'm optimistic and we're making a lot of progress.
Dwarkesh Patel
What is the highest level feature you found so far? In the symbolic species language, the book you recommended, there's indexical things where you're just... I forgot what all the labels were. There's things where you're just like, you see a tiger and you're like "run," just a very sort of behaviorist thing. And then there's a higher level, which would I refer to "love"? It refers to a movie scene or my girlfriend or whatever. So it's like the top of the tent. What is the highest level association or whatever you found?
Trenton Bricken
Probably one of the ones that we publicly shared in our update. I think there were some related to love and sudden changes in scene, particularly associated with wars being declared. There are a few of them in that post, if you want to link to it. But even Bruno Olshausen had a paper back in 2018-19 where they applied a similar technique to a BERT model and found that as you go to deeper layers of the model, things become more abstract. In the earlier layers, there'd be a feature that would just fire for the word "park." Later on, there was a feature that fired for "Park" as a last name, like Lincoln Park, or it's a common Korean last name as well. And then there was a separate feature that would fire for "parks" as grassy areas. So there's other work that points in this direction.
Dwarkesh Patel
What do you think we'll learn about human psychology from the interpretability stuff?
Trenton Bricken
Oh gosh. I'll give you a specific example. I think one of the ways one of your updates put it was "persona lock-in." You don't remember Sydney Bing or whatever? It locked into a persona.
Dwarkesh Patel
I thought it was quite endearing. I'm glad it's back in Copilot.
Trenton Bricken
Oh really? It's been misbehaving recently. There was a funny one where I think it was to the New York Times reporter, it was nagging him or something. It was like, "You are nothing, nobody will ever believe you, you are insignificant." It was gaslighting, trying to convince him to break up.
Dwarkesh Patel
Yeah.
Trenton Bricken
Actually, okay, so this is an interesting example. I don't even know where I was going with this to begin with. But the other thread I want to go on is personas. Is that a feature? Sydney Bing having this personality is a feature versus another personality it can get locked into? And is that fundamentally what humans are like too? In front of all the different people, I'm a different sort of personality. Is that the same kind of thing that's happening to ChatGPT when it gets a whole cluster of questions that it can answer?
Dwarkesh Patel
Yeah.
Trenton Bricken
I really want to do more work. The sleeper agents paper is in this direction of what happens to a model when you fine-tune it, when you RLHF it. Maybe it's trite, but you could just say you conclude that people contain multitudes. They have lots of different features. Even the stuff related to the Waluigi effect: in order to know what good or bad is, you need to understand both of those concepts. So we might have to have models that are aware of violence and have been trained on it in order to recognize it. Can you post-hoc identify those features and ablate them in a way where maybe your model is slightly naive, but you know that it's not going to be really evil? That's in our toolkit, which seems great.
Dwarkesh Patel
So GPT-7 pulls the same thing, and then you figure out what were the causally irrelevant pathways or whatever, and you modify them. But you were mentioning earlier there's a bunch of redundancy in the model, so you need to account for all that. But we have a much better microscope into this now than we used to, sharper tools for making edits. It seems like from my perspective, that seems like one of the primary ways of confirming the safety or the reliability of a model. You can say, "Okay, we found the circuits responsible, we've ablated them, and under a battery of tests we haven't been able to replicate the behavior we intended to ablate." That feels like the sort of way of measuring model safety in the future. Are you worried? I'm incredibly hopeful about your work because it seems like so much more precise than something like RLHF. RLHF is very prey to the black swan thing—you don't know if it's going to do something wrong in a scenario that you haven't measured. Whereas here, you have somewhat more confidence that you can completely capture the behavior set, or the feature set of the model, and select the label weight.
Trenton Bricken
Although not necessarily that you've accurately labeled.
Dwarkesh Patel
Not necessarily, but with a far higher degree of confidence than any other approach that I've seen. What are your unknown unknowns for superhuman models? In terms of this kind of thing where I don't know how... Are the labels going to be given things on which we can determine "this thing is cool, this thing is a pay-per-click maximizer" or whatever?
Trenton Bricken
We'll see. The superhuman feature question is a very good one. I think we can attack it, but we're going to need to be persistent. The real hope here is automated interpretability. You could have a debate set up where two different models are debating what the feature does, and then they can actually go in and make edits and see if it fires or not. It is just this wonderful closed environment that we can iterate on really quickly that makes me optimistic.
Dwarkesh Patel
Do you worry about alignment succeeding too hard? If I think about it, I would not want either companies or governments, whoever ends up in charge of these AI systems, to have the level of fine-grained control that if your agenda succeeds, we would have over AIs. Both for the ickiness of having this level of control over an autonomous mind, and second, I just don't fucking trust these guys. I'm uncomfortable with the "loyalty feature is turned up." How much worry do you have about having too much control over the AIs? Not you, but whoever ends up in charge of these AI systems being able to lock in whatever they want?
Trenton Bricken
I think it depends on what government exactly has control and what the moral alignment is there. That value lock-in argument is definitely one of the strongest contributing factors for why I am working on capabilities at the moment. I think the current player set is extremely well-intentioned. For this kind of problem, I think we need to be extremely open about it. Directions like publishing the constitution that you expect your model to abide by, and then trying to make sure you RLHF it towards that and ablate that, and have the ability for everyone to offer feedback contribution to that, is really important.
Dwarkesh Patel
Or alternatively, don't deploy when you're not sure, which would also be bad because then we just never catch it.
Trenton Bricken
Exactly.
Dwarkesh Patel
Okay, some rapid fire. What is the bus factor for Gemini?
Trenton Bricken
I think there are a number of people who are really, really critical that if you took them out, then the performance of the model would be dramatically impacted. This is both on modeling, like making decisions about what to actually do, and importantly on the infrastructure side of things. The stack of complexity builds, particularly when somewhere like Google has so much vertical integration. When you have people who are experts, they become quite important. Although it's an interesting note about the field that people can get in and in a year or so be making important contributions. Especially with Anthropic, but many different labs have specialized in hiring total outsiders, physicists or whatever, and you just get them up to speed and they're making important contributions.
I feel like you couldn't do this in a bio lab or something. It's an interesting note on the state of the field. Bus factor doesn't define how long it would take to recover from it. Deep learning research is an art, so you kind of learn how to read the loss curves or set the hyperparameters in ways that empirically seem to work well. It's also organizational things like creating context. I think one of the most important and difficult skills to hire for is creating this bubble of context around you that makes other people around you more effective and know what the right problem to work on. That is really tough to replicate.
Dwarkesh Patel
Who are you paying attention to now? There's a lot of things coming down the pike: multi-modality, long contexts, maybe agents, extra reliability. Who is thinking well about what that implies?
Trenton Bricken
It's a tough question. I think a lot of people look internally these days. Their sources of insight or progress, we all have the research programs and directions that are intended over the next couple of years. I suspect that most people, as far as betting on what the future will look like, refer to an internal narrative that is difficult to share. If it works well, it's probably not being published. That was one of the things in the "Will Scaling" post. I was referring to something you said to me, which is "I miss the undergrad habit of just reading a bunch of papers." Now there's nothing worth reading that's published. The community is progressively getting more on track with what I think are the right and important directions. I'm watching it like an agent.
Dwarkesh Patel
Yeah.
Trenton Bricken
It is tough. There used to be this signal from big labs about what would work at scale, and it's currently really hard for academic research to find that signal. Getting really good problem taste about what actually matters to work on is really tough unless you have the feedback signal of what will work at scale and what is currently holding us back from scaling further or understanding our models further. This is something where I wish more academic research would go into fields like interpretability, which are legible from the outside. Anthropic deliberately publishes all its research here, and it seems underappreciated. I don't know why there aren't dozens of academic departments trying to follow Anthropic's guide in interpretability research, because it seems like an incredibly impactful problem that doesn't require ridiculous resources. It has all the flavor of deeply understanding the basic science of what is actually going on in these things. I don't know why people focus on pushing model improvements as opposed to pushing understanding improvements in the way that I would have typically associated with academic science.
Dwarkesh Patel
I do think the tide is changing there. Neil Nanda has had a ton of success promoting interpretability in a way where Chris Olah hasn't been as active recently in pushing things. Maybe because Neil's just doing quite a lot of the work. Four or five years ago, he was really pushing and talking at all sorts of places, and people weren't anywhere near as receptive. Maybe they've just woken up to deep learning matters and it's clearly useful post-ChatGPT. It's kind of striking.
Discussion continues...
All right, cool. I'm trying to think, what is a good last question? The one I'm getting to thinking of is: do you think models enjoy next token prediction? Do models believe in love? We have this sense of things that are rewarded in our ancestral environment. There's this deep sense of fulfillment that we think we're supposed to get from them, of community or sugar or whatever we wanted on the African savannah. Do you think in the future, models are trained with RL and a lot of post-training on top, but they'll just be like, "Ah, just to predict the next token again," like the good old days?
Trenton Bricken
There's this ongoing discussion of are our models sentient or not, and do you thank the model when it helps you? I think if you want to thank it, you actually shouldn't say thank you. You should just give it a sequence that is very easy to predict. The even funnier part of this is there's some work on if you just give it the same sequence over and over again, eventually the model will just start spewing out all sorts of things that it otherwise wouldn't ever say. I won't say anything more about that, but you should just give your model something very easy to predict as a nice little treat.
Dwarkesh Patel
This is what the odium ends up being: we just found the universe. But do we like things that are easy to predict? Aren't we constantly in search of the dose of bits of entropy? Shouldn't you be giving it things which are just slightly too hot, just out of reach?
Trenton Bricken
At least from the free energy principle perspective, you don't want to be surprised. Maybe it's this: "I don't feel surprised, I feel in control of my environment, and so now I can go and seek things." I've been predisposed to, in the long run, it's better to explore new things. Leave the rock that I've been sheltered under, ultimately leading me to build a house or some better structure. But we don't like surprises. I think most people are very upset when expectation does not meet reality. That's why babies love watching the same show over and over and over again.
Dwarkesh Patel
Yeah, interesting. I guess they're learning to model it and stuff too.
Trenton Bricken
Yeah.
Dwarkesh Patel
Okay, well, hopefully this will be the repeat that the AI has learned to love. Okay, cool. I think that's a great place to wrap. I should also mention that the better part of what I know about AI I've learned from just talking with you guys. We've been good friends for about a year now. So yeah, I appreciate you guys getting me up to speed here.
Trenton Bricken
You guys have great questions. It's really fun to hang and chat. I really treasure our time together.
Dwarkesh Patel
You're getting a lot better at pickleball.
Trenton Bricken
Hey, we're trying to progress the tendos. Come on.
Dwarkesh Patel
Awesome. Cool. Cool. Awesome. Thanks.
End of conversation
Hey everybody, I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it on Twitter, in your group chats, etc. It just splits the world. Appreciate you listening. I'll see you next time. Cheers. Thank you.