John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

Dwarkesh Patel: Today, I have the pleasure to speak with John Schulman, who is one of the co-founders of OpenAI and leads the post-training team here. He also led the creation of ChatGPT and is the author of many of the most important and widely cited papers in AI and RL, including PPO and many others. John, really excited to chat with you. Thanks for coming on the podcast.

John Schulman: Thanks for having me on the podcast. I'm a big fan.

Dwarkesh Patel: Oh, thank you. Thank you for saying that. So the first question I had is we have these distinctions between pre-training and post-training beyond what is actually happening in terms of loss function and training regimes. I'm just curious, taking a step back conceptually, what kind of thing is pre-training creating? What does post-training do on top of that?

John Schulman: In pre-training, you're basically training to imitate all of the content on the internet or on the web, including websites and code and so forth. So you get a model that can basically generate content that looks like random web pages from the internet. The model is also trained to maximize likelihood where it has to put a probability on everything. The objective is basically predicting the next token given the previous tokens. Tokens are like words or parts of words. Since the model has to put a probability on it and we're training to maximize log probability, it ends up being very calibrated. So it can not only generate all of the content of the web, it can also assign probabilities to everything. The base model can effectively take on all these different personas or generate all these different kinds of content. Then when we do post-training, we're usually targeting a narrower range of behavior where we basically want the model to behave like this kind of chat assistant. It's a more specific persona where it's trying to be helpful. It's not trying to imitate a person. It's answering your questions or doing your tasks. We're optimizing on a different objective, which is more about producing outputs that humans will like and find useful, as opposed to just trying to imitate this raw content from the web.

Dwarkesh Patel: Okay. I think maybe I should take a step back and ask, right now we have these models that are pretty good at acting as chatbots. Taking a step back from how these processes were currently, what would the models released by the end of the year be capable of doing? What do you see the progress looking like five years from now?

John Schulman: I think the models will get quite a bit better. In the course of five years, I think even in one or two years, we'll find that you can use them for a lot more involved tasks than they can do now. For example, right now you could imagine having the models carry out a whole coding project instead of maybe giving you one suggestion on how to write a function. You could imagine giving the model high-level instructions on what to code up, and it will go and write many files, test it, look at the output, iterate on that a bit. So just much more complex tasks.

Dwarkesh Patel: And fundamentally, the unlock is that it can act coherently for long enough to write multiple files of code? What has changed between now and then?

John Schulman: I would say this will come from some combination of just training the models to do harder tasks like this. Right now, most of the training data is more like doing single steps at a time. I would expect us to do more training for the models to carry out these longer projects. Any kind of training, like doing RL to learn how to do these tasks, whether you're supervising the final output or supervising each step, is going to make them a lot better. Since the whole area is pretty new, there's just a lot of low-hanging fruit in doing this kind of training. Also, I would expect that as the models get better, they're just better at recovering from errors. They're better at dealing with edge cases. When things go wrong, they know how to recover. The models will be more sample efficient, so you don't have to collect a ton of data to teach them how to get back on track. Just a little bit of data, or their generalization from other abilities, will allow them to get back on track, whereas current models might just get stuck and get lost.

Dwarkesh Patel: I'm not sure I understood how the generalization helps you get back on track. Can you say more about that? I'm not sure I got how those two concepts are connected.

John Schulman: They're not directly connected. If you collect a diverse data set, you're going to get a little bit of everything in it. If you have models that generalize really well, even if there's just a couple examples of getting back on track, or even if in the pre-training there are examples of getting back on track, then the model will be able to generalize from those other things it's seen to the current situation. If you have weaker models, you might be able to get them to do almost anything with enough data, but you might have to put a lot of effort into a particular domain or skill. For a stronger model, it might just do the right thing without any training data or any effort.

Dwarkesh Patel: Do you have some intuition about this? Right now these models can maybe act coherently for five minutes. We want them to be able to do tasks that for a human would take an hour, then a week, then a month, and so forth. To get from each of these benchmarks, is it going to be that each one takes 10x more compute, analogous to the current scaling laws for pre-training? Or is it going to be a much more streamlined process because you're already more sample efficient and you can just go to the years of carrying out tasks?

John Schulman: At a high level, I would agree that longer horizon tasks are going to require more model intelligence to do well and are going to be more expensive to train for. I'm not sure I would expect there to be a really clean scaling law unless you set it up in a very careful way or design the experiment in a certain way. There might end up being some phase transitions where once you get to a certain level, you can deal with much longer tasks. For example, when people do planning at different time scales, I'm not sure they use completely different mechanisms. We probably use the same mental machinery if we're thinking about one month from now, one year from now, or 100 years from now. We're not actually doing some kind of reinforcement learning where we need to worry about a discount factor that covers that timescale. Using language, you can describe all of these different timescales, and then you can do things like make a plan. In the moment, you can try to make progress towards your goal, whether it's a month away or 10 years away. So I might expect the same out of models where there's some kind of capabilities that work at multiple scales.

Dwarkesh Patel: Okay, correct me if this is wrong, but it seems like that implies that right now we have models that are, on a per-token basis, pretty smart. They might be as smart as humans on a per-token basis, the smartest humans. The thing that prevents them from being as useful as they could be is that five minutes from now, they're not going to be writing your code in a way that's coherent and aligns with the broader goals you have for your project. If it's the case that once you start this long horizon RL training regime, it immediately unlocks your ability to be coherent for longer periods of time, should we be predicting something that is human level as soon as that regime is unlocked? And if not, then what is remaining after you can plan for a year and execute projects that take that long?

John Schulman: It's not totally clear what we're going to see once we get into that regime and how fast progress will be. That's still uncertain. I wouldn't expect everything to be immediately solved by doing any training like this. I would think there will be other miscellaneous deficits that the models have that cause them to get stuck or not make progress or make worse decisions than humans. So I wouldn't say I expect that this one little thing will unlock all capabilities, but it's not clear. Some improvement in the ability to do long horizon tasks might go quite far.

Dwarkesh Patel: Would you say it's plausible or quite likely that there will be other reasons why there might be bottlenecks? I'm also kind of curious, what would be the nature of the bottlenecks? It has all these representations from pre-training, now it can act coherently for a long period of time because of long horizon RL. What's remaining?

John Schulman: Maybe there's some other experience that human experts bring to different tasks, like having some taste or dealing with ambiguity better. I could imagine that if we want to do something like research, those kinds of considerations come into play. There are going to be just mundane limitations around the affordance of the model, like whether it can use UIs and the physical world, or having access to things. I think there might be a lot of mundane barriers that are probably not going to last that long, but would initially slow down progress.

Dwarkesh Patel: The websites that are designed for these AIs, once they're much more multimodal or at least trained on more multimodal data, will they be in any way different from the ones we have for humans? The UIs that will be needed, compensating for their strengths and weaknesses, how would that look different from the current UIs we have for humans?

John Schulman: That's an interesting question. I would expect that models will be able to use websites that are designed for humans just by using vision, when the vision capabilities get a bit better. So there wouldn't be an immediate need to change them. On the other hand, some websites that are going to benefit a lot from AIs being able to use them will probably want to design better UXs for AIs. I'm not sure exactly what that would mean, but probably assuming that our models are still better in text mode than reading text out of images, you'd probably want to have a good text-based representation for the models, and also a good indication of what are all the things that can be interacted with. But I wouldn't expect the web to get totally redesigned to have APIs everywhere, because I'd expect that we can get models to use the same kind of UIs that humans use. I guess that's been the big lesson of language models, right? That they can act in the similar affordances that humans have.

Dwarkesh Patel: The point you made earlier about this process could be more sample efficient because it could generalize from its experiences in pre-training of how to get unstuck in different scenarios. I'm curious what the strongest evidence of this kind of generalization and transfer you've seen is. The big question about the future abilities of models is how much generalization is happening. Is there something that feels really compelling to you, where you really learned something that you wouldn't expect it to learn from the generalization?

John Schulman: There have definitely been some interesting instances of generalization in post-training. One well-known phenomenon is if you do all your fine-tuning with English data, you'll automatically have the model also behave well in other languages. If you train the assistant on English data, it'll also do something reasonable in Spanish. Sometimes you might get the wrong behavior in terms of whether it replies in English or replies in Spanish, but usually you get the right behavior there as well. You get it to respond in Spanish to Spanish queries. That's one interesting instance of generalization where you just sort of latch onto the right helpful persona, and then you automatically do the right thing in different languages. We've seen some version of this with multimodal data where if you do text-only fine-tuning, you also get reasonable behavior with images. Early on in ChatGPT, we were trying to fix some issues in terms of the model understanding its own limitations. Early versions of the model would think they could send you an email or call you an Uber or something. The model would try to play the assistant and say, "Oh yeah, of course I sent that email," and obviously it didn't. So we started collecting a little data to fix those problems, and we found that a tiny amount of data did the trick, even when you mix it together with everything else. I don't remember exactly how many examples, but something like 30 examples showing this general behavior of explaining that the model doesn't have this capability, and that generalized pretty well to all sorts of capabilities we didn't train for.

Dwarkesh Patel: Okay, I still want to go back to this because I'm not sure I understood. If you have this model that is trained to be coherent for longer periods of time, does that imply that unless there are these other bottlenecks, which may or may not be there by next year, you could have models that are potentially human level? You're interacting with this as a colleague, like an ASI, interacting with a human colleague. You can tell them to go do stuff and they go get it done. What seems wrong with that picture? Is this the capabilities you think might be possible?

John Schulman: It's hard to say exactly what will be the deficit. When you talk to the models today, they have various weaknesses besides long-term coherence, like really thinking hard about things or paying attention to what you asked them. I wouldn't expect just improving the coherence a little bit to be all it takes to get to AGI. But I guess I wouldn't be able to articulate exactly what the main weakness is that'll stop them from being a fully functional colleague.

Dwarkesh Patel: It seems like then you should be planning for the possibility you would have AGI very soon.

John Schulman: I think that would be reasonable.

Dwarkesh Patel: So what's the plan if there's no other bottlenecks next year or something? You got AGI. What's the plan?

John Schulman: If AGI came way sooner than expected, we would definitely want to be careful about it. We might want to slow down a little bit on training and deployment until we're pretty sure we know we can deal with it safely and we have a pretty good handle on what it's going to do, what it can do. I think we would have to be very careful if it happened way sooner than expected, because our understanding is rudimentary in a lot of ways still.

Dwarkesh Patel: What would being careful mean? Presumably you are already careful, right? You do these evaluations before you're deploying.

John Schulman: I would say just maybe not training the even smarter version, being really careful when you do train it that it's not properly sandboxed and everything, maybe not deploying it at scale, being careful about what scale you deploy it.

Dwarkesh Patel: Okay, let's just play with a scenario. It happens next year, and you're not training a smarter system, and you're deploying somewhat in a measured way. I'm wondering, presumably if this is just intelligence was much easier than we expected and this is why it happened, and so you wait to deploy a little bit, now other companies have similar level of capabilities. What happens next? So you've waited to deploy. What are you waiting for? What is every company doing in this scenario?

John Schulman: The game theory is a little tough to think through. First of all, I don't think this is going to happen next year, but it's still useful to have the conversation. Maybe it's like two or three years instead. Two or three years is still pretty soon. I do think you probably need some coordination. Everyone needs to agree on some reasonable limits to deployment or to further training for this to work. Otherwise, you have the race dynamics where everyone's trying to stay ahead, and that might require compromising on safety. So you would probably need some coordination among the larger entities that are doing this kind of training.

Dwarkesh Patel: And so you're coordinating to pause deployment until what exactly? Until you figure out what's happening in the models?

John Schulman: Pause either further training, pause deployment, avoid certain types of training that we think might be riskier. Just setting up some reasonable rules for what everyone should do to limit these things.

Dwarkesh Patel: But limit to what end? Because at some point you're going to have to do something with the potential energy that's within this intelligence world. What is the plan to do? Suppose in two years we get the AGI and now everybody's freaking out, and so the AI companies have paused. Now what? What would be the plan to wait for?

John Schulman: I don't have a good answer to that. If everyone is going to coordinate like that, I think that would be an okay scenario. That would be a pretty good scenario because building these models is very capital intensive and there are a lot of complex pieces. It's not like everyone's going to go and recreate this stuff at home. Given the relatively small number of entities who could train the largest models, it does seem possible to coordinate. I'm not sure how you would maintain this equilibrium for a long period of time, but I think if we got to that point, we would be in an okay position.

Dwarkesh Patel: I'm curious, I'm not sure what happens next. Because fundamentally, the problem or the benefit is that we've got a ton of intelligence. You push it to the server and now we've got a bunch of intelligences, or they could push themselves to the server. And now we've got everybody coordinated, but I'm not sure what we do next in this world. Why does that set us up for a good outcome?

John Schulman: If we had everyone reasonably coordinated, we could figure out some things. If we felt like we had solved the technical problems around alignment well enough to be able to deploy really smart AIs that can act as an extension of people's will, but also prevent them from being misused in some way that would cause a catastrophe, then that would be great. We could go ahead and safely deploy these systems, and it would usher in a lot of prosperity and a new, much more rapid phase of scientific advancement and so forth. That would be what the good scenario would look like.

Dwarkesh Patel: That makes sense. But I'm curious, how would you know in a couple of years if all these actors, even in the best case scenario, have agreed to pause until we've figured out that we're building aligned systems that are not themselves going to attempt to take over a coup or not going to enable somebody else to do that? What would proof of that look like? What would evidence of that look like?

John Schulman: If we can deploy systems incrementally that are successively smarter than the ones before, then I think that's safer. I hope the way things play out is not this scenario where everyone has to coordinate and lock things down and safely release things, because it would lead to this big build-up in potential energy. I would rather some scenario where we're just continually releasing things that are a little better than what came before, while making sure we're confident that each diff is improving the safety and alignment in correspondence to the improvement in capability. If things started to look a little bit scary, then we would be able to slow things down. That's what I would hope for. If there's more of a discontinuous jump, the question is, how do you know if the thing you've got is safe to release? I can't give a generic answer. The type of thing you might want to do to make that more acceptable would be to do a lot of testing, like simulated deployment, red teaming of sorts. You'd want to do that in a way that you feel is much more likely to fail than the thing you're planning to do in the real world. You'd want to have a really good monitoring system so that if something does start to go wrong with the deployed system, you feel like it's going to be detectable immediately. Maybe you've got something watching over the deployed AIs and what they're doing, looking for signs of trouble. You'd want some defense in depth. You'd want some combination of the model itself seeming really well behaved and having an impeccable moral compass and everything, and you're pretty confident that it's extremely resistant to any kind of takeover attempt or severe misuse. And then you would also want to have really good monitoring on top of it so you could detect any kind of trouble.

Dwarkesh Patel: What are you keeping track of while you're doing long horizon RL, or when you eventually start doing it, that you could notice this sort of discontinuous jump before you deployed these systems broadly?

John Schulman: You would want to have a lot of evals that you're running during the training process.

Dwarkesh Patel: What specifically? How would you notice something like that? Does it make sense to train on long horizon RL knowing that this is something that could happen, or is it just a very low possibility? How do you think about this?

John Schulman: You'd want to be pretty careful when you do this kind of training if you see a lot of potentially scary capabilities, if those seem close. It's not something we would want to be scared of right now because right now it's hard to get the models to do anything coherent. But if they started to get really good, I think we would have to take some of these questions seriously. We would want to have a lot of evals that test them for misbehavior. That's for the alignment of the models. We want to check that they're not going to turn against us or something. You might also want to look for discontinuous jumps in capabilities. You'd want to have lots of evals for the capabilities of the models. Also, you'd want to make sure that whatever you're training on doesn't have any reason to make the model turn against you, which itself I think isn't the hardest thing to do. The way we train them with RLHF, that does feel very safe, even though the models are very smart, because the model is just trying to produce a message that is pleasing to a human. It has no concern about anything else in the world other than whether this text it produces is approved. Obviously, if you were doing something where the model is carrying out a long sequence of actions which involve tools and everything, then it might have some incentive to do a lot of wacky things that wouldn't make sense to a human in the process of producing its final result. But it wouldn't necessarily have an incentive to do anything other than produce a very high quality output at the end. You have these old points about instrumental convergence. The model is going to want to take over the world so it can produce this awesome piece of code at the end. If you ask it to write you a Flask app, it'll be like, "Oh yeah, first I need to take over the world." At a certain point, it's a little hard to imagine why for some fairly well-specified tasks like that, you would want to first take over the world. But of course, if you had a task like "make money," then maybe that would lead to some nefarious behavior as an instrumental goal.

Dwarkesh Patel: Okay, before we get back to that, let's step back and talk about today's RLHF systems and everything. But I do want to follow that third point, it's kind of interesting. Today's RLHF, the way in which it influences these models, would you characterize it in terms of human psychology? Is it a drive? Is it a goal? Is it an impulse? Psychologically, what kind of thing is being changed? And I'm not just talking about the persona of a chatbot, but just "don't talk that way, talk this other way" or "don't put those kinds of outputs."

John Schulman: There are probably some analogies with a drive or a goal in humans. You're trying to steer towards a certain set of states rather than some other states. I would think that our concept of a drive or a goal has other elements, like the feeling of satisfaction you get for achieving it. Those things might have more to do with the learning algorithm than what the model does at runtime when you just have a fixed model. There are probably some analogies, though I don't know exactly how close it is. To some extent, the models do have drives and goals in some meaningful way. In the case of RLHF, where you're trying to maximize human approval as measured by a reward model, the model is just trying to produce something that people are going to like and judge as correct.

Dwarkesh Patel: I've heard two ideas in terms of using that inner monologue type of thing to get better at reasoning, at least publicly. I'm curious which you think is more promising. One is that the model learns from outputting a bunch of potential trains of thought, and it learns to follow the one that leads to the correct answer and is trained on that before deployment. The other one is you use a bunch of compute to do inference in deployment, which involves the model talking to itself while it's deployed. Which one do you expect it to be closer to when it's really good at reasoning? Is it because it's doing just a bunch of inference calls, or is it just because you've trained it to do well at that?

Dwarkesh Patel: Well, I would say you could define reasoning as tasks that require some kind of computation at test time, or maybe some kind of deduction. So by definition, reasoning would be tasks that require test time computation and step-by-step computation. On the other hand, I would also expect to gain a lot out of doing some kind of training time computation or practice at training time. So I would think that you get the best results by combining these two things.

[Sponsor break for Nucleus Genomics]

Dwarkesh Patel: Right now, you have these two ways in which the model learns. It's either in training, whether it's pre-training or with the post-training, but most of the compute in training is spent on pre-training. It's just glossing over trillions of tokens, standing by as they all go by, almost like skimming trillions of tokens worth of information. If a human was subjected to that, they would be totally confused. It's not a very efficient way to learn. The other way is in-context learning, which is more sample efficient, but it's destroyed with each instance. I'm curious if you think there's a path for something in between those, where it's not destroyed with each instance, but it's also not as frivolous as just seeing trillions of tokens, where it's more deliberate and active.

John Schulman: So do you mean models having some kind of medium-term memory? Too much to fit in context, but much smaller scale than pre-training.

Dwarkesh Patel: I'm not sure if it's memory. It might be memory. But certainly, when I'm trying to prepare for this conversation, I think, "Well, I should understand this." So I look it up and I read it carefully, and I maybe think about it as I'm reading it. I'm not sure what that naturally corresponds to in terms of models. But I'm curious what that would look like.

John Schulman: I see. So it's not just a memory, but it's also somewhat like specializing to a task, putting a lot of effort into some particular project.

Dwarkesh Patel: And I'm not even sure it's specialization. More so, I'm thinking about, "I don't understand this part, so let me look into this part deeper. I already understand this." It's like specializing to your existing knowledge base.

John Schulman: Yeah, I see. So it's not just about training on a bunch of sources that are relevant, or fine-tuning on some special domain. It's also about reasoning, developing some knowledge through your own reasoning, and using some sort of introspection and self-knowledge to figure out what you need to learn.

Dwarkesh Patel: Yeah.

John Schulman: I would say that does feel like something that's missing from today's systems. People haven't really pushed too hard on this middle ground between large-scale training, where you produce a snapshot model that's supposed to do everything, and on the other hand, in-context learning. I think part of that is that we've just been increasing context length so much that there hasn't been an incentive for it. If you can go to a hundred thousand or a million context, that's actually quite a lot, and it's not the bottleneck in a lot of cases. But I agree that you'd probably also want to supplement that by some kind of fine-tuning. The capabilities you get from fine-tuning and in-context learning are probably somewhat complementary. So I would expect us to want to build systems that do some kind of online learning and also have some of these cognitive skills of introspecting on their own knowledge and seeking out new knowledge that fills in the holes.

Dwarkesh Patel: Is this all happening at the same time? Is it just a new training regime where all these things can happen at once, whether it's the long horizon training or this kind of training? Are they separate? Or is it just because the model is smart enough so it can both introspect and act on longer horizons, and you can get adequate reward on long horizon tasks?

John Schulman: Yeah, I would say if you're doing some kind of long horizon task, you're learning while you do the task. The only way to do something that involves a lot of steps is to have learning and memory that gets updated during the task. There's a continuum between short-term and long-term memory. So I would expect this capability, the need for it, would start to become clear when we start to look at long horizon tasks more. To some extent, just putting a lot of stuff into context will take you pretty far because we have really long context now. But you probably also want things like fine-tuning. As for introspection and the ability to do active learning, that might automatically fall out of the model's abilities to know what they know. Models have some calibration regarding what they know. That's why models don't hallucinate that badly, because they have some understanding of their own limitations. I think that same kind of ability could be used for something like active learning.

Dwarkesh Patel: How so? There's all these complicating RL procedures, many of which you've pioneered. How many of them will be relevant when you get to the point where the model itself is this smart, that it can act as its own environment and interact in a more online and stable way? Is the path for progress going to be more straightforward than the kinds of solutions that were required for RL in the past?

John Schulman: Well, I think policy gradient algorithms are not the most sample efficient algorithms, so that's probably not what you want to do at test time if you want to learn really fast. Though who knows, maybe it's not that bad. Something like motor learning in animals is probably something like a policy gradient algorithm. For example, you're learning how to shoot baskets. I think that probably takes thousands of tries to get more accurate, and there's probably something like a policy gradient algorithm underneath. But that's not going to be the fastest way to learn if you have a model trying to do a project or some kind of task. So I would think we would want to rely more on in-context learning, where you effectively have a learned algorithm. You've learned how to explore, you've learned how to try all the possibilities exhaustively, instead of doing the same thing over and over again making the same mistake. So yeah, I would say we'll be able to do things that look more like learned search algorithms, and that'll be the kind of thing that gets used in a particular task.

Dwarkesh Patel: Interesting. I want to step back and ask about your own history. At least at OpenAI, you led the creation of ChatGPT. At what point did you realize that LLMs are the path to go, and then that a chatbot, or some way to instruct them, would be a useful thing to do? Walk me through the whole lineage, from when this became your main focus and what the process was like.

John Schulman: So early on, before ChatGPT, we had these instruction following models. The idea was that we had base models and people could prompt them in elaborate ways, but they were also kind of hard to prompt. They basically do auto-complete, so you had to set up a very good prompt with some examples. So people at OpenAI were working on taking the base models and making them easier to prompt, so that if you just wrote a question, it would answer the question instead of giving you more questions. We had these instruction following models, which were kind of like base models but a little easier to use. Those were the original ones deployed in the API after GPT-3. Those were the next generation of models.

At the same time, there were definitely a lot of people thinking about chat. Google had some papers, like LaMDA and earlier Meena. They had these chatbots that were more like a base model really specialized to the task of chat, really good at chat. Looking at the examples from the paper, it was more used for fun applications, where the model would take on some persona and pretend to be that persona. It was not so functional, like "help me refactor my code."

I had worked on a project before looking at chat called WebGPT, which is more about doing question answering with the help of web browsing and retrieval. When you do question answering, it really wants to be in a chat because you always want to ask follow-up questions, or sometimes the model should ask a clarifying question because the question is ambiguous. So it was kind of clear after we did the first version of that that the next version should be conversational.

Anyway, we started working on a conversational chat assistant. This was built on top of GPT-3.5, which finished training at the beginning of 2022. That model was quite good at language and code, so we quickly realized that it was actually quite good at coding help, and that was one of the things we were excited about. We worked on that for most of the year. We had browsing as another feature in it, though we ended up de-emphasizing that later on because the model's internal knowledge was so good that browsing wasn't the most interesting thing about it.

We had it out for beta testing to friends and family for a while, and we were thinking about doing a public release. But at that time, GPT-4 finished training in August of that year. The flagship RL effort at OpenAI was the instruction following effort, because that was the models that were being deployed into production. The first fine-tunes of GPT-4 used that whole stack. Those models were really good, and everyone got really excited about that after seeing the instruct fine-tuned GPT-4s. They were really, really good. They would occasionally give you amazing outputs, but they were also a little unreliable. They would sometimes hallucinate a lot and give you pretty unhinged outputs, so it was clearly not quite ready for prime time, but it was obviously very good.

I guess people forgot about chat for a little while after that, about this alternative branch. But then we pushed it further, and we ended up mixing together all the datasets, the instruct and the chat data, to try to get something that was the best of both worlds. The chat models were clearly more easy to use. They automatically had much more sensible behavior in terms of the model knowing its own limitations. That was actually one of the things I got excited about as we were developing it. I realized that a lot of the things that people thought were flaws in language models, like blatantly hallucinating, could be not completely fixed, but you could make a lot of progress with pretty straightforward methods.

The other thing about chat was that when we had these instruct models, the task of "complete this text, but in a nice way or in a helpful way" is a pretty poorly defined task. I think that task is both confusing for the model and for the human who's supposed to do the data labeling. Whereas for chat, I think people had an intuitive sense of what a helpful robot should be like. So it was much easier to tell people what the model was supposed to do. As a result, the model had a much more coherent personality, and it was much easier to get pretty sensible behavior robustly.

Dwarkesh Patel: Interesting. Is it the case that anybody could have made ChatGPT using your publicly available fine-tuning API?

John Schulman: Not exactly. I mean, they could have... I don't remember the status of which models were available for fine-tuning. Assuming we had 3.5 available for fine-tuning at the time, you could have made something pretty decently close. But I don't think you would have been able to do just one iteration of fine-tuning where you have purely human written data and you fine-tune on that. You would want to do several iterations. If you're not going to do RL, which we did, you'd want to do some kind of iterative supervised fine-tuning where you have humans edit the model generated outputs. It's really hard to get people to... if you train on human generated data, even if it's really high quality, it's just hard for a model to fit that data perfectly because it might not be something a model is capable of outputting. So you need to do something iterative that looks a little bit more like RL. If you had done that, you could have gotten something pretty close, but that would have been kind of non-trivial.

We also had another instruction following model trained with RL that was released a little before ChatGPT. So I think if you put a chat-like wrapper on that, you would get something decently close. But if you just prompted it with chat, that model had some differences in strengths. It was pretty good at writing and poetry and so forth, but it wasn't as good at knowing its limitations and at factuality.

Dwarkesh Patel: Stepping back from 3.5, I think I heard you somewhere say GPT-2, you were super impressed compared to your expectations in 2019. Has AI progressed faster or slower than you would have expected?

John Schulman: I would say faster than I would have expected since GPT-2. I was pretty bought into scaling and pre-training being a good idea. But when GPT-2 was done, I wasn't completely sold on it revolutionizing everything. I only really pivoted what I was working on and what my team was working on after GPT-3. After that, we kind of got together and said, "Oh, yeah, let's see what we can do here with this language model stuff." But after GPT-2, I wasn't quite sure yet.

Dwarkesh Patel: Especially if the stuff we were talking about earlier with RL starts working better with these smarter models, would the fraction of compute that is spent on training, pre-training versus post-training, change significantly in favor of post-training in the future?

John Schulman: Yeah, there are some arguments for that. Right now it's a pretty lopsided ratio, but you could argue that the output generated by the model is higher quality than most of what's on the web. So it makes more sense for the model to think by itself instead of just training to imitate what's on the web. I think there's a first principles argument for that. We found a lot of gains through post-training, so I would expect us to keep pushing this methodology and probably increasing the amount of compute we put into it.

Dwarkesh Patel: The current GPT-4 has an ELO score that is like a hundred points higher than the original one that was released. Is that all because of what you're talking about with these improvements that are brought on by post-training?

John Schulman: Yeah, I would say that most of that is post-training.

Dwarkesh Patel: Interesting. So there are a lot of different separate axes for improvement.

John Schulman: Yeah. We think about data quality, data quantity, just doing more iterations of the whole process of deploying and collecting new data, and changing what kind of annotations you're collecting. There are a lot of things that stack up, but together they give you a pretty good effective compute increase.

Dwarkesh Patel: That's a huge increase. It's really interesting that there's this much room for improvement from post-training. What makes for somebody who's really good at doing this sort of RL research? I hear it's super finicky. What sort of intuitions do you have that enable you to find these ways to mess with the data and set up these environments?

John Schulman: I'd say I just have a decent amount of experience at this point from the different parts of the stack, from RL algorithms obviously, since I've worked on those since grad school, to the data collection and annotation process, to playing with language models. I'd say I just dabbled with these things. The people who do well at this kind of research have some view of the whole stack and have a lot of curiosity about the different parts of it. They also think about it from both sides: you want to be empirical and use experiments to update your views, but you also want to think from first principles somewhat, like "assuming that learning works, what would be the ideal type of data to collect?" and that sort of thing.

Dwarkesh Patel: Since there doesn't seem to be a model release since GPT-4 that seems to be significantly better, there seems to be the hypothesis that potentially we're hitting some sort of plateau, and that these models aren't actually generalizing that well. You're going to hit some sort of data wall beyond which the abilities unlocked by memorizing a vast corpus of pre-training data won't actually help you get something much smarter than GPT-4. What do you think of that hypothesis? Is it wrong?

I think we've talked about some examples generically about generalization, like Spanish to English and so forth. But one example I was thinking of was the idea that there is transfer from reasoning in code. If you train a bunch of code, it gets better at reasoning in language. Is that actually the case? Do you see things like that, which suggests that there's all this kind of positive transfer between different modalities? So once you train on a bunch of videos and images, it'll get smarter, and it'll get smarter from synthetic data. Or does it seem like the abilities that are unlocked are extremely local to the exact kind of labels and data you put into the training corpus?

John Schulman: Yeah, OK. I'll try to respond to all that. First, are we about to hit the data wall? I wouldn't draw too much from the time since GPT-4 was released. It takes a while to train these models and to do all the prep to train a new generation of models. So I wouldn't draw too much from that fact. I would say there are definitely some challenges from the limited amount of data, but I wouldn't expect us to immediately hit the data wall. I would expect the nature of pre-training to somewhat change over time as we get closer to it.

In terms of generalization from different types of pre-training data, it's pretty hard to do science on this type of question because you can't create that many pre-trained models. You can't do ablation studies at GPT-4 size scale. Maybe you can train a ton of GPT-2 size models, or maybe even a GPT-3 size model, with different data blends and see what you get. I'm not aware of any public results on ablations involving code data and reasoning performance and so forth. I'd be very interested to know about those results.

Dwarkesh Patel: If one of the things is that the model gets murderous, it's bigger, would an ablation on a GPT-2 level model, which suggests that there isn't that much transfer, how much evidence does that provide for the level of transfer on a similar set of domains in the GPT-4 level model?

John Schulman: Right. You might not be able to conclude that if transfer fails at GPT-2 size, then it's also going to fail at a higher scale. It might be that for the smaller models, they have to lean too much on memorization, whereas the larger models can learn how to do the right computation. I would expect this to be true to some extent.

Dwarkesh Patel: This might have a very simple answer, but bigger models, you train them on the same amount of data and they become smarter. Or conversely, to get the same amount of smarts, you have to train them on less data. Why is that the case? It's got more parameters, it saw less things, and now it's equally as smart. Why is that the case?

John Schulman: I don't think anyone has a good explanation of the scaling law with parameter count. I don't even know what the best mental model is for this. Clearly you have more capacity if you have a bigger model, so you should be able to eventually get lower loss. But I guess why are bigger models more sample efficient? I can give you a very sketchy explanation.

You could say that the model is like an ensemble of a bunch of different circuits that do the computation. It's doing a bunch of computations in parallel, and the output is a weighted combination of them. If you have more width of the model, or depth, with residual networks you end up with depth doing something similar to width in terms of updating what's in the residual stream. You could argue that you're learning all these different computations in parallel, and you just have more of them with the bigger model. So you have more chance that one of them is lucky and ends up guessing correctly a lot and getting up-weighted.

There are some algorithms that work this way, like mixture models or multiplicative weight update algorithms, where you have a weighted combination of experts with some learned gating. You could imagine something like that. Having a bigger model gives you more chances to get the right function. And of course, it's not just that you have a bunch of totally disjoint functions that you're taking a linear combination of. It's more like a library where you might chain the functions together in some way, so there's some composability. So I would just say the bigger model has a bigger library of different computations, including lots of stuff that's kind of dormant and only being used some of the time. But it has more space to look for those circuits to do something useful.

Dwarkesh Patel: I want to ask you about stepping back from the current research questions. I want to understand your sort of modal scenario of what happens for the next few years. I think towards the beginning of the conversation we were talking about the case in which it progresses really fast, but let's just take the modal scenario. You're unlocking long horizon RL at some point, but as you said, there are potentially other bottlenecks. So what's happening? How good are these models? How are they being deployed? What other modalities are part of them? At what stage are these being unlocked? I want to understand your broader picture of what the next few years look like.

John Schulman: I would expect new modalities to be added over time, pretty soon. I would expect the capabilities to generally keep getting better through a combination of pre-training and post-training, and that'll open up new use cases. Right now, AI is still not a huge part of the economy. There's a pretty small fraction of jobs that it can help with at all. I'd expect that to be higher over time, and not just from the models improving, but also from people figuring out how to integrate them into different processes. Even if we just froze the models at their current state, I think you would still see a lot of growth in how they're being used.

I would expect AI to be used much more widely, and for more technically sophisticated tasks. I gave the programming example earlier of doing longer projects, but also helping with various kinds of research. I would hope that we can use AI to accelerate science in various ways, because you can potentially have the models understand all the literature in a given field and be able to sift through tons of data, more than a person would have patience to do. I would hope that the form factor would basically be that people are still driving all this, and you have your helpful assistant that you can direct and point to lots of different problems that are useful to you. Everyone sort of has all these AIs helping them do more.

[Sponsor break for CommandBar]

Dwarkesh Patel: But obviously at some point they're going to be better than everyone at whatever they want to do. So what would that process look like? Right now they're clearly only helping you. At some point they're able to just do things for you, and maybe run entire firms for you. At that point, is it just going to be a smooth process?

Dwarkesh Patel: And at that point, the hope is that we have systems that are aligned with the user enough that they can count on the firm being run in the way they expect and so forth.

John Schulman: Yeah, I think we might not want to jump to having AIs run whole firms immediately. I mean, we might want to have people overseeing these important decisions and calling the shots. Even if the models are good enough to actually run a successful business themselves.

So to some extent there might be choices there. I think people will still have different interests and different ideas for what kind of interesting pursuits they want to direct their AIs at. People could do a lot. AI doesn't necessarily have an intrinsic desire, not yet, unless we put it in the system. So I think people can still end up being, even if AIs become extremely capable, I would hope that people are still the drivers of what the AIs end up doing.

Dwarkesh Patel: But I wonder if the economic equilibrium is so far from that. You have the equivalent of Amdahl's law in a firm: the slowest part of the process is the one that's going to bottleneck you. The AI makes all the non-human parts of the firm 10x more efficient, but the firm is still bottlenecked by that step. If one company decides to proceed by keeping humans in the loop on all the things you really want human oversight on, they'll just be outcompeted by other companies. If one country decides to go this route, other countries will beat it. I hope this is sustainable, but I wonder if it is.

John Schulman: Right. I think if we wanted to keep humans in the loop, which seems reasonable, and it turned out that firms with any humans in the loop were outcompeted by firms that didn't have any humans, then you would obviously need some kind of regulation that disallowed having no humans in the loop for running a whole company.

But there are so many companies in any country, let alone the world. I wonder if it's better to do the regulation on companies and say you've got to keep humans in the loop on important processes, but then you have to define what important processes are and monitor every single company. You also need collaboration from every single country that has firms.

Dwarkesh Patel: Versus if this is a problem, should it be solved before the model is even deployed, such that if you did decide to build a firm end to end on these models, it basically does what you want it to do and you don't need a human in the loop? Does that question make sense? I'm wondering how we actually monitor every single firm for a human in the loop, and what happens if China doesn't decide to do that.

John Schulman: Right. You would either have to have every country agree to this regulatory regime, or you would need all of the model infrastructure or the model providers to agree to this requirement. So it's definitely going to be non-trivial. It's a little hard to imagine this world before seeing anything like it.

For example, are we actually confident that AI-run companies are better in every way, or do we think they're better most of the time but occasionally malfunction because AIs are still less sample efficient in certain ways, like dealing with very wacky situations? So AI-run firms have higher tail risk because they're more likely to malfunction in a big way. There might be practical questions like that that determine how things play out.

Maybe if you just require people to be accountable for various liability, that would also change the incentives. If it turned out that AIs are better at running everything, completely benevolent, we've totally solved alignment, and they're better at being accountable to people than people are, then maybe it's okay having the AIs run the firms. But that might be pretty far out. I think we're more likely to be in a situation where they look better in the short term but still have some serious problems, and practical considerations push you more towards having humans in the loop for the near future.

Dwarkesh Patel: This is a problem you had to deal with today with RLHF, where you have to aggregate preferences across a lot of different humans. It'll be maybe more marked with future more powerful systems. When you say we want these eventual AI systems that are going to fully replace humans as part of these firms to be aligned, what does that mean? Does it mean they basically do what the user wants them to do? Does it mean they have to result in some sort of global outcome that we're happy with as the stakeholders? What concretely would that mean?

John Schulman: If the models are being used for these higher stakes use cases, then we would have to think about RLHF in a much different way than we are right now. We're not quite ready for that, or the current methods might not be completely sufficient. We would need to make compromises between the needs of the different stakeholders involved.

We have this document we're releasing called the Model Spec. It's about how we want our models to behave in the API and in chat. We try to talk about this issue where there are different stakeholders involved and sometimes there are conflicts between what they might want. In our case, we think of the stakeholders as the end user, the developer using the API, the platform which is OpenAI, and the rest of humanity including people who might not be users or customers.

Obviously, the user might ask the model to do something that we think is actively harmful to other people, so we might have to refuse that. By the way, this isn't the order of priority necessarily. Actually, you could also say maybe in the future the model itself will be a stakeholder. We're not going there yet.

We had to think through the trade-offs. The rough heuristic is that we mostly want the models to follow your instructions and be helpful to the user and the developer, but when this impinges on other people's happiness or way of life, it becomes a problem and we have to block certain kinds of usage. We mostly want the models to be an extension of people's will and do what they say. We don't want to be too paternalistic. We want to be kind of neutral and not impose our opinions on people.

Dwarkesh Patel: I got a chance to read the spec beforehand. It's a question of how well that transfers over to how the model itself behaves, but I was impressed with how sensible the trade-offs were. It explicitly stated the actual edge cases rather than the obvious things everybody agrees on.

John Schulman: Yeah, we wanted it to be very actionable so that it wasn't just a bunch of nice sounding principles, but each example tells you something about some non-obvious situation and reasons through that situation.

Dwarkesh Patel: Okay. Now I have a couple of questions about the state of the research itself. Famously in the social sciences, things are really hard to replicate. When you look at the average ML paper, does it feel like a really solid piece of literature, or does it feel often like it's the equivalent of p-hacking in the social sciences?

John Schulman: Everyone has their complaints about the ML literature, but overall I think it's a relatively healthy field compared to some others, just because it's largely grounded in practicality and getting things to work. If you publish something that can't be replicated easily, people will just forget about it. It's accepted that often you don't just report someone's number from their paper; you try to re-implement their method and compare it to your method on the same training data set.

If you publish methods that are really hard to implement or are really finicky, they'll tend to get forgotten. As a result, people actually try to open source their work a lot. There are various unfavorable incentives, like people being incentivized to make the baseline methods they're comparing to worse, or trying to make their method seem sophisticated mathematically. But overall, I feel like the field makes progress.

I would like to see a little more science and trying to understand things rather than more hill climbing on benchmarks and proposing new methods. There's been a decent amount of that recently, but we could use more of that. I think that's a good thing for academics to work on.

On a slightly different note, I'd be really excited to see more research using base models to do simulated social science. These models have a probabilistic model of the whole world, and you can set up a simulated questionnaire or conversation and look at how any traits are correlated with other traits. It'd be pretty cool to see if people could replicate some of the more notable results in social science, like moral foundations, by just prompting base models in different ways.

Dwarkesh Patel: With the rest of the research that happens at big labs, how much of it is increasing or decreasing the amount of compute you need to get a certain result as an actual compute multiplier, versus how much of it is things that are just making the learning more stable and building out the infrastructure? Since GPT-4, does it feel like with the same amount of compute you can train a much better model? Or does it feel like we've made sure that learning can happen better and in a more scalable way with GPT-5, but it's not like we can train GPT-4 with GPT-3.5 budget now?

John Schulman: Definitely there's always progress in improving the efficiency. Whenever you have a 1D performance metric, you're going to find that different improvements can kind of substitute for each other. Post-training and pre-training both improve the metrics, but they'll have a slightly different profile of which metrics they improve. If at the end of the day you have a single number, they're both going to substitute for each other somewhat.

For something like a human evaluation, what do humans prefer, we've definitely made a lot of progress on both sides, pre-training and post-training, in improving that.

Dwarkesh Patel: A couple of rapid fire questions about RLHF. Obviously RLHF is important to make these models useful. Maybe the "lobotomized" description is inaccurate, but there is a sense in which all of these models, once they're put in a chatbot form, have a very similar way of speaking. They really want to delve into things. They want to turn things into bullet points. They often seem to have this formal and dull way of speaking. There are complaints that they're not as creative, like with rhyming poetry. Is that a result of the particular way RLHF happens now? Is it because of who the raters are? Is it because of what the loss function is? Why is this the way all chatbots look?

John Schulman: I would say there's a decent amount of room for variation in exactly how you do the training process. We're actively trying to improve this and make the writing more lively and more fun. I think we've made some progress improving the personality of ChatGPT. It is more fun and better when you're trying to chit-chat with it, less robotic.

It's a kind of interesting question how some of the tics came about, like the word "delve." I've actually caught myself using it recently. I don't know if it rubbed off on me from the model. Actually, there might be some funny effects going on where there's unintentional distillation happening between language model providers. If you hire someone to do a labeling task, they might just be feeding it into their favorite chatbot, having the model do the task, and then copy-pasting it back. That might account for some of the convergence.

But also, some of the things we're seeing are just what people like. I think people do like bullet points and structured responses. People often like the big info dumps they get from the models. So it's not completely clear how much is just a quirk of the particular choices in the design of the post-training processes and how much is actually intrinsic to what people actually want.

It does seem persistently more verbose than some people want. Maybe during the labeling stage, the raters prefer the more verbose answer. I wonder if it's inherent because of how the training works—the stop sequence doesn't come up that often and it really wants to keep going. There might be biases in the labeling that lead to verbosity. We tend to train for one message at a time rather than the full interaction. If you only see one message, something that has a clarifying question or a short response with an invitation to follow up is going to look less complete than something that covers all possibilities.

There's also a question of whether people's preferences would change depending on how fast the model is streaming its output. If you're sitting there waiting for the tokens to come out, you're going to prefer that it gets to the point. But if it just gives you a dump of text instantly, maybe you don't actually care if there's a bunch of boilerplate—you'd rather just have it all there to skim.

Dwarkesh Patel: The reward model is such an interesting artifact because it's the closest thing we have to an aggregation of what people want, what preferences they have. When you think about models that are much smarter, one hope would be that you could just give a list of things we want that are not trivial and obvious, like a UN declaration of rights. On the other hand, I think I heard you make the point that a lot of our preferences and values are very subtle, so they might be best represented through these pairwise preferences. When you think of a GPT-6 or GPT-7 level model, are we giving it more written instructions, or are we still doing these sorts of subliminal preferences?

John Schulman: That's a good question. These preference models do learn a lot of subtleties about what people prefer that would be hard to articulate in an instruction manual. Obviously you can write an instruction manual that has lots of examples of comparisons, and that's what the Model Spec has—a lot of examples with some explanation. It's not clear what the optimal format is for describing preferences.

I would guess that whatever you can get out of a big data set that captures fuzzy preferences, you can distill it down to a shorter document that mostly captures the ideas. The bigger models do learn a lot of these concepts automatically from all the pre-training data—what people would find useful and helpful, what moral theories they have. But there's still a lot of room to latch on to a different style or a different morality.

When we align these models, what we're doing is latching onto a specific style, a specific morality. You still need a decently long document to capture exactly what you want.

Dwarkesh Patel: How much of a moat is better post-training currently? Companies distinguish themselves by how big their model is. Will it be a big moat for whoever has figured out all the finickiness with regards to all this data?

John Schulman: I think there's something of a moat because it's a very complex operation. You have to have a lot of skilled people doing it. There's a lot of tacit knowledge and organizational knowledge required. Post-training, to create a model that actually has all the functionality people care about, is pretty complicated and requires a pretty complicated effort. This requires an accumulation of a lot of R&D. So I'd say that makes it somewhat of a moat—it's not trivial to spin this up immediately.

It does seem like the same companies that are putting together the most serious pre-training efforts are also putting together the serious post-training efforts. So it seems somewhat possible to copy or spin up more of these efforts.

There's also one force that makes it less of a moat: you can distill the models, or take someone else's model and clone the outputs, or use someone else's model as a judge to do comparisons. The more big league people probably aren't doing that because it goes against terms of service policies and would be a hit to their pride, but I would expect some smaller players are doing that to get off the ground. That catches you up to a large extent and helps clear the moat.

Dwarkesh Patel: What is the median rater like? Where are they based? What are their politics? What is their sort of knowledge level?

John Schulman: It varies a lot. We've definitely hired raters with different skills for different kinds of tasks or projects. A decent mental model is just look at people who are on Upwork and other platforms like that—people doing odd jobs with remote work. It's a pretty international group. There's a decent number of people in the US. We hire different groups of people for different types of labeling, whether we're more focused on writing or STEM tasks.

People doing STEM tasks are more likely to be in India or other middle or lower middle income countries, whereas people doing more English writing and composition tend to be US based. There've been times when we needed to hire different experts for some of our campaigns. Some of the people are very talented. We even find that they're at least as good as the researchers at doing these tasks, and they're much more careful than us. I would say the people we have now are quite skilled and conscientious.

Dwarkesh Patel: With regards to the sort of plateau narrative, one of the things I've heard is that a lot of the abilities these models have to help you with specific things is related to having very closely matched labels within the supervised fine tuning data set. Is that true? If it can teach me how to use ffmpeg correctly, there's somebody who's figuring out the inputs and what flags you need to add, and a human is figuring that out and smashing that in. Do you need to hire all these labelers who have domain expertise in all these different domains? Because if that's the case, it seems like it would be a much bigger slog to get these models to be smarter over time.

John Schulman: You don't exactly need that, because you can get quite a bit out of generalization. The base model has already been trained on tons of documentation, tons of code with shell scripts and so forth. It's already seen all the ffmpeg man pages and lots of bash scripts. Even just giving the base model a good few-shot prompt, you can get it to answer queries like this. Training a preference model for helpfulness will somewhat generalize to STEM, even if you don't train it on any STEM. You not only don't need examples of how to use ffmpeg, you might not even need anything with programming to get some reasonable behavior in the programming domain.

Dwarkesh Patel: Maybe final question. We've touched on this in different ways, but to put it together. You say you're turning on much more multimodal data, presumably these things understand what screens look like and will be able to interact with them in a much more coherent way. And you're going to do this long horizon RL, so they'll be able to act as agents in systems who can be part of your workflow in a much more integrated way. What do you expect that to look like? What will be the next steps from there? Suppose by the end of the year or next year, you have something that's like an assistant who can work with you on your screen. Does that seem like a sensible thing to expect? And where does it go from there?

John Schulman: I would definitely expect things to move in that direction. It's unclear what's going to be the best form factor—whether it's like a Clippy that's on your computer helping you with something, or if it's more like a helpful colleague in the cloud. We'll see which kinds of form factors work the best. I would expect people to try all of them out.

I would expect the mental model of a helpful assistant or helpful colleague to become more real, where you can share more of your everyday work. Instead of just giving it one-off queries, you would have a whole project that you're doing and it knows about everything you've done on that project so far. You can tell it to proactively make suggestions. Maybe you can tell it to remember to ask you about something and if you've made any progress on it.

Proactivity is one thing that's been missing. I'd really love to see moving away from one-off queries, using the model like a smarter search engine, and more towards having a whole project that I'm doing in collaboration with the model. It knows everything I've done, it's proactively suggesting things for me to try, or it's going and doing work in the background.

Dwarkesh Patel: That's really interesting. By the way, final question: what is your median timeline for when it replaces your job?

John Schulman: Replaces my job? Maybe like five years. Pretty soon, yeah.

Dwarkesh Patel: Interesting. Okay, John, this is super interesting. Thanks so much for making the time. This seems like one of the parts of the AI process that are super important and people don't understand that much about. It was super interesting to delve into it and get your thoughts on it.

John Schulman: Yeah, thanks for having me on the podcast. It was fun to talk about all this stuff.

Dwarkesh Patel: [Closing remarks] Hey everybody, I hope you enjoyed that episode with John. He's just a very thoughtful guy, and it's super interesting to learn about the way in which these models become the kind that they are. Anyways, as you can see, I'm now doing ads on the podcast. So if you'd like to advertise, you can reach out at the link in the description. And of course, if you enjoyed the episode, it's really helpful if you can share it with other people who you think might enjoy it—your friends, group chats, Twitter, whatever else. See you on the next one. Cheers.