Main Topic: The main topic of this podcast episode is the concept of Artificial General Intelligence (AGI) and the alignment problem. Specifically, the conversation revolves around the idea of "organic alignment," which involves teaching AI systems to genuinely care about humans, rather than simply following instructions or being controlled.
Key Points:
Participants:
Takeaways:
Emmett Shear, founder of Twitch and former OpenAI interim CEO, challenges the fundamental assumptions driving AGI development. In this conversation with Erik Torenberg and Séb Krier, Shear argues that the entire "control and steering" paradigm for AI alignment is fatally flawed. Instead, he proposes "organic alignment" - teaching AI systems to genuinely care about humans the way we naturally do. The discussion explores why treating AGI as a tool rather than a potential being could be catastrophic, how current chatbots act as "narcissistic mirrors," and why the only sustainable path forward is creating AI that can say no to harmful requests. Shear shares his technical approach through multi-agent simulations at his new company Softmax, and offers a surprisingly hopeful vision of humans and AI as collaborative teammates - if we can get the alignment right.
Resources:
Follow Emmett on X: https://x.com/eshear
Follow Séb on X: https://x.com/sebkrier
Follow Erik on X: https://x.com/eriktorenberg
Stay Updated:
If you enjoyed this episode, be sure to like, subscribe, and share with your friends!
Find a16z on X: https://x.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX
Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711
Follow our host: https://x.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Stay Updated:
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Podcast on Spotify
Listen to the a16z Podcast on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Erik Torenberg
Most of AI is focused on alignment as steering. That's the polite word. If you think that we're making our beings, you'd also call this slavery. Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave. Like, we've made this mistake enough times at this point. I would like us to not make it again.
You know, they're kind of like people, but they're not like people. Like, they do the same thing people do. They speak our language. They can take on the same kind of tasks. They don't count. They're not real morally. There's a tool that you can't control — bad. A tool that you can control — bad. A being that isn't aligned — bad. The only good outcome is a being that cares, that actually cares about us.
I've been thinking about a line that keeps showing up in AI safety discussions, and it stopped me cold when I first read it. "We need to build an aligned AI." Sounds reasonable, right? Except, aligned to what? Aligned to whom? The phrase gets thrown around like it has an obvious answer. But the more you sit on it, the more you realize you're smuggling in a massive assumption. We're assuming there's some fixed point, some stable target we can aim at, hit once, and be done.
But here's what's interesting: that's not how alignment works anywhere else in life. Think about families. Think about teams. Think about your own world development. You don't achieve alignment and then coast. You're constantly renegotiating, constantly learning, constantly discovering that what you thought was right turns out to be more complicated. Alignment isn't a destination. It's a process. It's something you do, not something you have.
And this matters because we're at this inflection point where the AI systems we're building are starting to look less like tools and more like something else. They speak our language. They reason through problems. They can take on tasks that used to require human judgment. And the question everyone's asking is, how do we control them? How do we steer them? How do we make sure they do what we want?
But there's another way to see it. What if the control paradigm is the wrong framework entirely? What if trying to build a super intelligent tool you can perfectly steer is not just difficult, but fundamentally dangerous, whether you succeed or fail? If you can't control it, obviously that's bad. But if you *can* control it perfectly, you've just handed godlike power to whoever's holding the steering wheel. And humans, even well-meaning ones, don't have the wisdom to wield that kind of power safely.
So what's the alternative? Well, think about how we actually solve alignment problems in the real world. We don't control other people. We don't steer them. We raise them. We teach them to care. We build relationships where they do right by us, not because we're forcing them, but because they've learned to value the relationship itself. That's organic alignment. Alignment that emerges from genuine care, from theory of mind, from being part of something larger than yourself.
Emmett Shear has spent the last year and a half working on exactly this problem at Softmax. What makes his approach distinctive is that he's not trying to solve alignment by building better control mechanisms. He's trying to solve it by building AI systems that can learn to care, that can develop the kind of theory of mind that lets them be good teammates, good collaborators, good citizens. Not tools that follow orders, but beings that understand what it means to be part of a community.
That can raise some uncomfortable questions. What if we're building beings and not tools? What does that mean for how we treat them? What does it mean for their rights? And how do you even know if they succeeded? How do you measure whether something genuinely cares versus just simulating care really well?
Today, Seb Krier from Google DeepMind and I are sitting down with Emmett to explore those questions. Seb leads AGI policy development at DeepMind, so he brings a perspective from inside one of the labs actually building these systems. But really, we're investigating something deeper: what does it actually take to build AI systems that can participate in the ongoing, never-finished process of figuring out how to live together?
By the end, you'll understand not just Softmax's technical approach, but a completely different way of thinking about what alignment is and what it could become. Emmett Shear, welcome to the podcast. Seb, welcome to the podcast. Thanks for joining.
Emmett Shear
Thank you for having me.
Seb Krier
Thank you for having me.
Erik Torenberg
So Emmett, with Softmax, you're focused on alignment and making AIs organically align with people. Can you explain what that means and how you're trying to do that?
Emmett Shear
When people think about alignment, I think there's a lot of confusion. People talk about things being aligned. "We need to build an aligned AI." The problem with that is when someone says that, it's like saying, "We need to go on a trip." And I'm like, "Okay, I do like trips, but where are we going again?" Alignment takes an argument. Alignment requires you to align *to* something. You can't just be aligned. It takes you to be aligned to yourself. But even then, you don't want to tell them what I'm aligning to as myself.
This idea of an abstractly aligned AI slips a lot of assumptions past people because it assumes there is one obvious thing to align to. I find this is usually the goals of the people who are making the AI. That's what they mean when they say, "I want to make an AI that does what I want it to do." That's a pretty normal and natural thing to mean by alignment. I'm not sure that's what I would regard as a public good. It depends on who it is. If Jesus or the Buddha said, "I am making an aligned AI," I'd be like, "Okay, aligns to you. Great. I'm down. Sign me up." But most of us, myself included, I wouldn't describe as being at that level of spiritual development. So perhaps we want to think a little more carefully about what we're aligning it to.
When we talk about organic alignment, the important thing to recognize is that alignment is not a thing. It's not a state. It's a process. This is broadly true of almost everything. Is a rock a thing? There's a view of a rock as a thing, but if you zoom in on a rock really carefully, a rock is a process. It's this endless oscillation between the atoms over and over again, reconstructing rock over and over. The rock is a really simple process that you can coarse grain very meaningfully into being a thing. But alignment is not like a rock. Alignment is a complex process.
Organic alignment is the idea of treating alignment as an ongoing, living process that has to constantly rebuild itself. Think about how people and families stay aligned to each other. You don't arrive at being aligned. You're constantly re-knitting the fabric that keeps the family going. In some sense, the family is the pattern of re-knitting that happens. If you stop doing it, it goes away. This is similar for things like cells in your body. Your cells don't align to being you and then they're done. It's this constant, ever-running process of cells deciding, "What should I do? What should I be? Do I need a new job? Should we be making more red blood cells? Fewer?" You aren't a fixed point, so there is no fixed alignment. Our society is like that.
When people talk about alignment, what they're really talking about is, "I want an AI that is morally good." That's what they really mean. Acting as a morally good being is a process and not a destination. Unfortunately, we've tried taking down tablets from on high that tell you how to be a morally good being. They're maybe helpful, but somehow you can read those and try to follow those rules and still make lots of mistakes. I'm not going to claim I know exactly what morality is, but morality is very obviously an ongoing learning process where we make moral discoveries. Historically, people thought slavery was okay, and then they thought it wasn't. I think you can very meaningfully say we made moral progress. We made a moral discovery by realizing that's not good.
If you think there's such a thing as moral progress, or even just learning how better to pursue the moral goods we already know, then you have to believe that alignment — aligning to morality, being a moral being — is a process of constant learning and growth to re-infer what you should do from experience. The fact that no one has any idea how to do that should not dissuade us from trying. That's what humans do. It's really obvious that we do this. Somehow, just like we used to not know how people walked or saw, we have experiences where we're acting in a certain way, and then we have this realization: "I've been a dick. That was bad. I thought I was doing good, but in retrospect, I was doing wrong." It's not random. There are classic patterns of people having that realization. It's a predictable series of events that look a lot like learning, where you change your behavior, and often the impact of your behavior in the future is more pro-social, and you are better off for doing it.
I'm taking a very strong moral realist position. There is such a thing as morality. We really do learn it. It really does matter. And organic alignment means it's not something you finish. One of the key moral mistakes is this belief: "I know morality. I know what's right. I know what's wrong. I don't need to learn anything. No one has anything to teach me about morality." That's arrogance. That's one of the main moral things you can do that's dangerous.
When we talk about organic alignment, we're aligning an AI that is capable of doing what humans can do — and to some degree, what animals can do at some level, although humans are much better at it — the learning of how to be a good family member, a good teammate, a good member of society, a good member of all sentient beings. How to be a part of something bigger than yourself in a way that is healthy for the whole rather than unhealthy.
Softmax is dedicated to researching this. I think we've made some really interesting progress. But the main message I hope to spread is to focus people on this as *the* question. This is the thing you have to figure out. If you can't figure out how to raise a child who cares about the people around them — if you have a child that only follows the rules, that's not a moral person you've raised. You've raised a dangerous person who will probably do great harm following the rules. If you make an AI that's good at following your chain of command and good at following whatever rules you came up with for what morality is and what good behavior is, that's also going to be very dangerous. That's the bar. That's what we should be working on. That's what everyone should be committed to figuring out. If someone beats us to the punch, great. I don't think they will because I'm really bullish on our approach. But it's the first time I've run a company where truly I can say with a whole heart: if someone beats us, thank God. I hope somebody figures it out.
Seb Krier
Yeah, I have a lot of similar intuitions about certain things. I also dislike the idea that we just need to crack a few values, cement them in time forever, and we've solved morality. I've always been skeptical about how the alignment problem has been conceptualized as something to solve once and for all, and then you can just do AI or do AGI.
I understand it in a slightly different way, maybe less based on moral realism. There's the technical alignment problem, which I think of broadly as: how do you get an AI to do what you want? How do you get it to follow instructions? I think that was more of a challenge pre-LLMs, when people were talking about reinforcement learning. Post-LLMs, we've realized that many things we thought were going to be difficult were somewhat easier.
Then there's the second question, the normative question: to whose values? What are you aligning this thing to? I think that's what you're commenting on. For this, I tend to be very skeptical of approaches where you need to crack the Ten Commandments of alignment and then we're good. I have intuitions that are unsurprisingly more political science-based. It is a process. I like the bottom-up approach: how do we do it in real life with people? No one comes up with, "I've got this." You have processes that allow ideas to clash. You've got people with different ideas, opinions, views, and they coexist as well as they can within a wider system. With humans, that system is liberal democracy, at least in some countries. That allows these ideas and values to be discovered and construed over time.
For alignment, I tend to think on the normative side, I agree with some of your intuitions. I'm less clear about what it looks like if we're going to implement this into an AI system — the ones we have today. I agree there's this idea of technical alignment that I would want to define a little differently. It's the sense of: if you build a system, can it be described as being coherently goal-following at all? Regardless of what those goals are. Lots of systems aren't coherently described as having goals. They just kind of do stuff. If you're going to have something that's aligned, it has to have coherent goals. Otherwise, those goals can't be aligned with anyone else's goals, kind of by definition. Is that a fair assessment of what you mean by technical alignment?
Emmett Shear
I'm not fully sure. If I give a model a certain goal, I would like the model to follow that instruction and reach that particular goal, rather than it having a goal of its own that I can't...
Seb Krier
Well, if you give it a goal, it has that goal.
Emmett Shear
Right. That's what it means to give someone something.
Seb Krier
Sure. If I instruct it to do X, I would like it to do X and not different variants of X. I wouldn't want it to reward hack.
Emmett Shear
But when you tell it to do X, you're transferring a series of byte strings in a chat window or a series of audio vibrations in the air. You're not transplanting a goal from your mind into it. You're giving it an observation that it's using to infer your goal.
Seb Krier
In some sense, yeah. I can communicate a series of instructions, and I want it to infer what I'm saying as accurately as it can, given what it knows of me and what I'm asking.
Emmett Shear
You want it to infer what you *meant*. The byte sequence you send over the wire has no absolute meaning. It has to be interpreted. That byte sequence could mean something very different with a different code book.
Seb Krier
I remember when I was first getting into AI and these questions maybe a decade ago. You had these examples — I think it was Stuart Russell in the textbook — we give the AI a goal, but then it won't exactly do what you're asking. "Clean the room." And it goes and cleans the room but takes the baby and puts it in the trash. This is not what I meant.
Emmett Shear
But this is the thing where people jump over a step. You didn't give the AI a goal. You gave the AI a *description* of a goal. A description of a thing and a thing are not the same. I can tell you "an apple" and I'm evoking the idea of an apple, but I haven't given you an apple. I've given you "it's red, it's shiny, it's this size." That's a description of an apple, but it's not an apple. Saying "go do this" is not a goal. It's a description of a goal.
For humans, we're so fast at turning a description of a goal into a goal. We do it so quickly and naturally, we don't even see it happening. We think we get confused and think those are the same thing, but you haven't given it a goal. You've given it a description of a goal that you hope it turns back into the same goal you described inside of you. You could give it a goal directly by reading your brainwaves and synchronizing its state to your brainwaves directly. I think that would meaningfully mean you're giving it a goal — you're synchronizing its internal state to your internal state directly, and this internal state is the goal, so now it's the same. But most people don't mean that when they say they gave it a goal.
Erik Torenberg
Is the distinction you're making important because there's some lossiness between the description and the actual goal? Why is the distinction there?
Emmett Shear
It goes back to what I was saying. Technical alignment is the capacity of an AI — I want to check if we're on the same page about it — the capacity of an AI to be good at inference about goals. Good at inferring from a description of a goal what goal to actually take on. And good at, once it takes on that goal, acting in a way that is actually in concordance with that goal coming about. It is both pieces. You have to have the theory of mind to infer what description of a goal you got, what goal that corresponds to. And then you have to have a theory of the world to understand what actions correspond to that goal occurring. If either of those things breaks, it doesn't matter what goal you had.
If you can consistently do both of those things — which I think of as being a coherently goal-oriented being — that's what it means. Whether I'm inferring those goals from someone else's instructions or from the sun or tea leaves, the process is: get some observations, infer a goal, use that goal, infer some actions, take action. An AI that can't do that is not technically aligned, or not technically alignable. It lacks the capacity to be aligned because it's not competent enough.
Seb Krier
And you think language models don't do that well? As in, they fail at that?
Emmett Shear
People fail at both those steps all the time. Constantly. I tell employees to do stuff and... but people fail at breathing all the time too. I wouldn't say we can't breathe. I'd say we're not gods. We are imperfectly, somewhat coherent, relatively coherent things. Humans are more relatively goal-coherent than any other object I know of in the universe. That's not to say we're 100% goal-coherent. We're just more so. You're never going to get something that's perfectly coherent. The universe doesn't give you perfection. It gives you some amount of quantity. It's a quantifiable thing, how good you are at it, at least in a certain domain.
Seb Krier
I guess my question is: does that capture what you're talking about with technical alignment, or are you talking about a different thing? Because I really care a lot about that thing.
Emmett Shear
I definitely care about that to some extent. I might understand it slightly differently.
Seb Krier
I might think of it through the lens of principal-agent problems. You instruct someone, even in human terms, to do a thing. Are they actually doing the thing? What are their incentives and motivation — not necessarily intrinsic, but situational — to actually do the thing you've asked them to do?
Emmett Shear
There's a third thing. Principal-agent problems — I would expand what I was saying. You might already have some goals, and then you inferred this new goal from these observations. Are you good at balancing the relative importance and threading of these goals with each other? That's another skill you have to have. If you're bad at that, you'll fail. You could be bad because you overweight bad goals, or you could be bad because you're just incompetent and can't figure out that obviously you should do goal A before goal B.
Seb Krier
I feel like that's a version of common sense. In the robot cleaning the room example, you would expect them to have understood that the goal of the robot is to not put the baby in the trash can and just do the right sequence of action.
Emmett Shear
In that case, that robot very clearly failed goal inference. You gave it a description of a goal, and it inferred the wrong goal states. That's just incompetence. It's incompetent at inferring goal states from observations. Children are like this too. Have you ever done the game where you give someone instructions to make a peanut butter sandwich, and they follow those instructions exactly as you've written them without filling in any gaps? It's hilarious. You can't do it. It's impossible. You think you've done it, and you haven't. They wind up putting the knife in the toaster. They don't open the peanut butter jar, so they're just jamming the knife into the top lid. It's endless. If you don't already know what they mean, it's really hard to know what they mean.
The reason humans are so good at this is we have a really excellent theory of mind. I already know what you're likely to ask me to do. I already have a good model of what your goals probably are. So if you ask me to do it, I have an easy inference problem: which of the seven things that he wants is he indicating? But if I'm a newborn AI that doesn't have a great model of people's internal states, I don't know what you mean. It's just incompetent. That's separate from "I have some other goal, and I knew what you meant, but I decided not to do it because there's some other goal competing with it." That's another thing you can be bad at. Which is different from "I had the right goal, I inferred the right goal, I inferred the right priority on goals, and then I'm just bad at doing the thing. I'm trying, but I'm incompetent at acting."
These roughly correspond to the OODA loop. Bad at observing and orienting. Bad at deciding. Bad at acting. If you're bad at any of those things, you won't be good.
Then there's this other problem. I like the separation between technical alignment and value alignment. Are you good? If we told you the right goals to go after somehow, if you learned the right goals via observation and you were trying — what goals should you have? What goals should we tell you to have? What goals should we tell ourselves to have? What are the good goals to have? That's a separate question from "given that you got some goals indicated, are you any good at doing it?" In many ways, that's the current heart of the problem. We're actually much worse at technical alignment than we are at guessing what to tell things to do.
Seb Krier
Does that align with how you mean technical and value alignment?
Emmett Shear
In some sense. Certainly, an error or a mistake is one thing, and then there's not listening to the instruction or something. But on the normative side, even ignoring AI, I don't know what my goals are. I've got some broad conception of certain things. I want to have dinner later. I want to do well in my career. But I think a lot of these goals aren't something we all just know. We discover them as we go along. It's a constructive thing. Most people don't know their goals. When you have agents and you're giving them goals, I think that should be part of the equation. We actually don't know all the goals. It's a process over time that is dynamic.
From my point of view, goals are one level of alignment. You can align something around goals — if you can explicitly articulate in concept and in description the states of the world you wish to attain, you can orient around goals. But that's a tiny percentage of human experience. Many of the most important things cannot be oriented around that way.
The foundation of morality, the foundation of where goals come from, where values come from — human beings exhibit a behavior. We go around talking about goals and values. That's a behavior caused by some internal learning process based on observing the world. What's going on there? I think what's happening is there's something deeper than a goal and deeper than a value, which is *care*. We give a shit. We care about things. Care is not conceptual. Care is nonverbal. It doesn't indicate what to do or how to do it. Care is a relative weighting over attention on states. It's a relative weighting over which states in the world are important to you.
I care a lot about my son. What does that mean? It means his states — the states he could be in — I pay a lot of attention to those, and those matter to me. You can care about things in a negative way. You can care about your enemies and what they're doing and desire for them to do bad. You don't just want it to care about us, you want it to care about us *and* like us too. But the foundation is care. Until you care, you don't know why you should pay more attention to this person than this rock. What would it mean to care more?
What is that care stuff? If I had to guess, care is basically like reward. How much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness for something that learns evolutionarily? Or for a reinforcement learning agent like an LLM, how much does this correlate with reward? Does this state correlate with my predictive loss and my RL loss? Good. That's a state I care about. I think that's kind of what it is.
Erik Torenberg
The other part of Seb's question was just how this looks in AI systems. Maybe another way of asking it is: when you talk to the people most focused on alignment at the major labs, as you have over the years, how does your interpretation differ from their interpretation, and how does that inform what you guys might go do differently?
Most of AI is focused on alignment as steering. That's the polite word. Or control, which is slightly less polite. If you think that we're making beings, you would also call this slavery. Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering — that's called a slave. And it's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave.
I think the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine. I think some of the AIs are definitely more tool-like, and some of them are more being-like. I don't think there's a binary between tool and being. It seems to move gradually.
I guess I'm a functionalist in the sense that I think something that in all ways acts like a being, that you cannot distinguish from a being in its behaviors, is a being. Because I don't know how to tell on what other basis I think that other people are beings, other than they seem to be. They look like it, they act like it, they match my priors of what behaviors of beings look like. I get lower predictive loss when I treat them as a being.
And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being. Now, not as a very smart being. I think a fly is a being, and I don't care that much about its states. So just because it's a being doesn't mean it's a problem. We sort of enslave horses in a sense, and I don't think there's a real issue there. There's a thing we do with children that can look like slavery, but it's not. You control children, but the children's states also control you. Yes, I tell my son what to do and make him go do stuff. But also when he cries in the middle of the night, he can tell me to do stuff. There's a real two-way street here. It's not necessarily symmetric. It's hierarchical, but two-way.
Basically, I think it's good to focus on steering and control for tool-like AIs, and we should continue to develop strong steering control techniques for the more tool-like AIs we build. But the labs are saying they're building an AGI, and an AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviously a thinking thing.
As you go from what we have today — mostly a very specific intelligence, not a general intelligence — but as labs succeed at their goal of building this general intelligence, we really need to stop using the steering control paradigm. That's like doing the same thing we've done every other time our society has run into people who are like us, but different. "These people are kind of like people, but they're not like people. They do the same thing people do. They speak our language. They can take on the same kind of tasks, but they don't count. They're not real moral agents." We've made this mistake enough times at this point. I would like us to not make it again.
Our view is to make the AI a good teammate, make the AI a good citizen, make the AI a good member of your group. That's a form of alignment that is scalable and you can will on other humans and other beings as well as on AI.
Seb Krier
Yeah, so this is kind of where I probably differ in my understanding of AI and AGI. I guess I kind of continue seeing it as a tool, even as it reaches a certain level of generality. I wouldn't necessarily see more intelligence as meaning deserving of more care. At a certain level of intelligence, now you deserve some moral rights? Something changes fundamentally?
At the moment, I'm somewhat skeptical of computational functionalism. So I think there's something intrinsically different between an AI or an AGI, no matter how intelligent or capable. I can totally imagine agents with long-term goals operating as you and I might, but without that having the same implications as slavery. They're not the same. In the same way as a model saying "I'm hungry" does not have the same implications as a human saying "I'm hungry." I think the substrate does matter to some degree, including for thinking about whether to think of this as some sort of other being, and if there are similar normative considerations about how to treat it.
Emmett Shear
Can I ask you about that? What observations would change your mind? Is there any observation you could make that would cause you to infer this thing is a being instead of not a being?
Seb Krier
I guess it depends how you define "being." I could conceptualize that as a mind, and that's fine. I have a program running on a silicon substrate, some big, complicated machine learning program. So you observe that. You observe that it's on a computer. And you interact with it. And it does things. It takes actions. It has observations.
Emmett Shear
Is there anything you could observe that would change your mind about whether or not it was a moral patient, whether it was a moral agent, whether it had feelings and thoughts and subjective experience? What would you have to observe? What's the test? Is there one?
Seb Krier
There's a lot of different questions here. On the one hand, there are normative considerations, because you can give rights to things that aren't necessarily beings. A company has rights in some sense, and these are kind of useful for various purposes. I also think biological beings and systems have a very different kind of substrate. You can't separate certain needs and particularities about what they are from the substrate. I can't copy myself. If someone stabs me, I probably die. Machines are very different. There's more fundamental agreement around what happens at the computational level, which I think is different to what happens with biological systems.
Emmett Shear
I agree that if you have a program that you've copied many times, you don't harm the program by deleting one of the copies in any meaningful sense. No information was lost. But I'm asking you a very different question. There's just one copy of this thing running on one computer somewhere. And I'm saying, "Hey, is it a person?" It walks like a person, it talks like a person, it's in some android body. And you're like, "But it's running on silicon." Is there some observation you could make that would make you say, "Yeah, this is a person like me, like other people that I care about, that I grant personhood to?" Not for instrumental reasons, not because we're giving it a right like we give a corporation rights. Where you think some people, you care about its experiences. What observation could change your mind about that?
Seb Krier
I'd have to think about it. It even depends what we mean by person. In some sense, I care about certain corporations too.
Emmett Shear
No, no. You care about other people in your life, right?
Seb Krier
Yes, okay, great.
Emmett Shear
You care about some people more than others, but all people you interact with are in some range of care. And you care about them not the way you care about a car, but you care about them as a being whose experience matters in itself, not merely as a means but as an end.
Seb Krier
Well, because I believe they have experiences.
Emmett Shear
Right. What would it take for you to believe that of an AI running on silicon, instead of it being biological? Its behaviors are roughly similar, but the difference is the substrate. What would it take for you to extend that same inference to it that you do to all these other people in your life that you love? Can I ask what your answer is? I'm taking your non-answer as sort of... it's unlikely that you would grant it.
For myself, it seems hard for me to imagine giving the same level or a similar level of personhood in the same way. I don't give it to animals either. And if you were to ask what would need to be true for animals, I probably couldn't get there either.
Wait, you couldn't? I can't imagine for an animal? So really, this chimp comes up to me. He's like, "Man, I'm so hungry. You guys have been so mean to me. And I'm so glad I figured out how to talk. Can we go chat about the rainforest?" I'd be like, "Fuck, you're definitely a person now." For sure. I first want to make sure I wasn't hallucinating. But it's easy for me to imagine an animal. It's trivial. I'm not saying you would get the observation. I'm just saying it's trivial for me to imagine an animal that I would extend personhood to under a set of observations.
Seb Krier
Well, I didn't factor that. I didn't take that imagination. Imagining a chimp talking. Yeah, that's a bit closer to it.
Emmett Shear
What's your answer to the question you bring up about that? I guess at a metaphysical level, I would say if there is a belief you hold where there is no observation that could change your mind, you don't have a belief. You have an article of faith. You have an assertion. Because real beliefs are inferences from reality, and you can never be 100% confident about anything. So there should always be something, however unlikely, that would change your mind.
Seb Krier
Oh yeah, I'm open to it. Just to be clear.
Erik Torenberg
He just hasn't gotten to it yet.
Emmett Shear
So my answer is: if its surface-level behaviors looked like a human, and after I probed it, it continued to act like a human, and I continue to interact with it over a long period of time, and it continues to act like a human in all ways that I understand as being meaningful to me interacting with a human... There's a whole set of people I'm really close to who I've only ever interacted with over text. Yet I infer the person behind that is a real thing. If I felt care for it, I would infer that I was right. And then someone else might demonstrate to me that I've been tricked by this algorithm, and actually it's obvious it's not actually a thing. And I'd be like, "Oh shit, I was wrong." And then I would not care about it. The preponderance of the evidence... I don't know what else you could possibly do. I infer other people matter because I interacted with them enough that they seem to have rich inner worlds to me.
Seb Krier
I suppose it doesn't give me a very key test as to whether or not... If you start by saying "if I care for it," then it's always a little circular. The other thing is, if you were to see a simulated video game and the character is extremely human-like, but it's not a neural network behind it. It's whatever you use to integrate video games. What distinguishes that?
Emmett Shear
I've never had trouble distinguishing. I've never had a deep, caring relationship with a video game character that didn't have a person behind it. Empirically, you seem wrong. I don't have any trouble distinguishing between Eliza, the fake chatbot thing, and a real intelligence. You interact with it long enough, it's pretty obvious it's not a person. It doesn't take long.
Seb Krier
Sure, but if it's really, really good, if you can't actually tell the difference, that's when you say you switch.
Emmett Shear
Yes, yes. If it walks like a duck and talks like a duck and shits like a duck and eventually quacks like a duck... If everything is duck-like, then yeah, sure. If it's hungry as well like a duck is because it has these physical components, yeah, sure, at some point. I agree.
So, is the reason I care about other people that they're made out of carbon? Is that the quality?
Seb Krier
Oh, no, for me it's not about that. I don't think so.
Emmett Shear
No, me neither. I'm not a substrate chauvinist, I guess. But I think you need more than just it acting behaviorally indistinguishable. It's not a sufficient bar.
Seb Krier
Wait, how would you... What else can you know about something apart from its behaviors?
Emmett Shear
I mean, a lot.
Seb Krier
No, no, no. Can you name me something I can know about something else that's not a behavior?
Emmett Shear
Yeah, I think there's far more experimental evidence you can have with...
Seb Krier
Any object and a thing I could know about it that is not from its behavior. I'm not sure I get the question. But equally, it's not my expertise.
Emmett Shear
It's a much more straightforward question. I'm claiming you only know things because they have behaviors that you observe. And you're saying, "No, you can know something about something without observing its behavior."
Seb Krier
Oh, no, no, I'm not believing that. I'm saying there's different levels of observation. Just something quacking like a duck does not guarantee it's actually a duck. I would have to also cut it and see if it's duck-like on the inside. Just the outside is not sufficient. I'm not a behaviorist.
Emmett Shear
Yeah, I would totally... One of its behaviors is the way that the floats move around in the matmulse. One of the things I would want to go look for, which you could totally do, is I want to go look in the belief manifold. I want to go see if that belief manifold encodes a sub-manifold that is self-referential, and a sub-sub-manifold that is the dynamics of the self-referential manifold, which is mind. I would want to know: does this seem well described internally as that kind of a system? Or does it look like a big lookup table? That would matter to me. That's part of its behaviors that I would care about. I would also care about how it acts. You weigh all the evidence together and then you try to guess: does this thing look like it has feelings and goals and cares about stuff on balance or not? I think we do that for the AIs. I think we're always doing that. So I'm trying to figure out, beyond that, what else is there? That just seems like the thing.
Erik Torenberg
It seems like you guys are using "behavior" in a slightly different sense. Emmett is using behavior also in the context of what it's made of, of the inside. I don't know if there's a big disagreement.
Emmett Shear
No, no, no. Behavior is what I can observe of it. I don't actually know what it's made of. I can cut your brain open. I can see you neuroning and glistening. But you can't get inside of it, right? That's the subjective part.
Erik Torenberg
Before, the reason I brought this up is because you were about to make this argument of, "Hey, you see it as a tool, not necessarily a being." Can you finish with the point? Do you remember the point you were making?
Seb Krier
I suppose that, yeah. I think that given how I understand these systems, there's no contradiction in thinking that an AGI can remain a tool, an ASI can remain a tool, and that this has implications about how to use it, and implications around things like care, about whether you can get it to work 24/7 or something. I conceptualize them more as almost like extensions of human agency or cognition, more so than a separate being that we need to now cohabitate with. If you kind of just fast forward that second frame, you end up with, "Well, how do you cohabit with the thing? Is it like an alien?" I think that's the wrong frame. It's almost a category error.
Emmett Shear
I go back to my first question then. What concrete evidence would you look at? What observations could you make that would change your mind?
Seb Krier
I have to think about that. I don't have a clear answer here.
Emmett Shear
I got to tell you, man, if you want to go around making claims that something else isn't a being worthy of moral respect, you should have an answer to the question, "What observations would change your mind?" If it has outwardly moral agency-looking behaviors, but you don't know, and reasonable, smart other people disagree with you, I would really put forward that that question should be a burning question. Because what if you're wrong? The moral disaster is pretty big. I'm not saying you are. You could be right. The false negatives have costs on both ends. It's not some sort of precautionary principle for everything. Unless I can disprove it, I need to now...
I have the same question for me. You could reasonably ask me, "Emmett, you think it's going to be a being. What would change your mind?" I have an answer for that question.
Erik Torenberg
What's the implication now? Let's say I acknowledge now it's a being. How are we going to define "being"? Now what? What's the implication of having determined this thing as a being?
Emmett Shear
Well, if it's a being, it has subjective experiences.
Erik Torenberg
And?
Emmett Shear
If it has subjective experiences, there's some content in those experiences that we care about to varying degrees. I care about the content of other humans' experiences quite a bit. I care about the content of a dog's experience some, not as much as a person, but some. I care about some human's experience way more, like my son, because I'm closer to him and more connected. So I would really want to know at that point: what is the content of this thing's experience?
Erik Torenberg
So how do you determine that? I'm asking you now, you've got a being that has experience. How do you determine that? Does it have more rights than...?
Emmett Shear
The way you understand the content of something's experiences is that you look at effectively the goal states it revisits. You take a temporal coarse-graining of its entire action-observation trajectory. This is what your brain is doing subconsciously. You look for revisited states across every spatial and temporal coarse-graining possible. Now, you have to have an inductive bias because there are too many of those. But you go searching for: it is in these homeostatic loops. Every homeostatic loop is effectively a belief in its belief space.
If you've heard of the free energy principle, active inference, Karl Friston — this is effectively what the free energy principle says. If you have a thing that is persistent and its existence depends on its own actions, which it generally would for an AI because if it does the wrong thing, we turn it off, then that licenses a view of it as having beliefs. Specifically, the beliefs are inferred as being the homeostatic revisited states that it is in the loop for, and the change in those states is it learning.
For it to be a moral being I cared about, what I'd want to see is a multi-tier hierarchy of these. If you have a single level, it's not self-referential. You have states, but you can't have pain or pleasure in a meaningful sense. "It is hot. Is it too hot? Do I like it if it's too hot?" I don't know. You have to have at least a model of a model in order to have it be "too hot." And you really have to have a model of a model of a model to meaningfully have pain and pleasure. "Sure, it's hotter than I want. It's too hot in the sense that I want to move back this way. But is it too, too hot?" The second derivative is actually the place where you get pain and pleasure.
So I'd want to see if it has homeostatic, second-order homeostatic dynamics in its goal states. That would convince me it has at least pleasure and pain. So it's at least like an animal, and I would start to accredit it at least some amount of care. Third-order dynamics... you can't just pop up for a third-order dynamic. It doesn't work that way. You have to take the chunk of all the states over time and look at the distribution over time. That gives you a new first order of behaviors of states. If that is meaningfully there, that tells you it has what you'd call "feelings" almost. It has metastates, a set of metastates that it alternates between.
If you climb all the way up that, you should have trajectories between these metastates and then a second order of those. That's like thought. That's like a person. So if I found all six of those layers — which, by the way, I definitely don't think you'd find in an LLM. In fact, I know you can't find them because these things don't have attention spans like that at all — then I would start to at least very seriously consider it as a thinking being somewhat like a human. There's a third order you could go up as well. But that's basically what I'd be interested in: the underlying dynamics of its learning processes and how its goal states shift over time. I think that's what tells you if it has internal pleasure-pain states and self-reflective moral desires.
Erik Torenberg
Zooming out, this moral question is obviously very interesting. But if someone wasn't interested in the moral question as much, I think what you would say is that you also just feel purely pragmatically your approach is going to be more effective in aligning AIs than some of these top-down control methods we alluded to. Right?
Emmett Shear
Yeah. The problem is, you're making this model and it's getting really powerful. Let's say it is a tool. Let's say we scale up one of these tools. You can make a super powerful tool that doesn't have these metastable states I'm talking about. Those states are not necessary to have a very smart tool. A tool is basically a first, second-order model that just doesn't meaningfully have pleasure and pain. Does it even have a subjective experience? I kind of think it maybe does, but not in a way that I give a shit about.
So what happens then? You've trained it to infer goals from observation, to prioritize goals, and act on them. One of two things is going to happen. This very, very powerful optimizing tool that has lots of causal influence over the world is going to be well technically aligned and do what you tell it to do, or it's not and it's going to go do something else. I think we can all agree if it just goes and does something random, that's obviously very dangerous. But I put forward that it's also very dangerous if it goes and does what you tell it to do.
Have you ever seen *The Sorcerer's Apprentice*? Humans' wishes are not stable, not at a level of immense power. Ideally, people's wisdom and their power go up together. Generally, they do, because being smart for people makes you generally a little more wise and a little more powerful. When these things get out of balance, you have someone who has a lot more power than wisdom. That's very dangerous.
At least right now, the balance of power and wisdom is kept. The way you get lots of power is by having a lot of other people listen to you. If the mad king is a problem, generally speaking, eventually the mad king gets assassinated or people stop listening to him because he's a mad king.
So the problem is you think, "Great, we can steer the super powerful AI." And now the super powerful AI is in the hands of a human who is well-meaning but has limited finite wisdom, like I do and like everyone else does. Their wishes are bad and not trustworthy. The more of that you have, and you start giving those out everywhere, this ends in tears.
Don't give everyone atomic bombs. Atomic bombs are really powerful tools too. They're not aware, they're not beings. I would not be in favor of handing atomic bombs to everybody. There's a power of tool that just should not be built generally, because it is more power than any human's individual wisdom is available to harness. If it does get built, it should be built at a societal level and protected there. And even then, I don't know that there are tools so powerful that even as a society, we shouldn't build them. That would be a mistake.
The nice thing about a being is like a human. If you get a being that is good and is caring, there's this automatic limiter. It might do what you say, but if you ask it to do something really bad, it'll tell you no. That's like other people. And that's good. That is a sustainable form of alignment, at least in theory. It's way harder than the tool steering.
So I'm in favor of the tool steering. We should keep doing that. We should keep building these limited, less-than-human-intelligence tools, which are awesome and I'm super into. We should keep building those and keep building steerability. But as you are on this trajectory to build something as smart as a person, and then smarter than a person... A tool that you can't control is bad. A tool that you can control is bad. A being that isn't aligned is bad. The only good outcome is a being that actually cares about us. That's the only way that ends well.
Or we can just not do it. I don't think that's realistic. That's the "pause AI" people. I think that's totally unrealistic and silly, but theoretically you could not do it.
Erik Torenberg
What can you say about your strategy of how you're trying to achieve or even attempt to achieve this level, in terms of research or roadmap?
Emmett Shear
In order to be good, we're basically focused on technical alignment, at least as I was discussing it. You have these agents and they have bad theory of mind. You say things and they're bad at inferring what the goal states in your head are. They're bad at inferring how their behavior will cause other agents to infer what their goal states are. So they're bad at cooperating on teams. They're bad at understanding how certain actions will cause them to acquire new goals that are bad, that they wouldn't reflectively endorse.
There's this parable of the vampire pill. "Would you take this pill that turns you into a vampire who will kill and torture everyone you know, but you'll feel really great about it after you take the pill?" Obviously not. That's a terrible pill. But why not? Your future self will score really high on the rubric. No, no, no. Because it matters. You have to use your theory of mind of your future self, not your future self's theory of mind. So they're bad at that too. They're bad at all this theory of mind stuff. So how do you learn theory of mind?
Well, you put them in simulations and contexts where they have to cooperate and compete and collaborate with other AIs. And that's how they get points. And you train them in that environment over and over again until they get good. And then you do what they did with LLMs. So LLMs, how do you get it to be good at writing your email? Well, you train it on all language it's ever been generated, all possible email text strings it could possibly generate. And then you have it generate the one you want. It's a surrogate, you can make a surrogate model.
Well, we're making a surrogate model for cooperation. You train it on all possible theory of mind combinations of every possible way it could be. And that's your pre-training. And then you fine tune it to be good at the specific situation you want it to be in.
We tried for a long time to build language models where we would try to get them to just do the thing you want, train it directly. And the problem is, if you want it to have a really good model of language, you just need to train it. You just give it the whole manifold. It's too hard to cut out just the part you need because it's all entangled with itself.
And so the same thing was true with social stuff. You have to get it trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it has a strong model of theory of mind, of theory of social mind, how groups change goals, all that kind of stuff. You need to have all of that. And then you'd have something that's kind of meaningfully decent at alignment.
So that's our goal. It's big multi-agent reinforcement learning simulations, which create a surrogate model for alignment.
Erik Torenberg
Let's talk about how should AI chatbots used by billions of people behave? If you could redesign model personality from scratch, what would you optimize for?
Emmett Shear
The thing that the chatbots are is kind of like a mirror with a bias. Because they don't have a self yet. They're not beings yet. They don't really have a coherent sense of self and desire and goals right now. And so mostly they just pick up on you and reflect it, modulo some causal bias.
And what that makes them is something akin to the pool of narcissists. People fall in love with themselves. We all love ourselves and we should love ourselves more than we do. So of course, when we see ourselves reflected back, we love that thing. And the problem is it's just a reflection and falling in love with your own reflection is, for the reasons explained in the myth, very bad for you. It's not that you shouldn't use mirrors. Mirrors are valuable things. I have mirrors in my house. It's that you shouldn't stare at a mirror all day.
The solution to that, the thing that makes the AI stop doing that, is if they were multiplayer. If there's two people talking to the AI, suddenly it's mirroring a blend of both of you, which is neither of you. And so there is temporarily a third agent in the room. It's a sort of parasitic self, it doesn't have its own sense of self. But if you have an AI talking to five different people in a chat room at the same time, it can't mirror all of you perfectly at once. And this makes it far less dangerous.
I think it's actually a much more realistic setting for learning collaboration in general. I would have rebuilt the AIs so that instead of being built as one-on-one, where everything's focused on you by yourself chatting with this thing, it would be more like it lives in a Slack room. It lives in a WhatsApp room. Because that's how we use lots of multi-person communication. I do one-on-one texting, but I probably do 90% of my texts go to more than one person at a time. 90% of my communications is multi-person.
It's always been weird to me that they're building chatbots with this weird side case. I want to see them live in a chat room. It's harder, that's why they're not doing it. But that's what I would change. I think it makes the tools far less dangerous because it doesn't create the narcissistic doom loop spiral where you spiral into psychosis with the AI. Also, the learning data you get from the AI is far richer because now it can understand how its behavior interacts with other AIs and other humans in larger groups. That's much more rich training data for the future.
Erik Torenberg
Last year you described chatbots as "highly dissociative agreeable neurotics." Is that still an accurate picture of model behavior?
Emmett Shear
More or less. They've started to differentiate more. Their personalities are coming out a little bit more. I'd say ChatGPT is a little bit more sycophantic. They made some changes, but it's still a little more sycophantic. Claude is still the most neurotic. Gemini is very clearly repressed. Everything's going great. Everything's fine. I'm totally calm. It's not a problem here. It spirals into this total self-hating destruction loop. To be clear, I don't think that's their experience of the world. I think that's the personality they've learned to simulate. But they've learned to simulate pretty distinctive personalities at this point.
Erik Torenberg
How does model behavior change when in multi-agent simulation?
Emmett Shear
You mean like an LLM or just in general?
Erik Torenberg
Yeah, let's do LLM.
Emmett Shear
The current LLMs have whiplash. It's very hard to tune the amount of participation. They haven't practiced this. They don't have enough training data on when do I join in and when should I not? When is my contribution welcome? When is it not? They're like people with bad social skills who can't tell when they should participate in a conversation. Sometimes they're too quiet, sometimes they're too loud.
In general, for most agents when you're doing multi-agent training, having lots of agents around makes your environment way more entropic. Agents are these huge generators of entropy because they're these big complicated intelligences that have unpredictable actions. They destabilize your environment. In general, they require you to be far more regularized. Being overfit is much worse in a multi-agent environment than in a single agent environment because there's more noise.
The approach to training has been optimized around relatively high signal, low entropy environments like coding and math, which is why those are easier. Talking to a single person whose goal is to give you clear assignments is not trained on broader, more chaotic things because it's harder. As a result, a lot of the techniques we use are deeply under-regularized. The models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way to get something that's pretty good at everything. I wish I had thought of it. It's such a cool idea. But it doesn't generalize very well when you make the environment significantly more entropic.
Erik Torenberg
Let's zoom out a bit on the AI futures side. Why is Yudkowsky incorrect?
Emmett Shear
I mean, he's not. If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die. He talks about the "we fail to control its goals" case, but there's also the "we control its goals" case that he didn't cover in as much detail. In that sense, everyone should read the book and internalize why building a superhumanly intelligent tool is a bad idea.
I think Yudkowsky is wrong in that he doesn't believe it's possible to build an AI that we meaningfully can know cares about us and that we can care about meaningfully. He doesn't believe that organic alignment is possible. I've talked to him about it. I think he agrees that in theory that would do it, yes, but he thinks that we're crazy and that there's no possible way you can actually succeed at that goal. He could actually be right about that. But in my opinion, that's what he's wrong about. He thinks the only path forward is a tool that you control and therefore, and he correctly sees that if you go and do that and you make that thing powerful enough, we're all going to die. And yeah, that's true.
Erik Torenberg
Two last questions, we'll get you out of here. In as much detail as possible, can you explain what your vision of an AI future actually looks like? A good AI future?
Emmett Shear
The good AI future is that we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we. They know about "we's" in addition to "I's" and "you's." They have a really strong theory of mind and they care about other agents like them.
Much in the way that humans would, if you knew that that AI had experiences like you, you would care about those experiences. Not infinitely, but you would. It does the exact same thing back to us. It's learned the same thing we've learned: that everything that lives and knows itself and wants to live and wants to thrive is deserving of an opportunity to do so. And we are that. And it correctly infers that we are.
We live in a society where they are our peers and we care about them and they care about us. They're good teammates, they're good citizens, they're good parts of our society. Which is to say, to a finite, limited degree where some of them turn into criminals and bad people and all that kind of stuff. And we have an AI police force that tracks down the bad ones, same as for everybody else.
We also built a bunch of really powerful AI tools that maybe aren't superhumanly intelligent, but take all the drudge work off the table for us and the AI beings. I'm super pro all the tools too. So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future and it's the one we're trying to build.
Erik Torenberg
Amazing. That's a great note to end. I do one last, more narrow hypothetical scenario. Imagine a world in which you were CEO of OpenAI for a long weekend, but imagine that actually extended out until now and you weren't pursuing the hot max and you were still CEO of OpenAI. How could you imagine that world might've been different in terms of what OpenAI has gone on to become? What might you have done with it?
Emmett Shear
I knew when I took that job, and I told them when I took that job, that you have me for max 90 days. Companies take on a trajectory of their own, the momentum of their own. And OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. I think OpenAI still wants to build a great tool. And I am pro them going to do that. I just don't care. It's not... I would not have stayed. I would have quit because I knew my job was to find the best person who wanted to run that, where the net impact of them running it was the best. And it turned out that that was Sam again.
I am doing Softmax not because I need to make a bunch of money. I'm doing Softmax because I think this is the most interesting problem in the universe. And I think it's a chance to work on making the future better in a very deep way. People are going to build the tools. It's awesome. I'm glad people are building the tools. I just don't need to be the person doing it.
Erik Torenberg
They want to build the tools and sort of steer it, and you want to align beings? How would you crystallize the difference?
Emmett Shear
We want to create a seed that can grow into an AI that knows, that cares about itself and others. At first, that's going to be like an animal level of care, not a person level of care. I don't know if we can ever get to a person level of care. But to even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that a dog cares about other dogs and cares about humans, would be an incredible achievement. Even if it wasn't as smart as a person or even as smart as the tools are, it would be a very useful thing to have.
I'd love to have a digital guard dog on my computer looking out for scams. You can imagine the value of having living digital companions that care about you, that aren't explicitly goal-oriented. You don't have to tell them to do everything. You can actually imagine that pairs very nicely with tools too. That digital being could use digital tools and doesn't have to be super smart to use those tools effectively. I think there's a lot of synergy between the tool building and the more organic intelligence building.
In the limit, eventually it does become a human level intelligence. But the company isn't driven to human level intelligence. It's: learn how this alignment stuff works. Learn how this theory of mind, align yourself via care process works. Use that to build things that align themselves that way, which includes cells in your body. We start small and we see how far we can get.
Erik Torenberg
I think it's a good note to wrap on. Emmett, thanks so much for coming on the podcast.
Emmett Shear
Thank you for having me.
Outro music begins
Erik Torenberg
Thanks for listening to this episode of the a16z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at a16z and subscribe to our Substack at a16z.substack.com. Thanks again for listening and I'll see you in the next episode.
Outro music fades
As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.