--:-- Sonal: Hi, everyone. Welcome to the A6NZ podcast. I'm Sonal, and as your primary host, editor, and showrunner here of the past seven years, I wanted to share some of my very favorite, just a few selected episodes for you on this feed as I get ready to make a change and as we get ready to relaunch this show anew on this feed.
--:-- The second of these three episodes is a conversation that originally took place spring-summer 2019, pre-pandemic — ah, that we could remember that time — between A6NZ co-founder Mark Andreessen and myself interviewing author, consultant advisor, and former pro poker player Annie Duke on one of her first appearances here. It's a conversation quite unlike her other conversations, and it's one of my favorites because Mark and I did several of these interviews together where we really bring out different aspects based on our own interests combined.
--:-- Here's a fun behind-the-scenes fact about this episode: Mark had to leave midway for an urgent meeting, so I had to rebalance the episode to account for both of our segments together. But we all talk super fast together, barely caught our breaths, and cover such a range of topics relevant to both businesses and individuals — basically all of you — covering skill versus luck, outcomes versus process, asking the right questions on how to communicate about probabilities instead of just deadlines on a team, and on how not making a decision is also a decision and why that matters.
--:-- Be sure to also check out in this feed the follow-up conversation with Annie, and that time it was Jeff and me, on the specifics of how to decide as well, where she covers the practical and organizational implementations of what we cover in this episode. Because the A6NZ podcast became an early go-to for book authors to release their books, Annie agreed to share her next book with us first. So that's what we do in the next episode on how to decide.
--:-- Overall, these evergreen episodes are really all about innovation, which is another signature theme in defining the identity of the A6NZ podcast.
--:-- Sonal: Hi, everyone. Welcome to the A6NZ podcast. I'm Sonal. And today, Mark and I are doing another one of our book author episodes. We're interviewing Annie Duke, who's a professional poker player and World Series champ and is the author of *Thinking in Bets*, which is just out in paperback today. The subtitle of the book is *Making Smarter Decisions When You Don't Have All the Facts*, which actually applies to startups and companies of all sizes and ages, quite frankly. I mean, basically any business or new product line operating under conditions of great uncertainty, which I'd argue is my definition of a startup and innovation. So that will be the frame for this episode.
--:-- Annie's also working on her next book right now and founded HowIDecide.org, which brings together various stakeholders to create a national education movement around decision education, empowering students to also be better decision makers. So anyway, Mark and I interview her about all sorts of things in and beyond her book, going from investing to business to life. But Annie begins with a thought experiment, even though neither of us really know that much about football.
--:-- Annie Duke: So what I'd love to do is kind of throw a thought experiment at you guys so that we can have a discussion about this. I know you guys don't know a lot about football, but this one's pretty easy. You're going to be able to feel this one.
--:-- Do this thought experiment: Pete Carroll calls for Marshawn Lynch to actually run the ball. So we're betting on someone who we know is really good. Well, they're all really good, but we're betting on the play that everybody's expecting. The default. This is the default. This is the assumed rational thing to do. Right.
--:-- So he has Russell Wilson hand it off to Marshawn Lynch. Marshawn Lynch goes to barrel through the line. He fails. Now they call the timeout. So now they stop the clock. They get another play now. And they hand the ball off to Marshawn Lynch. It's what everybody expects. Marshawn Lynch, again, attempts to get through that line. And he fails. End of game. Patriots win.
--:-- My question to you is, are the headlines the next day "Worst call in Super Bowl history"? Is Chris Collinsworth saying, "I can't believe the call. I can't believe the call"? Or is he saying something more like, "That's why the Patriots are so good. Their line is so great. That's the Patriots line that we've come to see this whole season. This will seal Belichick's place in history"? It would have all been about the Patriots.
--:-- So let's sort of divide things into like, we can either say the outcomes are due to skill or luck. And luck in this particular case is going to be anything that has nothing to do with Pete Carroll. And we can agree that the Patriots line doesn't have anything to do with Pete Carroll. Belichick doesn't have anything to do with Pete Carroll. Tom Brady doesn't have anything to do with Pete Carroll as they're sealing their fifth Super Bowl victory.
--:-- So what we can see is there's two different routes to failure here. One route to failure, you get result-ing. And basically what result-ing is, is that retrospectively, once you have the outcome of a decision, once there's a result, it's really, really hard to work backwards from that single outcome to try to figure out what the decision quality is. This is just very hard for us to do. They say, "Oh my gosh, the outcome was so bad. This is clearly, I'm going to put that right into the skill bucket. This is because of Pete Carroll's own doing." But in the other case, they're like, "Oh, you know, there's uncertainty. What could you do?" Weird, right?
--:-- Annie Duke: Okay. So you can kind of take that and you can say, "Aha, now we can sort of understand some things." Like, for example, people have complained for a very long time that in the NFL, they have been very, very slow to adopt what the analytics say that you should be adopting. Right. And even though now we've got some movement on like fourth down calls and when are you going for two point conversions and things like that, they're still nowhere close to where they're supposed to be.
--:-- Sonal: So they don't make the plays corresponding to the statistical probabilities?
--:-- Annie Duke: No. In fact, the analytics show that if you're on your own one yard line and it's fourth down, you should go for it no matter what. The reason for that is if you kick it, you're only gonna be able to kick to midfield. So the other team is basically almost guaranteed three points anyway. So you're supposed to just try to get the yards. Like when have you ever seen a team on their own one yard line on fourth down and be like, "Yeah, let's go for it"? That does not happen.
--:-- Okay, so we know that they've been super slow to do what the analytics say is correct. And so you sit here and you go, "Well, why is that?" And that thought experiment really tells you why. Because we're all human beings. We all understand that there are certain times when we don't allow uncertainty to bubble up to the surface. And there are certain times that we do. And it seems to be that we do when we have this kind of consensus around the decision.
--:-- Sonal: So if I'm a human decision maker, I'm gonna choose a path where I don't get yelled at.
--:-- Annie Duke: Yeah, exactly. So basically we can kind of walk back and we can say, "Okay, are we allowing the uncertainty bubble to the surface?" And this is going to be the first step to understanding what really slows innovation down, what really slows adoption of what we might know is good decision making because we have conflicting interests, right? Making the best decision for the long run or making the best decision to keep us out of a room where we're getting judged.
--:-- Mark Andreessen: Yelled at or possibly fired.
--:-- Mark Andreessen: So let me propose the framework that I used to think about this and see if you agree with it. So it'd be a two by two grid and it's consensus versus non-consensus and it's right versus wrong. And the way we think about it, at least in our business, is basically consensus right is fine. Consensus non-consensus right is fine. In fact, generally you get called a genius. Consensus wrong is fine because you just, you know, it's just the same mistake everybody else made. You all agreed, right? Non-consensus wrong is really bad. It's horrible. It's radioactively bad.
--:-- And so, as a consequence of that, and maybe this gets to the innovation stuff that you'll be talking about, but as a consequence of that, there are only two scripts for talking about people operating in the non-consensus directions. One script is they're a genius because it went right. And the other is they're a complete moron because it went wrong. Is that, does that map?
--:-- Annie Duke: That's exactly right. And I think that the problem here is that what does right and wrong mean? In your two by two, wrong and right is really just "did it turn out well or not?" And this is where we really get into this problem because now what people are doing is they're trying to swat the outcomes away. And they understand, just as you said, that on that consensus wrong, you will have like a cloak of invisibility over you. Like you don't have to deal with it.
--:-- So let's think about other things besides consensus. So consensus is one way to do that, especially when you have like complicated cost benefit analyses going into it. I don't think that people, when they're getting in a car, are actually doing any kind of calculation about what the cost-benefit analysis is to their own productivity versus the danger of something very bad happening to them. Like, well, as a society, someone's done this calculation, we've all kind of done this together, and so therefore, like, getting in a car is totally fine. I'm going to do that. And nobody second guesses anybody. If somebody dies in a car crash, you don't say, "Wow, what a moron for getting in a car." No.
--:-- Another way that we can get there is through transparency. So if the decision is pretty transparent... Another way to get there is a status quo. So like a good status quo example that I like to give, because everybody can understand it, is you have to get to a plane and you're with your significant other in the car and you go the usual route. So you go your usual route, like you go the literally, this is the route that you've always gone. And there's some sort of accident, there's bad traffic, you miss the plane. And you're mostly probably comforting each other in the car. It's like, "What could we do?" You know?
--:-- But then you get in the car and you announce to your significant other, "I've got a great shortcut. So let's take the shortcut to the airport." And there's the same accident, whatever, horrible traffic, you missed the flight. And that's like that status quo versus non-status quo decision.
--:-- Sonal: Right. You're going against what's familiar and comfortable.
--:-- Annie Duke: Exactly. If we go back to the car example, when you look at what the reaction is to a pedestrian dying because of an autonomous vehicle versus because of a human. We're very, very harsh with algorithms. For example, if you get in a car accident and you happen to hit a pedestrian, I can say something like, "Well, Mark didn't intend to do that." Because I think that I understand your mind is not such a black box to me. So I feel like I have some insight into what your decision might be. And so more allowing some of the uncertainty to bubble up there. But if this black box algorithm makes the decision, now all of a sudden I'm like, "Get these cars off the road." Never mind that the human mind is a black box itself.
--:-- Mark Andreessen: Of course.
--:-- Annie Duke: But we have some sort of illusion that I understand sort of what's going on in there, just like I have an illusion that I understand what's going on in my own brain.
--:-- And you can actually see this in some of the language around crashes on Wall Street, too. When you have a crash that comes from human beings selling, people say things like "the market went down today." When it's algorithms, they say it's a "flash crash." So now they're sort of pointing out like this is clearly in the skilled category. It's the algorithm's fault. We should really have a discussion about algorithmic trading and whether this should be allowed. When obviously the mechanism for the market going down is the same either way.
--:-- So now if we understand that, so exactly your matrix. Now we can say, "Well, okay, human beings understand what's going to get them in the room." And pretty much anybody who's living and breathing in the top levels of business at this point is going to tell you process, process, process. I don't care about your outcomes. Process, process, process. But then the only time I ever have like an all hands on deck meeting is when something goes wrong.
--:-- Like, let's say that you're in a real estate investing group. And so you invest in a particular property based on your model and the appraisal comes in 10% lower than what you expected. Like everybody's in a room, right? You're all having a discussion. You're all examining the model. You're trying to figure out. But what happens when the appraisal comes in 10% higher than expected? Is everyone in the room going, "What happened here?"
--:-- There is the obvious reality, which is like, we don't get paid in process. We get paid in outcomes. Poker players, you don't get paid in process. You get paid in outcome.
--:-- Mark Andreessen: Incentive alignment. It's not completely emotional. There's also an actual, there's a real component to it.
--:-- Annie Duke: Yeah. So two things. One is you have to make it very clear to the people who work for you that you understand that outcomes will come from good process. That number one. And then number two, what you have to do is try to align the fact that as human beings, we tend to be outcome driven to what you want in terms of getting an individual's risk to align with the enterprise risk. Because otherwise you're going to get the CYA behavior.
--:-- And the other thing is that we want to understand if we have the right assessment of risk. So one of the big problems with the appraisal coming in 10% too high there. Could be that your model's correct. It could be that you could have just a tail result, but it certainly is a trigger for you to go look and say, "Was there risk in this decision that we didn't know was there?" And it's really important for deploying resources.
--:-- Sonal: I have a question about translating this to say non-investing context. So if in the example of Mark's matrix, even if it's a non-consensus wrong, you are staking money that you are responsible for. In most companies, people do not have that kind of skin in the game. Right. So how do you drive accountability in a process driven environment that the results actually do matter? You want people to be accountable, yet not overly focused on the outcome. Like how do you calibrate that?
--:-- Annie Duke: So let's think about how can we create balance across three dimensions that makes it so that the outcome you care about is the quality of the forecast. So first of all, obviously this demands that you have people making forecasts. You have to state in advance: "Here's what I think. This is my model of the world here where all the places are going to fall. So this is what I think." So now you stated that and the whether the outcome is quote unquote good or bad is how close are you to whatever that forecast is. So now it's not just like, "Oh, you won to it or you lost to it." It's "Was your forecast good?" So that's piece number one: make sure that you're trying to be as equal across quality as you can and focus more on forecast quality as opposed to like traditionally what we would think of as outcome quality.
--:-- So now the second piece is directional. So when we have a bad outcome and everybody gets in the room, when was the last time that someone suggested, "Well, you know, we really should have lost more here." Nobody's saying that. But sometimes that's true. Sometimes if you examine it, you'll find out that you didn't have a big enough position. It turned out, okay, well, maybe we should have actually lost more. So you want to ask both up, down, and orthogonal. So could we have lost less? Should we have lost more? And then the question of should we have been in this position at all?
--:-- Mark Andreessen: So in venture capital, after a company works and exits, let's say it sells for a lot of money, you do often say, "God, I wish we had invested more money." You never, ever, ever, ever — I have never heard anybody say on a loss, "We should have invested more money."
--:-- Annie Duke: See, I wouldn't be great if someone said that. Like, wouldn't you love for someone to come up and say that to you? That would make you so happy.
--:-- Mark Andreessen: What would be the logic of why they should say that? I still don't get the point.
--:-- Annie Duke: Exactly. Why does that matter? I don't really understand that. So let's, can I just like simple in a poker example. So let's say that I get involved in a hand with you and I have some idea about how you play. And I have decided that you are somebody that if I bet X, you will continue to play with me. Let's say this is a spot where I know that I have the best hand. But if I bet X plus C that you will fold. So if I go above X, that I'm not going to be able to keep you coming along with me. But if I bet X or below that you will. So I bet X, you call, but you call really fast in a way that makes me realize, "Oh, I could have actually bet X plus C." You hit a very lucky card on the end. And I happened to lose the pot. I should have maximized at the point that I was a mathematical favorite.
--:-- Mark Andreessen: Your model of me was wrong, which is a learning independent of the win or the loss.
--:-- Annie Duke: Exactly. So you need to be exploring those questions in a real honest way. Right. Because it has to do with how you size future bets.
--:-- Sonal: This is exactly like a company betting on a product line. Correct. And then like ticking, like, you know, what the next product line is going to be and then not having had the information that would then drive a better decision making process.
--:-- Annie Duke: Right. So think about the learning loss that's happening because we're not exploring that. The negative direction is. And now you should do this on wins as well. So if you do ever discuss a win, you always think like, "How could I press? How could I have won more? How could I have made this even better? How could I do this again in the future? Should we have won less? We oversized the bet and then got bailed out by a fluke. We should have actually had less in it." And sometimes not at all, because sometimes the reasons that we invested turned out to be orthogonal to the reasons that it actually ended up playing out in the way that it was. And so had we had that information, we actually wouldn't have bet on this at all because it was completely orthogonal. Like we totally had this wrong. It just turned out that we ended up winning. And that can happen. I mean, obviously that happens in poker all the time.
--:-- But what does that communicate to the people on your team? Good, bad, I don't care. I care about our model. I want to know that we're modeling the world well and that we're thinking about how do we incorporate the things that we learn? Because we can generally think about stuff we know and stuff we don't know. There's stuff we don't know we know, obviously. So, well, we don't worry about that because we don't know we don't know it. But then there's stuff we could know and stuff we can't know. It's things like the size of the universe or the thoughts of others or what the outcome will actually be. We don't know that.
--:-- Sonal: I have a question about this, though. What is a time frame for that forecast? So let's say you have like a model of the world, a model of a technology, how it's going to adopt, how it's going to play out. In some cases, there are companies that can take like years to get traction. You want to get your customers very early to figure that out, right? So you can get that data. But how much time do you give? Like how do you size that timeframe for the forecast? So you're not constantly updating with every customer data point. And so you're also giving it enough time for your model, your plan, your forecast to play out.
--:-- Annie Duke: You have to think about very clearly in advance, "What's my time horizon? How long do I need for this to play out?" But also don't just do this for the big decisions, because there's things that you can forecast for tomorrow as well, so that you end up bringing it into just the way that people think. And then once you've decided, okay, this is the time horizon of my forecast, and you would want to be thinking about what are forecasts we make for a year, two years, five years for this specific decision to play out. And then just make sure that you talk in advance at what point you'll revisit the forecast. So you want to think in advance, "What are the things that would have to be true for me to be willing to come in and actually revisit this forecast?" Because otherwise you can start, as you just said, like you can turn into like a super bad-
--:-- Sonal: You're like a leaf in the wind.
--:-- Annie Duke: I mean, right, exactly. Because then you're like one bad customer and you suddenly like over-rotate on that when in fact it could have been-
--:-- Annie Duke: So if you include that, if you include that in your forecast, "Here are the circumstances under which we would come in and check on our model," then you've already got that in advance. So that's actually creating constraints around the reactivity, which is helpful.
--:-- Mark Andreessen: Two questions on practical implementation of the theory. So what I'm finding is more and more people understand the logic of what you're describing because people are getting exposed to these ideas and kind of expanding in importance. And so more and more people intellectually understand this stuff. But there's two kind of, I don't know, so call it emotion-driven warps or something that people just really have a hard time with.
--:-- So one is they — you understand, this could be true of investor, CEO, product line manager in a company, you know kind of anybody in one of these domains — which is you can't get the non-consensus right results unless you're willing to take the damage, right, the risk on the non-consensus wrong results. But people cannot cope with the non-consensus wrong outcome. They just emotionally cannot handle it. And they would like to think that they can and they intellectually understand that they should be able to, but as you say, when they're in the room it's such a traumatizing experience that it's the touching the hot stove. They will do anything in the future to avoid that.
--:-- Is that just — and so like one interpretation would be that's just simply flat out human nature. And so to some extent the intellectual understanding of here doesn't actually matter that much because there's an emotional override. And so that would be a pessimistic view on our ability as a species to learn these lessons, or do you have a more optimistic view of that?
--:-- Annie Duke: I'm going to be both pessimistic and optimistic at the same time. So let me explain why. Because I think that if you move this a little bit, it's a huge difference. You sort of have two tacks that you want to take. One is how much can you move the individual to sort of train this kind of thinking for them? And that means naturally they're thinking and forecast a little bit more that when they do have those kinds of reactions, which naturally everybody will, they right the ship more quickly so that they can learn the lessons more quickly.
--:-- I mean, I actually just had this happen. I turned in a draft of my next book, the first part of my next book to my editor, and I just got the worst comments I've ever gotten back. And I had a really bad 24 hours. But after 24 hours, I was like, "You know what? She's right." Now, I still had a really bad 24 hours. And I'm the like, "Give me negative feedback," like queen, because I'm a human being, but I got to it fast. Like I sort of got through it pretty quickly after this. I mean, I, you know, the, you know, on the phone with my agent saying "I'm standing my ground, this is ridiculous." And then he got a text the next day being like, "No, she's right." And then I rewrote it. And you know what? It's so much better for having been rewritten. And now I can get to a place of gratitude for having the negative feedback, but I still had the really bad day. So it's okay. It doesn't go away.
--:-- Mark Andreessen: Yeah.
--:-- Annie Duke: And it's okay. Like we're all human. Like we're not robots. So number one is like, how much are you getting the individuals to say, "Okay, I improved 2%." That's so amazing for my decision-making and my learning going forward.
--:-- And then the second through line is what are you doing to not make it worse? Because obviously for a long time, people like to talk about, "I'm results oriented." I mean, it's like the worst sentence that could come out of somebody's mouth.
--:-- Mark Andreessen: Why is that the worst? I've heard that a lot.
--:-- Annie Duke: Because you're letting people know that all you care about is like, did you win or lose? That's fantastic. Be results oriented all you want. You should pay by the piece. You will get much faster work. But the minute that you're asking people to do intellectual work, results oriented is like the worst thing that you could say to somebody. So I think that we need to take responsibility and the people in our orbit, we can make sure at minimum that we aren't making it worse. And I think that that's pessimistic and optimistic. I don't think anyone's making a full reversal here.
--:-- Mark Andreessen: So the second question then goes to the societal aspect of this. And so we'll talk about the role of the storytellers or, as they're sometimes known, the journalists.
--:-- Annie Duke: And the editors.
--:-- Mark Andreessen: I love it.
--:-- Annie Duke: And the editors and the editors and the publishers.
--:-- Mark Andreessen: And so the very first reporter I ever met when I was a kid, Jared Sandberg at The Wall Street Journal, the internet was first emerging. Like there were no stories in the press about the internet. And I used to say, like, there's all this interesting stuff happening. Why am I not reading about any of it in these newspapers? And he's like, "Well, because the story of something is happening is not an interesting story." He said there are only two stories that sell newspapers. He said one is, "Oh, the glory of it." And the other is, "Oh, the shame of it." And basically he said it's conflict. So it's either something wonderful has happened or something horrible has happened. Those are the two stories.
--:-- And then you think about business journalism as kind of our domain. You got to think about it. You're like, those are the only two profiles of a CEO or founder you'll ever read. It's just like, "What a super genius for doing something presumably non-consensus and right." Or "What a moron. Like, what a hopeless idiot for doing something non-consensus and wrong." And so since I've become more aware of this, like it's actually gotten very hard for me to actually read any of the coverage of the people I know, because it's like the people who got non-consensus right, they're being lavish with too much phrase. And the people got non-consensus wrong, they're being damned for all kinds of reasons. The traits are actually the same in a lot of cases.
--:-- So I guess as a consequence, like if you read the coverage, it really reinforces this bias of being results oriented.
--:-- Annie Duke: Like it's not our fault that people don't want to read a story that says, "Well, he tried something and it didn't work this time."
--:-- Mark Andreessen: Yes, exactly.
--:-- Annie Duke: But it was mathematically pretty good.
--:-- Mark Andreessen: If we go back to Pete Carroll, this is a pretty great case. If we think about options theory, just quickly the paths preserve the option for two run plays. So if you want to get three tries at the end zone instead of two, strictly for clock management reasons, you pass first. And that's not going to kick off ESPN SportsCenter that night. And so optimistic or pessimistic that the narrative, the public narrative on these topics will ever move.
--:-- Annie Duke: I am super, super pessimistic on the societal level, but I am optimistic that if we educate people better, we can equip them better for this. So I really focused on how do we make sure that we are equipping people to be able to parse those narratives in a way that is more rational. And particularly, you know, now there's so much information and it's all about the framing and the storytelling. And it's particularly driven by what's the interaction of your own point of view. We could think about it as partisan point of view, for example, versus the point of view of the communicator of the information and how is that interacting with each other, you know, in terms of like how critically are you viewing the information, for example.
--:-- I think this is another really big piece of the pie and somewhat actually related to the question about journalism, which is that third dimension of the space. So we talked about two dimensions, which is sort of outcome quality and how are you allowing that you're exploring both downside and upside outcomes in a way that's really looking at forecast? How are you thinking directionally so that you're more directionally neutral? But then the other piece of the puzzle is how are you treating omissions versus commissions?
--:-- Sonal: So one of the things we know with this issue of resulting is, here's a great way to make sure that nobody ever results on you: don't do anything. If I just don't ever make a decision, I'm never going to be in that room with everybody yelling at me for the stupid decision I made because I had a bad outcome. But we know that not making a decision is making a decision. We just don't think about it that way.
--:-- Annie Duke: And it doesn't have to just be about investing. You can have a shadow book of your own personal decisions. I remember I was giving advice to someone who was about 23, newly out of college, had been in a position for a year and was really unhappy. He asked me, "I don't know what to do. Should I change jobs?" I did the time traveling trick. I said, "Imagine it's a year from now. Do you think you'll be happy in this job?" He said, "No." So I said, "Maybe you should go find another position."
--:-- This is what he said to me, and it shows how much people don't realize that the status quo is a decision. He said, "But if I go find another position and spend another year learning the ins and outs, and it turns out I'm not happy there, I'll have wasted my time." I said, "Let's think about this. The job you're in, which is a choice to stay, you've told me is 100% going to make you sad in a year. If you go to the new job, yes, it's more volatile, but at least you've opened the range of outcomes." But he didn't want to do it because staying didn't feel like he was choosing it. He felt that if he went to the other place and ended up sad, it would be his fault, a bad decision.
--:-- Sonal: That's so profound. In my case, it was a decision I didn't know I had made to not have kids. It's still an option, but it's probably not going to happen. My therapist told me that my not deciding was a choice. I was blown away. It allowed me to examine what was going on in that framework so I wouldn't do that in other arenas where I might actually want something. Or maybe I don't, but at least it's a choice with intentionality behind it.
--:-- Annie Duke: Well, I appreciate you sharing. I think people should be sharing this kind of stuff so they feel like they can talk about these things. In my book, I have examples of how you make choices about raising your kids when it feels so consequential, when you're making decisions for other people. You're trying to decide, should I have kids or shouldn't I? This school or that school? Who am I supposed to marry? Where am I supposed to live?
--:-- We can talk about investing in financial instruments, but we all have resources we're investing: our time, energy, heart, friendships, relationships. If you choose to have children, you're deploying resources with some expected return, some good, some bad. If you choose not to, that's a different deployment toward other things.
--:-- Sonal: And you need to know that there are limits. Everything isn't a zero-sum game.
--:-- Sonal: But evolution has approached the world as a zero-sum game, and our toolkit makes it one. We need to view everything as a zero-sum game when it comes to those tradeoffs and resources, because you are losing something every time, even in a non-zero game. I don't feel the world is a zero-sum game in terms of activities we engage in—we can both win. But it's a zero-sum game between you and the other versions of yourself that you don't choose.
--:-- Annie Duke: Exactly. Or an organization and the other versions of itself it doesn't choose. There's a set of possible futures that result from not making a decision as well.
--:-- Sonal: On an individual decision, let's put things into three categories: clear misses, near misses, and hits. Some are clear misses—throw them out. Some I agonize over and do a lot of analysis on. The ones that become a yes go into the hit category. The other is a near miss. What happens with near misses is they just go away.
--:-- On any given decision, say an investment decision, if I explain why I didn't invest in a company, it sounds incredibly reasonable. You can only see in the aggregate, across many decisions, that I tend to have a bias toward missing, toward saying, "We're not going to do it so I don't have to stick my neck out." It's hard to spot because I can tell a good story on any individual decision.
--:-- The way to combat that, to get people to think about what we really care about—forecast, not outcomes—is to keep a shadow book. The anti-portfolio should contain all your near misses, but you also have to take a sample of the clear misses, which nobody ever looks at. The near misses tend to be in your periphery if they happen to be big hits.
--:-- Annie Duke: So here's the problem. The good news is we have actually done this. We call it the shadow portfolio. The way we do it is we make the investment, then take the other equivalent deal of that vintage and size that we almost did but didn't, and put that in the shadow portfolio. We're trying to do an apples-to-apples comparison.
--:-- Sonal: In finance theory terms, the shadow portfolio may well outperform the real portfolio.
--:-- Annie Duke: Yes, because the shadow portfolio may be higher variance—higher volatility, higher risk, and therefore higher return.
--:-- Sonal: Correct. Because the fear is that the ones hitting are less spiky, less volatile, less risky.
--:-- Annie Duke: What's wonderful is that when you decide not to invest in a company, you model out why. It's often a single flaw we've identified. Like, "We would do it except for X," where X looks like something potentially existentially bad. That's written in there. Then just make sure people sample the ones rejected out of hand.
--:-- Sonal: We never do that. Let me ask you how to do that. We reject 99 for every one we do.
--:-- Annie Duke: You just take a random sample. As long as it's kept in view a little bit, it pushes against your model. It gets people to have the right kind of discussion. It communicates to the people around you, "I care about your model."
--:-- Sonal: Let me ask you a different question about groups of decisions—portfolios of decisions. Early on in the firm, I had a discussion with a friend. He looked at me and said, "You're thinking about this all wrong. You're thinking about invest or not. That's the wrong way. You should be thinking, is this one of the 20 investments of this class of size that you're going to put in your portfolio?" When you're evaluating an opportunity, you're talking about that opportunity, but it's very hard to abstract that from the broader concept of a portfolio.
--:-- Annie Duke: What I would suggest is doing some time traveling. When people are down in the weeds, say, "Let's imagine it's a year from now. What does the portfolio look like of these investments of this kind?" I'm a big promoter of time traveling—asking, "What does this look like in a year? In five years? Are we happy? Are we sad? If we imagine we have this, what percentage do we think will have failed?" We understand any one of these could have failed. That allows you to get out of what feels like the biggest decision on earth and see it in the context of everything going on.
--:-- Sonal: One of the most powerful things my therapist gave me was such a simple construct. Doing certain things today is like stealing from my future self. It blew my mind.
--:-- Annie Duke: It's so beautiful. It seems hokey and self-help-y, but I had never thought of it. We're on a continuum—so-and-so in the past, so-and-so today, so-and-so this woman in the future I haven't met yet. The idea of stealing from her...
--:-- Sonal: That's a really lovely way to put it.
--:-- Annie Duke: Isn't it so good? She's a fucking awesome therapist. I like talking publicly about therapy because I like lessening the stigma. I'm very open about it. There's no reason to hide it.
--:-- Sonal: Some of the ways we deal with this prospectively is employing good decision hygiene. One is time traveling, imagining what this will look like in the future so it's metabolized into the decision. Two is making sure you have pushback once consensus is reached. "Great. Now let's go disagree with each other." Then, to avoid the consensus problem, elicit input not in a room with other people. When someone has a deal to bring to everybody, it goes to people individually. They write their thoughts individually, then it comes into the room. That reduces the pylon effect.
--:-- Annie Duke: That reduces the effects of consensus.
--:-- Sonal: This is how you come up with your forecast of the future, which is then memorialized. That memorializing acts as a prophylactic. It gives you your forecast, which is what you're trying to push against. You're trying to change the attitude so the forecast is the outcome we care about. It acts as a prophylactic for emotional issues. You can say, "We all talked about this. We had our red team. We had a good steel man. We thought about why we were wrong. We questioned if someone had the outside view. By eliciting information individually, we were less likely to be in the inside view." All that good hygiene protects you. You're going to have a bad 24 hours, for sure. But you can get out of it more quickly, more often, and get to a place where you can say, "Moving on to the next decision. How do I improve going forward?"
--:-- Annie Duke: Building on that, returning to my optimism/pessimism question: if society isn't going to move on these issues but we can move as individuals, one form of optimism is that more of us can move as individuals. Another is that there will always be room in these probabilistic domains for the rare individual who can actually think about this stuff correctly. There will always be an edge. There will always be people much better at poker than everybody else.
--:-- Sonal: I think that's for sure. Most people simply can't or won't get there. A few people in every domain might take the time and have the discipline to get all the way there, but most can't or won't.
--:-- Annie Duke: In some ways, maybe that's okay. From an evolutionary standpoint, that kind of thinking was selected for a reason. It's better for survival, likely better for happiness. Don't touch the burnt stove twice. Run away when you hear rustling in the leaves. Don't sit around and say, "It's a probabilistic world. I have to figure out how often that's a lion that's going to eat me." Most people shouldn't be playing in the World Series of Poker. People come up to me and say, "I play poker, but it's just a home game." I say, "What are you saying? Just a home game?" There are different purposes to poker. You probably have a great time doing it. It brings you enjoyment. You don't have an interest in becoming a professional. Be proud of that. I play tennis. I'm not saying, "I'm just playing USTA 3.5." I'm really happy with my tennis. People have different things they love. I would love to spread this kind of thinking more, but of course some people will end up in this category more than others. That's okay.
--:-- Sonal: One thing I get asked all the time is, "We can't really do this because people expect us to be confident in our choices."
--:-- Annie Duke: Don't confuse confidence and certainty. I can express a lot of uncertainty and still convey confidence. Ready? "I'm weighing these three options: A, B, and C. I've done the analysis. Here it is. I think option A will work out 60% of the time. Option B, 25%. Option C, 15%. Option A is the clear winner." I just expressed so much uncertainty, but also a lot of confidence. I've done my analysis. This is my forecast. All I ask people to do before they bank the decision is ask, "Is there some piece of information I could find out that would reverse my decision and actually cost something?" Not that would make it go from 60 to 57—I don't care about modulating that much. I care that it would actually change your decision.
--:-- Sonal: Organizations can bake that into their process, not just in forecasting but in arriving at the decision, so that next time, right or wrong, they make a better decision. If the answer is yes, go find it. Sometimes the answer is yes but the cost is too high—time, opportunity cost. So you just don't. Then you say, "We all recognized that if we found this out, it would change our decision, but we agreed the cost was too high, so we didn't." If it reveals itself afterwards, you're not surprised.
--:-- Annie Duke: You've talked about how people should use confidence intervals in communicating, which I love because we're both ex-PhD psychology people—neither finished. One thing I struggle with in an organizational context, translating this to a big group of people, is how do you communicate a confidence interval and all the variables in an efficient, compressed way? Part of communication in organizations is emails and quick decisions. You can have all the process behind the outcome, but how do you convey that to people who weren't in the room?
--:-- Sonal: I think there's a simpler way to express uncertainty: using percentages. Sometimes you can only come up with a range. For example, if I'm talking to my editor in a quick email, I'll say, "You'll have the draft by Friday 83% of the time. By Monday, you'll have it 97% of the time." Those are inclusive. It's another way of doing a confidence interval without making it so wonky. I'm letting her know most of the time you'll get it on Friday, but I'm building in that my kid gets sick or I have trouble with a section. I set expectations that way.
--:-- Annie Duke: That's fantastic.
--:-- Sonal: We've been trying to do forecasting for timelines for podcast editing and episodes. I feel frustrated because I have frameworks—if there are accents, more than two voices, complex room tone, feedback, sound effects. I know the factors that go into my model, but I don't know how to put a confidence interval in our pipeline spreadsheet for all the content coming out.
--:-- Annie Duke: One way to do it is think about the range. What's the earliest you could get it? Put a percentage on that. Then think about the latest day they'll get it. Put a percentage on that. What's wonderful is that you've set expectations properly so you're not getting yelled at on Friday, "Where the hell's the draft?" Half the battle is that because we think we have to give a certain answer, it ends up boy who cried wolf. If I tell her I'll get it on Friday and 25% of the time I'm late, she stops putting stock in what I say. Number two is that you infect other people with this in a good way. It moves them off black-and-white thinking.
--:-- Sonal: One thing I love thinking about is the difference between a deadline and giving a range. We ask ourselves and others, "Are you sure?" way too often. It's a terrible question because the only answer is yes or no. What should we be asking?
--:-- Annie Duke: "How sure are you?"
--:-- Sonal: I have a quick question. Earlier you mentioned uncertainty. How do you as an organization build that uncertainty in by default?
--:-- Annie Duke: We talked about time traveling and its usefulness. One thing I like to think about is not overvaluing the decision right at hand. Think about how you're going to figure out the best path. The goal you want to reach defines the best path. If you're standing at the bottom of a mountain you want to summit, all you can see is the base. You're worried about how to get the next little bit. But if you're at the top, you see the whole landscape. You can see what the best path looks like. We want to find a cognitive way to get there. That's through good time traveling, backcasting, and premortem.
--:-- Let's look backwards instead of forwards. "This is now the headline. Let me think about why that happened." For a simple weight loss goal: "I want to lose a certain amount of weight in six months. It's the end of six months. I've lost it. What happened? I went to the gym. I avoided bread. I didn't eat sweets." You have this list. Then pair that with a premortem: "I didn't get to the top of the mountain. I failed to lose the weight." Then list all the things you can do to counter-program against that. That reveals different things. Some are luck—can I reduce the influence of luck? Some are decisions—"I went into the break room every day and there were donuts. I couldn't resist." Now you can think about how to counter that, bring other people in. Then there's stuff out of your control—"It turned out I had a slow metabolism." You become much less reactive and more nimble because you've seen the whole landscape, both good and bad.
--:-- Sonal: But people are very loath to do premortems because imagining failure feels like failure. People say, "No, you should do positive visualization." Even in brainstorming meetings, everyone says, "Don't dump on an idea." But the point is you don't have to dump on an idea and kill the winnowing of options. As part of the process, you should be premorteming it.
--:-- Annie Duke: There's wonderful research by Gabrielle Oettingen that I recommend. Across domains, she found that when people do positive fantasizing, the chances they actually complete the goal are lower than if they do negative fantasizing. Research shows that time traveling and backwards thinking increases identifying reasons for success or failure by about 30%. You're more likely to see what's in your way.
--:-- For example, she did a simple study with college students. "Who do you have a crush on that you haven't talked to yet?" One group did positive fantasies: "I'm going to meet them, ask them out, it's going to be great, we'll live happily ever after." Another group did negative fantasizing: "What if I ask them out and they say no? I was really embarrassed." She revisited them four months later to see which group had actually gone on a date. The ones who did negative fantasizing were much more likely to have gone out on the date.
--:-- Sonal: That's fantastic. In teams, we tend to view people as naysayers, but we don't want to think of them as downers. I suggest dividing those into two processes. Have the group individually do a backcast. Have the group individually write a narrative about a premortem. That changes the rules of the game. Being a good team player is now identifying the ways you fail.
--:-- Annie Duke: I love what you said. Having two modes as a way to get into two mindsets, where you're not stopping people from feeling like a team player. That's the issue. "Don't crap on my goal." What they're really saying is, "You're not being a team player." So change the rules of the game.
--:-- Sonal: You had a line in your book about how regret isn't unproductive. The issue is that it comes after the fact, not before. I don't want people to think about how they feel right after the outcome because they'll overweight regret. You want to think about regret before you make the decision. You have to get it within the right timeframe. In the moment of the outcome, when you're feeling sad, stop and say, "Am I going to care about this in a year?" Think about yourself as a happiness stock. If we can get that 10,000-foot view on our own happiness, we can get to that regret question a lot better.
--:-- You don't need to improve that much to get really big dividends. You make thousands of decisions a day. If you can get a little better at this stuff—de-bias a little bit, think more probabilistically, wrap your arms around uncertainty to free yourself from the emotional impact of outcomes—it will have a huge effect on your future decision-making.
--:-- Annie Duke: Well, that's amazing, Annie. Thank you so much for joining the a16z podcast.
--:-- Sonal: Thank you very much. Yes, thank you.
