--:-- Sonal: Hi, everyone. Welcome to the A16Z podcast. I'm Sonal. Today, we have another one of our early book launch episodes for a new book coming out next week by expert decision strategist and leading World Series of Poker player, Annie Duke. You can catch the podcast we did with her a couple years ago for the paperback release of her first book, *Thinking in Bets*. That episode was me and Mark interviewing Annie and was titled "Innovating in Bets," as is perhaps also one of the signature themes of this podcast.
--:-- Sonal: But in this episode, we talk about her new book, *How to Decide*, which picks up where the last book left off. The discussion that follows covers lots of useful strategies, tools, and mindsets for helping all kinds of people and organizations decide under conditions of uncertainty. Annie is interviewed by A16Z managing partner Jeff Jordan, who was previously CEO and then executive chairman of OpenTable, former GM of eBay North America, and much more.
--:-- Sonal: They begin by quickly covering common pitfalls in decision making, then share specific tools – what not to do and what to do – including how to operationalize good decision hygiene into teams and when to spend time deciding or not, especially when not all decisions are equal and some may seem bigger and more impactful, whether it's investing in life decisions like getting married or business decisions, such as what product to invest in or what strategy to pursue or what market or what investment.
--:-- Sonal: As a reminder, none of the following is investment advice, nor is it a solicitation for investment in any of our funds. Please be sure to read a16z.com/disclosures for more important information.
--:-- Jeff Jordan: So Annie, as the author of one of my favorite books, what motivated you to do a sequel, your new book *How to Decide: Simple Tools for Making Better Choices*?
--:-- Annie Duke: When I think about what *Thinking in Bets* was about, it was really the way that our decision making gets frustrated by this kind of discorrelation between decision quality and outcome quality. Toward the end of that book, I did a little bit of exploration about how you might become a better decision maker given the uncertainty, but it was mostly a "why" book. This one is really trying to lay out for people how you actually create a solid and high-quality decision process that's going to do two things.
--:-- Annie Duke: One is get a pretty good view on the luck, which you need to do. You need to be able to see it for what it is. Obviously, you can't control it, but you can see it. The other thing, and I think this was something that was really fun, I got to really dig deep into this problem of hidden information. When we're making decisions, we just don't know a lot because we're not omniscient and we also aren't time travelers. I got to do this really deep exploration into how you might actually improve the quality of the beliefs that inform your decisions. It was a topic I covered a tiny bit in *Thinking in Bets*, but here we do a super deep dive.
--:-- Jeff Jordan: It is a super deep dive. And why I love your books is it's so germane to what we do in our day job, which is make decisions under extreme uncertainty. So to recap, why trying to learn from experience can go sideways.
--:-- Annie Duke: Both of my books kind of start in the same place and then they diverge. I think that's because it's the most important place to start. What I talk about at the beginning of this book is what I call the paradox of experience. Obviously, experience is necessary to become a better decision maker. You do stuff, the world gives you feedback. You do more stuff, the world gives you feedback. Hopefully along the way you're becoming a better decision maker given that feedback.
--:-- Annie Duke: The problem is that any individual experience we might have can actually frustrate that process. We can learn some pretty bad lessons when we take any individual piece of feedback. So experience, while necessary for learning, is also one of the main ingredients that makes us worse decision makers. It comes from this problem that in the aggregate, if you get 10,000 coin flips, we can say something spectacular about the quality of our decisions and what we should learn. But that's not the way our brains process information. Our brains process information sequentially, one at a time. Because we're getting these outcomes one at a time, we're taking really big lessons from something that's really just one data point.
--:-- Annie Duke: The two main ways that frustration occurs is because of resulting, which I cover quite a bit in *Thinking in Bets*, where we use the quality of an outcome to derive the quality of the underlying decision. You can run red lights and get through just fine, and you can run green lights and get in accidents. These things are correlated at one, but with resulting, we act like they are. The other problem is hindsight bias. We aren't really good at reconstructing our state of knowledge at the time we made a decision. Once we know the outcome, we not only view that as inevitable, but we'll also think we knew that outcome was going to occur. None of which are true. Those two things combined are really problematic.
--:-- Jeff Jordan: You had this beautiful imagery of "decision forestry," which resonated with me. I sort of think of them as cognitive illusions. What those illusions are creating for us is the idea as if it's the only outcome that could have occurred. In reality, at the moment we make a decision, there are all sorts of different ways the future could turn out. When we're at the moment of a decision, we can see all those branches of the tree. But after the future unfolds, we take a cognitive chainsaw to that whole tree and start lopping off the branches we happen not to observe. We forget about all the counterfactual worlds. How do you keep the forestry from lopping off the branches?
--:-- Annie Duke: There are two tools, actually three. The first has to do with trying to reconstruct what the actual state of knowledge you were in. When you think about what did I know beforehand and what did I know afterwards, you can start to reasonably see what information was informing the decision at the time. When you go through this process, you'll spot, "No, wait a minute, that was something that revealed itself after the fact." That can be very helpful.
--:-- Annie Duke: Another thing that can be very helpful is to go through this process of thinking about the two-by-two matrix of the relationship between decision quality and outcome quality. There's a quadrant: good decision, good outcome – which you can think of as an earned reward; good decision, bad outcome – that would be bad luck; bad decision, good outcome – dumb luck; bad decision, bad outcome – just desserts. When you think about any outcome in your life, if you do that over time, you'll see certain patterns about which quadrant you're really filling in a lot. If you see that you're really only putting things into good-good and bad-bad, you need to start seeing how luck is influencing you.
--:-- Annie Duke: The other thing you want to do is to start thinking about the good-good quadrant, because we are asymmetrically willing to go and try to find some luck in there. If you have a bad outcome, you already feel bad. It's kind of nice to go in and deconstruct that and analyze process and really look at the quality of the decision. If you find some bad luck in there, you get a little relief. You get off the hook. Luck is giving you a way out. So we're pretty eager to explore those bad outcomes.
--:-- Annie Duke: What we're not so eager to do is, when we have a good outcome, apply the exact same process. To actually spend some time thinking, "What was luck? Was there a better way?" The reason we don't want to look at that is because we feel pretty good. If you find out you won because of luck, that's a door you don't want to have open. I put a lot of focus on using this tool of really digging into that one quadrant. To think about which quadrant it fits in, you have to apply this other tool, which you can do in retrospect: explore what are the other things that could have happened. If you don't understand those counterfactuals, it's very hard to appropriately place any outcome into the right quadrant. I have tools in the book that help you reconstruct these things retroactively.
--:-- Jeff Jordan: It's kind of interesting. The investment community often tries to capture the thinking at the time through the investment memo, which records, "Okay, these are the potential outcomes we can envision. Here are the probabilities of the different outcomes. In total, we were willing to make this bet, even though there are some outcomes that are pretty unattractive."
--:-- Annie Duke: That absolutely, if you think about a knowledge tracker, that's what you're doing. You're trying to reconstruct an investment memo. It's better than nothing. But what you really want to be doing is doing this stuff prospectively. You want to have some record of not only what you thought at the time, but also exactly what you said: what are the ways we think this could turn out? What are the payoffs to each of those possibilities? How probable do we think those are? You can look at generally two things: what's the expected value, and what's my downside risk. Then you can compare options to each other.
--:-- Annie Duke: What I think is really important about creating this evidentiary record at the time of the decision, as opposed to trying to reconstruct it, is that it's not actually an extra step. People talk about decision journals, which feels like work because it feels like an extra step. The fact is that a really great decision process is going to produce this evidentiary record naturally. What the evidentiary record is giving you – what that investment memo is supposed to give you – is your expectations of the world. Not just, "Do I think I'm going to win or lose at this probability?" but also, "What do we think is going to be true of the world in general?"
--:-- Annie Duke: What I find in my work is that when people lose, they'll do these process dives. The problem is when there's a big win, they're like, "We won. Yep. Exactly."
--:-- Jeff Jordan: When an investment goes bad, you do spend time trying to say, "Okay, what can I learn? What can I do differently?" And then when it goes well, you just spike the ball in the end zone and do a dance.
--:-- Annie Duke: We really are just spiking the ball. But there's so much to be learned from the wins as well. I would argue actually more. Particularly when the power is up, in a lot of ways, there's more to be learned from the wins than the losses. You can win for all sorts of reasons that you didn't expect. Yet we spend a lot more time exploring the losses that were for reasons we expected than the wins that might have been for reasons that were unexpected. Maybe we could have cleaned up the process, or there was information we were missing that we could have applied. We're losing a lot of the learning time. We're not being very efficient. The other problem with that is that it has downstream effects that are quite bad. I'm going to do things that are very consensus. I'm going to want everybody to agree with me.
--:-- Jeff Jordan: Yeah, that resonates a lot. So you take on using a pro-con sheet. It was funny. I was cleaning up offices a couple years ago, and I found sheets in different places and aggregated my career decisions. I came to the conclusion that they were pretty much worthless. So you come to the same conclusion in the book. Why are pro-con sheets worthless?
--:-- Annie Duke: A pro-con list is actually a decision tool. If you have a choice between that and nothing, I think a pro-con list is very slightly better than nothing. But here are the problems. The first is that it's flat. It lacks any dimension. It's a side-by-side list: here are the pros, here are the cons. I don't understand how you would weigh one side against the other without adding some dimension. That dimension would be two things: one is magnitude – how bad? The other dimension that's missing, which is terrible, is probability.
--:-- Annie Duke: In that sense, I'd rather just use a decision tree. For an option I'm considering, I want to think about what are the reasonable possibilities, what are the payoffs for those, and what are the probabilities of those things occurring. Then I can add that dimension back in. Without that dimension, it's not a great tool for comparing one option to another. I can't calculate any kind of weighted average. If I'm choosing between two colleges, is the one with more pros the one I'm supposed to go to? I really don't know because I don't have this dimension.
--:-- Annie Duke: The third problem, which I think is actually the most dire, is that what we're really trying to do is reduce the effect of cognitive bias. Pros and cons lists actually amplify all of that stuff. It's a tool of the inside view. For people listening, I imagine some people saying, "No, when I go to make a pros and cons list, I haven't decided yet." I have news for you: the minute you start thinking about a problem, you've already started deciding. Regardless of whether you've made that explicit, you've already started to get yourself to a conclusion. When you go to do a pros and cons list, it's just going to amplify the conclusion you already want to get to. I think it's just not a very good tool.
--:-- Jeff Jordan: My worst career decision by a mile was joining a company called Real.com right at the beginning of the internet era. It was being purchased by Hollywood Entertainment, which ran the Hollywood Video stores. It was a bad decision. I unwound it in a year. I got scars. But when I went and saw the pros and cons, the pros were aspirational and the cons were delusional. I clearly had decided before I started the list.
--:-- Annie Duke: Yes, exactly. When we start to use something that feels objective, like a pros and cons list, we get that feeling of, "Now I can have confidence that it's a really good decision." One of the things I'm very wary of is that certain things can come into a decision process that feel like they're certifying the process. We end up with this combo of a decision that isn't really better, but that we feel is much more certified.
--:-- Jeff Jordan: I love the tools you described using the decision trees, the prospective gathering of information. Then you took your "how" into an interesting direction. I really enjoyed the part on spending your decision time wisely. It's a book about making great decisions, and then you start talking about all the decisions you shouldn't apply it to.
--:-- Annie Duke: I know. I spent the first six chapters laying out what a pretty robust decision process would look like. Then I sort of take a hard left and say, "Okay, so now that you know, mostly you shouldn't be doing that." Which I know sounds a little bit odd. But it's this meta-skill of understanding that you can't take infinite time to make decisions because opportunities expire and you're losing the ability to do stuff in between. We want to think about what types of decisions merit taking time and what types of decisions merit going fast. It turns out that most of the decisions you're going to make on a daily basis are ones you should be going fast on, much faster than you actually do. In some ways, I think people sort of have it reversed.
--:-- Jeff Jordan: Throw out a couple examples, because that's where it really came alive to me.
--:-- Annie Duke: Okay. Let me ask you this. Pre-pandemic, what's your guess on the average amount of time that an adult in America takes on what to watch on Netflix, what to wear each day – at the moment it's sweatpants, but we'll ignore that – and what to eat? If you're my mother-in-law, she spent a half hour every time we went to a restaurant. She's not even that much of an outlier. If you add it all up over the course of a year, the average adult is spending between six and seven work weeks on just those three decisions. She's looking at the menu, quizzing the wait staff, asking everybody else at the table what they're going to order, trying to go back to the chef, looking on Yelp.
--:-- Annie Duke: Here's my question for you. Let's say we ate a meal together and you were trying to decide between two dishes. What are two dishes you would have a hard time deciding between?
--:-- Jeff Jordan: Fish and a good veggie stew.
--:-- Annie Duke: Okay. So you're trying to decide between those two things. If you're my mother-in-law, you're quizzing everybody. Let's imagine you ordered the veggie stew and it came back and it was really yucky and you didn't even finish it because it was so gross. Now let's imagine it's a year later and I say, "Hey Jeff, how are you feeling right now? Happy or sad? Do you remember that horrible veggie stew you had a year ago? How much of an impact did it have on your happiness today?"
--:-- Annie Duke: Okay. So let's imagine I catch up with you in a month and I say, "Hey Jeff, feeling happy or sad right now? Do you remember that horrible veggie stew you had a month ago? How much of an effect on your happiness today?"
--:-- Annie Duke: What if I take you a week later?
--:-- Annie Duke: Now, if it had been the fish and it had been bad, the week may have been a...
--:-- Jeff Jordan: Maybe, but not the veggie stew.
--:-- Annie Duke: Okay. So what I just walked through with you is something I call the happiness test. I use happiness generally as a proxy for "are you reaching your goals?" because we're generally happier when we're reaching our goals. You can substitute any goal you have in there. This is a way for us to figure out how fast we can go. Basically, the shorter the amount of time in which your answer to the question "Did it affect your happiness at all?" is "no," the faster you can go. Why? Because there's a trade-off between time and accuracy. In general, the more time we take with a decision, the more time there is for us to map things out, calculate expected values, figure out what the volatility might be, gather information, get more data. Generally with time, we should be increasing our accuracy. That's why we can speed up.
--:-- Annie Duke: I'm assuming no food poisoning here. When we look at the worst of those outcomes, it has no effect. It's neither here nor there. That means we can take on the risk of saying, "I'm going to spend less time because I'm willing to risk the fact that I might increase the probability of the worst outcomes, because it doesn't really matter to me."
--:-- Jeff Jordan: And then you make another point: you can repeat the decision next day at the restaurant and order the fish instead of the tasteless stew.
--:-- Annie Duke: That's the other thing you can look at: when you have these low-impact decisions that are quickly cycling and they repeat very quickly – like what to watch on Netflix, what to wear, what to order in a restaurant – we should go really fast for two reasons. One is you're going to get another crack at it in like four hours. The other is that one of the things we don't know well, although we think we do, is our own preferences. We've all had that experience of having a goal, achieving it, and realizing that wasn't really what we wanted in the first place. There are certain types of decisions where it's just really helpful to get some feedback from the world.
--:-- Annie Duke: When we can cycle these decisions really quickly, I'm not too worried about making sure I'm making the best possible decision in terms of accuracy. What I'd rather do is get a lot of cracks, get a lot of at-bats, so that the world can start giving me information back more quickly and I can start cycling that feedback a lot faster. Then I'm going to build much better models of the world, my own preferences, my own goals, my own values, what works and doesn't work. When I do actually make a decision that really matters, my models of the world are going to be more accurate by having just done a whole bunch of stuff really fast and not really cared whether I won or lost.
--:-- Jeff Jordan: That makes perfect sense. Now, one of the chapters that I loved was decision hygiene. I found this book fascinating from the perspective of both an investor and a former operator. For investors, it's obvious. You're making two or three investments a year. You're seeing hundreds of companies. How do you decide? But as an operator, there are a few decisions you make each year that are super, super important. In particular, the ones I used to labor over was, "Okay, you have to commit, you have to invest your product resources, your most viable asset, your engineers, into specific deliverables. Is it going to be A, B, or C?" That's the most important decision I made all year, other than possibly people decisions. Explain a little bit how you can maintain great decision hygiene.
--:-- Annie Duke: The decision hygiene stuff and the ideas of predicting these intermediating states of the world apply so much in a startup environment. The nature of a startup is that you do have very little information and you're making pretty big bets on a future that by definition is going to be somewhat contrarian. Making sure you don't get into this kind of groupthink – I'm not saying don't believe in yourself, of course – but this is actually a way to have more belief in yourself because the quality of the decisions that come out of a good decision hygiene process are going to be so much better. That becomes much more important in a situation where we have a paucity of information. It starts to close feedback loops more quickly for you, which also increases the quality of your models and information. I actually can't think of a place where this is more important than in a startup environment.
--:-- Annie Duke: Let me start with the premise of why you need some decision hygiene. I don't have control over luck. What I can do is make decisions that reduce the probability of a bad outcome. Even if I make a decision that's only going to have a bad outcome 5% of the time, I shall observe it 5% of the time. Luck is what determines when I observe that bad outcome. That's one side of the puzzle. The other side has to do with how you construct your decision process. What do you think your goals are? What do you think your options are? What do you think your resources are? What do you think those possibilities are for any given option? What do you think the probabilities of those things occurring are? Your whole process is built on this foundation. That whole house is sitting on top of a foundation, which is your beliefs.
--:-- Annie Duke: By beliefs, I don't mean things like religious beliefs. I mean, what are your models of the world? How do you think the world operates? What are the facts that you have? What's the knowledge that you have? That foundation has two problems. One is that a bunch of the things we believe are inaccurate. It's like cracks in the foundation. The other is that we don't know very much. It's like a flimsy foundation. The solutions to both problems are the same: we need to start exploring that universe of stuff that we don't know. That's where we run into new information that helps us beef up our foundation. It's also where we happen to run into corrective information, things that can correct the inaccuracies in the things we believe.
--:-- Annie Duke: The other thing that helps us do, when we're talking before about the pros and cons and being caught in your own cognitive bias, is to realize that a lot of the cure to those kinds of problems is to get other people's perspectives. Two people can be looking at the exact same data and come to very different conclusions. That's what a market is: different perspectives colliding. Having set that stage, one of the best things you can do for your decision-making is finding out what other people know and what their perspectives are on the problems you're considering. The problem is that without really good decision hygiene, you're not actually going to be able to execute on that properly.
--:-- Annie Duke: Let's figure out how to get this into a team setting. Basically, human beings are very tribal and we like to agree with each other more than we actually do. Our opinions are really infectious. In order for you to know that you disagree with me, what is the thing that you need to know from me first?
--:-- Jeff Jordan: What do you think? Right, exactly.
--:-- Annie Duke: This is where we get into this huge problem in interpersonal communication. When people ask for feedback, pretty much 100% of the time, they tell the person what they believe first. "I'm thinking about a particular sales strategy or whatever," and I will lay out for you not just the information that you need, but I also tell you my opinions on that. "Give me your unbiased opinion." Now that I've biased you hugely.
--:-- Annie Duke: So the reason your decision hygiene point was so interesting to me is you called out one of my tools that I used as an operator, which was quarantining in group settings. I found an open table that if I walked in and put a strategic choice on the table, there was one and a half people in there who would drive the discussion, and their opinion would always carry the day.
--:-- Annie Duke: So I developed a tool where I would pre, very important big time strategic decisions. I would ask everyone to send me their list of prioritizations and then I would aggregate them and then feed that back to the group to heighten the contradictions essentially. The quiet person who didn't really want to put a contrary point of view and spar with the other person, all of a sudden the data's on the table because they quarantined the gathering of it. And then I found the conversation was so much better than just throwing it open and having the charismatic, loquacious, opinionated person carry the day every time.
--:-- Annie Duke: I could quarantine my opinion, but as soon as someone else talks, as you just so nicely put, it's like everybody else is infected anyway. I'm just a really huge fan of pre-work. Figure out what it is that you're trying to get feedback about, give everybody the same information, and then actually elicit those responses. Now, the more specific, the better. So I like them to rate it, right? Give me it on a scale of zero to five, because then I can find out like Jeff is a four and Annie's a two, and maybe Jeff and Annie need to have a conversation because it turns out that there's quite a bit of dispersion of opinion there.
--:-- Annie Duke: What this allows you to do is, first of all, it actually disciplines your decision process because you have to think about what are the things that matter to this decision that I'm trying to elicit opinions about. And let me be clear. It's not that I don't think people should provide rationales. I think those are actually quite important. It's just that they need to have something that's much more precise. It's like a point estimate, right? Because I need to see where the dispersion is and then let them give the rationale.
--:-- Annie Duke: I used to give a hypothetical budget. You have a million dollars. Here are the 12 ideas you can invest behind, deploy your budget, and each person would deploy. And then all of a sudden, you've got something that's really powerful and you've got, oh, you love this idea and you hated this idea. Let's discuss the idea.
--:-- Jeff Jordan: Right, which I love.
--:-- Annie Duke: So exactly. So you can actually see that they disagree with each other or see that they do agree with each other. It also makes you actually think about what are the component parts of this decision that really matter. You can start to actually create for yourself almost like a little bit like a checklist, but here are the things that we need to pay attention to and that I actually need to get the feedback on. So what Kahneman would call these are mediating judgments. You're thinking about what are the mediating judgments for any broader category that you might be judging on, and that helps you to really discipline the decision process.
--:-- Annie Duke: You then bring that together in one doc and you sort it into here are areas of agreement, here are areas where there's some dispersion. People get to read that prior to coming into the meeting. So they've actually seen sort of the full slate now of what the different opinions are in the group. This does really great things for your meetings. It makes them much more efficient, much more productive. You're not surfacing all that stuff in the room, which just takes a long time. And by the way, you're not going to surface all of it anyway. So that's bad.
--:-- Annie Duke: But the other thing is that now you can come in and you can say, here are areas where we generally agree, yay us, but let's not talk so much about the fact that we agree, which is what happens in a lot of meetings where you'll say something, Jeff, and then I'll go, I agree with Jeff and let me tell you why. And then somebody else is like, yes, and I have more color to add to that because everybody sort of wants credit for that idea. But we don't care now because we already found out we all agree.
--:-- Jeff Jordan: Yeah, yes.
--:-- Annie Duke: Right. The earth is round. Cool. Right. But now it turns out that Annie thinks the earth is flat over here. Okay. So now what are we going to do? Jeff thinks the earth's round, Annie thinks the earth's flat. And that's where you really want to be focusing your time on places where there's dispersion. And you want to focus that time in a way where it's not about convincing anybody of your opinion. It's about just informing the group. And then if anybody sort of agrees with you, I'll say, hey, you know, Sonal, you also agree that the earth is round. Is there anything you want to add to that? So you'll get to say your piece. And then Annie, you believe the earth is flat. Is there something you didn't understand?
--:-- Annie Duke: Now notice in no way am I, is anybody saying you're wrong or you haven't thought about it this way or whatever. It's I get to tell you, here's something I don't understand. And then we sort of get to the point where I say, okay, explain your position. There's really amazing things that come out of that process. Thing number one is you get much more comfortable with the idea that everybody doesn't have to agree. Number two is people have different mental models. And so you get to expose everybody to those different perspectives and the different facts people are bringing to the table. So the whole group becomes more informed, which is awesome.
--:-- Annie Duke: The third thing is that the person who is conveying their position becomes more informed. Why? Because in the process of having to defend why I believe the earth is round, I discovered that I actually can't explain that very well. So maybe I have to go Google some stuff or look it up and there's going to be good stuff that comes out of that because I'm going to be more likely to actually moderate because I'm sort of poking around in my knowledge a little bit.
--:-- Annie Duke: And then the last thing I think that comes out of this that's really good is that once you get into this idea of convey versus convince, you realize that you don't need to agree to decide. You need to inform to decide. And that the idea that all of you would be on equal agreement about whether you should do something or not is completely absurd because we don't have to, because that's the whole point. If you thought that that was the goal, why do you have more than one person on the team?
--:-- Jeff Jordan: Yeah. You want a diversity of opinion. And if you don't tease out the different opinions and you make an inferior decision. I actually thought this was one of my management secrets, but you just outed it in your soon-to-be-a-best-selling book.
--:-- Annie Duke: Yeah. So actually what's interesting about that problem, I think that teams often act like a pros and cons list where we have the intuition that more heads are better than one. So when we bring more heads into a decision, you have this decision that feels much more certified. But what we know is that when you allow people to make these decisions in sort of committee style, like in a team meeting, that the decision quality often isn't better. And there's lots and lots of science that shows this.
--:-- Sonal: So one of the things in venture that is often cited as a challenge in decision-making is the feedback loops can be forever. What's your take on that feedback loop in decision-making?
--:-- Annie Duke: Yeah. So basically my take is that there's actually no such thing as a long feedback loop, which I know sounds weird, right? Because obviously you're saying like we invest in a company, we find out how it exits like 10 years from now, isn't that a really long feedback loop? But the thing is, I mean, going back to this idea that when you make a decision, it's a prediction of the future. It's not like you're just predicting what the exit is going to be. You're predicting a whole bunch of intermediating states of the world. And that might be just like, for example, like, what is the arc of the ability to attract talent for this particular founder? Just like as an example, right?
--:-- Jeff Jordan: You know, obviously, he's going to fund it the next round. I mean, good example, the funding management team.
--:-- Annie Duke: Right. If you knew for a fact that they weren't going to be able to hire a good team, you would invest in them. So it's really good to sort of make predictions about those things and make them probabilistically, because as you're making these types of forecasts, and now over the court in a much faster time period, you're starting to see when we say that there's a 60% chance that this intermediating state of the world is going to exist? Does it actually exist 60% of the time?
--:-- Annie Duke: Because in the end, the thing that's so important to understand is that you are saying that you're an expert at the market that you're investing in. So you want to be explicit about the things those predictions that you making about that market both near term and far term so that you don have to wait around 10 years because the thing is you're going to have to make another investment in between you can't just make the investment wait 10 years get your feedback and then make another investment and now if you're actually being explicit in the way that you're thinking about those things you can actually create much tighter feedback loops it's just aggregated get it into a set of milestones right there's no reason you can't do that out in the world.
--:-- Annie Duke: One of the knocks that people will say about poker is, oh, but you get really fast feedback. And so poo on you. And I'm like, well, yeah, except that it's really just a compressed version. There's the end of the hand, which is what you're thinking about. But in between the start of the hand and the end of the hand, there's all sorts of predictions that I'm making in between.
--:-- Jeff Jordan: I've been an investor for nine years. The feedback loop is 10. I've made 35, 40 decision, if I deferred any learning to the end, it would be pretty wasteful. And there's another psychological thing we fight. The phrase is your lemons ripen first. So if your company goes for 10 years, there's a pretty good probability it's going to have a good outcome. But the ones that die after, can't raise the next round, can't have the management team, that's when your negative outcomes manifest before your positive outcomes. And psychologically, you have to manage through that.
--:-- Annie Duke: Yeah. So this actually, I think, gives you a tool to be able to do that because you have no secondary way to be right. Like, how am I doing in terms of like calibrating around how likely I think this company is to fail? You know, in what ways is it going to fail? What does that actually look like?
--:-- Annie Duke: The other thing that comes from that is that when you make yourself sort of break this into its component parts, when you actually force yourself to do that, I think it actually improves the knowledge that goes into it because you have to start thinking about what are the things that I know? What are the things that I could find out? What are the perspectives that I could consider? What are the mental models that I could apply that will help me with this prediction? Because it is now recorded. It is part of that evidentiary record, which we have already said is incredibly important. That allows you to have that look back. And because you know that you're accountable to it, I think it actually improves the accuracy of the original decision because it makes you be more fox-like rather than hedgehog-like because you know that there's going to be a look back.
--:-- Annie Duke: Basically, fox-like thinking is looking at the world from all sorts of different perspectives, applying lots and lots of different mental models to the same problem to try to get to your answer. And hedgehog is like you approach the world through your one big idea. So you can think about like an investing, you have like one big thesis instead of looking at it from all sorts of different angles. Generally, what you find is that fox-like thinking is generally going to win the day. And this is something like Phil Tutlock, I'm sure a lot of people are familiar with super forecasting, talks a lot about. So apart from the fact that you can speed up the learning cycle, I think it actually improves the decision in the moment, the knowledge that at some point someone's going to look back at it.
--:-- Jeff Jordan: I think that's absolutely true. And it's a good tool. And we may start implementing that at the firm really soon.
--:-- Sonal: You know, as investors, we have the, we get the benefit of being able to make a basket of decisions, you know, diversification. A lot of the people are making decisions are making like one decision. What is the impact to optionality and how do you deal with that one decision?
--:-- Annie Duke: So first of all, here's the secret. Your decisions are a portfolio because you make many of them in your life. And I understand one decision like this particular product decision, but that's actually kind of like a false segregation because you're kind of working across different decisions. But I do understand that some decisions you're making feel like they're much higher impact. Like when we go back to the happiness test, obviously, like when you're sort of putting your eggs in one product basket, this is something that if it goes wrong, it's going to have a very big effect on your ability to achieve your long-term goals. But that doesn't mean that you can't think about how can we just sort of move fast? And then how would we then apply this to making a higher quality decision about something like that?
--:-- Annie Duke: So one of the things that we want to think about besides impact, when we're considering how fast we can go is optionality, which is really just if we're on a particular route, how easy is it for us to exit? Can we get off the route? Because obviously when we choose a particular option, we're forgoing other options and there's obviously opportunity costs to not choosing those options. And what we're doing is we're saying this option compared to others is gonna work out better for me a higher percentage of the time than other options that I might choose. But we know that after you choose something, sometimes new stuff reveals itself or the world tells you some things that maybe this isn't a road that you want to be on. So then the question just is, how easy is it for me to get off the road?
--:-- Annie Duke: So one of the things that we want to look at is what people call type one or type two decisions or Jeff Bezos says two-way door or one-way door decisions, that when you have a two-way door, when it's easy for you to quit and either go back and choose an option that you previously rejected or choose a new option that you hadn't previously considered, that we can go faster. Because really, it's a way to mitigate the downside, right? If I'm kind of on a bad route, I can at least get off and try to figure out how to get onto another route. So that would be like going on a date, super quittable. I can leave in the middle if I want. Getting married a little harder, less quittable.
--:-- Annie Duke: Right. Right. So, you know, taking taking a few classes online, much easier to sort of quit than like actually committing to a particular college or renting more equitable than buying.
--:-- Jeff Jordan: By the way, it turns out it turns out doing online classes and going to college is now the same thing.
--:-- Jeff Jordan: My children will tell you that.
--:-- Annie Duke: That is so true. So but the more equitable something is, the faster we can go because we can when we can quit. Obviously, that mitigates the effect of observing the downside outcomes. The other thing we can do is actually think about portfolio theory, but for decisions that we don't think of as investments, even though all decisions are investments, which is sometimes we don't need to choose among them. So you can date more than one person at once, right? I actually don't need to choose between these two options. I could actually do both at once and then I can kind of figure out which one's working better. And, you know, we do this with like A-B testing and marketing. That happens in software development, where you're sort of trying to decide between two features and you develop them in parallel and you test them with one set of users and another set of users are seeing different features.
--:-- Jeff Jordan: A number of the businesses do business locally. We'll add restaurants in San Francisco and LA, delivers groceries in San Antonio. You can charge, you can have different pricing approaches in the different markets and just learn. I mean, no one in San Antonio is going to know what you did in Montpellier, Vermont.
--:-- Annie Duke: You know, just so try it out and you learn and learn and learn and learn, then you go national.
--:-- Jeff Jordan: Exactly.
--:-- Annie Duke: When we can do things in parallel, obviously we're also better off. And then the other thing is sometimes you have an option that isn't quittable, but you can still quit it because you can negate it. So that would be like, let's say that I'm invested in a stock and it's totally illiquid, have no ability to sell it. If I could find a stock that's perfectly negatively correlated with the stock that I own and I buy that in an equal amount, I've now solved my problem. So I've quit it even though it wasn't liquid. That's just hedging. So if you can find something that's kind of negatively correlated with the first thing, then you can actually go faster. So that you have to think about in advance, right? This thing is pretty illiquid. It's going to be hard for me to exit. Is there something where if new information reveals itself, I can kind of just negate that decision. And if you can do that, then you can also go faster.
--:-- Annie Duke: So now that we've sort of understood, like there's the impact of the decision, and then we have this optionality thing, like, can you quit? Can you hedge? We can now get to this idea of decision stacking, which helps us when we have to make this big bet, is to say, what are the things that I can do before that are going to help me to gather information so that when I do have to make that big bet, that's going to be hard to reverse, my model of the world is going to be better. So how can I start to use this idea of making some little low impact decisions just to kind of see what's going on, to do some things in parallel? I can blunt it in order to start building better models of the world so that when I do actually put this out into the world, then I know something more about the market.
--:-- Annie Duke: So when you know that you're going to have one of those on the horizon, I mean, they normally don't just like hit you by surprise. It's like, oh, crap, I've got this decision to make. It's just really good to try to stack these other types of decisions in front of it. Because when you do actually have to make that decision tree, when you are actually trying to figure out what the user uptake of something is going to be or whatever, what people are willing to pay for something, your model is just going to be so much stronger for having thought about what are the things that I could do in front of that really big decision. De-risking, trying to get all these little nuggets of directional information to give you higher confidence in the really big decision.
--:-- Jeff Jordan: Yeah. And so you can apply this in all sorts of different places. But the classic thing is dating before you marry. One of the things that I find is that when people aren't 90% sure that it's the right path, that they're pretty reticent to actually execute on it. But we have to make lots of decisions where we're 60%. And by the way, when we estimate ourselves to be 60% on something, we're overestimating that because we're just deciding under uncertainty. It's just kind of how it is. We don't have a lot of information. So once you have an option that appears to be significantly better than the other ones, you just have to do a final step, which is to say to yourself, is there some information that I could find out that would cause me to flip this option in relation to the other options that I have under consideration? And now it just becomes really simple. If the answer is yes, you can just say, can I afford to go get it? You might not be able to because of time or money. And if the answer is yes, I can afford to go get it, go get it. If the answer is no, look, this is the state that we're always making decisions under. I don't have a time machine. My decision-making would be much better if I had a time machine. Sadly, I have none.
--:-- Sonal: That's the next book, The Time Machine.
--:-- Annie Duke: The Time Machine, right. I know, right, exactly.
--:-- Sonal: This has been a fascinating session. Thank you for spending the time with us on the A16Z podcast.
--:-- Annie Duke: I am so grateful to have gotten to come on and to get to discuss this stuff with you. I had so much fun. I've been looking forward to this conversation for quite a while.
--:-- Sonal: Well, I'm so excited because it did get delayed a little bit due to a small misprint.
--:-- Jeff Jordan: That wasn't a small misprint. That was a big misprint.
--:-- Annie Duke: And now I have an eBay collector's item, which I'm the perfect person to know how to monetize.
--:-- Jeff Jordan: Yeah, right.
--:-- Annie Duke: So for people who don't know is that books get printed in sort of 20 page sections, rather that get bound together and really a lot to do with COVID. One section got printed twice and one was totally missing.
--:-- Sonal: I was just questioning my mental facilities while reading.
--:-- Annie Duke: But no, don't worry. It's been repaired.
--:-- Sonal: Excellent. October 13th, when the book is out, you will get an appropriate copy.
--:-- Annie Duke: That'll be awesome. I can't wait.
