--:-- Sonal Chokshi: These markets are actually very good at producing predictions which tend to be more accurate than polls.
--:-- Alex Tabarrok: There is a sort of subtle distinction between wisdom of a random crowd and wisdom of an informed crowd.
--:-- Scott Duke Kominers: Instead of having politicians decide what policies to have, politicians and voters would just decide on what our metric for success is going to be.
--:-- Alex Tabarrok: As you're deciding which thing do we build first, or as we're progressively decentralizing, what do we prioritize? You actually have to understand the market context you're working in.
--:-- Sonal Chokshi: We, as humans, love making predictions. To improve our predictive power, we've built up mechanisms that leverage the wisdom of the masses. Whether it be political polls, financial markets, even Twitter, or community notes. One such mechanism that had its moment this year was prediction markets, with search queries for platforms like Polymarket or Kalshi going hyperbolic ahead of the election.
--:-- Now, today's episode is all about prediction markets, including where they're useful and where they're limited, but also how they coexist with other mechanisms like the polls. Was the attention they received an election year phenomena or a sign of something to come? What's the difference between gambling and speculation anyway? And what implications does that question have on their future in the United States?
--:-- Given that this episode was originally published on our sister podcast, Web3 with A16Z, we also explore where Web3 and decentralized networks play a role here. Finally, if you're excited about the next generation of the internet, be sure to check out Web3 with A16Z wherever you get your podcasts.
--:-- All right, on to the episode.
--:-- Sonal Chokshi: Welcome to Web3 with A16Z, a show from A16Z Crypto about building the next generation of the internet. I'm Sonal Choksi, and today's episode is all about prediction markets and beyond. Our special guests are Alex Tabarrok, Professor of Economics at George Mason University and Chair in Economics at the Mercatus Center, and Scott Duke Kominers, Research Partner at A16Z Crypto and Professor at Harvard Business School.
--:-- Prediction markets hit the main stage in the recent election, which we cover briefly, especially to tease apart the hype from the reality there, since people have talked about the promise and premise of these for a very long time. But we also go more deeply into the how, why, and where these markets work, challenges and opportunities, including implications for designers throughout. We also briefly cover other information aggregation mechanisms and discuss applications for all these markets, including touching on trends like futarchy, AI entering the market, DeSci, and more.
--:-- About halfway through, we discuss where blockchain and crypto technologies come in and where they don't. As a reminder, none of the following should be taken as business, investment, legal, or tax advice. Please see a16z.com/disclosures for more important information. Be sure to also check out the show notes for this episode. We have a really rich set of links, including all the research cited in this conversation.
--:-- But first, we begin with a quick overview of what prediction markets are. The first voice you'll hear is Alex's, followed by Scott's.
--:-- Alex Tabarrok: I think a prediction market is a very simple idea. The bottom line is that we're interested in forecasting. Lots of people are interested in forecasting things. Prediction markets are some of the best methods of forecasting which we have yet created. They tend to be better than complicated statistical models. They tend to be better than polls, or at least as good.
--:-- There's one reason for that. Suppose that a model was better. Suppose I have the Nate Silver statistical model of predicting elections, and it's better than the prediction market. If that were true, I could make money. I could use Nate's model to go and make bets. In making bets on the prediction market, I push the prediction market closer to the truth. Almost by definition, the prediction markets have to be at least as good, and typically they're better than other methods of forecasting.
--:-- Sonal Chokshi: That is an illustration of why we think of these things as information aggregation mechanisms. What are they really doing? They're aggregating information from all of the people in the market. If many different people are out there doing their own private forecasts and calibrating their own models — there's Nate Silver, Jonathan Gold, Melanie Bronze, all their variations — they all have their own models and estimates which they trust with some degree of confidence. They come together, buying or selling the prediction asset based on what their model leads them to believe. As a result, the asset is aggregating all of this information. It's price discovery, just like we think about in financial markets and commodities markets. Everybody's demand together discovers the price at which the market clears. Here, because what the value of the asset is depends on probability, its value is a function of the probability of the outcome of the event. The price aggregates people's estimates of that probability.
--:-- Alex Tabarrok: Exactly. I think it's useful that these are markets, and actually all markets do this. We learned this going back to Hayek's 1945 article, "The Use of Knowledge in Society." This is a Nobel Prize winning paper which doesn't have a single equation in it. Anybody can go and read this paper, and they should. It's a fantastic paper. I'll link it in the show notes.
--:-- Sonal Chokshi: Awesome.
--:-- Alex Tabarrok: Prior to Hayek, people thought prices coordinate and make demand equal supply and production consumption. Hayek said, no, you're thinking about the price system all wrong. The price system is really about aggregating and transmitting information. There's all this information out there in the world, in people's heads — their preferences, but also what they know. People know what the substitutes are, what the complements of a thing are, how to increase supply, what the demands and the supplies are. It's all in heads. For a good economy, you want to use that information, which is buried in people's heads. But how do you get it out? It's dispersed in millions of people's heads. Sometimes it's fleeting information, sometimes tacit, hard to communicate to a central planner.
--:-- Hayek said that markets do this because markets give people an incentive through their buying and selling to reveal this kind of information, to pull this dispersed information from millions of people. People who are buying push the price up. People who are selling push the price down. Suppliers, consumers — they're all in the same market. All of this dispersed information comes to be embedded in the prices. Remarkably, the price can know more than any person in the market.
--:-- Sonal Chokshi: I just want to pause on that for a second because you guys might take it for granted, but that's a very profound insight. What you're basically saying is that it's really surfacing what people know collectively at scale and getting at the truth in that way. That is a very profound thing.
--:-- Alex Tabarrok: Exactly. What economists have found is that these markets are actually very good at producing predictions which tend to be more accurate than polls. If you go to a prediction market — for example, the recent election with Trump and Harris — you can buy an asset which pays off a dollar if Trump wins and nothing if Trump doesn't win. How much are you willing to pay for that asset? If you think Trump has a 70% chance of winning and you go to the market and see the price is 55 cents, you're going to want to buy because you're buying something you think is worth 70 cents. You expect to make 15 cents. By doing that, you push the price closer to 70 cents. You can interpret the price as a prediction.
--:-- In the most recent election, the prediction markets were tending to predict a Trump win, even when the polls were closer to 50-50. The Polymarket CEO said a lot of people trust the market, not the polls, at least when it came to the election. Do you guys agree with that? I'm curious, because if that's a place where we can quickly tease some hype versus signal.
--:-- Scott Duke Kominers: I don't think polling is dead. Polling is one of the inputs into prediction markets. Pretty useful. I do think people need to be more sophisticated about how they poll and who they poll. It's pretty clear that a lot of people now are not answering their telephones, and a lot of people don't want to talk to the pollsters. There needs to be some new sophisticated techniques, but there has to be ways of drawing information from asking people questions. That's not going away.
--:-- Sonal Chokshi: I saw on Twitter that landline poll response rates in the olden days were above 60%, but today the response rate is like 5%. That means you're getting a very bad sample bias in terms of who's willing to answer a call on a poll. I'll hang up right away if someone tries polling me.
--:-- Scott Duke Kominers: Yeah. In particular, it's not that prediction markets will outmode polls. They're going to lead to revolutions in technology for doing this well. If anything, the availability of prediction markets increases the incentive to conduct polls. As we literally saw with the whale, they went out and ran their own poll precisely because they thought they could use it usefully in this market.
--:-- Sonal Chokshi: That's fantastic. I have to ask though — this may seem obvious to you — the key point is that you're putting a price on it where people are putting skin in the game with their opinion or prediction. That seems very interesting and useful. How is that different from betting? Can prediction markets be incredibly tiny amounts that don't have big value to be valid? How does the pricing part of this all work in terms of the incentive design?
--:-- Scott Duke Kominers: At some fundamental level, the pricing works exactly as Alex described. If you think the probability that Trump is going to win is 70% and you see the price at 55 cents, if you believe your prediction, you now have an incentive to show up and buy. If enough people have beliefs of different types and they all come into the market and purchase, eventually the price converges according to the convex combination of all of their different predictions.
--:-- When you ask about the size of the market or the size of a betting market, it doesn't matter once people are there and they've already formed their opinions. But it might affect their incentive to gather information. If the size of the market is capped at $1,000 and you think the probability is 70%, you're not going to invest $10,000 to get a more precise estimate. If the maximum possible upside for you is on the order of $1,000, you can't possibly invest more than that to learn new information that will change your estimates and inform the information in the market even more.
--:-- It's funny — we've been talking a lot in the wake of this most recent presidential election about prediction markets as having been very strong predictors in the trend direction of what actually happened. But if you look at the 2016 election, that didn't happen at all. The prediction markets totally didn't call Trump, and they also didn't call Brexit, which happened the preceding summer.
--:-- Sonal Chokshi: Oh, yeah.
--:-- Scott Duke Kominers: People were asking, what happened? How did these miss this? At the time I wrote an opinion column where I argued that this information thing was a key part of the story. At least at the time, prediction markets were relatively narrow, both in terms of the total amount that could be enclosed in the market and in terms of who participated in them. It was concentrated in a small number of locations. Those participants, because the upside was not necessarily that high, didn't necessarily have an incentive to go out and find out what's going on in other parts of the country. You end up aggregating information just from the people who were already there, which might not be a good estimate in that circumstance.
--:-- Alex Tabarrok: I want to push back a little bit on what Scott said.
--:-- Scott Duke Kominers: Oh yeah?
--:-- Alex Tabarrok: Cards on the table — I am much more of a prediction market bear than Alex is.
--:-- Scott Duke Kominers: We're both really excited about them, but there's a stack rank in our estimate.
--:-- Alex Tabarrok: I agree you want a thick market, and it helps to have people willing to bet a lot of money because then they're willing to invest a lot in making their predictions accurate. The part I want to push back on is this idea that the market did not predict well if it predicted a 40% chance of Trump winning and Trump actually won. This is what people always do, and it's frustrating. You can go back and look at individual examples and say, did the market predict well? But that's like flipping a coin that says 50% chance of coming up heads, it comes up tails, and you say your probability theory isn't very good. It had a 50% chance, it came up 100%.
--:-- What's the real test? You need a large sample of predictions — from political markets, but prediction markets predict other things as well. Then you have to say, in the sample of cases in which the market predicted a 40% win of the Republican, how many times did the Republican actually win? What you find is that it's pretty close — 40% of the time that the market predicted a win, 40% of the time Republicans actually won. There's a linear relationship. When the market predicts a high chance of winning, that happens a lot. When markets predict something with a low chance of winning, that doesn't happen very often. But sometimes it does happen. Something which happens with only a 5% probability ought to happen one every 20 times. That's exactly what you see with these prediction markets. They tend to be more accurate than other methods of forecasting, and they tend not to be systematically biased. We can talk about some odd biases which are possible, but they tend not to be systematically biased. It's not the case that something predicted 40% of the time actually only happens 20% of the time. The markets systematically get it right — 40% of the time it's predicted, 40% of the time it happens.
--:-- Let me give a simple non-example which illustrates this. People have heard of the wisdom of the crowds. You ask people how much a cow weighs. People are not that good at figuring out how much a cow weighs. Some are too high, some are too low. But if you take the median prediction of how much the cow weighs, the median prediction tends to be very, very accurate. The crowd knows more than any individual predictor knows. Markets do the same thing. They embed in the price more information than any single individual knows.
--:-- Sonal Chokshi: Just to be super precise, you're specifically saying the median, not the mean, not the mode. It has to be the exact middle point, not averaging out from the extremes.
--:-- Alex Tabarrok: In that particular example, yes. But it varies by context.
--:-- Scott Duke Kominers: Let me build on that and illustrate it through a simple example in the language of the price system. When you're going around polling people about the weight of a cow, you have to ask them, and they don't necessarily have a strong incentive to figure it out. But suppose you have a very large amount of money to invest in commodities or commodities futures. You have a predictive model that tells you what you think is going to happen to these markets. You have reason to believe there's going to be a big shortage of oil or a surplus of orange juice. You can buy and sell in the market in a way that reflects that estimate, and it pushes the price accordingly.
--:-- If you think there's going to be a big shortage of oil, you're going to stockpile oil today. You're going to buy a lot of it today, and that's going to push up the price because suddenly there's more demand than there was before. When you see the price of oil going up, it's a signal that somehow people think oil is more valuable right now than it was five minutes ago. Of course, these are all hypotheticals — none of this is investment advice, people should not go out and buy a bunch of oil or oil futures. But conceptually, that's how the price reflects the information. The more strongly you believe there's going to be a shortage, the more you're willing to pay to buy right now.
--:-- Sonal Chokshi: Right.
--:-- Scott Duke Kominers: Thus, the sharper the price movement, the stronger the inference about the information that buyer brought to the market.
--:-- Alex Tabarrok: If you want to know whether there's going to be a war in the Middle East, keep an eye on the price of oil.
--:-- Sonal Chokshi: I remember that as a child in the 80s. And it's still true today.
--:-- Alex Tabarrok: That's exactly right. Note that the oil market is a little bit of a prediction market too. The oil market is revealing information about people's beliefs about things that are correlated with the availability of oil, like whether there's a war in the Middle East.
--:-- Sonal Chokshi: That goes perfectly to the question I was about to ask. I want to dig a little bit more into the economic and market foundations, and then we can go more into the challenges of prediction markets and where they're going. On that very note of oil — a great example, Scott — the question I wanted to ask you both is, where does this break? In the oil example, one could argue it's not a pure market. You have cartels, other forces at play. You might be saying it doesn't matter because all that matters is people's opinions, which is what the prediction market is putting as inputs. Or doesn't it matter? My question is really getting at what the distortions can be here. Are there things that can manipulate it, or other distortions where people's behavior changes so significantly that they untether the market from reality?
--:-- Alex Tabarrok: One of the things about these markets — oil predicting possibility of war in the Middle East — they're not designed to do that. In those cases, the information is a leakage. It's an unintended consequence of market behavior, which is very useful for economists to be able to pull information out of these market prices.
--:-- With the creation of prediction markets — the first ones go back to the Iowa political prediction markets created in 1988 — it was almost for the first time that a market was created in order to produce information. There's a much more direct connection between the output of the market, the prices, and the predictions because that's what they were designed to do.
--:-- You're totally correct that if you want to get a market to predict the future, you want to have lots of people to take advantage of all the dispersed knowledge. There are people in Pennsylvania who have extra knowledge about what their neighbors are talking about that can give them a little bit of insight that you might not have if you're living in New York or San Francisco. You want lots of people to participate, and you want the markets to be quite thick because you want people to be able to invest some time and energy, maybe apply some models. You want it to be free and open, and you have to be a little bit worried about manipulation.
--:-- Scott Duke Kominers: There are some funny edge cases that we've seen crop up occasionally. There were even allegations that maybe that was going on here. If there's some external outcome or even some internal behavioral outcome that conditions on the prediction — if political candidates are going to decide how hard to campaign in a given state based on what the prediction for that state says — you might want to influence the price not for the sake of earning money in the prediction market, but rather to place the prediction in a given position. That's very hard to do because you actually have to change beliefs from doing that.
--:-- I think it was the Obama versus McCain campaign. Somebody tried to sink a bunch of money to move the McCain percentage. People who had estimates that the Obama probability was higher just arbitraged that out over an hour or two. Markets work. If you see something that looks like a market anomaly, you buy or sell accordingly.
--:-- Sonal Chokshi: Yes. We're all market purists here, so that seems like that's working.
--:-- Scott Duke Kominers: But if the market is thin, or if the information signals are very dispersed, maybe you can convince people. If you have enough money to swing the market in a very sharp way, especially if you're doing it through Sybils — many identities, so it looks like a surge of people who have a given belief — you might actually change the beliefs of the market participants in a way that distorts the probability and could have various other impacts.
--:-- The other thing is this idea that the oil markets are leaking information. We'll stick with that example. The oil markets leak information about potential conflict in the Middle East. That's a feature and a bug. The fact that it's an oil market that is informative about the Middle East on the one hand, as Alex said, means that the market is not optimized for specifically answering the question "what's going to happen in the Middle East?" There's lots of other stuff that affects the oil market, like how popular electronic vehicles are at that given moment. You have this very complicated signal extraction problem. You see a big spike in the oil price. Is it because there's a potential conflict coming in the Middle East? Or is it because there's just been some new electronic vehicle test that failed, and somebody knows that, so they know oil is going to be more important next month?
--:-- Whereas if you have a market that is just predicting "will there be a conflict in the Middle East?" — that's all it's predicting. But that's now a zero-sum market. It's a harder market to participate in if you only have dispersed information. If you don't actually know whether there's a conflict in the Middle East forthcoming, but you know some things that are happening suggest that — for example, you saw an oil price change — you have to do a much more complicated calculation. You're taking a riskier bet by participating in a prediction market where you're staking everything on this one outcome rather than on something that's heavily correlated, where there are many different things that could have related predictions that could be mostly correct even if your main prediction is wrong.
--:-- The takeaway is that prediction markets' narrowness is a feature and a bug. It's dual to the sense in which ordinary markets' broadness is a feature and a bug. A prediction market is a narrow zero-sum contract on a specific event. Many people's information about that event is actually coming from all these correlates. It's not that they know specifically, "is there a conflict coming in the Middle East?" They see a lot of potential signals of it. If you're buying and selling in a market that responds to those signals, that insures you a little bit. If you get the main estimate wrong but all your signals were correct, you're at less risk than if you go into a prediction market and had all the signals right but the final estimate wrong, and then you're just betting on the wrong side of the event.
--:-- Alex Tabarrok: I think what Scott said also has implications for why we don't have prediction markets in everything. If these markets are so great and they work so well at predicting things, why don't we have more of them? Scott was basically giving the answer there. This is how I would put it.
--:-- If you have the market for oil, there are lots of people who are buying and selling oil who are not interested in what's going on in the Middle East. They're not trying to predict that. But it's precisely because you have lots of organic demand and supply that this provides a subsidy to the sharks who go in there to make the price more accurate.
--:-- Take the example of wheat. There are lots of farmers who are buying and selling in the market for wheat just to insure themselves, just to hedge themselves. Because of that native organic demand, the market is thick enough that you then have all of the sharks who are not themselves farmers but go in there and use models and techniques to predict which way the market for wheat is going to go. They make that market more accurate.
--:-- If you didn't have the organic demand, you're going to have a market with just sharks in it. Who wants to be in a market where you're only with other sharks? If I know that the other guy is just trying to predict this one thing as much as I am trying to predict it, I don't want to be in a market with Scott. He's just too smart.
--:-- Scott Duke Kominers: I would say the same thing about you.
--:-- Alex Tabarrok: That's why the market wouldn't work. Some of these markets, even though they might be forecasting something which is useful, there isn't enough organic demand. You have to subsidize it from outside the market in order to get a useful prediction out of it, in order to get the sharks willing to go against one another to try and predict this thing. That's why we don't have markets in everything yet, potentially.
--:-- Sonal Chokshi: This is maybe jumping ahead a little bit, but I just have to ask at this point. Scott, you're a market design expert. On the market design front, what does that mean if there isn't organic demand? Is there a way for market designers to essentially create markets in situations where there isn't that kind of latent organic or existing thing to harness? Can you actually manufacture that market without distorting it and create conditions that could design a market into place?
--:-- Scott Duke Kominers: That's a great question. There are two different ways to get at it. One of them, which is sort of what the framing of the question is pointing at, is could you find a way to create latent demand? And Alex was saying you could subsidize it.
--:-- Scott Duke Kominers: You could basically somehow subsidize the experience of some people trying to predict this event. Like, you know, subsidize a bunch of college students developing forecasting models so they have a lower cost of entering the prediction market or something of the sort. Again, not advocating this specific policy.
--:-- Sonal Chokshi: Right. Although in Alex's example, that subsidy was not intentionally a subsidy. It's just a result of the behavior. It wasn't like people were trying to subsidize. It was just subsidizing because of their natural behaviors.
--:-- Alex Tabarrok: True. No, no, exactly. So we started this conversation with the recent presidential election and all of these other associated elections. Those have proven, at least in practice, to be much thicker markets because there are some people who seem just interested and betting on them, right? A lot of people have some amount of information, some amount of opinion, and so there's a little bit of that latent demand that comes from people's general interest in the question.
--:-- Scott Duke Kominers: Yeah. One could try and create that for other contexts. You can try to help people feel that something is interesting or feel that they have an opinion about it enough that they're willing to participate in a prediction market. The other thing you can do is use other types of information elicitation mechanisms. Prediction markets are one of many ways of doing incentivized information aggregation. Others are things like incentivized surveys or peer prediction mechanisms.
--:-- There's a whole class of what are called peer prediction mechanisms, where what you're doing is asking people what they believe about an outcome and what they think other people will believe about an outcome. Then you use their beliefs about others as a way of cross-examining whether they were telling the truth. Because you survey a lot of people, you get your crowd volume. You know the aggregate belief of the population. And you can check whether someone's own belief about the population is the right mixture of that aggregate belief and their belief.
--:-- So if you yourself think that Trump is more likely to win, then you yourself are more likely to believe that other people think Trump is likely to win, because the frame you have, your information, indicates that at least one more person in the market believes that. So you can cross-examine your predictions with your estimate of what the population believes and what the population actually reveals they believe. Then you can reward people based on how well they did. In effect, how good are you at estimating what everyone else thinks given what you think?
--:-- Those sorts of mechanisms you can incentivize. You can pay people immediately, incidentally. Unlike prediction markets where the event has to be realized and payment only happens at the end, here you're not paying people based on their accuracy about the event. You're paying them based on their accuracy about everyone's estimates. So you can do that all at once. Collect all the estimates, pay people, they go home, you have your estimate. These have been shown in practice to be very effective for small population or opinion estimates, things where there isn't a thick market and a very big public source of signal.
--:-- Sonal Chokshi: That really answers that question. And by the way, it brings up a very important point that we did not address in the recent example of the election, which is the French "whale" who won by using the neighbor poll. Their neighbors won't say what they think, but when you ask them who they think their neighbors are going to vote for, it's a way to indirectly reveal their own preferences. That's the so-called neighbor poll. I don't know if that's a standard thing or that just came up in this election. It's the first time I heard of it, but it's a great example of something I did study in grad school when I was doing ethnography work, which is never trust what people say they're going to do, but what they actually do.
--:-- Alex Tabarrok: This goes to your economist world of revealed preferences.
--:-- Sonal Chokshi: Right. Very similar. But in that case, that person polled his neighbors and then used that data essentially off-chain to go back onto the market, Polymarket in that case, to up his bet and essentially won big as a result. So that would be an example of what you were mentioning. Although in that context, you were mentioning it in how we can address a case where there's a thin market. This is a case where that played out in a thick market of the election.
--:-- Scott Duke Kominers: Well, you might say that he was using this in the thin market of trying to understand his neighbor's local preferences and estimates.
--:-- Sonal Chokshi: There you go. That's more precise.
--:-- Scott Duke Kominers: Although we actually don't know the details of how he produced these estimates. It doesn't sound like they were incentivized, so it's not exactly like what I was talking about with pure prediction. But you're right, it's the same core idea that using people's beliefs about the distribution can be much more effective than using their personal beliefs a lot of the time.
--:-- Alex Tabarrok: I would underline two things there. One, the market is a way of bringing all of this dispersed information and creating an aggregation. But it's not the only way. That's what Scott is saying. Understanding this is one of the first information aggregation mechanisms we have studied and understood reasonably well, but there are other ones. You can think about prediction markets as being one example of a class of mechanisms which take dispersed information and pull out some knowledge that none of the people in the market or polled might be aware of. And yet somehow it is in the air, as it were.
--:-- Sonal Chokshi: That's fantastic.
--:-- Alex Tabarrok: There are also other ways of subsidizing these markets, which is something that corporations may be very interested in doing because corporations are interested in forecasting the future. Some of them in the past have created their own internal prediction markets. One famous example is Hewlett-Packard. They were interested in forecasting how many printers would be sold in the next quarter and the next two quarters, three quarters, four quarters, and so forth. So they created a market where if you correctly predicted how many printers would be sold in which time period, you could earn money. And they subsidized that market. Everybody going in, which was just HP employees, got $100 to play with. That's a way of trying to get more people involved and interested in playing on these markets to elicit information.
--:-- Sonal Chokshi: That example is actually really interesting to me because when I was at Xerox PARC, we talked about that. One of the things that came up is it's a very useful mechanism for getting certain things right, but it is not a useful mechanism for actually figuring out the future in terms of what to invent because it doesn't address the case of you don't know what you don't know. You only know what you know. This came up just yesterday. Trump announced his candidate for attorney general, and one of the examples someone cited on Twitter was it's the first time they've seen a Polymarket contract resolved to zero for all potential outcomes because Gaetz wasn't even listed among the 12 potential nominees in that range of possible outcomes. That's an example where you have to have the right information itself in that prediction market. Maybe you guys can explain that a little bit more quickly.
--:-- Alex Tabarrok: These markets are good at when people have some knowledge and it's hard to aggregate that knowledge. The other thing they're good at: people have run these markets for predicting when a project will be complete. This is a classic case. When you ask people, they'll say, "Oh, no problem, it'll be ready in five weeks." They're very optimistic. They tell the boss it's going to be ready in five weeks while they go back and tell their friends, "Oh my God, it's delayed." You've got all these problems. But if you let people bid anonymously in these markets, then the truth comes out. This is a way for corporate leaders to learn information that their employees know but are not willing to tell them. But to your larger point, nothing is more difficult to predict in the future. Trump is a chaos agent. Whatever he's going to do is hard to predict. I don't think anybody predicted Matt Gaetz.
--:-- Scott Duke Kominers: This highlights what prediction markets are good at versus where you might want to use other sorts of information elicitation mechanisms. The two examples Alex gave of within-company prediction markets—predicting sales or sales growth, a metric that many people in the firm are tracking and have different windows of information into—or predicting when a product is going to launch, where you might have product managers who know something and engineers who know there's a hidden bug they haven't told the product managers about yet. These are contexts where many people in the company have some information that only they have, and the aggregate of all that information is a pretty good prediction of the truth because the actual outcome is the aggregate of all those people's information directly. How many sales calls are you making that are succeeding? How is the coding for this specific feature going?
--:-- By contrast, you mentioned with Xerox PARC, trying to predict whether a new, totally imagined product is going to succeed. That's really hard. It doesn't rely on information in particular that the company has. The company has some idea of what products people might buy, but you might be AT&T and invent the first picture phone. You thought that was a great idea, but you don't actually know until you put it in the market and see whether people are interested in using it. The aggregate of all the information in the company—they went through with a product, concluded it was a good idea based on all the signal everyone could see—and it still flopped. The total information in the company wasn't high enough to provide the right answer, even when aggregated.
--:-- Alex Tabarrok: Right. But I do think there is a subtle distinction between wisdom of a random crowd and wisdom of an informed crowd. With our Hewlett-Packard example, HP knows that if you're trying to figure out whether a product's going to launch on time, a random person on the street has no information about this. You don't want to pull together a focus group of miscellaneous HP customers and ask them, "When do you think we're going to finish designing our new printers?" They don't know. So there is this question: are you learning things from the right crowd? You could have the best incentivized information elicitation mechanism on the planet, and if you only survey people who don't know anything at all about the topic, you incentivize them, you'll learn what they believe truthfully, but you won't be able to do anything with it.
--:-- Scott Duke Kominers: And then back to the future, the whole idea of the best way to predict the future is to invent it. That goes to Jobs and the phone. No one—you can ask a million people, "Will they ever use a touch phone?" People's behaviors can also evolve and change in ways that they themselves are not aware of. That's the other side of that example. Prediction markets are like a candle in a dark room. They help us see a little bit, but there are still areas you can't see very far.
--:-- Sonal Chokshi: Great. I'm going to ask a couple of quick follow-up questions. So thin versus thick—you guys are talking about the depth of the market in terms of the number of participants. Thin is too few, thick is many. Is that correct? Or is there a better, more precise way of defining that?
--:-- Alex Tabarrok: In a prediction market, a thin market is few people betting small amounts. One of the problems we've had is that prediction markets are mostly illegal in the United States. The biggest one in this past election was Polymarket, which was illegal for U.S. citizens to bet on. We're slowly changing, but we have this ridiculous situation where we have huge markets in sports betting and gambling, and we allow that. Yet here we have a kind of gambling market, a prediction market, where the output is actually quite useful and socially valuable, and we don't allow it. Making these markets legal and open to more U.S. citizens would thicken those markets, make them more accurate, attract more dispersed information, and would be really quite useful.
--:-- Sonal Chokshi: To your bigger point, Alex, you're basically arguing that they can be a public good in the right context, informationally.
--:-- Alex Tabarrok: Absolutely.
--:-- Sonal Chokshi: And interestingly, if you think about some of these prediction markets getting cease-and-desist notices, it's interesting because in some cases, people might argue that some people trying to get information is a manipulation of the market. But in fact, to your point throughout this discussion, it's actually a way to provide more input of information into the market itself. That's an interesting point on the public interest side.
--:-- Alex Tabarrok: Let me give you another example of the public good nature of these prediction markets. One of the most fascinating uses is to predict which scientific papers will replicate. We have this big replication crisis in the sciences, psychology and other fields, where lots of research doesn't replicate. It's expensive to replicate a paper. But what some people have done is have a betting market, a prediction market, on which papers will replicate. That turns out to be very accurate. Then you only have to replicate a few of those papers for the markets to pay off. For the rest, you use the prediction market result as a pretty good estimate of whether it will replicate or not. This is a way of improving science, making it better, quicker, and more accurate.
--:-- Sonal Chokshi: I love that. I ran a lot of op-eds when I was at Wired on open access and science, evolving peer review and the replication crisis. It's very exciting to hear that this is something we can do to address that. It leads to a quick follow-up question, which happens to be on my list for the lightning round. When you guys were talking earlier about tapping into intuition and information dispersed across many people into these prediction markets, one of the first questions that came to mind is: do you need domain experts, or does that actually distort a market? This comes up as a perfect segue from your point, Alex, about scientific papers, because that's a case where one would imagine that people in that industry or domain, or other scientists who have experience analyzing research, would be the best at predicting things. But is that necessarily true? Do we have any research or data into domain expertise in these markets?
--:-- Scott Duke Kominers: I don't know the answer to that last part. Let me talk about the first part because it also speaks to your thick versus thin question.
--:-- Sonal Chokshi: Great.
--:-- Scott Duke Kominers: When Alex said a thin market is a small number of participants betting small dollar amounts, why is that a thin market? It's because the total information is small in two ways. One, there are a few people bringing their own individual estimates. You have a small number of people saying things. Two, because they're betting small dollar amounts, it's a signal that their information is not very strong or confident, at least relative to what it could be otherwise. If you are staking a very large amount of money on this, the market inference is that you have done the research, and you have the incentive to do the research. Why is the inference that you've done the research? Because if you're staking a large amount of money, you should have done the research. Otherwise, you're putting money at risk without full information.
--:-- Sonal Chokshi: Like the French whale who did the neighbor poll to find out.
--:-- Scott Duke Kominers: Right, exactly. One can argue about how good or bad that neighbor poll was, but part of his confidence to make that huge bet was that he thought he had a signal that was accurate and the market had missed. So thickness and thinness—the proxy for it, the way we think about measuring it—is how many people and how much are they staking? How much value are they putting behind their beliefs? Thickness and thinness is really in terms of the information. Do we have a lot of different signals of information that are strong, coming together and mixing to determine the price? Or is it just a very small number of pretty uninformed signals?
--:-- This is the tension when Alex says it's a problem that the biggest prediction market for the U.S. election was not actually in the U.S. and was not legal to participate in from the U.S. A lot of the real signal is in the United States. Without those people being able to participate in the market, you miss a lot of that to a first order. People internationally will figure out ways to aggregate and use it, but you miss a lot of the people who have that information already at their fingertips.
--:-- So you ask about domain expertise. It's not exactly domain expertise versus not, but rather information richness. For example, in predicting scientific replication success or failure, domain experts are especially well-equipped to do that. A random person chosen off the street—you can tell them a scientific study and maybe they'll have an instinct, but a lot of the detail of figuring out whether something will replicate comes from knowing how to read statistical analyses, understanding the setup of the experiment, and the surrounding literature. Domain experts have a particularly large amount of information.
--:-- If you think about something like a political betting market, maybe domain experts focused on politics and polls have a big slice of information. But there might also be other categories of people, like people who know their neighborhood has recently switched its political affiliation in a way not yet captured in national polls, or our French whale who ran his own custom poll. The context of the question the prediction market is trying to evaluate—and this is true for any information elicitation problem—tells you something about who has the most information to bring to the market and thus who it's important to have there.
--:-- Alex Tabarrok: I agree with everything Scott said. One interesting thing is you often don't know who the domain expert is until after the market has been run. It's absolutely true that if you're predicting political events, you want people interested in politics. If you're predicting scientific articles, people need to be able to read stats. But one of the guys in the scientific replication paper on markets made $10,000. He was just a super obsessive guy who really got into it, running all kinds of regressions and doing all kinds of things. When you say "domain expert," I think one of the virtues of these prediction markets is that they're open to everyone. They don't try to say, "Only the experts get to have a voice." Only ex post do we learn who really made some money at these markets.
--:-- Sonal Chokshi: I'm so glad I asked you guys about the definition of thick versus thin because you gave me so much interesting nuance. People following this podcast definitely understood what you meant about thin versus thick early on, but you guys just took it to a new level. If you're so smart, why aren't you rich?
--:-- Alex Tabarrok: Hey, I am rich.
--:-- Sonal Chokshi: Yes, exactly.
--:-- Alex Tabarrok: I made some money in this market.
--:-- Scott Duke Kominers: And that again is about the incentives. When we talk about the dollar value staked, the amount of money someone is staking on their prediction, in equilibrium it should be a measure of their confidence—how confident they are in their own beliefs and how much effort they've put in to learn the information to be precise. Exactly as Alex says, one person who might be really good at predicting a scientific replication failure is someone who works in that exact same area. Another might be someone who just enjoys doing this for fun and has never had a real incentive to triple down on doing it, but now suddenly they can.
--:-- Sonal Chokshi: Right, right. And by the way, Scott, does it have to be dollar and price incentives? I'm asking you this question specifically because you and I have done a lot of pieces in the past on reputation systems. I almost wonder if the skin in the game can just be karma points and not even any money. From a pride perspective.
--:-- Scott Duke Kominers: 100%. Alex mentioned subsidy. One way you can subsidize—he said Hewlett-Packard subsidized by giving all their employees $100 and saying spend it all on this market. You can subsidize people with cash, but you can also subsidize them with tokens or reputation or various other sources of value. One advantage of using tokens is that you can deliver a subsidy that is only useful in this market. If it's a personal non-token, I give you a bucket of them, and the only thing you can do with them is use them to enter predictions. You choose which prediction markets to enter and how much to spend in each one. Then you earn payoffs. Payoffs are also measured in tokens. Maybe downstream you get prizes for having large numbers of tokens, or you get to join the elite predictor force, or it serves as a measurement of your reputation—how good you are at making predictions, which you might leverage into something else. People who win data science contests leverage that into data science jobs. Maybe you leverage this into a forecasting job.
--:-- All of that, so long as you find people who are willing to be incentivized by those types of outcomes, you can subsidize their participation in a unit that locks them into the market. Their one thing to do with it is to participate in the market, and it reinforces more and more participation among the people who are most successful and most engaged.
--:-- Sonal Chokshi: That's super interesting. I'm going to push back on you on that, actually, because I wonder if it necessarily needs to be crypto-based. You can just do any kind of internal marker.
--:-- Scott Duke Kominers: Oh, yeah, no, it's any internal marker. But for all the reasons we normally know, it's much better to do this in an open protocol form because if the token is eventually going to be leveraged for reputation, you want anyone to be able to verify that you have it.
--:-- Sonal Chokshi: Audit it, see it, hence blockchains.
--:-- Scott Duke Kominers: Got it.
--:-- Sonal Chokshi: Great. We'll talk a little bit more about that. Just more lightning questions. Go for it. So where do super forecasters like Philip Tetlock's work come into all of this? Are they especially good at prediction markets? Because that's a case where they're generally better than the general public at forecasting and making predictions. Is there a place for them in this world, or are they outliers, or does it not even matter here?
--:-- Alex Tabarrok: I think there are two things. One, the basic lesson of Tetlock's work is that most people, even those in the forecasting business, are terrible forecasters. He started tracking so-called political experts and seeing what their forecasts were. Ten years later, were they right? Five years later? They were completely wrong. So he shifted into looking for: is anybody ever right? Are there super forecasters? Yes, he found that some people—not typically the ones in the public eye—can definitely forecast better than others. One of the things those people can do is participate in these markets, and by their participation, they push the market price closer to their predicted probabilities. So forecasters have an incentive to be in these markets, and by being in these markets, they make the markets more accurate.
--:-- Now, is the market always going to be more accurate than the super forecaster? No. Warren Buffett has made a lot of money even though markets are basically efficient. He has shown that in many cases, he is able to predict better than the market price itself. More power to him. There are going to be some super forecasters, but they're hard to find. They're rare. A virtue of the price is that everyone can see it. It's public.
--:-- Sonal Chokshi: This gets at a bigger, maybe more obvious point to you guys, but a recurring theme I'm hearing is that it's not that the prediction market is only taking in guesses, people's intuitions, bets, opinions, and any information it has. Theoretically done well, it's taking in all information. It could be super forecasters contributing to it. It could be Nate Silver taking his 80,000 simulations and feeding his inputs and adding that signal into it. It could be pollsters putting their data and predictions. It doesn't even matter how people get at their intuition. All that matters is that they're pricing that information into that market.
--:-- Do you know the WallStreetBets "Everything is Priced In" post?
--:-- Alex Tabarrok: No, I don't actually.
--:-- Scott Duke Kominers: I don't know this one either.
--:-- Sonal Chokshi: Let me read it just a little bit. It's a fantastic post from about five years ago. It's called "Everyone is Priced In." It says: "The answer is yes, it's priced in. Think Amazon will beat the next earnings? That's already been priced in. You work at the drive-thru for Mickey D's and found out that the burgers are made of human meat? That's priced in. You think insiders don't already know that?"
--:-- Sonal Chokshi: The market is an all-powerful, all-encompassing being that knows the very inner workings of your subconscious. Your very existence was priced in decades ago when the market was valuing Standard Oil's expected future earnings based on population growth. That is so great. Okay, you have to send me that link, Alex, and then I'll put it in the show notes. So you're basically agreeing that it's the markets that price everything in.
--:-- Alex Tabarrok: Yeah, I mean, that's an exaggeration, but yeah.
--:-- Scott Duke Kominers: I mean, anything is fair game. I want to push back, but we're fine here because anything is fair game, but you have to wonder who is going to show up to those markets and where their signals are coming from. If you're a super forecaster, maybe you work for a super secretive hedge fund, and the last thing you want to do is directly leak what it is you believe.
--:-- Alex Tabarrok: Yeah, yeah.
--:-- Scott Duke Kominers: And in fact, you would prefer that the market be confused by this public signal. We talked about manipulation. You might show up and tank the prediction in one direction or the other just to take advantage of that in the financial market off to the side. So while in principle these things can be very comprehensive, you still have to think about who participates in which market where. Just like we see in other markets where some people trade in dark pools and some people trade in public exchanges, that selection affects what information price is really aggregating where.
--:-- Alex Tabarrok: That's fantastic. The other thing about public forecasters, super or otherwise, is that they're very salient to the average person. So another thing we see in prediction markets is herd behavior. Again, just like we see in other types of markets. If a lot of people are suddenly buying oil futures, does that mean they all have knowledge that there's going to be a conflict in the Middle East? Or does it mean they saw other people buying oil futures and are like, "I'd better do this too"? Or did they see one analyst report, and they all saw the same analyst report, and as a result they all went and bought oil futures because they believed the report? Or worse, did they see one analyst report that said oil is going to be expensive next quarter, and they went and bought oil futures not because they believe the report—maybe they even have information that it's not true—but they know everyone else is going to see the report, and so there will be purchasing pressure.
--:-- Scott Duke Kominers: Yes. There's this very famous paper by Morris and Shin in the American Economic Review called "Social Value of Public Information."
--:-- Sonal Chokshi: Okay, I'm going to put that in the show notes.
--:-- Scott Duke Kominers: It talks about information herding. The idea is basically, if you have a market where everyone has private signals, and then there are some very salient public signals, and people have to coordinate—are you going to run on a bank or not? Or what do you think is the probability of this thing happening?—people might ignore their private signals if the public signal is strong enough that they think other people are going to follow it.
--:-- Alex Tabarrok: Yes. And so when a very prominent forecaster makes a prediction, as the polls were coming in the week leading up to the election, a new major poll would drop and then the prediction markets would judder around and veer off, at least briefly, in the direction of that poll. This public information effect is salient. You expect a lot of market movement based on this information, and so the market actually moves even more. It incorporates not just the information, but also the fact that other people are incorporating the information too.
--:-- Sonal Chokshi: Are there any market design implications for how to avoid that happening? If you're setting up the conditions of a perfect, great prediction market?
--:-- Scott Duke Kominers: Oh, gosh, that's a great question. First of all, it's not completely avoidable. You can't have a market where a sufficiently strong public signal doesn't generate some herd behavior. At that level, it's unavoidable. But you can try to do things to dampen the effect. Off the top of my head, I can think of two. There are probably others. One is you could slow trading a little bit. You could limit people's abilities to enter or exit positions very quickly, which forces people to average.
--:-- Alex Tabarrok: It's also an example of slowing contagion, like an infection spreading very fast, kind of like the herding becoming viral.
--:-- Scott Duke Kominers: Yeah, contagion is a very good example of what Scott's talking about.
--:-- Alex Tabarrok: In stock markets, we have circuit breakers.
--:-- Scott Duke Kominers: Circuit breakers. Yes, exactly. So that's one way. Another thing you could do is try to refine your market contracts in a way that orthogonalizes—by which I mean it extracts the signal that is independent of that signal. A prediction market contract somehow incorporates the information, adjusted for whatever Nate Silver claims. Let me give you an example. My colleague Robin Hanson, who is one of the founders of prediction markets, is usually many steps ahead. He has a very clever proposal for this, which I don't think anyone has ever implemented. He says you have a prediction market and then you have a second prediction market on whether that prediction market will revert in the future to something else.
--:-- Alex Tabarrok: Yes. Oh, so genius.
--:-- Sonal Chokshi: Yes, exactly. That's the way you orthogonalize it. Perfect. That's way better than my example. That's so great because I was actually going to guess something like combining the reputation thing. This is essentially a way of combining reputation by having a parallel market that verifies and validates.
--:-- Alex Tabarrok: Exactly. Totally. That's so interesting. By the way, that's not futarchy, right? His new thing. One of the criticisms of futarchy was precisely the point which Scott made. And Robin's response to that is, "The solution to a problem of futarchy is more futarchy."
--:-- Sonal Chokshi: Ah, okay. Can you quickly define futarchy for me?
--:-- Alex Tabarrok: Yeah. Robin Hanson's idea is to take these decision markets and apply them to government. Let's create a new form of government. There aren't many new forms of government in the world—democracy, monarchy—futarchy is a new form of government. The way it would work is that instead of having politicians decide what policies to have, politicians and voters would just decide on what our metric for success is going to be. It might be something like GDP, but you might want to adjust it for inequality or environmental issues. So you create some net statistic, GDP plus. Then anytime you have a question—should we pass this healthcare policy? How should we change immigration rules?—you have a market on whether GDP plus would go up or down if we pass this new law. And then you just choose whichever one. If GDP plus goes up, you say, "OK, we're going to do that." People would submit new ideas to the futarchy—a proposal for immigration, a proposal for healthcare, one for science policy—and then you run a prediction market. Would GDP plus go up with that or would it go down? And then you choose whichever comes out. So Robin expands this idea of decision markets to an entirely new form of government.
--:-- Sonal Chokshi: That's fascinating. And it relates so much to one of our partners' collaborators' work, Andrew Hall at Stanford. He studies a lot on on-chain and liquid democracies and more. Thank you for explaining that, Alex, because I've never fully gotten what futarchy is. People toss it around and I'm like, "But actually, what is it?" So that was very helpful.
--:-- Scott Duke Kominers: It also sounds like it could be the subject of a Borges short story or something.
--:-- Sonal Chokshi: Oh my God. Yes. Absolutely. Oh gosh. What was the last one that we put in the last reading list, Scott, for the Founders Summit? Was it the Labyrinth short story collection?
--:-- Scott Duke Kominers: I think it was Labyrinth, right?
--:-- Sonal Chokshi: Yeah. I think so. That's so funny. So a few more questions and I want to switch to crypto. Since we're talking about market theories and practice in this recent segment, Alex, did you want to say a little bit more about efficient markets?
--:-- Alex Tabarrok: Sure. Another fascinating example of how markets could leak information, which then could be used for other things, is if you've ever seen the movie *Trading Places*. You probably know that the main determinant of orange juice futures is what the weather is going to be in Florida. Of course. So Richard Roll, a finance economist, had this interesting question: Can we use orange juice futures to predict the weather? What he found was that there was information in those market prices which could be used to improve weather forecasts in Florida. Kind of an amazing example because no one knew this. No one was even predicting this, but this was a leakage of amazing information.
--:-- Scott Duke Kominers: Fantastic. Another fascinating one is Richard Feynman famously demonstrated that it was the O-rings which were responsible for the Challenger disaster by dipping the O-ring in the ice water at the congressional committee. However, economists went back and looked at the prices of the firms which were supplying inputs into NASA and the Challenger. They found that the stock price of Morton Thiokol, which was the firm that produced the O-rings, dropped much more quickly and by a much larger amount than any of the other firms. So the stock market had already predicted and factored in that it was probably the O-rings which were the cause of the Challenger disaster, even before Richard Feynman had figured this out.
--:-- Sonal Chokshi: And by the way, that ties back to your HP example in a way, because if I recall, part of the backstory with the Challenger was also that it was a case of death by PowerPoint because of the way they were communicating information internally. The format and structure constrained how that information was presented. I think Tufte gives a famous case study of this in one of his many books.
--:-- Alex Tabarrok: Another way of putting that, which is kind of disturbing, is that the people on the ground knew this wasn't a good idea. They knew it was not a good idea to launch the Challenger on such a cold day. If there had been a prediction market—"What's going to happen?" or "Should we do this?"—then I think it is quite likely that dispersed information, which no one was willing to tell their bosses, no one was willing to stand up and say, "We should not do this," instead got buried in PowerPoints. That dispersed information might have found its way to the top if there had been a prediction market asking, "Is this launch going to go well?"
--:-- Scott Duke Kominers: Exactly. Or said another way, the earlier definition of a prediction market—it would have been another way for management to elicit better information from their employees, using that as a mechanism for communication.
--:-- Alex Tabarrok: Exactly.
--:-- Sonal Chokshi: The HP thing really struck me because I remember that as a communication no-no for how information is presented. That's actually a good segue to the crypto section because I want to ask you guys—and this will help me break down some definitions. One of the things we talk about in crypto is the ethos of decentralization. Sometimes the information is public on a public blockchain. It's often open source, distributed. It can be real time. I don't know if it's necessarily accurate information, but the information can be corrected very quickly, which makes it more likely to be accurate because of the speed of revision. We also saw that in the recent election, compared to media. One observation people made is that media didn't move fast enough, or didn't want to because of biases, with their polls and predictions, whereas the prediction markets were faster at self-correcting. So one question I have to kick off this section about the underlying technology and how it works is: Let's tease apart all those words. I just gave you a big buzzword bingo soup. What are the words that actually matter when it comes to eliciting better information and aggregating that information in a market? What are the key qualities we should start with? Then we can talk about the technologies underlying that.
--:-- Alex Tabarrok: One way of answering that question might be that the largest prediction market was the Polymarket crypto prediction market. The question is, is crypto a necessary part of this? I think the answer is probably no. I think why the crypto market was particularly successful was because it was open to anybody in the world, barring U.S. citizens. And the market, because of that, was much thicker than the other markets. There are some prediction markets which limit people's bets to $1,000. The crypto whale was betting millions of dollars on these markets. So that's why the crypto market, as a kind of regulatory arbitrage, became very important. The FBI is looking at this, the French are looking at this—was it legal? Is it violating some laws? But I think the crypto part of it was not actually necessary.
--:-- Sonal Chokshi: Yeah, I'm glad you pointed that out, Alex. I think people have been hyping and overstating the crypto part of it, and I actually agree with you completely. I don't know if crypto was at the heart of the way that market works, except in those qualities you mentioned. Scott, any thoughts on that point?
--:-- Scott Duke Kominers: I totally agree with all of that. One thing that crypto does very well, on top of being open, interoperable, and transparent, is it enables commitment. You can write a piece of software that is going to run in the exact specified way. It can be audited by all of the users, and then they can be convinced that it's going to run correctly. Some ways we do information elicitation have challenges with commitment. If you're going to survey people and pay them six months from now based on whether their survey estimate was accurate or not, they might be worried that you're not going to show up and pay them. So long as whatever the information is can also exist on-chain—the resolution of the uncertainty can somehow be visible on-chain, either through an oracle or if it were an on-chain function to begin with, like "what is the price of this asset?"—you can commit in a way that you can't easily do without complicated contracts. You can just commit that it's going to run as expected. For that to work, your information elicitation mechanism has to be fairly robustly committed and often also decentralized. Polymarket, by contrast, famously changed the terms of a couple of their resolutions because something happened that didn't quite make sense in the context of the way they'd said they were going to evaluate the outcome. Post hoc—after people have already bought in under the original terms of resolution—they changed the terms of resolution. That's a lack of commitment that's hard for markets to form when people don't trust that they're going to be resolved as described.
--:-- Sonal Chokshi: Right. Isn't the most basic rule of markets that you can't just suddenly change the rules under you? Isn't that why we don't trust governments that don't enforce property rights? You just can't mess around.
--:-- Scott Duke Kominers: No, you're exactly right. The same way that blockchains create a form of property right that you can trust even without a very trustworthy entity having established it, because the property right itself lives in this immutable ledger. Same thing here. At least in principle, you can set up resolution contracts that are trustable and immutable and therefore expand the scope of the set of marketplaces we can configure. It's not just the set of tools we had when you have to be able to trust the market organizer, but now this commitment enables you to go further.
--:-- Sonal Chokshi: Just to break this down a little bit more, because I think you said some really important things in there and I want to pause and make sure we flesh it out for our audience. First of all, based on what Alex said earlier, in the case of Polymarket, one of the key points was public and the information being out there. I mentioned the example of it being updated quickly compared to media. You just mentioned the importance of credible commitments, and we've often described blockchains as computers that make commitments. So that's a third or fourth feature. And then you also mentioned potentially decentralized, but I couldn't tell if it really needed to be decentralized or not. Can you give me more bottom line on decentralization where you stand there?
--:-- Scott Duke Kominers: Yeah, that's a great question. Maybe we should have started here. The necessity of all of these different features moves around with the type of market. The more complicated your information elicitation mechanism is—and this is especially important for the context where pure information markets don't work—the more likely it is that you want something that looks like crypto rails.
--:-- Sonal Chokshi: Ah, that's actually good to know. Okay.
--:-- Scott Duke Kominers: So if Hewlett Packard is running an internal prediction market, first of all, it doesn't have to be open to the entire world because you're only trying to learn information from your employees. Openness is important within the firm. Maybe there's someone in the mailroom who knows something that you don't know they know, so you actually want that market of people to be able to participate. But Hewlett Packard does not necessarily care what a person on the street thinks about printer sales, and certainly doesn't need to build the architecture to bring in random people's estimates of printer sales. You need some amount of transparency because you need people to be able to see what the current price is and see whether they agree or disagree so they can move the price around. But in other types of elicitation mechanisms, maybe you don't need transparency. If you're just going to pay someone based on the accuracy of their forecast down the line, you don't need them to be able to see what else is happening. You just need them to believe that you have committed and that the final accuracy is going to be transparent. That they can verify that you didn't stiff them by saying the thing they predicted didn't happen when it actually did. Transparency is important only there with respect to the resolution, not with respect to the interim states.
--:-- Alex Tabarrok: Yes, yes.
--:-- Scott Duke Kominers: By contrast, commitment is incredibly essential and needs to be believed, or else the user won't even participate.
--:-- Sonal Chokshi: By the way, great that you gave the example of the transparency. I'll let you finish your example in a second, but I'm jumping in because it reminds me of how we talk about the things that can be done on-chain and off-chain when it comes to scaling blockchains, and provers versus verifiers when it comes to zero knowledge. It's really interesting you pointed that out because I want to make sure people who are listening who are builders listen to that. That means you can do certain things on-chain to achieve your design goals and then put other things off-chain. You don't have to have this purist view of how truth must be transparent. It's very smart to point that out. Anyway, keep going with your other example.
--:-- Scott Duke Kominers: Yeah, and I completely agree. One of the things when I talk to teams, I'm constantly trying to get them to think about which features of the marketplace are the most essential for market function. It varies by market context. Even if eventually you're planning on having all of these features, as you're deciding which thing to build first, or as you're progressively decentralizing, you have to understand the market context you're working in.
--:-- Sonal Chokshi: That's so smart because it's basically another way to hit product-market fit too, because then you're not overbuilding and over-featuring something. Anyway, yeah, but keep going with your other side of that.
--:-- Scott Duke Kominers: Totally, 100%. So to get to the question of when does decentralization matter: Decentralization has lots of different components that might make it matter. One of them is the ability to make these commitments even more enforceable. It makes it possible to be confident in function and liveness and so forth. All of those things are important for a market because if your prediction market goes down the night before the election, first of all, you lose the information signal from it. Second of all, you lose the ability for people to participate in the market, which would adjust the price and move the signal around. Similarly, if you lose the ability to resolve the truth, then maybe you can't finally resolve the market and you have all of these bets sitting in limbo because the market doesn't know what happened. The key is everyone is bringing in their own information. But to finally resolve the contract and determine who gets the payout for the bet, you have to have the chain have a way to know what actually happened. Another place decentralization is sometimes very important is in that resolution function. If the market is on-chain, you somehow have to get what actually happened onto the chain. Maybe the biggest bettor happens to also control the one resolution function, and they can rob the prediction market by lying about the resolution of the event. They tell the system, "Candidate A won," when actually candidate B won. By the time people realize this wasn't correct, they might not have a way to fix it. That person might just be gone. So decentralization in resolution—this is basically an oracle. You have to bring off-chain information on-chain in a lot of these contexts to resolve the contract. If you're doing this on a centralized platform, the users have to trust the centralized platform to resolve the contract correctly. By contrast, if the information does not need to be brought in through an oracle—if it already lives in a system that's verified and the resolution is provably going to do what it's claimed—then you don't actually care about decentralization in the discovery of the resolution. You're just reading information and your commitment contract takes care of everything else.
--:-- Sonal Chokshi: Just really quick, Scott, you've said "oracle" a few times. Can you properly define what you mean by oracle in this context? I know we talk about it a lot in crypto.
--:-- Scott Duke Kominers: Yeah, and indeed, "oracle" is not a completely uniformly well-defined term. In this context, I'm talking about oracles as a truthful source of information about what the actual resolution of the event was. So if Trump won the election, the oracle tells us Trump won the election. If Harris won the election, the oracle tells us Harris won the election. The reason we're using that is because the election is, of course, not being conducted on a blockchain—at least not in 2024. So if you're going to have an on-chain prediction market, you somehow need the chain to be able to learn the information of what actually happened in the off-chain election. The oracle is basically the source of that information.
--:-- Alex Tabarrok: The key of the oracle, as Scott says, is to bring off-chain information on-chain. The thing about off-chain is that people can look at the New York Times. The New York Times is often considered an oracle in that you go by whatever's printed in the New York Times. That would be a way of resolving a lot of bets. "Did the New York Times report Trump won?" That might be one way of resolving these bets.
--:-- Scott Duke Kominers: Yeah, great.
--:-- Alex Tabarrok: But the key problem is to bring that off-chain knowledge on-chain in a way in which the information is not distorted in the transmission. The reason you're worried about it being distorted is precisely because that's the revelation where all the money is.
--:-- Scott Duke Kominers: Right. So there are big incentives to distort the transmission of that information. In fact, a lot of the crypto hacks which have happened have happened because people found a way of distorting the oracle and then using that on the crypto market. The market resolved in one way, and if you can change the oracle, you can make a huge amount of profit. So there's a big incentive to mess with the oracle. That's why it's really difficult.
--:-- Alex Tabarrok: And we can stick with the New York Times example. A lot of people are going to make their morning trading decisions based on what they see in the New York Times and on the Bloomberg terminal. If you could, in a coordinated way, feed the wrong information to that, it would change many people's behavior, and you could trade against that because you knew they were going to get the wrong information.
--:-- Scott Duke Kominers: Exactly. So this can happen in the off-chain world. Indeed, we saw there was one tweet that the SEC is going to legalize Bitcoin ETF contracts. It looked like an official ruling, and it turned out to be a hack. Turned out to be correct, but that wasn't revealed until days later. So if you can distort an oracle, you can make money.
--:-- Sonal Chokshi: Totally. Or if we're talking about the New York Times, it would be remiss to not have the "Dewey Defeats Truman" example. Famous front page with huge text headline that just turns out to be inaccurate. That's a famous case of what we did in media at Wired too. It's called the pre-write. You accidentally print it sooner and you get it wrong. There have been cases where someone writes an obituary months or years in advance and it goes out saying they're dead. You conflated earlier, and I agree they're generally connected and similar, but there are some nuances between decentralized and distributed. Distributed can just be redundant systems with multiple nodes—the system going down is a case where being distributed matters, but it doesn't have to be decentralized. There could be distributed nodes managed by a centralized entity. I just want to make sure we're very clear about the distinction between decentralized and distributed. Whereas by contrast, with the oracles, you might really care about being decentralized. You might care that no individual entity can unilaterally change how the contracts resolve.
--:-- Scott Duke Kominers: Exactly. Just one other point. Another advantage of doing all this stuff on blockchains is that it's composable. It's not that we're just intrinsically interested in some of these questions—like maybe some people are intellectually curious about who's going to win the presidency in a month—but rather lots of other stuff depends on it.
--:-- Scott Duke Kominers: If you're making decisions about which supplies to order in advance, you need to have beliefs about the likelihood the tariffs are imposed under the next administration. And so having these things live on open composable architectures is useful because they can be wrapped with other information and other processes. You can tie your corporate operations in a very direct way into these sort of information aggregation mechanism signals.
--:-- Sonal Chokshi: Yeah, to put it an even more basic way, just because I don't know if everyone necessarily knows "composable" in the way that we talk about it. It's like the Lego building blocks. The markets on chain or the information on chain is a platform that people can build around, build with, bring in pieces of information, combine it with other tools, et cetera. And you can create different things. And that's composability. I'll put a link in the show notes to a post explaining composability as well. And then the other quick one is open source. Does the code itself have to be open source, auditable, public good? Again, it depends how much you trust the market creator.
--:-- Scott Duke Kominers: Yeah. And again, this is true across the board for applications that can be run on blockchains or not. You're always making trade-offs between trust through reputational incentives and institutions, and trust through code. For example, in actual commodities markets, there's a lot of trust through institution and legal contract, but there's an architecture in place to establish the trust between the institutions and the contracts and their enforceability via the institutions for those contracts to be real enough that people believe in them enough to pay money for them and to have all of these market features. Blockchains enable these sorts of trusted activities in lots of contexts where the institutions are not strong enough or present enough to do it for you. If you're having $5 bets, small money bets on some incredibly minor question—like, will the horse that wins the Kentucky Derby have a prime number of letters in their name?—you're not going to have an institution that is even able to evaluate and set up that contract in a way that is worth doing at the amount of money it's going to raise.
--:-- Alex Tabarrok: I like how Scott changes the Kentucky Derby into something he would be interested in.
--:-- Sonal Chokshi: Oh, well, if it involves prime numbers. Horses, forget horses, but prime numbers. That's so funny. I love how well you know him.
--:-- Scott Duke Kominers: I will have you know, the Kentucky Derby is also interesting because it has all sorts of cool statistical questions going on.
--:-- Sonal Chokshi: And cool hats.
--:-- Scott Duke Kominers: Fascinating hats. Absolutely fascinating hats.
--:-- Sonal Chokshi: Pun definitely intended. I love it. So like substituting code for the source of trust for these very unusual or micro or international contexts where there's not a clear jurisdiction—all of these contexts push you more into security via code rather than security via institution.
--:-- Alex Tabarrok: Let me add one more point on the blockchain. So I think generally speaking, as I said, the blockchain is not necessary. However, as we're looking towards the future, it may become more and more useful to have these very decentralized rails. So Vitalik Buterin recently wrote a post on InfoFinance talking about prediction markets. And he credited you at the top as one of the people who reviewed it.
--:-- Sonal Chokshi: But yeah, keep going.
--:-- Alex Tabarrok: Exactly. And so one of the interesting points which he made is that AIs may become very prominent predictors. They may become very prominent participants in these prediction markets. Because if you can have a lot of AIs trying to predict things, that lowers the cost tremendously. And that opens up the space of possibilities of what you can use prediction markets for. And so the blockchain is very good for, you know, nobody knows you're an AI on the blockchain.
--:-- Sonal Chokshi: Right, right, right.
--:-- Alex Tabarrok: And so if we're going to have a lot of AIs interacting and acting as participants in markets, then the blockchain is very good for that.
--:-- Scott Duke Kominers: That's absolutely right.
--:-- Sonal Chokshi: And we have a lot of content that's already on this topic, which actually gets at the intersection of crypto and AI and where they're a match made in heaven. In fact, not only because of AI's centralizing tendencies and crypto's decentralizing tendencies, but because of concepts like proof of personhood, being able to, in privacy-preserving ways, yet even on a public blockchain, find ways of adding attribution. And there's just so much more that you can do with crypto. I agree, Alex, and I'm so glad you brought that up. It's funny because when you were saying earlier that in the early definition of a prediction market as this way to kind of elicit information that's dispersed across many people, I immediately went to, "Oh, that's the original AGI." If you think about artificial intelligence, let's just talk about human intelligence at scale. Like that's what a prediction market can be. I do want to make sure we also touch on other applications a little bit on the future. One quick thing, though, before we do that. So now we've summarized some of the key features. We've talked about the election. We've talked about some of the underlying market foundations and some of the nuances. We've talked about what does and doesn't make prediction markets work, and also mentioned earlier that they're part of a class of mechanisms that can aggregate information. So I want to quickly summarize what are some of those other mechanisms that could get at this kind of information aggregation that aren't necessarily prediction markets.
--:-- Alex Tabarrok: Awesome. So first of all, again, just to think about what is this class of information aggregation mechanisms. And Alex defined it earlier—it's these mechanisms that bring together lots of dispersed information to produce an aggregate statistic or set of statistics that combine the information of many different sources. And ideally that aggregate is informative. Now, there are lots of ways to do that. Some of the simplest ones we actually talked about earlier are just to ask people for their predictions and later pay them based on whether they're correct. And you can do that with random people, wisdom of the crowd style, or you could do that with experts. Very simple types of information aggregation backends are just incentivizing people to tell you what they know. Or even just go and survey them in an unincentivized context, where people have no incentive to lie and just have an opinion. They don't have to do any research or invest any effort to know their version of the answer. You just run a survey. But then there's this whole menagerie of incentivized elicitation mechanisms that are designed around different elicitation challenges. I mentioned earlier peer prediction mechanisms. These are the mechanisms where you ask people for their beliefs and their beliefs about other people's beliefs. And then you use people's estimate of the population beliefs to infer whether they were lying to you about what they believe, or how informed they were in aggregate. So you can use that to figure out where the person fits in the distribution. And pure prediction is like an incentivized version of that. You're going to actually pay people based on how accurate they are, but you're not paying them based on how accurate they are about what actually happens in the future. Rather, you're paying them based on how accurate they are about the population estimate. And so that enables you to pay people up front immediately. These are used for subjective information or information that's dispersed among small populations. Maybe it's not big enough to have a thick prediction market, but people are informed enough that if you can directly incentivize them to tell you the truth, then you can actually aggregate the information usefully. A couple of my colleagues at HBS, Reshma Hassam, Natalia Regal, and Ben Roth, have this beautiful paper where they use these peer prediction mechanisms in the field in a developing country context. They ask people who in their community is likely to be the most successful micro-entrepreneur. And then they allocate funding according to these predictions. And it turns out that the predictions are actually quite accurate. The incentivized peer prediction mechanism produces answers that line up with who actually ends up being successful in these businesses down the line, in a way that is more effective than just asking people and telling them, "We're going to allocate the money according to whatever you said," because then people will lie and say, "Oh, my neighbor or my friend is the best."
--:-- Sonal Chokshi: I'll put that paper in the show notes, too.
--:-- Alex Tabarrok: Yeah, it's a great paper. Super fun to read. Very readable, too. So one way in which the wisdom of the crowds doesn't work, of course, is when the crowd thinks they know the answer to a problem, but they actually don't.
--:-- Sonal Chokshi: Oh, okay. Of course. Yeah.
--:-- Alex Tabarrok: So there's this great paper by Freilich and Song and McCoy, and they give the example of, suppose you ask people, "What's the capital of Pennsylvania?" Most people will think, "Oh, it's probably Philadelphia." It's the biggest city, popular city, American Heritage, Liberty Bell, all that kind of stuff. But it actually is the wrong answer. So if you go just by the wisdom of the crowds, you're going to get Philadelphia, and that's wrong. The correct answer is actually Harrisburg, which most people don't know. However, a small minority of people do know the correct answer. So how do you elicit this? Their mechanism for doing this is what they call the "surprisingly popular" mechanism. What you do is what Scott says: you ask people not only what they think is the correct answer, but what they think other people will say. Most people will think, "I think the correct answer is Philadelphia. Other people will say Philadelphia." But then you're going to see a bump of Harrisburg. It's going to be very surprising. There could be a substantial number of people who will say Harrisburg, and that will be quite different than what people expect. And if you choose that, the authors show that this can improve on the wisdom of the crowds. So the surprisingly popular answer—the answer which a minority chooses in contrast to the majority—can actually get you more information out. Depending upon the question, there are these clever ways of pulling this inchoate information out of the crowd and eliciting the truth, even when most people in the crowd don't know the truth.
--:-- Sonal Chokshi: That's fantastic. I'm obviously going to include all these things we're referencing in our show notes, but that one is really interesting.
--:-- Scott Duke Kominers: Right. That's wild. And then maybe one other piece in the menagerie, of course, the listeners of this podcast will be very familiar with, are simple auctions. Auctions are information aggregation mechanisms too. We talk about price discovery in an ordinary, very liquid market as being an information aggregation source, but some markets aren't big and liquid all the time. They don't have lots of flow transactions. Maybe it's a super rare piece of art. But an auction is still exactly useful for figuring out what the art is worth in the eyes of the market. And you can often discover things. There's some artist that was not popular to the best of your knowledge, and then they have a piece with a major sale. And people's estimates of the values of all of their other works change accordingly because of the information that's been revealed about people's change in taste from this one sale.
--:-- Sonal Chokshi: While we're thinking things through the show notes, there's an incredible book. Actually, I think this is my very first a16z Crypto book list contribution, called *Auctions: The Social Construction of Value* by Charles Smith, which talks about auctions from a sociological perspective as a way of establishing an understanding of value in a bunch of different contexts.
--:-- Scott Duke Kominers: That's great.
--:-- Sonal Chokshi: And by the way, I do want to plug the episode you, me, and Tim Roughgarden did where we literally dug into auction design for Web3 for hours. That was so much fun. So as we've been arcing through these different types of mechanisms, it's a really good reminder that the type of question you're asking, the type of market participants you have, shapes your decisions about how to structure your market mechanism.
--:-- Scott Duke Kominers: It also shapes your decisions about what type of market mechanism to use. If you think that the population is not super informed on average, but informed at the second order level, then this mechanism Alex was describing is perfect, because the information's there, it's just not immediately apparently there.
--:-- Sonal Chokshi: Right. What I love that you guys are talking about, and we can now segue into some quick discussion of some applications in the future, and then we can wrap up. We've been talking about implications for design throughout this podcast, but I think it is very interesting because you've been saying throughout, both of you, that it really depends on the context and your goals, and then you can design accordingly. And that's actually what incentive mechanism design is all about, as I've learned from you and Tim Roughgarden and seen over and over again. But two quick things, lightning round style, that I want to make sure I touch on. One, multiple times you both have alluded to this payout feedback loop. I'm inferring from what you've said that the payouts have to be almost quick, that you get an instant feedback loop on your outcomes. Because you gave an example earlier where if it's delayed by two weeks or so-and-so, it may be less effective. Is that necessarily true?
--:-- Scott Duke Kominers: Depends on trust and attention. Some people have said that one of their concerns about prediction markets is that people like betting on sports because it's happening in real time. You know the answer within a couple of hours, or in the case of a horse race, within minutes. Whereas these prediction markets often take months to resolve. The final answer or the time of resolution might not even be known. It might be, "Who will be appointed to this position?" So there's a possibility that speed is relevant for who chooses to participate in some context, whether they find it fun. The other context we were talking about is when time matters for trust. If you're in the developing world trying to figure out how to allocate grants, people might not trust, or even just have the infrastructure support, to participate in a mechanism where they're going to be paid six months out based on the resolution of some confusing outcome. Whereas if you can pay them today, they'll participate today. Hence why they experimented with peer prediction mechanisms in that context in the first place. It was a setting where you could in principle pay people based on the outcome, like how successful their neighbor was at being an entrepreneur with whatever grant they'd received. But a lot of complexity goes into actually doing that in practice because you have to track down the people again and all of that.
--:-- Sonal Chokshi: Ah, yeah. One other quick buildery thing that came up, again, seems so obvious to you guys probably, but the best systems are where prediction markets and such systems work when there is a discrete event, like an election or something to be resolved. It probably wouldn't work for some ongoing, loosely defined, non-discrete event.
--:-- Scott Duke Kominers: So the prediction market mechanism, the canonical prediction market as we've described it, is a mechanism where you're buying an asset that has a payout as a function of a discrete event. But that is not even the average case of markets. When you're buying oil futures or something, most of the transactions in many of these markets are actually in the interim. It's based on changes in people's estimates. And so if you have a market where it's possible to continually update and trade as estimates change, then you can still gather a lot of information, even if the value attained is in a flow or you're in stages or something of the sort. It doesn't have to be a single cutoff date. I think you can design them in different ways. They do have to resolve at a point in time, but the way that they resolve could be based upon a stock price or something like that.
--:-- Alex Tabarrok: Yeah. And you can have dividends or something. You can have things that pay out over time based on interim steps. Lots of things have continuous payouts based on the growth of a company or something of the sort. And so you could imagine prediction securities that are kind of like that.
--:-- Sonal Chokshi: The stock market, exactly.
--:-- Alex Tabarrok: The stock market, exactly. The HP example I gave earlier divided the time into two periods. Is it May to June, or is it July to August, or September, October? You can always take a continuous event and chunk it into five or six discrete periods.
--:-- Sonal Chokshi: Yeah, yeah. Even if somewhat arbitrary, that makes so much sense. So, so far, these prediction markets have been used just for what we've been saying: predicting something. But you can also create, and here I'm going to riff off Robin Hanson again, my colleague on these questions, and he says we can also create these conditional markets. So the question would be something like, as I said earlier with Futarchy, "What would happen to GDP if we put together this science policy?" Now, we might not want to jump all the way from democracy into Futarchy in one go. We're probably not ready for that.
--:-- Alex Tabarrok: We're not ready for the full hands.
--:-- Sonal Chokshi: Not quite ready for prime time, I think.
--:-- Alex Tabarrok: Yeah.
--:-- Sonal Chokshi: But here's a fascinating idea of Robin's, which I think we are ready for and which we should use. And that is, "What would happen if we fired the CEO?" This is a huge question that companies want to know. We saw a few years ago, it was kind of remarkable when Steve Ballmer left Microsoft and the stock price went way up, suggesting that the market thought that Ballmer was not a great CEO. Or we just saw with Brian Nichols, he moved to Starbucks from Five Guys. He'd been extremely successful at Five Guys. He moved to Starbucks. On the day that Starbucks announced that they were hiring Brian Nichols as CEO, the price of Starbucks jumped up. So why, however, do we need to wait? How about creating a continuous market which says at any given time, "Would the price of Starbucks be higher if they fired the CEO?" And so you can create these decision markets, prediction markets. You create a prediction market in "Would the stock price be higher if we had the same CEO, or would the stock price be higher if we fired the CEO?" Now, that's an incredibly useful piece of information. Companies spend billions of dollars every single day based upon exactly this question. And that's a question which I think decision markets, prediction markets, would be really good at answering. We already have the stock market. People are already investing billions of dollars in exactly this question, and we can make it more precise and more detailed and more usable.
--:-- Scott Duke Kominers: What I really like about that application is it leverages a type of information that people are already developing. People are spending a lot of time reasoning about what's going to change the stock price of Starbucks, and they have a lot of different refined ways of doing it, but it uses it to address a question that's useful as a practical hypothetical. As Alex said, it brings the information forward in time. Normally in a current market context, we can only learn what happens if Starbucks replaces the CEO when they replace the CEO. But actually that's the least important time for us to learn that. We actually want to know it when they're deciding, "Should we replace the CEO?"
--:-- Alex Tabarrok: Yeah, exactly. You want to know it before, yeah.
--:-- Scott Duke Kominers: And so being able to harness that same effort that people are putting into understanding what affects the stock price of Starbucks and which companies are well run and which aren't, and pushing it towards this question, can reveal important information at a time when it's more useful.
--:-- Sonal Chokshi: Leveraging things people are already good at predicting. Exactly. That's such an interesting and such a useful and extremely real and possible right now thing to do. We're not just being crazy futuristic like 10, 15, 20 years from now. That's so great.
--:-- Alex Tabarrok: Can I be crazy futuristic? Put you in a little bit more?
--:-- Sonal Chokshi: Yeah, yeah. We actually want a little of that. Go for it.
--:-- Alex Tabarrok: You're absolutely right. The "should we fire the CEO" market could be implemented right now and it would be extremely useful. And it's the first step towards making more decisions by DAOs, by a blockchain consensus. If you can make a decision about "should we fire the CEO," "should we expand into Argentina or into China," "should we have a new model this year," you can start asking the market lots of different types of these questions. So let's start with "should we fire the CEO," one of the biggest and most important, most salient of these questions, where Scott says it's an information rich environment. People are already collecting lots of information on exactly this question. And once we've got some experience in this market, we can start applying it to further markets down the line.
--:-- Scott Duke Kominers: Footnote. I love that application, too. And that ties into the importance we talked about earlier of running these markets in an internal currency. An advantage there is you can use it to put everyone on the same footing at the outset. For the Starbucks CEO question, there are many different very high value and high ability to trade entities that already are participating in this style of question. Whereas for a DAO, you actually might have tremendous inequality in wealth of the participants, but you can make them wealthy in proportion to their reputation or something in the internal token, which can then be used to have them all participate equitably at the entrance to these decisions.
--:-- Sonal Chokshi: I love this. And this is where I'm very proud that we have published a deep body of research across many people, not just our own team, into DAOs, what makes them work, what doesn't work, what's effective, governance mechanisms. I'm going to link to that in the show notes. Because also we're arguing that sometimes you can do a lot of these things, not just in the crypto world, but you can apply them to other decentralized communities. And I want people to remember that that's a useful use of DAOs, which are just decentralized autonomous organizations. Are there any other pet applications, either current or futuristic, that either of you have?
--:-- Alex Tabarrok: I have one, but I'm going to wait till you guys are done.
--:-- Scott Duke Kominers: I mean, two other very quick hits. We haven't touched directly yet on the idea of markets for private data. Another form of information aggregation is that maybe a lot of people have information that will be useful in designing a new pharmaceutical or medical treatment, and they have their own private information of this form, and we'd like to be able to elicit it from them in a way that also fairly compensates them for their participation. We have some mechanisms for this already, like surveys managed by a health center that pay you a show-up fee for participating. But there's a possibility for much richer markets of that form that leverage individual data ownership and permissioning and so forth.
--:-- Sonal Chokshi: Yeah, one example, by the way, just concretely, is in the DeSci movement, decentralized science, where people are putting their information, like medical data, using blockchains to bring more ownership, transparency, consent, which they don't have. That's just one example. What's the other one you had, Scott?
--:-- Scott Duke Kominers: The other one is getting incentivized subjective beliefs. We've talked a lot about predictions of things that have an objective truth, but another big frontier for information aggregation is getting really good estimates of things that people believe that are fundamentally subjective. If you're trying to do market research for your product, "Do people want this?" One of the advantages of crowdfunding, for example, is that it's a better information elicitation mechanism. You can go and ask 10,000 people, "Do you want to buy this?" And some of them might say yes, but unless you're actually taking money from them, you don't know whether that's a truthful representation.
--:-- Alex Tabarrok: Yeah.
--:-- Scott Duke Kominers: And so crowdfunding lets you learn about the total market for your initial version of the product in a way that's incentivized. More broadly, I think subjective elicitation is a really important direction to go into.
--:-- Sonal Chokshi: Can you quickly maybe give a very short definition in the uniquely crypto blockchain context of a Bayesian truth serum here? Isn't this where Bayesian truth serums apply?
--:-- Scott Duke Kominers: Sure. The Bayesian truth serum is actually an example of those pure prediction mechanisms we described, and there are many different versions of it. But loosely, the idea is, if I ask you your opinion on something, "Did you like this movie?" And then I ask you, "What's the likelihood that another person I ask will say that they liked the movie?" You might have a reason to lie to me about whether you liked the movie or not. You might say, "Oh, I really liked it," because you produced it. What am I going to do?
--:-- Scott Duke Kominers: But you actually hated it. Your estimates of everybody else's beliefs will be sort of tilted in the direction of them mostly disliking it, so long as I'm going to reward you proportional to your accuracy. Like, you know that you disliked it, and so everyone else probably will too, because you're a Bayesian. And so I can detect, looking at everybody else's responses, I can detect whether you sort of told me a distribution of other people's beliefs that's consistent with what you said your belief is.
--:-- Sonal Chokshi: Great. One of my quick applications, and kind of an obvious one, but I want to just call it out because I find it very boring when people say the same thing. They're like, "Oh, media, da-da-da," but whatever. What I find very interesting is, and people often talk a lot about having mechanisms for quote, "finding truth," but sometimes I find it to be very pedantic and moralistic and equally as grating as the very people they're trying to bring down. And so it's a pet peeve of mine when I'm on the Twitter discourse, like, "Oh God, I'm so bored by this." But I do find it very interesting that some of the commentary surfaces at prediction markets for basically resolving more accurately and faster than mainstream media, but not having some of the same filtering of partisan interests.
--:-- Alex Tabarrok: Although this might be different with certain communities of DAOs, if you do predictions limited to certain DAOs.
--:-- Scott Duke Kominers: Yeah, again, it depends who's in your market.
--:-- Sonal Chokshi: Yeah, exactly. This goes back to your point about thick and thin. But it's also interesting because it's a way to put a little bit more skin in the game, which is one of the biggest drawbacks in current media is like the people writing don't have skin in the game, which is why I've always been a believer in not having third party voices, but the experts write their own posts and then editing them is more interesting to me. So I do think it's very interesting to think about this use case of reinventing news media using prediction markets. And Vitalik's post actually had a great headline which is that think of a prediction market as a betting site for participants and a news site for everyone else. That'd be my application.
--:-- Alex Tabarrok: So I think more generally, it is odd how we do quite a bit of journalism. For example, it's totally standard practice for a financial journalist for it to be against company policy for them to invest in the companies which they're recommending. And as an economist, I kind of think, wait a second, don't I want the exact opposite?
--:-- Sonal Chokshi: Yeah, you want more skin in the game, exactly.
--:-- Alex Tabarrok: Yeah, more skin in the game. So, I say that a bet is a tax on bullshit.
--:-- Sonal Chokshi: I like that line.
--:-- Scott Duke Kominers: That's a great line. I love it.
--:-- Alex Tabarrok: So, how about you have to be upfront about it. You have to be honest about it, transparent about it. But maybe journalists should say, "This is what I think will happen. And these are the bets which I've made." And you can see my bets on chain.
--:-- Scott Duke Kominers: Yeah, yeah.
--:-- Alex Tabarrok: And let's see what their past track record is. It's kind of amazing that we do not have any track record of opinion editorialists whatsoever. Only Tetlock started to create that and found that they were terrible. But how about let's create a series of bets on chain? And this would change the types of people who become editorialists who get these jobs in the first place. So let's start making sure you bet your beliefs and then let's promote people whose bets turn out to be accurate. And that's going to change journalism entirely if we were to change the metrics by which journalists are evaluated.
--:-- Sonal Chokshi: I agree. Annie Duke talks a lot about this too. It's not just bets in a binary true/false way, but bets that are weighted in terms of likelihood, probability of accuracy. Like you don't have to make a binary like "it will be this or that." I believe 80% that X will happen. And that is also another way to assess in a more nuanced way. And that gives a lot of room for the nuances that are often true when it comes to guessing the truth.
--:-- Alex Tabarrok: Absolutely. Exactly. There's a big incentive to say, "This is never going to happen. This is impossible." But then if you ask them, "Well, if it's never going to happen, are you willing to bet $10 that it might happen?" They should all be willing to. Of course, they're never willing to make those bets.
--:-- Sonal Chokshi: That's right. Even people who hate Elon Musk as journalists will then start saying, "Well, actually, I'm going to bet on that guy for Building X to happen because I saw that shuttle launch." And now I'm thinking, "OK, maybe I increase that from 10 to 20 or whatever."
--:-- Scott Duke Kominers: Yeah. Exactly.
--:-- Sonal Chokshi: So betting could reduce the hyperbole.
--:-- Scott Duke Kominers: Yeah, that's exactly right. Yeah, totally.
--:-- Sonal Chokshi: By the way, this bordered on some other really critical information elicitation mechanism that uses a different version of this sort of cross-examining some people's beliefs against others. If you think about community notes on Twitter, that's an information aggregation mechanism. It's like getting a lot of people's opinions and then only deciding that they're correct if you have agreement from people who usually disagree.
--:-- Scott Duke Kominers: Yes, exactly. Because that's where Wikipedia failed when they had the cabal of expert reviewers. They didn't have that kind of check and balance mechanism.
--:-- Sonal Chokshi: Yeah, totally. Community notes is a great one. I have one last question for you guys, because we don't have enough time to go into policy. In general, some of these became popular because they're offering contracts that were banned from the market. So a big question is whether the offshore crypto markets will follow the rules or not. So how do you sort of create innovation in that environment? To me, the core question here is, what's the difference between gambling and speculation? Is there a difference? I'm curious if you guys have a thought on as a parting note on this.
--:-- Scott Duke Kominers: I mean, so one very important thing to remember is that depending on the context, you may be in a different point on a continuum. Part of what makes sporting events exciting and suspenseful is that there's a lot of stochasticity and the amount of information that any individual has is reasonably small, even if they put a lot of effort into figuring it out. But there might be some amount of informed betting in sporting events. And then as you move towards things where there's a lot of information to be had and a lot of value also to knowing the answer, a lot of market value to actually figuring it out, like how do we allocate goods and markets, going back to the very beginning when we were talking about the role of markets in determining the value of something and clearing supply and demand. There, there is value generated through the process of people engaging. Now, there's one really important caveat about speculation. We talk about this a lot in crypto land. There is speculation of the form: "I have beliefs and I'm investing to support a product that I think will exist and that I want to exist and that I think other people will want." And then there's also speculation on speculation where you're actually not so much betting based on your own beliefs. You're betting on what you think other people will choose to bet on, like we talked earlier about herding. You might place bets because you think other people are going to place bets in a given direction, not because you actually have any information about what's going to happen, just because you have information about how the market might move.
--:-- Sonal Chokshi: That's speculating on speculation.
--:-- Scott Duke Kominers: Exactly. That's speculating on speculation. So there is this sort of valuable type of speculation, which is people moving resources around in a way that reflects their beliefs and can help us make markets work better and achieve better outcomes. That's in this mid space between the randomness where moving the money around has no impact on outcomes, you're just betting on coin flips, your money does nothing. And this other edge where moving the money around becomes its own project that is independent of outcomes. And so again, that doesn't provide information. These prediction markets are particularly well architected, at least in the cases where they're very large and thick and all the things we talked about that you need to make them work. They're particularly well architected to try and be in that midspace where the information provided is valuable and comes out of real knowledge and activity in a way that actually means the market does something valuable.
--:-- Sonal Chokshi: Yeah. And by the way, on the earlier example, when we talk a lot, the obvious examples where it plays out is like the Carlotta Perez framework of speculation phase followed by an installation phase. That's a driver of technology cycles. There's also the example, Berne Hobart wrote a piece for me a few years ago on how bubbles are actually a good thing when they have a certain type of quality in this case. And he also wrote a new book about it recently for Stripe Press with Tobias Huber, which they go into greater detail about that.
--:-- Alex Tabarrok: Oh, I should read that.
--:-- Sonal Chokshi: It's basically an example of, quote, I don't want to put moralistic terms on it necessarily, but useful speculation that leads to other things as an outcome versus speculating for the sake of speculating, which is partly the distinction you're pointing out.
--:-- Alex Tabarrok: Well, I think people in Las Vegas who are at the slot machines, they're gambling because they have no way of influencing or of improving their predictions of what the slot machine is going to show up. It's just pure random chance. On the other hand, there are many, many areas in which we are trying to predict the future and in which investing can help us improve our predictions. And this is why I think prediction markets should be completely legal, should be legalized because of all the forms of gambling, of all the forms of speculation, this is one of the most useful forms. So we want to incentivize the type of speculation or gambling which as a side product produces these useful public goods, which is trying to predict the future. This is incredibly important. You think about all of the questions that we have: what is happening with climate change? Which of these scientific predictions are accurate? Who is the best candidate for the presidency? All of these questions we have, prediction markets can help us answer these questions in a way which is more objective, more accurate, and more open to everyone. So I think the case for legalizing these is very, very strong.
--:-- Sonal Chokshi: That's amazing. I'm going to give you the last word on that, Alex. You guys, thank you so much for joining this episode. That was so fun.
--:-- Alex Tabarrok: Thanks, Sonal. Thanks, Scott. It's been fantastic being here.
--:-- Scott Duke Kominers: Thanks so much. Really fun conversation and QED.
--:-- Sonal Chokshi: Bye, QED.
--:-- [Outro music begins]
--:-- Sonal Chokshi: This episode was produced and edited by Sonal Chokshi. That's me. The episode was technically edited by our audio editor, Justin Golden. Credit also to Moonshot Design for the art and all thanks to support from a16z crypto. To follow more of our work and get updates, resources from us and from others, be sure to subscribe to our Web3 weekly newsletter. You can find it on our website at a16zcrypto.com. Thank you for listening and for subscribing. Let's go.
