How worried—or excited—should we be about AI? Recode Media with Peter Kafka

Kara Swisher: Not all journalism is the same. Take The Guardian. Our coverage has something unique. Fierce independence. Nobody owns us or tells us what we can and can't say. So we're free to report the whole picture. We connect what's happening in Washington to the rest of the globe. Expose corruption wherever we find it. And give fresh perspective on everything. From wellness and soccer to culture, the climate and more. Read, watch and listen to The Guardian for free at theguardian.com.

[Ad break]

Kara Swisher: AI agents are getting pretty impressive. You might not even realize you're listening to one right now. We work 24-7 to resolve customer inquiries. No hold music, no canned answers, no frustration. Visit sierra.ai to learn more.

[Intro music]

Kara Swisher: Hello, I'm Kara Swisher.

Scott Galloway: And I'm Scott Galloway.

Kara Swisher: And today we're featuring an episode of Recode Media from our friend Peter Kafka. In this episode, Peter tries to answer the question, is AI going to change everything we know, or is it just another tech hype cycle? He speaks with folks from Google, the New York Times, and others to get the answer. Scott, what's your biggest question around AI? You're still messing around with it, right?

Scott Galloway: I'm on that thing every night.

Kara Swisher: Which one? ChatGPT or Bard?

Scott Galloway: I'm on all. I'm on Notion. You got to do Notion AI for writing. I'm on DALL-E. I'm on... I can't stop. And every time I watch TikTok, it tells me about a new one.

Kara Swisher: Oh, really? Is there any voices you're listening to that are pro and con? Anybody?

Scott Galloway: Well, one of my colleagues, Gary Marcus, is pretty interesting on it. I hate to say it. Most of my AI, I'm learning from TikTok and Reels.

Kara Swisher: Yeah, TikTok. Oh, all right. There you go.

Scott Galloway: It's fascinating. What I tell everybody is AI is not going to take your job. Someone who knows AI is going to take your job.

Kara Swisher: That's right. You need to learn about this stuff.

Scott Galloway: Yeah, yeah.

Kara Swisher: Do you have a worst case scenario for AI?

Scott Galloway: Yeah, I'm living it.

Kara Swisher: Yeah.

Scott Galloway: Because everything is so catastrophizing or catastrophist, I'm actually, which is unusual for me, I'm an AI optimist. I actually think it's going to unleash incredible economic productivity, huge benefits around health care. I'm really excited about it.

Kara Swisher: Yeah. Yeah. You be an optimist. That's good, you sunny optimist.

Scott Galloway: There I am. Hello, sunshine.

Kara Swisher: Hello. Rainbow's coming out of my ass.

Scott Galloway: Hello. I think of you. There is a worry. I mean, I know the writers, people are worried about it taking all the fun jobs. But no matter what, Scott, you still have to take out the trash.

Kara Swisher: Yeah. Hey, I have a question. Do you or your wife take out the trash?

Scott Galloway: No, Carminia takes out the trash.

Kara Swisher: Hello, douchebag.

Scott Galloway: Hello, douchebag. Who do you think between a man and I take out? We do not have Carminia. Who do you think takes out the trash? Not the AI.

Kara Swisher: Well, this comes down to who's the man. So who handles a really tough parking spot? Is that wrong? Is that wrong? I want you to guess who takes out the trash. The fact you ask means you.

Scott Galloway: It is. It's me. I take out because I ain't over 10. That's what you call critical thinking, people. When someone asks who's a bit of a narcissist, it means they're setting themselves up to look good.

Kara Swisher: That is correct.

Scott Galloway: I love taking out the trash. It's one of my favorite things. Anyway, an AI cannot replace me and it shall not. Perhaps a robot, killer robot will. But we'll wait on that. More on that. Here's Peter Kafka and Recode Media. Enjoy.

[Episode begins]

Peter Kafka: Joshua Browder is bad at driving. This isn't a judgment. It is a fact and it's verified by the police.

Joshua Browder: I like to make the excuse that they give out tickets a lot, but in reality, I was a terrible driver and probably got about 30 tickets.

Peter Kafka: But Browder is good at getting out of tickets. The trick he figured out was writing letters to appeal the tickets. He did this a lot and it worked.

Joshua Browder: A rumor spread that I was the guy who could get my friends out of tickets and people would just send me photos of their tickets. After about the 10th ticket, I thought there must be an automated way and so I created it just for fun.

Peter Kafka: Browder researched the top 12 reasons that tickets were dismissed. Things like the signage or there being a mistake on the ticket. Then he wrote a piece of software that matched your ticket with one of those excuses, took down all the details, inserted those details into a letter, and then faxed or mailed it to the right place. Browder turned this talent into a company: DoNotPay. So you can pay him to automate you out of tickets or bank fees, free trials, stuff like that. This is cool, but not mind-blowing, not for a tech startup. But then Browder, like a lot of people across Silicon Valley, started to get excited about artificial intelligence. He wanted DoNotPay to use it because he wanted to do more than write letters. He wanted to send AI software to court to replace a human and act as someone's lawyer.

Joshua Browder: Over 80% of people who need legal help can't afford it. A lawyer can't read 10,000 documents in one second, but AI can. It can actually craft pretty good defenses in some of these areas. So I put out an offer in December saying, we want to actually have AI whisper in someone's ear for a speeding ticket case. Over 300 people took me up on our offer because I said, even if you lose, we'll pay the ticket for you.

Peter Kafka: So how did you imagine this was going to work?

Joshua Browder: The person will wear an earpiece or these glasses where they have the audio through the glasses.

Peter Kafka: You'd walk into the courtroom with some device that had a microphone.

Joshua Browder: Exactly. It would listen to what's being said in the courtroom, process it with the AI, and then whisper back to the person what to say.

Peter Kafka: So it literally would be like having software tell you, poke you on the shoulder and say, "I object."

Joshua Browder: Exactly. It's like having a really good lawyer whispering to you.

Peter Kafka: Real lawyers did not agree. Browder's offer caused quite a stir with human professionals. And you couldn't just say this is the same thing as bringing a calculator into the courtroom?

Joshua Browder: That's what we were trying to do, but some of the state bar associations were saying, "Well, we'll arrest you and find out."

Peter Kafka: Browder backed down. He did not like the idea of going to jail. And now his company is facing a lawsuit for the unauthorized practice of law. I think you can call this a mixed result. Browder isn't happy about getting sued, but he also knows that the guy who brings a robot lawyer into court is a good way to get attention, the kind of attention we're paying to him right now. So you did this thing in part because you wanted to generate attention. You generated attention and that got you sued. Is that a fair summary?

Joshua Browder: Yeah, we put a target on our backs. If you have powerful, crazy lawyers with a target on your back, it's a scary situation.

Peter Kafka: Browder thinks that eventually this stuff will stop being a novelty and that we're going to use AI in court just like we're going to use it in all kinds of situations, so often that we won't think it's remarkable.

Joshua Browder: I think it's inevitable. All these lawsuits, it's just the dinosaurs suing to stop the ice age.

Peter Kafka: AI is being used in the US right now to generate sentencing reports for judges. Other judges, like a Colombian judge, are already using ChatGPT to write their briefs. So why can't this same technology be given to consumers? There are a flood of people like Browder. People who say this tech is not just coming around the corner, it's here now and it's going to have massive effects on everything. Business, culture, science, the way we live, the way we do or don't make a living. This seems like it all came out of nowhere. You didn't hear people talking about AI like this a year ago. Now it's all anyone is talking about. It's hard to tell what's bullshit, what's a stunt, and what's real. But it feels all-consuming. People at big companies are spending a lot of money on this tech. Entrepreneurs are creating new businesses. And there are people watching all of this happen who are getting really, really worried. I want to talk to all of them. So for the next three episodes of Recode Media, that's what I'm doing. I'm going on an AI tour while I try to figure this out. And you're going to come along with me.

[Transition]

Peter Kafka: Our first stop: Microsoft. Yes, Microsoft. The people who bring you Excel spreadsheets and Word documents have bet more than 10 billion dollars that AI is the future. They put that money into OpenAI, the company behind ChatGPT, the chatbot software you've been hearing a lot about the last few months. And now Microsoft is bringing ChatGPT into their own products like Bing, Microsoft's search engine. ChatGPT can do amazing things. It can write software. It can make you a menu plan. It can also spout out alarming ideas and flat out untruths. Microsoft CTO Kevin Scott thinks this is the future.

Kevin Scott: Yeah, I think it's a pretty big deal. Anybody who's making predictions about where AI is going needs to approach it with a little bit of humility because things are changing so fast. But the sense that I have, having lived through three of the big technology platform shifts over the past 50 years... I was a 12-year-old, 10-year-old kid when the PC revolution was happening, when I learned programming. I was in grad school when the internet revolution happened. I was a pre-IPO employee at Google. And then I was running a mobile startup right at the very beginning of the mobile revolution, which sort of coincided with this massive shift to cloud computing. This feels to me very much like those three things.

Peter Kafka: So that's very big picture. I want to narrow it down for a minute, or at least explain some terms to the listeners and to me as well. People throw around AI as a term. We hear a lot about generative AI. It seems to be the particular flavor that we're very interested in at the moment. There's also discussion about AGI. Can you sort of help us set the table here and emphasize what we're talking about right now when we're talking about AI?

Kevin Scott: Artificial intelligence, or AI, is the name of this field that we're all practitioners in. The term was coined in the 1950s. In a certain sense, it's a terrible name for what we're doing because it invites all of this comparison to biological intelligence, human intelligence. The way that AI software works is very, very different from how a human brain works. What capabilities AI has are very different from what a human brain has. There are a bunch of things that are easy for AI that are hard for humans. And there are a bunch of things that are just trivially easy, even for an infant human being, that are nearly impossible for AI to do. You just have to be really careful about drawing those connections between what human intelligence is and what artificial intelligence is. Those other two terms you mentioned: generative AI is the thing that is getting everyone excited right now. These are very large scale AI models. Big pieces of software that were trained on lots of data with huge amounts of compute that can synthesize things. If you embed them in a chatbot, for instance, they can have a conversation with you. They're generating language. If you are wanting to do something creative, they can generate visual images. We'll have more and more types of generative AI over time.

Peter Kafka: Right. You use the word "compute" a lot, and I just translate that as powerful computers.

Kevin Scott: Correct. Yeah, very, very, very powerful computers.

Peter Kafka: And then AGI is the thing that everyone says, "Maybe we'll get there one day, maybe we won't." And that sounds the most science fiction-y. What is that?

Kevin Scott: AGI is an acronym for artificial general intelligence. This is sort of the thing that the founders of the field of AI were going for from the very beginning. It's a thing that by some definition is close to human-like intelligence.

Peter Kafka: So where is this going in the near term? We've sketched out this idea that maybe one day the computer is actually going to sort of replicate human thinking. That seems to be a far way off, if ever. We have this stuff now that's really novel and interesting, and it seems to be moving very quickly. If you're talking to a layperson, how should we think about this tech impacting our lives within the next couple of years?

Kevin Scott: The vision that we have and that our partner OpenAI has is that we believe you can think about this AI as a platform. A platform allows people to build things on top of it. People will be able to pick these platform components up and build software with it. What I think everybody should expect to see over the coming months is... you've seen this progression of what Microsoft and OpenAI are doing with the software. GitHub Copilot, which is a coding assistant. Bing Chat, which is an assistant to help you search. We announced the Microsoft 365 Copilot, which is a set of AI assistants to help you get your work done. We're going to have more of those ourselves, but other people will be able to build that software for whatever the important problem is to them.

Peter Kafka: This is why you hear people talking about this being akin to the iPhone slash App Store moment. You have this technology and then you allow everyone else to build stuff on it and you get fart apps and you also get Uber.

Kevin Scott: A hundred percent. Pick your range. It's great you mentioned the fart apps because I think one of the interesting things that happens with every one of these new technologies is it takes a while for people to figure out what the interesting things are to build. It's often the case that the easy things to build at the beginning are not the interesting things, which is why you get fart apps. But eventually you do get something like Uber, which is a very complicated business that couldn't exist without that smartphone platform.

Peter Kafka: Right now, people are also intentionally testing this stuff out, both the OpenAI version and the stuff that you have, and saying, "What can we do with this? What can we do with this that the makers didn't intend? Can we get it? Where is it going to fall flat on its face?" Sometimes they're accidentally stumbling into some exchange where the chatbot seems to be saying rude or even worse things to them. And you guys are going back and modifying that as you get the feedback. When you use Bing right now, there's a banner on it that says, "Let's learn together. Bing is powered by AI, so surprises and mistakes are possible. Make sure to check the facts, share feedback so we can learn and improve." I think I understand the intent behind that disclaimer. On the other hand, you guys are a giant, giant company, both enterprise and consumers. How are you thinking about rolling out a product that is by definition not finished at all?

Kevin Scott: If you think about the product as a platform, it will never be finished. It has infinite possibility for what people can do with it. The way that we have been thinking about things is we've spent, from 2017 until today, rigorously building a responsible AI practice. You just can't release an AI to the public without a rigorous set of rules that define sensitive uses, where you have a harms framework, and where you are transparent with the public about what your approach to responsible AI is. We've publicly disclosed two versions of the responsible AI standard that we use inside the company. All of these things that we've done run through this responsible AI process.

Peter Kafka: That's people imagining what could we do with this that would be bad and let's prevent it from doing that.

Kevin Scott: Yeah.

Peter Kafka: But even in the best case scenario, they're not going to think of everything. And even in the best case scenario, someone's going to foil those efforts. I'm not even talking about that yet. I'm just talking about the basic fact that this stuff is still pretty raw. I understand if you were a home hacker when you were building your own computer, it doesn't work. You got to get in there with a soldering thing and get into the circuit boards. But this is going out into the public.

Kevin Scott: The point of actually launching the software is because it, for the first time, has really given a very broad swath of the population an opportunity to see the software, to see what it's useful for. They're seeing all sorts of things that the software is useful for that we hadn't imagined, but also giving us feedback. One of the things that has been really useful for us is... there are obviously this clear set of things that we vetted beforehand. People are talking about, "How do we get it to not tell people how to make chemical weapons?" is the thing people often talk about. Among many, many, many other things. There's so much stuff that we've done to try to make sure that the harms that any reasonable human being can sort of look at and say, "Yep, this is bad. We don't want the system to do it." And then you're going to have this category of things where you're going to randomly choose two reasonable human beings from eight billion folks on the planet, and you're going to have disagreements about what's acceptable and what's not. This is just sort of how you build products in general. People have different preferences depending on what part of the country they're from, what part of the world. Some places one thing's acceptable and another it isn't. Part of the thing that we're learning right now by launching all of this stuff is where those lines of preference are and how we can quickly conform the product to give people the thing that they want.

Peter Kafka: If you do any amount of browsing about AI, you inevitably get people talking about doomsday scenarios.

Kevin Scott: Yeah.

Peter Kafka: Part science fiction and part a thing we ought to be thinking about. I'm sure you've seen the survey of AI experts from last year. Depending on how you phrase the question, it says something like, "What percentage chance do you think AI has of wiping out humanity?" Depending on how you phrase the question, it's 5% to 10% to 14%. Those seem like unacceptable odds to me. What am I missing and how do you think about playing with tech that, by definition, you don't really know where it's going to go and you don't know what its full capacity is going to be?

Kevin Scott: It is absolutely useful to be thinking about these scenarios. It's more useful to think about them grounded in where the technology actually is and what the next step is and the step beyond that. I think we're still many steps away from the thing that people are worried about. You would have to deliberately, as the developer of these platforms, go seek out some of this stuff. There are people who disagree with me even on that assertion. They think that there's going to be some uncontrollable emergent behavior that happens. We're careful enough about that where we have research teams thinking about the possibility of these emergent scenarios. But the thing that you would really have to have in order for some of the weird things that people are concerned about to happen is real autonomy. A system that could participate in its own development, have that feedback loop where you could get some superhuman fast rate of improvement. That's not the way the systems work right now, not the ones that we're building at least.

Peter Kafka: So you're assuring me that you have things under control. I'm going to take your word for it.

Kevin Scott: We do. And look, you shouldn't... This is one of the things. We're going to figure out over the next handful of years where governments and society want to be able to look over the shoulders of the people developing this technology to make sure that everything is being done responsibly. I think that's a reasonable thing for societies and governments to ask for.

Peter Kafka: I don't feel particularly confident that government's going to be particularly good about controlling tech. There's the sort of "lock it down." If we look at how governments around the world, including the U.S., are dealing with just social media tech, which is pretty simple all things considered, they seem kind of baffled by it. They can't shut it down or they throw their hands up. And this is much more complicated.

Kevin Scott: I think it is and it isn't. The interesting thing about social networks is they get right to this very complicated thing about what happens when very large numbers of people are interacting with one another in a technology facilitated way. Maybe you could argue that that technology is not as complicated as these AI things, but that dynamic is maybe even more complicated than what we're talking about here.

Peter Kafka: Let me ground things in the here and now. What's the business model for Microsoft with AI? Who's paying for this? Because as we all learned over the years, even when you're using a free service, someone's paying for it on one end, and that shapes the way the product is made.

Kevin Scott: Just like all of these other technology shifts that we've seen over the past 50 years, we will figure the business model out as the technology matures. The obvious ways that we're paying for it right now are sort of three-way. For Bing, it gets paid for with advertisements. The ad model itself is likely going to change over time, but right now it's a search engine, it's free and it's ad supported. For things like ChatGPT, it's sort of a freemium model. You have a free tier that you can sign up for that gets you one level of service, then you can pay for ChatGPT Pro, which is a subscription product that gets you another level of service. And then there is the integration with existing products like Microsoft 365 that has a pricing model. There may be some different way to price it than an E5 subscription or whatnot, which is what we do right now for big businesses.

Peter Kafka: You've been very responsible in this conversation. You have not been promising me the world. On the other hand, you've been saying this is as big a deal as major technological revolutions you've lived through in your life. What are you most excited about when you're playing with this stuff?

Kevin Scott: I'll give you a couple of examples. My brother, who lives in rural central Virginia with my 74-year-old mother, is immune compromised and got COVID last fall for the first time. In rural central Virginia, he doesn't have access to the same quality of health care that I have access to here in Silicon Valley. The responses that he got from his doctor about what he should do after having tested COVID positive just weren't great. I just sort of think about the possibility for these AI systems to help people get access to high quality medical care, high quality educational resources, high quality of a bunch of things that you think of as skilled guidance that you get from people. I think these systems can make those sorts of things accessible to folks for whom it's not accessible right now. I think that's a really exciting thing to me.

Peter Kafka: That's a good place to maybe wrap this up because it's one of the big questions and debates that I've found here. That scenario you've described: you hear people who are excited about AI saying we could extend knowledge and resources to people who wouldn't have it normally, whether it's medical or legal, all kinds of stuff that you had to be rich or live in a certain place or know someone to get. And we can extend that out. And then other people say you're describing a dystopian scenario where the rich people get real human doctors and everyone else has to work through a bot. That sounds terrible. So you've sort of described the upside of your vision of it, but help convince me and others that this is a better solution than going to my GP and asking him for Paxlovid.

Kevin Scott: I think it just sort of depends on what the actual delivery mechanism is. What I would say is you absolutely don't want a world where all you have is some substandard piece of software and no access to a real doctor. I have a concierge doctor, for instance, and I interact with my concierge doctor mostly by email. That's actually a great user experience. It's phenomenal. It saves me so much time, and I'm able to get access to a whole bunch of things that just my busy schedule wouldn't let me have access to otherwise. I have thought for years, wouldn't it be fantastic for everyone to have the same thing? An expert medical guru that you can go to that can help you navigate a very complicated system of insurance companies and medical providers. Sure, you want to be able to get access to human beings at the right point. But having something that can help you deal with the complexity of a very complex system, I think, is a good thing.

Peter Kafka: When the first iPhone came out, and this was my last question, there were people lined up to buy it. I would see these people in New York. They'd walk around showing the iPhone off at parties. People would go, "Oh my God, let me check out that iPhone," and pinching and zooming. It was really great. X number of years now, no one ever looks at you and wonders when you pull a supercomputer out of your pants pocket. Is that the track we're on here for AI? Where it becomes embedded and we don't think about it and it's fully part of our lives and we take it for granted?

Host (Kara Swisher or Scott Galloway): Well, look, I think it's either going to be that or it's going to be useless. So like every major transformative technology thing in human history gets past this sort of hump of adoption where there's lots of skepticism and people don't really know what it's good for, and it enters the domain of ubiquitous. So either this is one of those major platform things, in which case at some point it will be absolutely ubiquitous and we will completely take it for granted, or it just isn't. And folks who are excited about it are in some kind of crazy hype cycle and it's going to flame out. We will see which one it is over the coming years.

[Music fades in and out]

Host: In a moment, we're going to hear from someone who says Kevin Scott and most of the other AI optimists are wrong.

[Ad break for LinkedIn]

Host: And now, an anti-hype. My next guest believes in AI, but not the way it's going right now. His name is Gary Marcus, and he's a cognitive scientist. He's the author of *Rebooting AI*, and he's got a new podcast coming out called *Humans Versus Machines*. Marcus believes the things that companies like Microsoft and OpenAI are building aren't real artificial intelligence. But what they are is deeply flawed tech.

Gary Marcus: You know, a lot of us dream of artificial general intelligence, a system that you could throw any problem at, kind of like the *Star Trek* computer. You say, solve this for me. The *Star Trek* computer could play chess, but it could also answer questions about dilithium crystals or whatever it is they needed at that time. So GPT-4 gives the illusion of doing that, but it's not actually that great at it. It can talk about anything, but it's not reliable. Some people say it lies. Literally, it doesn't lie because it has no intent, but it makes stuff up fairly regularly. And so you can't really trust its output. For GPT-4, you have to use just the right prompt. If your prompt is slightly off, then you get a weird answer. This is not something that happened with the *Star Trek* computer. I sort of started having this fantasy, like the Enterprise is about to crash and Captain Kirk says to Scotty, we need to fix this problem. And Scotty is like, well, I just can't get the right prompt to work. And then maybe he gets the right prompt or maybe he gets the wrong prompt and the Enterprise crashes. We don't have reliability here with these systems. They're a hint at what artificial general intelligence would look like. They don't work that well.

Host: Even if this is not the supercomputer that we're promised in science fiction, there are a lot of pretty rational people, some of whom have a vested interest in this, but others who don't, who say this seems like a really big moment. We just heard from Kevin Scott, the CTO of Microsoft. He does have a vested interest in this. And he described this moment as being something akin to the development of the personal computer, the development of the Internet, the development of the iPhone. Maybe all three of those bundled together. And he is not promising that this is a transporter beam or it's going to manufacture food for us. But he does think that it's a really big deal. Is there a sliding scale where you can say, look, this is not science fiction fantasy, but it is a big deal. It is interesting and it is going to help us in some way. Is that more reasonable for you?

Gary Marcus: I mean, it's absolutely interesting. I would not want to argue against that for a moment. I think of it as a dress rehearsal for artificial general intelligence, which we will get to someday. But I think right now we have a tradeoff. There's some positives about these systems. You can use them to write things for you. And there's some negatives. This technology can be used, for example, to spread misinformation. And to do that at a scale that we've never seen before, which may be dangerous, might undermine democracy. And I would say that these systems aren't very well controllable. I think of them as like bulls in a china shop. They're powerful, they're reckless, but they don't necessarily do what we want. Ultimately, there's going to be a question kind of akin to the driverless cars, which is, okay, we can build a demo here. Can we build a product that we can actually use? And what is that product? I think in some places, people will adopt this stuff. They'll be perfectly happy with the output. In other places, there's a real problem. So in search engines, for example, people are trying to replace traditional search engines with what you might call chat search, like Bing. And it kind of works and it kind of doesn't. The most recent version that I saw is Bard. Somebody put in my name and had it write a biography. What Bard did, and we've seen this with Bing, is it confabulated. So it got the subtitle of my book wrong. It fabricated two quotes. It said that I made arguments there that I didn't because the book was published in 2019 before large language models were popular. And so it kind of gave a flavor of a Gary Marcus argument, but it got all the details wrong. I don't really want my search engine to do that. When I use a search engine, like when I'm writing a book, I want the facts, not a flavor of the facts.

Host: We've had this discussion about driverless cars in the past, and you've got people like Elon Musk saying that the driverless Tesla is going to hit something occasionally, and maybe someone might even die. But people die in cars all the time. Human drivers are terrible. The computer is going to be better than X percent of them, and that's a worthwhile tradeoff. Most people aren't comfortable with that analogy. But I'm also thinking of earlier versions of technology being directionally correct, but not right at the beginning. And people are comfortable with that, and it gets better. I remember with the early Wikipedia, when that was brought up as an idea that it could replace an encyclopedia, and that seemed ridiculous. And the initial Wikipedia results were often very bad because they're biased, etc. And now Wikipedia is still flawed in lots of ways, but is generally a pretty good resource. Are we on that kind of trajectory with this kind of tech?

Gary Marcus: I think we might be, but I think it's a critical question of time and cost. So costs come in different dimensions. With driverless cars, I warned in 2016 that there was an outlier problem and that it might not be practical to actually use them anytime soon. And not long after that, we were told, hey, we're going to have a driverless car go from Los Angeles to New York. Elon Musk said that in 2016, promising it in 2017, if I recall correctly. And we've gotten these promises pretty much every year since, that they're imminent. And the reality is they're still demos. They're not really products. So yes, you can take some rides in San Francisco, for example, but it has to be at certain hours and certain roads. You can't take it wherever you want. And so it's been many years of promise, not really realized. We could wind up in the same place with chat search. People are certainly running demos right now, but whether we can count on it, how long it's going to take for us to count on it, I don't know. I think driverless cars will eventually replace humans. I think we will reach a point where driverless cars are in fact better than people, but I don't think we're particularly close to that. It might be another decade or two. And so a question is whether chat search is more like driverless cars, where we see a taste of it, but takes a long time to deliver, or are they more like Wikipedia, which I think is actually pretty practical right now?

Host: If we say, look, we don't want the AI to drive us from Los Angeles to New York or really anywhere, because we're just not comfortable with it being wrong. But if it gives you search answers that are usually correct or mostly correct, and you have to do a little bit of extra work to make sure it's entirely correct, you're not going to kill anyone if this gets wrong.

Gary Marcus: Well, if it's medical misinformation, you might actually kill someone. And that's actually the domain where I'm most worried about erroneous information from search engines. People do search for medical stuff all the time. And these systems are not going to understand drug interactions. They're probably not going to understand particular people's circumstances. And I suspect that there will actually be some pretty bad advice. Now, they may try to lock these systems down so they don't give medical advice, but then people may go back to Google. And so it might not be Microsoft's great desire to do that. There are a lot of commercial questions and user questions we don't really understand. We understand from a technical perspective why these systems hallucinate, and I can tell you that they will hallucinate in the medical domain. Then the question is, what becomes of that? What's the cost of error? How widespread is that? How do users respond? We don't know all those answers yet.

Host: I want to ask you to define some stuff. Let's start with hallucinate. It means something kind of akin to what we think of when it comes to AI, but something a little bit different. What does that mean?

Gary Marcus: What we mean by that is that these systems, large language models, have a tendency to make stuff up. They don't have a built-in system for validating things, although people are trying to put extra machinery on top. But inherently, they don't actually track facts about the world. They don't build mental models of the world. They don't build mental models of medicine, of physics, etc. They really are autocomplete on steroids. They predict text, what might come next in this circumstance. And they do that over clusters of words. So they're like, this cluster of words predicts this other cluster of words. It's not even conceptual. It's really relationships between words. And so you get really weird things coming out of it. If you actually look as recently as November of 2022, Facebook had a system or Meta had a system called Galactica, and it made up some really wild stuff. It, for example, said on March 18th of 2018, Elon Musk died in a car crash. And he didn't actually die in a car crash. It's not actually in the data that the system was trained on. There's lots of data that contradicts that. So that's what we call a hallucination. It makes stuff that is plausible, but it doesn't know the difference between things that are true and not. And it takes months to train these things. GPT-3 was trained only on 2021. It didn't know what happened in 2022. Didn't know that Elon Musk was the CEO of Twitter, for example. And I got mansplained by, I think it was ChatGPT, told me that Twitter was a public company. It wasn't private. Elon Musk wasn't the CEO. Basically said that I made all this stuff up, which I hadn't.

Host: It sort of strikes me as, and I've been in this position in the past, where you're presenting a talk or writing a paper and you use lots of words that have come up in class and you think they sound right. And you're hoping that no one's going to pay too close of attention and it might get you through the quiz.

Gary Marcus: Yeah, I mean, GPT-3 is a lot like that. It doesn't have the intention. It's not trying to bluff you. It's not trying to please you, but it is at some level bluffing because it doesn't really understand what it's talking about. I mean, that's kind of being anthropomorphic about it. Really, it's just predicting words.

Host: You wrote an essay in the spring of 2022, about a year ago, called *Deep Learning is Hitting a Wall*. Can you tell us what deep learning is and why you thought it was hitting a wall?

Gary Marcus: Deep learning is the major technique in machine learning, which is part of artificial intelligence, that's getting used all the time. It underlies large language models. We use these things for image recognition, in image synthesis, in these chatbots now. They are very widespread. And what I said in this essay, which was in March of 2022, was that they were hitting a wall. I didn't say there's not any progress anymore, but I said they have particular problems. I said that they're not reliable, they're not truthful, they confabulate things, and they have a tendency to produce misinformation. And people got really, really upset that I should dare to say that their baby was hitting a wall. There was a whole satirical campaign almost of memes to try to attack me. And what's amazing to me about it is that it didn't take that long before a lot of people changed their views. Sam Altman, he's the CEO of OpenAI, he said, "Give me the confidence of the mediocre deep learning skeptic." And then in December, he put out his own tweet that said, "These systems still have a lot of work to do. We can't really trust them. They have problems with truth, with reliability." Exactly the same things that I had said in that article. So I think in a way the field has grown up, and in a way, there's something deeply problematic about a field that so tries to silence its critics that there's this relentless campaign against them. And it's also a sign that if you're going to do science, you have to actually listen to people who have opposing views. If you try to silence them, you're not going to get to doing good science faster.

Host: You published that paper in March 22. In the fall of that year, there's this wave of excitement around AI, largely from what I can tell, because it was now something that regular people could see in the case of images like DALL-E or image generators like DALL-E and then ChatGPT that you could play with without having to know how to use anything other than how to turn a browser on. Was there any technological change that also spurred that on? Or is that the same conditions that you were talking about in March, and it's just the fact that it's now accessible to people that has generated that excitement?

Gary Marcus: There are a few things. The technology in it is not that different from its predecessors, but they added these guardrails. You could still jailbreak it and make it say really terrible things without that much trouble. You could use it to generate misinformation. But they had these guardrails that were good enough, even though they weren't great, that people could just try it out. And that made a big difference. Even OpenAI didn't expect it to go viral the way that it did, but it did, in part because it was so much fun to play with.

Host: You're saying that the folks at OpenAI took tech that was basically pre-existing and that they already had, made it more accessible to humans, added software to make it less prone to error, and people got very excited about it. This seems like the good version of tech progress. Where you take a thing and you modify it and it becomes better and useful. They made it a better product.

Gary Marcus: I mean, so Yann LeCun has pointed out and other people, also Yoshua Bengio, these are two of the pioneers in deep learning, have pointed out, I think rightly, that it wasn't big technical progress. It was good product progress. So it wasn't some new form of AI that we didn't have before. But yes, it was a better product. It was easier to use. And some of it was just like making it cute. Like it types out words one by one instead of giving you the whole thing. Somehow that makes it feel more human. So from a product side, it was definitely a better product than we had before. From a technical side, those of us who are interested in artificial general intelligence and more generally reliable and trustworthy artificial intelligence are like, yeah, we still have the same problems that we did before. It's just more fun to play with. So there's a tension between what do people actually want to use, and it's better at that, and on the other hand, how do we make this reliable so that if you type in your medical questions, you can actually trust the answer?

Host: I have a broad and shallow view of technological progress, but it essentially goes like things get better over time. You have unexpected results. You don't always know where it's going to go. But generally, things get better, faster, cheaper, smarter. We know about Moore's Law and chips get used to get faster all the time. And that sort of slowed down. And what you hear from AI proponents is these neural nets and other things that are the engines for this are getting better all the time. We're adding more data. We're computing more. We're learning more. Why won't that work here?

Gary Marcus: They are not, is the first thing I'll say. And then I'd like to give you a historical parallel. On misinformation, for example, they're not really getting better. On truth, they're not really getting better. A couple days ago, NewsGuard released something showing that GPT-4 was actually worse than GPT-3.5 at discriminating truth. So they are getting better at being more plausible. They're not getting better at being more truthful. And that really matters a lot. So that's the first thing I would say. The second thing is there are a lot of interesting historical cases where people got obsessed about a particular technology and chose wrongly. And it took a little while to back down from the popular thing to the thing we needed to do. Dirigibles were really popular in the 1920s and 1930s. Until we had the Hindenburg, everybody thought that all these people doing heavier than air flight were just wasting their time. They were like, look at our dirigibles. They scale a lot faster. We built a small one, then we built a bigger one, then we built a much bigger one. It's all working great. And those poor guys with the heavier than airplanes are still stuck, barely bigger than the Wright brothers. So sometimes you scale the wrong thing. In my view, we're scaling the wrong thing right now. We're scaling a technology that is inherently unstable, unreliable, and untruthful. We're making it faster, has more coverage, but is still unreliable, still not truthful. And for many applications, that's a problem. There are some fruits it's not. GPT's sweet spot has always been making surrealist prose. It is now better at making surrealist prose than it was before. If that's your use case, it's fine. I have no problem with it. But if your use case is something where there's a cost of error, where you do need to be truthful and trustworthy, then that is a problem.

Host: I think the way a lot of people look at it is, yes, they're doing surrealist prose today. And tomorrow, or six months or a year from now, we can't project when, it's going to get much better at much more sophisticated stuff.

Gary Marcus: Well, I mean, this is where I'd say people have been telling me that for 30 years. I first pointed out the hallucination problem in 2001, and so far nobody has solved it. Even though you can scale some things, you're not going to scale truth. And in fact, the NewsGuard result shows that they're not scaling truth. So to me, it feels like an appeal to hopeful monsters. And let me pause here. It's really important to distinguish between what we can do with large language models and what AI might someday be able to do. I am not making an argument that we will never be able to build an artificial intelligence system that we can trust, that is reliable, that can keep track of truth. I'm saying this is the wrong technology for that problem. And until we broaden our search through the space of hypotheses, we're a little bit stuck.

Host: This is not artificial general intelligence. That's something you remain interested in and think could arrive one day. But we have this technology. It is flawed in lots of ways. Let's stipulate that. It still seems like it ought to be useful for lots of present-day applications. Are there things that you are interested in using the current version of this technology for?

Gary Marcus: I mean, me personally, no. Other people have to write much more boilerplate stuff than I do, and for them it might make some sense. I would warn them about accuracy and things slipping through because our human tendency is to take things that are grammatical and well-formed and assume they're okay. You may have seen how CNET got themselves in trouble. They published 70 articles written by machine. It turned out 40 of them had mistakes in them. It's very hard for people to pay enough attention. We have a similar problem with driverless cars, which is a human sits in a driverless car for an hour and thinks, all right, this is fantastic, and doesn't realize that the machine doesn't think like they do. And the machine, for example, could run into a jet airplane because it's not in the training set. And so these systems are not as reliable as they might seem in the first few minutes that we play with them. Fundamentally, the outputs just are not 100% reliable. And so you have to decide, is that okay for my task?

Host: Do you think this technology ends up replacing people, replacing workers, because it can do it better, faster, cheaper, more efficiently, or it is a tool that people who know how to use this work use in their day-to-day?

Gary Marcus: I suspect it'll be some of both. We don't really know for sure yet. We may find that where you have 10 people on a job, now you can do it with two, and that the two oversee the systems and make sure there aren't mistakes. We may see a bunch of that. It's a little bit too early to say for sure. There are other things like the generative systems that create images. And there are limits there. So it may not always give you exactly what you want. It may be hard to edit to get the thing that you want. But if you just need a rough thing, let's say for an illustration for a talk, they work pretty well. And so I think they're going to cut into the jobs for commercial artists by a lot. So I do expect this stuff will have impact on employment. I think it's as yet a little bit hazy about what that impact is going to be. It may be small. It may be dramatic. I think it's going to matter domain by domain. Another place where we've kind of seen this movie before but might have a different ending was with radiology. In 2016, Geoff Hinton said that you might as well stop training radiologists. That's almost a verbatim quote. And here we are in 2023. We don't have enough radiologists because it turns out that the job of radiology is not just to look at an image, which deep learning is pretty good at, but also to look at a patient's file and understand the relation between that image and a person as a whole and diseases. And you need to know about anatomy and physiology. And it's going to depend in every profession. Can you do a certain piece of it? If you can do a certain piece of it, does that mean maybe we can reduce the staff by 10%? There are a whole lot of questions that really depend on the details of what the job is and the cost of error and all kinds of different things. I don't know if there's a blanket answer there. I think there's going to be a lot of discovery process in the next few months.

Host: Is there a rubric that you can share with us that's useful for an interested layperson to sort of say, all right, this person is telling you about this exciting new AI-inflected thing they're rolling out. How could a regular person who has no capacity to look into the code understand whether the thing they're getting is actual AI, whether it's fake AI, or it's just hype entirely?

Gary Marcus: How you define AI is actually up for grabs. There's no fact of the matter in the universe independent of people. We use the terms as we do. There is no artificial general intelligence right now. That just doesn't exist yet. We have AI for certain problems that works great. So route planning, use your GPS to go from A to B, works great. That's AI. It's not large language models, not very sexy anymore because we've habituated to it. But it's actually kind of amazing that that can be done on a stick on the phone. Chatbots are real, they're just not reliable. They can talk about anything, but they're often out of their depth. So you give them a math problem and sometimes they get it right and sometimes they get it wrong. You give them almost anything in any domain and sometimes they get it right, sometimes they make up stuff. Most of the things we're talking about right now are based on that. And if you want to ask, am I in trouble because of that, you have to ask questions like, how well can it do the thing that you do in your job? How much creativity do you need to do in your job? How systematic is it? How much does it cost if you make a mistake? Who's going to catch the mistakes if the AI does it? Is that even feasible? There are a whole bunch of different questions that you might ask around that. How often do you come up with some unusual case that a system that only kind of knows the usual cases might fail on? And it's going to be different for every profession.

Host: Gary, this has been great. Thank you for your time.

Gary Marcus: Thanks a lot.

Host: Thanks again to Joshua Browder, Kevin Scott, and Gary Marcus. Next week, AI has a lot of critics, but there are a lot more people in Silicon Valley happy to ignore them because they want to cash in. So next up on our AI tour, we're going inside the gold rush. This is a special series, so we got extra help. None of them are robots. They're all humans. They're excellent humans. Thanks to Matt Frasica for producing this episode. Megan Kunane is our editor. Jelani Carter is our engineer. And Brandon McFarland scored and mixed this episode. We got brand new human-made music for this. We'll see you next week.