--:-- [Theme music fades out]
--:-- Sean Rameswaram: Welcome to Today Explained. I'm Sean Rameswaram, and I'm joined by Josh Dzieza from The Verge, who just wrote a big piece about the people behind artificial intelligence.
--:-- Josh Dzieza: Thanks for having me on the show.
--:-- Sean Rameswaram: We're used to thinking of artificial intelligence as knowledge generated by machines. You can get ChatGPT to write an email for you. You can ask mid-journey what Pope Francis would look like in a puffer jacket. But it turns out there's a vast network of human labor powering AI.
--:-- Josh Dzieza: That's right. There are people training AI every day, sometimes all day, just clicking away on images, on pixels, so that the AI can get better at identifying things the way we humans do.
--:-- Sean Rameswaram: We're going inside the AI factory on Today Explained. Support for this show comes from ServiceNow. AI is only as powerful as the platform it's built into, don't you know? That's why it's no surprise that more than 85% of the Fortune 500 use the ServiceNow AI platform.
--:-- Josh Dzieza: While other platforms duct tape tools together, ServiceNow seamlessly unifies people, data, workflows, and AI, connecting every corner of your business. And with AI agents working together autonomously, anyone in any department can focus on the work that matters most.
--:-- Sean Rameswaram: I've never met an AI agent. Learn how ServiceNow puts AI to work for people at servicenow.com. Support for this show comes from Macy's. Layering is an art form. All it takes is a bit of intention, style, and of course, the perfect cold weather staples.
--:-- Josh Dzieza: So find the perfect look for you with the help of Macy's Fall Fashion Guide, your one-stop shop for fall's latest trends and must-haves, curated by their experts. From leather to denim to cozy knits, find everything you need to piece together your new fall look.
--:-- Sean Rameswaram: You are listening to Today Explained. I'm Sean Rameswaram, and I'm joined by Josh Dzieza from The Verge, who just wrote a big piece about the people behind artificial intelligence.
--:-- Josh Dzieza: It is about the human labor behind artificial intelligence. You know, it's often said that AI learns from data, finds patterns in data, but that data has to be curated, sorted, labeled, sometimes made by humans.
--:-- Sean Rameswaram: So I wrote about those humans. It's often called data annotation, sometimes data labeling. The work is pretty weird, and there's a huge range in what you might be doing.
--:-- Josh Dzieza: Let's say you log on to your platform, and you might be labeling clothes in social media photos. You might be sorting TikTok videos based on whether they're fast-paced or slow-paced or something. Or you might be labeling food and saying, yes, that's Diet Coke.
--:-- Sean Rameswaram: Or you might be looking at chatbot responses and saying, you know, this is incorrect or this is profane or, you know, too long or totally off the wall.
--:-- Josh Dzieza: So there's a huge range in the types of jobs you might be doing. What they have in common is they tend to be sort of small. Like there's one thing you're doing over and over and over and also have extremely high quality standards.
--:-- Sean Rameswaram: Like, let's say you are outlining vehicles or something like that. You have to outline it to the pixel.
--:-- Josh Dzieza: Is this like the kind of thing that I do when I'm trying to log into a website? And it's like, how many of these pictures have cars in them?
--:-- Sean Rameswaram: Exactly. It's a lot like Captcha. That was actually a method and still is a method of kind of getting this work for free. You know, by definition, it's something I can't do yet.
--:-- Josh Dzieza: So when I do a captcha, I'm helping the back end of some website trade AI?
--:-- Sean Rameswaram: Exactly. And if you may, you may have noticed over the years that captchas have gotten harder. That's because the AI has gotten better.
--:-- Josh Dzieza: So you need blurrier, weirder images to sort of raise the bar and also to improve AI.
--:-- Sean Rameswaram: So in a way, you and I and all of us are AI annotators.
--:-- Josh Dzieza: Yes. Yeah. And annotators are just people who do it, you know, full time for pay.
--:-- Sean Rameswaram: Did you see people doing this kind of work?
--:-- Josh Dzieza: So I did this kind of work. You did it yourself?
--:-- Sean Rameswaram: Yes, I did it myself as a way to meet people who are doing this kind of work. It's all online for the most part.
--:-- Josh Dzieza: So did you like apply for a job? Did you cheat on the verge?
--:-- Sean Rameswaram: I made all of $1.50, I think. But yes, it was my second job for a couple months.
--:-- Josh Dzieza: But the application is very easy. You just have to speak English and, you know, have an email address. You fill out some basic information and then you get a welcome email and you're invited into a Slack channel.
--:-- Sean Rameswaram: And then you have to start training to actually work. You have to learn what data annotation is and then do kind of a training module for each task. It's sort of a video game.
--:-- Josh Dzieza: So these courses, they're like instructions. So you have to read them carefully and understand each and every bit of each. And those instructions can come with scenarios and they can come with some questions or quizzes.
--:-- Sean Rameswaram: A project has, let's say, three or four courses. You have to start the first one, you finish it, go to the next one, and so forth.
--:-- Josh Dzieza: Now you can start working on this thing for money. Give me like a day. What was your shift like?
--:-- Sean Rameswaram: So it was extremely difficult. You know, I thought I was going to kind of log in and see what kind of jobs were out there and, you know, get invited to these channels and move past this fairly quickly.
--:-- Josh Dzieza: But I kept flunking the training for the first task I would try to do. I can give you an example.
--:-- Sean Rameswaram: Like one of the early ones I was doing was just labeling clothing. And the instructions were something like, label the items of clothing that are real clothes that can be worn by real people or something like that.
--:-- Josh Dzieza: It was just like seemingly quite self-explanatory. So I just sort of clicked proceed past the instructions and got started and failed immediately.
--:-- Sean Rameswaram: One of the things that took me up at first was like, there was a magazine that had some photos of clothes in it. It like well that not you know you can wear a magazine.
--:-- Josh Dzieza: But like to an AI these systems are really literal. They not very smart. And so it all just pixels. It doesn't understand what a magazine is or what a reflection is.
--:-- Sean Rameswaram: And so you need to label images of clothes and reflections of clothes in mirrors and things like that.
--:-- Josh Dzieza: And so that was sort of the first curveball. But then it just goes on from there. It's like label costumes, but not suits of armor.
--:-- Sean Rameswaram: And where you draw that line is the difference between, you know, having a job and getting fired.
--:-- Josh Dzieza: These sorts of weird distinctions that get drawn. The full instructions were over 40 pages, and you have to kind of keep referring back to those as you do your work.
--:-- Sean Rameswaram: You talk about failing. Do you still get paid if you fail?
--:-- Josh Dzieza: No, I mean, you'll get paid for the tasks that you completed, but then, you know, you just get booted out.
--:-- Sean Rameswaram: It says your low quality has, you know, gotten you suspended from this task, and you have to go back and start training again on some new thing and try to qualify.
--:-- Josh Dzieza: Wow. So it's really in your interest to read the instructions, it sounds like.
--:-- Sean Rameswaram: Yeah. And workers, I found, because the instructions are not well-written. They're just inhumanly complex.
--:-- Josh Dzieza: And so they end up teaching each other, doing a lot of free labor, honestly, doing YouTube tutorials or Google Meets where they try to teach each other what these instructions actually mean.
--:-- Sean Rameswaram: Is it steady work, Josh? Do you get as many tasks as you want? Is it like dependable income?
--:-- Josh Dzieza: No. So this is one of the things that surprised me. I mean, it's obviously unsteady at the level of like, if you don't read the instructions really carefully and you do something wrong, you're going to get banned.
--:-- Sean Rameswaram: And so that is very precarious. But it's also just unsteady. Even if you're the best annotator in the world, there's like a really spiky demand for this sort of work.
--:-- Josh Dzieza: You know, there'll be a period where there's a bunch of well-paying tasks on there and you can work as much as you want, and then they'll disappear and you don't know why.
--:-- Sean Rameswaram: And you have no work or you can only do tasks for a penny or something like that.
--:-- Josh Dzieza: And then they'll come back. So I spoke to a lot of people and this was, people were frustrated at the low pay.
--:-- Sean Rameswaram: Even more than that, people were frustrated that it's like steady enough that you can almost depend on it, but not enough that you aren't constantly without work.
--:-- Josh Dzieza: And so they would, you know, I talked to people who developed these habits of waking up every three hours in case something well-paying appeared.
--:-- Sean Rameswaram: And then if there was staying up for 36 hours straight, just sort of labeling.
--:-- Josh Dzieza: I talked to one guy who was just labeling elbows and knees. He didn't know why, but it was paying well.
--:-- Sean Rameswaram: And he just wanted to do it while it lasted because then you might be out of work for a week.
--:-- Josh Dzieza: Elbows and knees? Yeah. There's a lot of stuff on there that you just have no idea what it's for.
--:-- Sean Rameswaram: And that was one of them where it was just like photos of crowds. And it was like, label all the elbows and knees.
--:-- Josh Dzieza: So, okay, so you're just sitting there labeling elbows and knees for 36 hours straight for how much money?
--:-- Sean Rameswaram: It's super variable. Each task pays some amount of money. But the workers I talked to, they were getting paid for something like that, like a couple bucks an hour, as low as $1 an hour.
--:-- Josh Dzieza: You cannot pay all the bills. It's a side hustle. Maybe just one bill, maybe it's for the internet bill, and then that's it.
--:-- Sean Rameswaram: And do they have any idea why they are labeling elbows and knees for a dollar an hour, potentially 36 hours straight?
--:-- Josh Dzieza: No. Well, they know they're training AI, and they know it's for some company, but they don't know who's AI or what they're training it to do unless they can kind of guess that it's a self-driving car or something.
--:-- Sean Rameswaram: But the elbows and knees, no, they don't know because there's just layers and layers of anonymity in the system.
--:-- Josh Dzieza: So each project, all they know about the platform is that it's called Remote Tasks.
--:-- Sean Rameswaram: And then each project is named something totally cryptic, like Pillbox Bratwurst or something, just like non sequitur code names.
--:-- Josh Dzieza: And so they have no idea what it's for, really.
--:-- Sean Rameswaram: In a minute, on Today Explained, what and who all this labeling is for, really.
--:-- Josh Dzieza: Support for today's show comes from Adio. Adio is an AI native customer relationship management system built specifically for the next era of companies.
--:-- Sean Rameswaram: They say it's extremely powerful, adapts to your unique data structures and scales with any business model.
--:-- Josh Dzieza: According to Adio, Adio takes less than a minute to set up. And within seconds, you will have your email and calendars synced.
--:-- Sean Rameswaram: You'll see all your relationships in a fully fledged platform, they say, all enriched with that actionable data.
--:-- Josh Dzieza: And they say you can build AI-powered automations and use its research agent to tackle some of your most complex processes to seize so that you can focus on what matters, building your company.
--:-- Sean Rameswaram: You can join industry leaders such as Flatfile, Replicate, Modal, and so much more. You can go to atio.com slash todayexplained to get 15% off your first year.
--:-- Josh Dzieza: That's A-T-T-I-O dot com slash todayexplained.
--:-- Sean Rameswaram: Let's be honest. Are you happy with your job? Like, really happy?
--:-- Josh Dzieza: The unfortunate fact is that a huge number of people can't say yes to that.
--:-- Sean Rameswaram: Far too many of us are stuck in a job we've outgrown, or one we never wanted in the first place.
--:-- Josh Dzieza: But still, we stick it out, And we give reasons like, what if the next move is even worse?
--:-- Sean Rameswaram: I've already put years into this place.
--:-- Josh Dzieza: And maybe the most common one, isn't everyone kind of miserable at work?
--:-- Sean Rameswaram: But there's a difference between reasons for staying and excuses for not leaving.
--:-- Josh Dzieza: It's time to get unstuck. It's time for Strawberry.me. They match you with a certified career coach who helps you go from where you are to where you actually want to be.
--:-- Sean Rameswaram: Your coach helps you get clear on your goals, create a plan, build your confidence, and keeps you accountable along the way.
--:-- Josh Dzieza: So don't leave your career to chance. Take action and own your future with a professional coach in your corner Go to strawberry slash unstuck to claim a special offer
--:-- Sean Rameswaram: That strawberry slash unstuck
--:-- Josh Dzieza: Why did the elbow cross the road Let me tell you it's quite humorous.
--:-- Sean Rameswaram: Today Explained, we are back with Josh Dzieza. Josh, you just told us that these people who are slaving away, training AI 36 hours straight, a dollar an hour, whatever it is they don't know exactly what they are doing it for.
--:-- Josh Dzieza: So AI needs tons of examples to learn from. And so autonomous vehicles is a great example of something where this thing is out in the world steering around a multi-ton piece of metal.
--:-- Sean Rameswaram: The stakes are really high. You can't have it get confused. It's super dangerous.
--:-- Josh Dzieza: There was a case a couple years ago where an Uber self-driving car killed a woman in Arizona.
--:-- Sean Rameswaram: It could recognize pedestrians, it could recognize bikes, but it struggled to figure out what was happening with a person walking a bike along a street, not near a crosswalk.
--:-- Josh Dzieza: It didn't have enough data on it. And so the demand for data for self-driving cars is super high.
--:-- Sean Rameswaram: If you think about how many times you're driving and you go past construction or just something unexpected happens, you need to have data on it.
--:-- Josh Dzieza: So there's thousands and thousands of people whose job it is to get data from these cars and go through and say, you know, here's a pedestrian, here's a traffic cone, here's a pothole.
--:-- Sean Rameswaram: That's basically how it works with any machine learning system. You know, whether it's language or image recognition, you need training data and you need someone to make sure it's the right training data and to put tags on it and provide that human input.
--:-- Josh Dzieza: And where are these data annotators based typically? They're all over the world because you need so much of this data.
--:-- Sean Rameswaram: The pay tends to be fairly low. And so you have a lot of people in India, the Philippines. Kenya is a big hub. Venezuela, because you often get paid in U.S. dollars.
--:-- Josh Dzieza: And so if there's a place where the currency is crashing and people can do the work and there's fast internet, the work tends to go there.
--:-- Sean Rameswaram: How long have we been outsourcing our data training?
--:-- Josh Dzieza: It's been at least a decade, you know, probably more.
--:-- Sean Rameswaram: One of the turning points happened in kind of the late 2000s.
--:-- Josh Dzieza: You know, you've always needed some form of data curation, but before that it was often done by a researcher and their grad students or something.
--:-- Sean Rameswaram: But with increasing computational power, it became possible to train on more data.
--:-- Josh Dzieza: And so in the late 2000s, you have people start to use labeled data sets of millions of images instead of, you know, a couple of thousands.
--:-- Sean Rameswaram: We downloaded nearly a billion images and used the crowdsourcing technology like Amazon Mechanical Turk Platform to help us to label these images.
--:-- Josh Dzieza: When you reach that scale, people start going overseas because you need people who will work for less.
--:-- Sean Rameswaram: Together, almost 50,000 workers from 167 countries around the world helped us to clean, sort, and label nearly a billion candidate images.
--:-- Josh Dzieza: Will the need for these data annotators eventually dry up? Is this job sort of a finite experiment?
--:-- Sean Rameswaram: There are different views on that.
--:-- Josh Dzieza: There's certainly people in the AI industry who think we're going to reach a breakthrough where the AI is going to be so smart that it doesn't need human input anymore.
--:-- Sean Rameswaram: It's going to become super intelligent.
--:-- Josh Dzieza: There are a lot of other people who disagree with that.
--:-- Sean Rameswaram: And certainly historically, what has happened is annotation is always kind of getting automated.
--:-- Josh Dzieza: Like if you look at those early image recognition systems, like that's automated. AI can tell the difference between an image of a cat and a dog.
--:-- Sean Rameswaram: But it enables new technologies like self-driving cars. And now you need even more people doing even more and more complicated forms of annotation.
--:-- Josh Dzieza: And that has been the way it's gone.
--:-- Sean Rameswaram: And, you know, you can certainly see a world where these language models are out in the world and all the things they're supposed to be doing, like giving health advice or legal advice, are complex, changing high stakes fields, and you're going to need even more human annotation there.
--:-- Josh Dzieza: So is this future of like, you know, perpetual human collaboration with AI going to lead us to some ideal where the cars will drive themselves perfectly?
--:-- Sean Rameswaram: Or the, I don't know, the robot doctors will know my knee from my elbow.
--:-- Josh Dzieza: I guess I should talk about sort of how brittle these systems are.
--:-- Sean Rameswaram: That's the word that's used to describe their knowledge, the state of their knowledge.
--:-- Josh Dzieza: When you're training something to be accurate, for example, you have people who are rating it for accuracy, but one, maybe they're not rating it correctly because it's very time consuming and often impossible to fact check every written response.
--:-- Sean Rameswaram: Often responses are open to interpretation or just too complicated.
--:-- Josh Dzieza: And two, you don't know that it's learning the right patterns as opposed to learning to talk like whatever text people have labeled as accurate sounds like.
--:-- Sean Rameswaram: So one of the risks that I think we're seeing now is it's become these language models particularly have become extremely good bullshitters.
--:-- Josh Dzieza: Like you may have seen the case of the ChatGPT lawyer who submitted some legal filings citing cases that he asked ChatGPT for.
--:-- Sean Rameswaram: The lawyer cited more than half a dozen relevant court decisions to make his case for why the lawsuit had precedent.
--:-- Josh Dzieza: The only problem, none of those decisions were real.
--:-- Sean Rameswaram: The program even reportedly told him yes when he asked it to verify that the cases were legitimate.
--:-- Josh Dzieza: Sounds like a trash lawyer though, honestly.
--:-- Sean Rameswaram: Yes, I would certainly not consult ChatGPT for legal advice.
--:-- Josh Dzieza: And the question is, will it ever get there if you just throw enough annotation at it, enough data at it is there going to be a point where it learns what is true or false or what the legal reasoning or something like that?
--:-- Sean Rameswaram: Or is it going to continue to just sort of be a better and better mimic?
--:-- Josh Dzieza: And you're always going to have that possibility that it's going to make some catastrophic error.
--:-- Sean Rameswaram: That's an open question.
--:-- Josh Dzieza: And also as an open question of how you're going to have people who can continue to oversee these models as they get so good at mimicking people.
--:-- Sean Rameswaram: Yeah. Like you need a very good lawyer all of a sudden who can critique an AI model that is good at making up legal advice.
--:-- Josh Dzieza: And what about the other side of this, just like the treatment of workers?
--:-- Sean Rameswaram: I mean, you mentioned people working 36 hours straight. If if Google might be behind the contract job that someone in Kenya has that's paying them a dollar an hour to annotate elbows, are they cool with working people like that?
--:-- Josh Dzieza: 36 hours straight for like a dollar an hour? That is a question for Google.
--:-- Sean Rameswaram: But I can say that some of their annotators in the U.S., the people who are reading search results and YouTube results through the platform Appin, have been protesting their conditions, saying that they're underpaid, that they don't have health benefits.
--:-- Josh Dzieza: Raiders are why Google search results are so good. They make sure that people like you and me get the information we need every single time.
--:-- Sean Rameswaram: And no one working for Google should be struggling to pay their rent.
--:-- Josh Dzieza: Google's defense has been that they are paid fairly, but there tends to be in the industry not a lot of attention on this kind of work.
--:-- Sean Rameswaram: Part of it, I think, stems from the sense that it won't be needed for long, that the AI will get good enough that you don't need annotators anymore.
--:-- Josh Dzieza: And so it's not really a job so much as just like some temporary work that you're calling on someone to do.
--:-- Sean Rameswaram: And what happens after that is not really your concern.
--:-- Josh Dzieza: And so I think there's a sense where companies just sort of don't even really think of it as a labor issue, that they're just kind of buying a bunch of data.
--:-- Sean Rameswaram: That may be changing. I've seen just sort of in papers, people say, you know, these annotators were paid the median wage, wherever they're based, or things like that.
--:-- Josh Dzieza: I think there is, you know, when attention is brought to this situation, there often is a push to do better, but it's pretty uneven.
--:-- Sean Rameswaram: And there's just not a lot of transparency in the data pipeline.
--:-- Josh Dzieza: And so even if you want to do better, it's hard.
--:-- Sean Rameswaram: You know what it sounds like, Josh? It sounds like it might just be easier to pay people to do jobs.
--:-- Josh Dzieza: Did that occur to you at any point while you were clicking through whatever data that you were annotating?
--:-- Sean Rameswaram: That did occur to me many times while I was annotating.
--:-- Josh Dzieza: There's one where it was quite acute, where I was tracing pallets in like a warehouse for some kind of self-driving forklift.
--:-- Sean Rameswaram: And just the amount of really kind of excruciatingly detailed labor that was going into figuring out how to drive a forklift around to automate one job of forklift driver was pretty staggering.
--:-- Josh Dzieza: I mean, there must have been hundreds, if not thousands, of people working on this thing around the world, just tracing pixel by pixel each pallet and each pallet hole in these dark warehouses.
--:-- Sean Rameswaram: I guess that the hope of these companies is that once you've done all that work, you have this thing that can do it forever.
--:-- Josh Dzieza: But I don't know that that's true because, you know, the world keeps changing and throwing up new edge cases.
--:-- Sean Rameswaram: And somewhere in this world, that used to be a good union job.
--:-- Josh Dzieza: Right, exactly.
--:-- Sean Rameswaram: What were you hoping people would take away from your piece?
--:-- Josh Dzieza: I think there are a couple of different things and a couple of different reasons why it's important to look at this work.
--:-- Sean Rameswaram: The first is just kind of the labor issues that it raises. You have these potentially extremely profitable technologies that rely on often low-paid labor around the world that is often not discussed.
--:-- Josh Dzieza: And kind of the second thing that I wasn't expecting to find but found is that the work is kind of structurally precarious in a way that a lot of, even for gig work, like gig work is notoriously precarious.
--:-- Sean Rameswaram: But the way AI development works, where you need a ton of data to train your model, and then you need like a bit of more specific data to fill in some edge case, and then nothing for a while, and then a ton more data means that if this is going to be a fixture in an AI economy, there's going to be a lot of time people are not working.
--:-- Josh Dzieza: And there's going to be times when lots and lots of people need to work. And the way it's set up right now, the workers pay the cost of that.
--:-- Sean Rameswaram: They're the ones who are unemployed whenever they're not needed and then they're expected to be kind of on demand when they are needed.
--:-- Josh Dzieza: I think also just a better understanding of the way these systems work.
--:-- Sean Rameswaram: I think it's easy to, especially with something like ChatGPT, when it can tell you that it's an AI trained by open AI using reinforcement learning and all about itself that, you know, act in these very human-like ways, that there's a tendency to think it can reason like a human.
--:-- Josh Dzieza: But it's important to think about the fact that a lot of that stuff was written there manually by humans and then reinforced by humans.
--:-- Sean Rameswaram: And there's a sense in which seeing the humans in the system kind of makes you realize how inhuman these machines are and that they have some pretty glaring weaknesses.
--:-- Josh Dzieza: I don't trust them, Josh.
--:-- Sean Rameswaram: I think that's wise for the time being.
--:-- Josh Dzieza: Josh Jezza does investigations at The Verge. You can read his work at theverge.com. His piece that inspired our episode today was titled Inside the AI Factory, and it also ran on the cover of a recent issue of New York Magazine.
--:-- Sean Rameswaram: The show today was produced by Amanda Llewellyn. It was edited by Amina Al-Sadi and fact-checked by Laura Bullard. We were engineered by Patrick Boyd.
--:-- Sean Rameswaram: I'm Sean Rameswaram, and this is Today Explained.
