There’s a sticky issue scientists have to deal with – science is carried out by humans. We humans have flaws (and how) and they can end up in our work. Fortunately, science is waking up to research bias. In the meantime, here’s what to look out for.
Learn more about your ad-choices at https://www.iheartpodcastnetwork.com
See omnystudio.com/listener for privacy information.
No summary for this episode yet.
This is an iHeart Podcast. Guaranteed human. Living with an autoimmune condition isn't easy, and every journey is different. That's why Season 5 of Untold Stories, Life with a Severe Autoimmune Condition, from Ruby Studio and Argenix, shares powerful firsthand stories from people with conditions like MG and CIDP. Hosted by Martine Hackett, these conversations dive into what resilience really looks like through setbacks, breakthroughs, and finding strength in community. Listen on the iHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Banking with Capital One helps you keep more money in your wallet with no fees or minimums on checking accounts and no overdraft fees.
Just ask the Capital One Bank Guy. It's pretty much all he talks about, in a good way. He'd also tell you that this podcast is his favorite podcast too. Thanks, Capital One Bank Guy. What's in your wallet? Terms apply. See CapitalOne.com slash bank. Capital One N-A, member FDIC. Welcome to Stuff You Should Know. A production of iHeartRadio. Hey, and welcome to the podcast. I'm Josh Clark, and there's Charles W. Chuck Bryant, and Jerry's here. Jerry's back, everybody, looking well-rested and sun-kissed and everything. And this is Stuff You Should Know. She's like a beautiful, juicy orange. That's right.
That's right, Chuck. That's a really apt description. Ready to be squoze. I wish I could squeeze her, but we're still not squeezing. No, not in this pandemic. Are you crazy? Are you out of your mind? No, no squeezing. Yeah, even Robert Plant wouldn't let you anywhere near him. Okay, Chuck, figure out that joke. Robert Plant, the squeeze? My. My. Lemon. The lemon song? No. till the juice runs down my leg. Yeah, that's the lemon song, right? No, I don't think so. I think it is. I don't think it is. Okay. I think it's, I don't think it is the lemon song, man. I think it's a whole lot of love. It's a whole lot of love.
All right. It's maybe the dirtiest thing that was ever said in like a top 10 song. Okay. Regardless. I'll just let the emails take care of this. I really think, no, the Lemon Song is, um... Is, ooh, I love those lemons. Right. No, the Lemon Song is all about how you have friends, like you want to have friends, and friends are good to have. Okay. Yeah. I may be all wrong. No, I think it's all I'd love. Yeah, it is. I'm 100% sure, buddy. All right, well, I encourage you not to Google the lyrics then. Well, we could ask our good friend and Stuff You Should Know writer, Ed Grabinowski, the Grabster.
Oh, look at that segue. Because he is in a band and has been for a while. We've mentioned it before, Space Lord, which has just a super cool Zeppelin-esque sound to them. And they just released. They cover some Zeppelin, too. They do. Here and there. And they just released a new single, which you can find on Bandcamp by searching Space Lord. Not the Space Lords. No, not Vinny and the Space Lords. Yeah, Space Lord. Just look for Space Lord with some cool graphics, and you know that's Ed. You'll know it's the Grabster. Yeah, good stuff. We also have a game out that Trivial Pursuit made. Yeah, we should plug our own stuff every now and then.
We just did. Yes, it is a co-branded game with Trivial Pursuit from Hasbro, and it is not a Trivial Pursuit game that you are used to. It is a Stuff You Should Know game that Trivial Pursuit was happy to co-brand with. So just what I don't want is emails that are like, guys, this isn't Trivial Pursuit. This is some other different game. You're always worried about the emails, aren't you? I'm going to just ignore them, let them roll off my back. I know. Like I'm disappointed in you guys for this. I haven't even listened to the episode, but I'm disappointed about this. You just got one of those.
Did you see that? It just rolls off your back. Yeah, those are always great. I didn't listen, but. Here's what was wrong. I wrote that person back, actually. I was like, we actually kind of did exactly what you hoped we would do. And they're like, oh, sorry for being presumptuous. Anyway. Oh, all is forgiven. Yeah. So we're talking today about bias, Chuck. And I want to set the scene a little bit because, you know, one of the things that I'm always like harping about is like the death of expertise, right? Oh, sure. And it's a real problem. Like this idea that science can't be trusted, that people who go and spend a decade or more learning about a specific thing that they go out and become an expert in.
Or that's their profession. That's their training. that those people, what they have to say is basically meaningless or that it's no better than somebody on the Internet's opinion about that specific subject that that person spent 10 or 12 years being trained to be an expert in. Like, that kind of stuff to me is, like, super dangerous. Like, there's an erosion of something, and it's an erosion of intelligence to start with. But it's also an erosion of just believing in facts and knowing that you're not being taken for a ride or hustled. It is a huge, enormous problem that we're just beginning to wake up to and is still unfolding.
It's not like it happened and now we're like reeling from it. It's still happening in real time. And it is a massive, huge issue. One of the biggest issues that humanity faces, I think, because it encompasses so many other large issues like climate change, existential risks, the pandemic, politics, all of them kind of fall under this erosion of belief in facts and that there are people out there who know more than you do. It's a big problem. Yeah, imagine being someone who studied and researched something intensely for 10 or 15 years when presenting facts to be met with, I don't know about that.
That's a response I hear a lot in the South. Yeah, or that they saw something on YouTube that flatly contradicts that. And it's like, it doesn't matter. What you just said is ridiculous, that somebody posted something on YouTube and that that has as much weight as what somebody who spent 10 or 12 years studying this very thing has to say about it, like knows exactly what they're talking about, has to say about it. It's maddening. Yeah, there's something about people from the South in general, I think, that are in this group that I have literally heard that response from a lot of different people when I've been like, oh, no, no, no, here are the facts, actually.
And then when presented with something that they can't refute, they say, I don't know about that. And like, that's it. That's the end of the conversation. That's different than the people I've encountered. The people I encountered, like their brow furrows and they start pointing fingers and their tone goes up. Like, are you hanging out at the country club or something? I think it's different types of people. You know, there's ignorance and then there's also people that actually think they're better informed that will fire back with YouTube clips. Right. So the reason I brought that up is because one of the reasons that that is being allowed to exist, that that does exist.
I think it's a reaction to something else that's going on simultaneously, which is there are a lot of experts out there who are performing really sloppy science, sometimes outright fraudulent science. And they're frittering away whatever faith the general public or society has in their expertise and in their profession. And there are a ton of scientists out there. I would say the vast majority by far of scientists are legitimate, upstanding, upright dedicants to science, right? That's where they place their – that's where they hang their hat. That's where their heart is. That's what they believe in and that's what they work to support.
But science has like kind of a problem, Chuck, in that it's allowing way too much for bias, which is what we're going to talk about, to creep into science. and undermine science and basically produce papers that are just useless and trashed. And there's a whole lot of reasons for it. But it's something that needs to be addressed if we're ever going to get back on a footing with a faith in experts and expertise and just facts that there are such things as objective facts. Yeah, I mean, a lot of times it's financially related, whether it's a lack of funding, a desire for more funding, a desire just to keep your lab running and people paid on staff, which, you know, all this stuff is understandable.
You want to keep doing this work, but you can't let that get in the way. It's like in Rushmore at the end when Margaret Yang faked the results of that science experiment because she didn't want it to be wrong, you know? I don't remember that part. Was that like a deleted scene? No, no, no. It was in the end when they meet up and he's flying the – I think he's flying the kite with Dirk. Right. And she's talking about her science fair project and he was really impressed with it. And she was like, I fake the results. And the reason why was because she didn't want to be wrong. And I think a lot of times people will get into a certain body of research or data to – because they want to prove a certain thing.
And if they can't, it might be really hard to live with that. So that weighs into it. Money for personal gain. Advancing your career. You know, publish or perish. That whole thing. Like, we're going to talk about all this, but there are a lot of reasons that it's been allowed to creep in. But all of it is at the disservice of the fundamentals of what they base their careers on to begin with. Yeah, it's at the disservice of science itself, right? Because the whole point of science and then scientific publishing, the whole publishing industry, is to basically create a hypothesis, test your hypothesis, and then share the results with the world.
And that's ideally what would happen because you're building this body of scientific knowledge. But money and corporate interests and academic publishing have all kind of come in and taken control of this whole thing. And as a result, a lot of the stuff that gets published are trash papers that shouldn't be published. A lot of the really good papers that don't come up with sexy results don't get published. And then, like you said, people using science for personal gain. There are a very small cadre of thoroughly evil people who are willing to use their scientific credentials to create doubt in the general public to prevent, like, people from understanding that climate change is real for 20 years.
Or that fossil fuels actually do contribute to anthropogenic climate change. But what we're mainly focusing on is like bias in the sense that people carrying out studies are human beings and the human beings are flawed. We're just flawed. And we bring those flaws to our studies and that you really have to work hard at rooting those flaws and those biases out to produce a really good, thorough scientific study with good, reliable results that can be reproduced by anybody using the same methods. and that science is just starting to wake up to the idea that it is really biased and it needs to take these into account in order to progress forward from the point that it's at right now.
Which is tenuous, I think. Perhaps more tenuous than ever. The point that science is at? I think so. Science isn't going away. It's not going anywhere. It's probably the greatest achievement humans have ever come up with, right? No. It's not going anywhere, but it is a terrible position that it's in. And it's going to take some genuine leadership in the scientific community from a bunch of different quarters and a bunch of different fields to basically step up and be like, guys, this is really bad and we need to change it now. And a lot of people need to be called out. And science typically shies away from naming names and calling out by name fraudulent scientists because scientists seem to like to suppose the best in people, which is not always the case.
Right. And having said all of this, we could root out every bias and really clean up the scientific publishing community 100%. And there's still a certain set of people in this country and in the world who that wouldn't matter to and would still shut down facts because it doesn't fit their narrative. So they can't be helped. For sure. But Chuck, those people have always been there, right? And they're always going to be there. Exactly. It's just contrarians. They are, you can call them free thinkers, you can call them stubborn, you can call them purposefully ignorant. Who knows? They're always going to exist.
The problem that this crisis that science finds itself in right now is that it's allowed that population to grow and grow. And, like, people who otherwise didn't ever really question science have been allowed to kind of trickle into that fold. And that those are the people that we should be worried about. The ones who would know better if they believed in science again. Right. And our way into this is to talk about different kinds of biases. In True Stuff You Should Know fashion, a top ten that is not a top ten. That's exactly right. We ate into at least three in this intro. And hopefully, shining a light on some of this stuff, people will at least be more aware of different biases.
Well, yeah. You know, the first one is good old confirmation bias. I mean, these aren't ranked because confirmation bias would probably be number one as far as people's awareness of it. But there are different examples that people use for confirmation bias. And I kind of enjoyed the one from the HowStuffWorks article, even though it was from 1903. After x-rays were discovered in Germany, there was a French scientist named René Blondelotte. Yeah? Yeah. He looked at x-rays. He said, moi aussi. Well, who said, hey, I see N-rays. I've discovered N-rays. And everyone's like, what's an N-ray? He said, well, it's like a corona when electricity discharges from a crystal, and you can only see it in your peripheral vision.
And American Robert Wood laid the wood and said, I'm going to come to your lab and check this out and secretly remove the crystals during one of the experiments. and Blondelot still saw these N-rays. And so that's confirmation bias. He wanted to see those N-rays. And then later, even though it was disproved, other French scientists supposedly published papers or published papers based on that research because they wanted it to be true. So that's what confirmation bias is. It's when you're starting out with a hypothesis that is going to shape the methodology of your study to confirm it. Right.
And then it can also occur where you're interpreting info to fit your hypothesis. So you're seeking out stuff that supports your hypothesis, and then the stuff that's just there in front of you, the results that are there in front of you, you're like, aha, this thing proves that those N-rays actually exist. Or this phenomenon cannot be due to anything but N-rays. Therefore, N-rays exist. All of it's confirmation bias. And like you said, that's number one because that's not just a scientific bias. I mean, like that is – that's like every human uses confirmation bias in that it's twofold. We avoid contradictory information because we don't like to be wrong.
And we find information that confirms our point of view because we like to be right. That's right. That's confirmation bias. And it's everywhere among everyone. That's right. Although I will say, I know it happens a lot politically, but myself and the people that I congregate with question their own leaders as much as they do leaders from the other parties. Oh, that's good. It's very good to do. And I don't know. There shouldn't be sacred calves in politics. That's a bad jam. Well, no, and it's like I've always been like at the forefront of calling out my own party's wrongs and saying, no, no, no, you need to do better than that.
Whereas I see a lot of other people in other situations truly bury and ignore those things because they don't, you know, just don't want to face that. Yeah, and it's not like – it's not even like I don't want to face it. It just doesn't fit their worldview, so they just don't include it. It just gets tossed out. But the point is, it's not an active process necessarily. Right. I think we should probably take our first break. I think so too, Chuck. All right. We'll be right back and talk about sampling bias right after this. you know everyone living with a rare autoimmune condition has their own story to tell and that's why in season five of untold stories life with a severe autoimmune condition a ruby studio production in partnership with argenics you'll hear powerful real life perspectives that's right This podcast explores stories of what life is really like with MG or CIDP.
Host Martine Hackett sits down with people who faced it all. The early signs and symptoms, the search for answers, and the strength it takes to keep moving forward. Yeah, and this season the stories go even deeper, showing us what resilience truly looks like through setbacks, breakthroughs, and the communities that make all the difference. So listen to Untold Stories, Life with a Severe Autoimmune Condition, on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Attention parents and grandparents. If you're looking for a gift that's more than just a toy, give them something that inspires confidence and adventure all year long.
Give them a Guardian bike, the easiest, safest, and number one kid's bike on the market. Yeah, with USA-made kid's specific frames and patented safety technology, kids are learning to ride in just one day with no training wheels needed. It's why Guardian is America's favorite kid's bike, and the New York Times and Wirecutter's top pick three years in a row. That's right. My daughter has a Guardian bike, and she loves it. And that thing was really easy to put together. And get this. This holiday season, Guardian is offering their biggest deal of the year, over 40% in savings on all bikes, plus $100 in free accessories.
Guardian bikes have become one of the most sought-after gifts of the season, and inventory is going fast, so don't wait. Join over a half a million families who've discovered the magic of Guardian. Visit guardianbikes.com to shop now. The universe is full of mysteries. Black holes, quantum physics, galaxies. On TikTok, millions of people are learning more about the universe around them every day. Scientists break down complex theories, demonstrate experiments, and connect dots between the cosmos and our daily lives. One scroll might reveal the concepts on the fabric of space-time, the next an optical illusion.
It's discovery on a massive scale, where millions learn something new every day. This time of year, it's sensory overload everywhere. But one feeling we're all chasing right now is cozy. And Bombas has the socks, slippers, tees, and basically everything you need to get you there. Yeah, and I got to tell you, there's nothing like new socks. It's almost therapeutic. They have sock scientists at Bombas, and they found a way to channel that energy into everything, from slippers with sink and cushioning to satisfyingly weighty tees. And you know, that feeling doesn't stop after one wear. Those things stay comfortable.
Yeah. And Bombas is also stepping up their slipper and slide game, too, this season. They've got new shapes, new styles, fluffy things, suede things, a little something for every foot. If there's one thing Bombas knows, it's feet. That's right. And you know what the best part is? For every pair of Bombas that you purchase, Bombas donates one to someone facing homelessness on your behalf. So anytime you get something cozy, someone else does, too. So head on over to Bombas.com slash S-Y-S-K and use code S-Y-S-K for 20% off your first purchase. That's B-O-M-B-A-S dot com slash S-Y-S-K, code S-Y-S-K at checkout.
All right, Chuck, we're back and we're coming back with something called sampling bias, which is, it turns out, a subtype of a larger thing called selection bias. Oh, and one other thing we should say, we kind of got into it before I could say this. There are different stages in a study where bias can occur. It can happen in like the planning, the pre-study phase, and it can happen during the actual study, and then it can happen after the study as well. And so when we're talking about any kind of selection bias, including sampling bias, This is pre-study bias, where when you're actually setting up the study, this is where the bias is going to happen.
Yeah, and you know what? I think it also bears saying that bias is, you have to work really hard to avoid it because it's almost like a disease that's always trying to get involved. And it's not like, just do better, everybody, and quit being biased. It's like it's way more complicated than that because it is always knocking at the door, like you said, in all three phases, trying to sneak in there. And it takes a lot of work in all three phases to avoid it. So it's not as I don't want it to come across as easy as us just saying, like, you shouldn't do that. Stop it. No, but the first step is to recognizing that there's a lot of bias and different kinds of bias that are just sitting there waiting for a scientist.
And then if you start admitting that it's there, you can start being on the lookout for it and you can start adjusting for it. And then other people who read your papers or read news articles about your papers can be on the lookout for that kind of thing. Yeah, so exactly. Sampling bias is your sample set not being accurate and a good representation of the whole. A lot of times you'll find this in either studies that are really small scale because you don't have a large sample and you don't have the kind of money like near you. Like maybe you work for university. So you work with university students as your first sample set who are not indicative of anything but, you know, people 18 to 21 years old or so.
Now, remember we talked about weird Western educated industrialized rich and democratic. Yeah, that's exactly the thing. It's like, I mean, it's a decent place to start if you don't have much money and you want to get the ball rolling. It's not like, oh, you shouldn't do university studies at all using students. But those findings definitely don't represent the wider nation. And it needs to grow and get more funding if you want to actually have a legitimate claim to something. Another way that sampling bias can come up is from, like, the group that you're recruiting from. Like if you're doing a strictly online survey, but you're trying to apply your findings to the wider society, that's just not going to happen because there's so many people who aren't Internet savvy enough to take an Internet survey.
Like by nature, you are a little savvier than the average person if you're hanging out on the Internet and taking a survey. And then also kind of tangential that I like to tell myself that at least. And then tangential of that is something called self-selection bias, which is where the people who say, let's say you're doing a study on wellness and, you know, what eating tuna can do for your health. People who are interested in wellness and health are going to be much more likely to volunteer for that study than people who couldn't care less about health and have no desire whatsoever to further science's understanding of what makes you healthy.
So you would have to go out and find those people and recruit them rather than just relying on the people who volunteered based on the flyer you put up in the student union. Or, you know, study all financial demographics or poll all financial demographics rather than just – and, you know, sometimes it's a methodology in which they do try and recruit people, steers them in that direction unknowingly. that I know in the article, they talked about the 1936 presidential campaign with Roosevelt and Alph Landon, Republican Alph Landon. They were doing polling with like country clubs, rosters, and people who drove cars and stuff at the time, that was kind of a luxury.
So, it was all out of whack. Everyone's like, Landon's going to win in a landslide. It's because you just kind of basically stuck your polling to, you know, I don't know about wealthy individuals, but people who are a little more well-off. And I think we talked about that in our polling episode, that fiasco with polling. I also saw one more, too, that I want to mention because it has a really great anecdote attached to it. It's called survivorship bias, where when you're studying something, say like business or something, you're probably going to just be looking at the extant businesses, the businesses that have survived 20 years, 30 years, 50 years, or something like that, and you're not taking into account all of the failures.
So when you put together like a prognosis for business in America, it might have a sunnier outlook than it should because all you're looking at are the ones that manage to survive and thrive. And that's survivorship bias. And did you see that anecdote about World War II fighter pilots? It was actually pretty funny because they studied planes that had been fired upon but managed to get back safely. and they were like, well, let's look at all these different bullet holes and where this plane was hit and let's beef up all those areas on the body. And a mathematician named Abraham Wald said, no, those are the places where they got shot and did okay.
What you should really do is find these planes that actually went down and beef up those sections of the plane. And that's survivorship bias. It's just, it's failing to take into account the failures that have to do with what you're trying to study. What about channeling bias? Channeling bias is another kind of selection bias. Did you get this one? It wasn't the best example of channeling bias. Yeah, I mean, I got it. Okay. All right. Did you not get it? I got it, but it took a lot of work before I finally did. Well, it's basically when, let's say you have a patient, and their degree of illness might influence what group they're put into.
So, if a doctor, if a surgeon was trying to study outcomes of a particular surgery, they might, because they're surgeons and they want to help people out, they may perform that surgery on maybe younger, healthier people who might have better outcomes than someone who is in a different, like, higher age group. Right. And the article just kind of ends it there. And I was like, so? What's the problem? And I finally found this example where it says like, okay, let's say you're studying a new heartburn medicine or something, or something to treat like GERD. And it's new, it's hardcore, it's cutting edge.
And the people who are likeliest to get this new hardcore antacid are the ones who are probably in worse shape, right? So say they're on the verge of going to the ER anyway. Well, if you look back at all of the people who have ever been prescribed this new hardcore antacid, you're going to see like a lot of them ended up in the ER, even though it had nothing to do with this hardcore antacid. And then similarly, the people who have so-so GERD, it's not particularly bad, they'll probably be prescribed the old drug, the standby that everybody knows is fine, that's going to work. So if you compare the old drug and the new drug, it looks like the old drug is super safe, but the new drug will put you into the ER.
whereas that's channeling. You've channeled different people with different prognoses into different groups and they're kind of pitted against each other in an effort to obscure the truth. If you wanted to really know the genuine health outcomes for that antacid, you would have to give it to people with not so bad GERD and people with really bad GERD and see what happens. See if the ER visits continue for people who wouldn't otherwise be going to the ER. Well, yeah, you want to find the outcome It's become for everyone. Right. And not just for you. Like if you're debating surgery and you're like, oh, well, it shows really good outcomes.
You're like, well, yeah, but who are they operating on? Right, right. Yes. So I would like to invite anyone who got what I was saying or got channeling because of what I was saying, I invite you to email and let me know. I'm doing like a little bit of surveys here, and I'd like to know if I confuse things more or make it more understandable. I know it's funny either way. I got that part. I'm just trying to figure out if it's understandable. But here with your methodology, talking about a stuff you should know, listener, who by nature is smarter than your average bear. Well, I'm not going to publish it.
I'm going to file drawer it either way. Oh, what a teaser. question order bias is the next one that is and this is mainly obviously with when you're just doing a like polling and stuff or like an online survey or you know it could be just asking people a set of questions like in a social science research set and the way you order things can affect the outcome and this isn't anything at all like everything from the brain's tendency to organized information into patterns to the brain simply paying attention more and being more interested early on. Like I know there was one, the General Social Survey, was a big long-term study of American attitudes.
And in 1984, they were asked to identify the three most important qualities for a child to have. And they were given a list of these qualities. Honesty was just listed higher on the list. When it was, it was picked 66% of the time. when it was further down on the list, it's 48% important. And that's simply because people are just reading this list that are like, you know, these look good to me. Honesty is important. And then, yeah, by the time they got down, you know, three-quarters of the way through the list, they'd started thinking about what they were going to have for dinner. Yeah, people get pooped when you're giving them lists of stuff.
Or you can prime people and get them all sort of worked up. Like if you have a question like, during the Trump administration, How mad were you at that guy about stuff he did? And you're like super mad. And then you're like, well, how did you feel generally about how your life was affected during his administration? You might say, it was awful. Whereas if they hadn't have asked that first question, they were just like, what was your life like from 2000? I'm blocking out the dates. What? 2016 to 2020. Yeah. You might say, well, you know, it wasn't, it was okay. I ate a lot of sandwiches. Just over those four years, though.
Yeah, so like you said, that's priming, which is a big thing that you have to worry about when you're doing any kind of survey. So there's some of the ways that you can combat that. You can randomize your question order. Sometimes you'll have a survey where one question is predicated on a previous question. So one thing you might want to do is ask that set of questions in a few different ways so that you can kind of maybe compare the answers to all three, add them up and divide them by three and there's your average answer kind of thing. There's a lot of things you can do to kind of, I guess, manipulate to demanipulate your respondent when you're doing a survey like that.
Manipulate to demanipulate. Look it up. You won't find anything on it, but you could look it up still. Oh, it's a Roxy Music album. Interesting. Wow, Chuck. Wow. Nice work. Yeah, that was great. What's next? So with question order bias, we've entered the during the study kind of bias. This is why you're actually conducting the study, and so is interviewer bias. and interviewer bias it's kind of like well question order bias has to do more with the study design but it's a bias that emerges during the study interviewer bias just straight up is in the middle of the study and it has to do with the person actually asking the questions in an interview it can also I think apply to somebody conducting like the clinical trial on a drug, if they know whether somebody is getting placebo or not, it might affect their behavior.
But ultimately what it is, is the person who's wearing the white lab coat is influencing the outcome of the study just through their behavior, through their tone of voice, through the way that they're asking a question. Sometimes it can be really overt. And like, let's say, like a super devout Christian is, you know, doing a study on whether, how many, what part of the population believes Jesus saves. And they might be like, you know, do you think Jesus saves is the question? Don't you? Like, it seems like it, huh? That kind of thing would be a pretty extreme example, but it's sometimes how you understand things is in the absurdities, you know?
Yeah, I thought this example in the House of Works article was kind of funny. It was about, just like a medical questionnaire where the interviewer knows that the subject has a disease that they're talking about. And they may probe more intensely for the known risk factors. And they gave smoking as an example. And it said, so they may say something like, are you sure you've never smoked? Never, not even once. Admit it. Like, if I heard that coming from a researcher, even without knowing a lot about this, I would say, what kind of a researcher are you? Yeah. It seems like you're looking for an answer.
You should say you are ethically compromised. Or even facial expressions or, you know, body language, all that stuff weighs in. I don't know why. Arched eyebrow. Like, why don't they just have the robots, Alexa or Google or Siri or somebody ask them? Well, that's one good thing about something like an Internet survey is, like, it's just questions. And as long as you design the questions and you randomize their presentation, like, it's going to be fairly helpful in that respect. But then it's got its own pitfalls and pratfalls. You can attract a lot of people who are just taking it to mess with you.
Right. There's a lot of problems with all of it. But, again, if you're aware of all the problems, you can plan for them. And then even if you can't plan for them or control them, you can write about it in the actual study and be like this study. I remember running across studies before where they're basically like there are, you know, there was a kind of bias that we couldn't control for. So, we can't really say whether it affected the outcome or not. And I thought, wow, this is really refreshing and like even daring kind of like I was thrilled. But you don't see that very often. But that, from what I understand, is the direction that science is going toward now.
Well, and the reason you don't see that, and it's something we'll talk about, is what actually ends up getting published. It may be less likely to get published if they're like, hey, you know what? Dude. You know what I'm saying? Yeah, I know. So let's do recall and acquiescence bias because they're very much related, and then we'll take a break. That's our plan. What do you think of it? Everyone says, sounds good to me. All right. So this is also during study, and this is so in the very much the way that an interviewer can influence the outcome. The participant can actually influence the outcome, too, especially if they're being asked questions or they're being asked to self-report.
There's a couple of ways that us just being humans can foul up the works on the findings of a study. The first is recall bias. Yeah, this is when you're, obviously, you're trying to recall something from the past. And it's amazing what might jump out at you from your past when probed with a certain question, certain correlations that really have nothing to do with it. But you may be like, oh, well, you know what? Now that I think back, I remember around that time, I was watching a lot of Dancing with the Stars. I kind of binged that show. So maybe that's why I had homicidal tendencies. I don't think you need a study to prove that.
I think that's just intuition, you know? Yeah. But yeah, so and if enough people do that, especially if there's something out kind of in like the zeitgeist about that, how like people who watch too much Dancing with the Stars want to kill other people. Like a number of your participants might recall that thing, whereas other people who don't watch Dancing with the Stars aren't going to recall that. And so in the same way that survivorship bias influences it, those people who don't have that memory to recall, that memory can't possibly be included in the study results, which means that dancing with the stars is going to kind of percolate to the top as like a major risk factor in homicidal tendencies.
That's not good. You don't want dancing with the stars unfairly canceled. You want it to be canceled because it is terrible. I've never seen it. I'm sure it's great if you're into dancing. I haven't either. But watch, we're going to be asked to be on. Oh, my God. And they'd have to change the name of the show to Dancing with the Mid-Level Internet Famous. Right, exactly. Wow. That'd be a good show. I'd have to dust off my jazz shoes. It would be us and Chocolate Rain and, you know. I love that guy. Tay-Johan Day is his name. Yeah, we actually met him that time, remember? He was great, man. Another thing, Chuck, that has to do with recall bias is that we just tend to have faultier memories.
with stuff that makes us look bad, like, say, unhealthy habits. Oh, sure. So if you're doing a study on junk food and health outcomes, and you interview a bunch of people who are in terrible health, and all of them are like, oh, I only eat, like, Cheez-Its, like, once in a blue moon or something like that, and the researcher writes that once in a blue moon Cheez-Its. like the results of the study are going to suggest that it takes just a very small amount of Cheez-Its to put you in the hospital with long-term chronic health conditions. Right. And that is a problem with recall bias. Like it's the participants affecting it.
In this case, because they just aren't paying attention or aren't really thinking about, no, you've eaten a lot of Cheez-Its and it takes a lot of Cheez-Its to put you in the hospital. not a very little amount. It's not the best example, but it kind of gets the point across, I think. Now, is this part of acquiescence bias? No, that was the end of recall bias. Acquiescence bias is different, but it's certainly related. Both of them kind of fall under an umbrella of participant bias. Yeah, and acquiescence bias, I feel like there's the opposite, too. I just don't know if they have a name. Because acquiescence bias is generally like people want to be agreeable and they want to answer in the affirmative and they want to, especially they found, if you are maybe less educated, you might be more willing to just go along with something and say, yeah, sure, yeah, yeah, to maybe appear smarter or just to be more agreeable.
I do think it's the opposite can happen too, especially with political research and social studies in that I think there are also people that are like, oh, you're from the what? Well, yeah, sure, I'd love to be interviewed. And then they go into it with a sort of opposite mentality where they're completely disagreeable no matter what anyone says or asks. Yeah, I didn't run across that, but I'm absolutely sure that that is a bias out there. But you can avoid these by doing it more smartly, right? More smartly? Yeah, there's ways that you can frame your questions. Like people don't like to admit that they didn't actually vote in American democracy.
Some people. So instead of saying there was a Pew suggestion where they said rather than saying like did you vote in the last election, a lot of people who didn't vote are going to be like, sure, yeah, of course. Why would you ask that? instead you would phrase it as like in the 2012 presidential election did things come up that prevented you from voting or were you able to vote? Right. And you would probably actually want to train your researcher to use that same intonation to make it seem casual either way. Like you want to give the person a sense of comfort that they're not being judged no matter how they answer.
Give them a back door. That's a good way. That's a good way to get around acquiescence bias. Yeah, absolutely. Yeah. The old backdoor policy. That's right, where you can squeeze a lemon. Right. All right, are we taking a break? I think we're mandated by the FCC to do that after that joke. All right, well, we'll be back and finish up with our final two biases right after this. You know, everyone living with a rare autoimmune condition has their own story to tell. And that's why in Season 5 of Untold Stories, Life with a Severe Autoimmune Condition, a Ruby Studio production, in partnership with Argenix, you'll hear powerful, real-life perspectives.
That's right. This podcast explores stories of what life is really like with MG or CIDP. Host Martine Hackett sits down with people who faced it all. The early signs and symptoms, the search for answers, and the strength it takes to keep moving forward. Yeah, and this season the stories go even deeper, showing us what resilience truly looks like through setbacks, breakthroughs, and the communities that make all the difference. So listen to Untold Stories, Life with a Severe Autoimmune Condition, on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. attention parents and grandparents if you're looking for a gift that's more than just a toy give them something that inspires confidence and adventure all year long give them a guardian bike the easiest safest and number one kids bike on the market yeah with usa made kids specific frames and patented safety technology kids are learning to ride in just one day with no training wheels needed.
It's why Guardian is America's favorite kids bike and the New York Times and Wirecutter's top pick three years in a row. That's right. My daughter has a Guardian bike and she loves it. And that thing was really easy to put together. And get this, this holiday season, Guardian is offering their biggest deal of the year, over 40% in savings on all bikes, plus $100 in free accessories. Guardian bikes have become one of the most sought after gifts of the season and inventory is going fast. So don't wait. Join over a half a million families who've discovered the magic of Guardian. Visit guardianbikes.
com to shop now. The universe is full of mysteries, black holes, quantum physics, galaxies. On TikTok, millions of people are learning more about the universe around them every day. Scientists break down complex theories, demonstrate experiments, and connect dots between the cosmos and our daily lives. One scroll might reveal the concepts on the fabric of space-time, the next an optical illusion. It's discovery on a massive scale, where millions learn something new every day. You buy insurance for peace of mind, but the policies you trust often cause the biggest financial shocks. Every year, millions of claims are denied because people's policies quietly excluded what happened.
Insurers know every detail. Policyholders rarely do. So for just 27 cents a day, my policy advocate reads your policies and explains in plain language where you're vulnerable. They don't sell insurance. They give you the same understanding insurers have had for decades. Before you trust your policy to protect you, let My Policy Advocate tell you what it really says. Go to MyPolicyAdvocate.com. I also want to apologize to all the parents who listen with their six-year-olds. these days. Oh, my daughter's sick. She doesn't care about what we do. That's great. So it's still just flying over her head, right?
I mean, she didn't even listen. She liked Movie Crush a little bit. Well, some kids that are six listen. And hey, shout out to all of you guys listening. I know. I'm always, whenever I see that, whenever someone writes in that says their kid my daughter's age actually listens, I'm like, really? Oh, yes. This is my daughter. She loves it. Right. Yeah, and she voted in the last election too. I'm like, uh. My daughter likes to watch videos of kids playing with toys on YouTube kids. Is she into that now? Those are the worst videos. I'm starting to get on her now just in terms of taste. I'm like, hey, you can watch something, but watch something with a story that's good.
It's like, this is garbage. Yeah, haven't you ever seen Chips? She goes, I like it. I can totally see her saying it just like that. Very defiant and happy. That was a great impression. All right, publication bias is one. And we kind of poked around it earlier a little bit with the whole publisher perish mentality. Can I add something more to that real quick before we get into publication bias? You don't mind? Add something to what? To just talking about publication in general. Oh, yeah. So, I don't think that it's fully grasped by most people. It certainly wasn't by me until really diving into this, that the academic publishing industry has a stranglehold on science right now.
In a very similar effect that 24-hour cable news had on, like, journalism to where it was like it became this voracious beast that was willing to just spit out money constantly. Feed it. In exchange for, yeah, feed it. Give me more stories. Give me more. Give me more pundits. Give me, like, that was the rise of pundits. Pundits didn't really exist prior to that. They just hung out on the editorial pages of newspapers. And then 24-hour news came along, and there's not possibly enough news stories, like good news stories, to keep going for 24 hours. So you have to talk about the news stories and analyze them, and then you start getting into who's wrong and all that stuff.
The publishing industry is very much like that now where it's this beast that must be fed. And so there can't possibly be that many high-quality scientific papers. So scientific papers have just kind of dipped down in quality. And then one of the other things that the publishing industry has done is said, we really like studies that have results. They're called positive results where it turned up that you found a correlation between something or the compound you tried on that tumor shrunk the tumor. Those are what we're interested in. That whole furthering of science with positive and negative outcomes, just to say this did work, this doesn't work, don't bother trying it.
We don't care about that kind of stuff. And that's a huge issue for the scientific community. They have to get control of the publishing community again if they're going to come out from under this dark cloud. Oh, yeah. I mean, they found in 2010 a study about papers in social sciences especially were about two and a half, I'm sorry, 2.3 times more likely to show positive results than papers in physical sciences even. And so some bodies of research are even more apt to publish positive results. And that means if you're going, you know this going into your profession, and you know this going into your set of research.
And it's, you know, that's when it becomes sort of put up or shut up time as far as standing firm on doing good work, even if it doesn't get published. Right. Right, and so that confirmation bias can really come in where you start, hopefully inadvertently, but certainly not in all cases, inadvertently start cherry-picking data to get a positive outcome where there really wasn't one there before. Or you use a kind of a weird statistical method to suss out the correlation between the variables so that you can have a positive outcome. Because if you're not publishing papers, like your academic career is not progressing and you can actually like lose jobs.
So you need to be published. The publishing industry wants your paper, but they just want positive outcomes. So, a high-quality, well-designed, well-executed study that found a negative outcome to where they said, well, this compound we tried didn't actually shrink the tumor. That's going to be ignored in favor of a low-quality paper that found some compound that shrunk a tumor just because they like positive outcomes. It's ridiculous. Yeah, and I mean, that kind of goes hand-in-hand with the last one. You know, there's a lot of overlap with these and a lot that work sort of in concert with one another.
And file drawer bias is, you know, it is what it sounds like. It's like you got a negative outcome. And whether or not you were being funded by a company that definitely doesn't want that information getting out there, or if it's just as a result of it being less likely to be published because it doesn't have a positive outcome, you just stick it in the file drawer and it goes bye-bye. Right. And again, like part of the point of science and scientific publishing is to generate this body of knowledge. So if you're about to do a study, you can search and say, oh, somebody already tried the same exact thing and they found that it doesn't work.
I'm going to not try to reproduce that. I'm just going to not go with it and move on to try something else. It's a huge waste of resources otherwise. And then also, if you aren't publishing that kind of stuff, you're missing out on. Well, I mean, you're missing out on the real data if the bad data is file-drawed. Like you're missing out on the truth. You're missing out on the whole picture, right? And also, again, it's not just that, oh, the poor negative outcomes, they need to be included too. Yes, that's true. But you're also promoting positive outcome studies that actually aren't good studies.
There's this thing called the Proteus effect where the initial studies, these initial papers on a subject, In 70% of cases, a follow-up study that seeks to reproduce them can't reproduce them. They don't come to the same findings, the same conclusions, which suggests that a study was really terrible. If it can't be reproduced or if it's reproduced, somebody comes to a different finding, different conclusion. That's not a good study. So the idea of publishing positive and negative outcomes together would definitely kind of slow that whole crazy 24-hour news cycle positive outcome study. Yeah, I don't see how it's even legal to bury, not bury, but I guess just not even, just a file drawer, a study that included like a drug having negative effects.
And I know that Congress has stepped up to try and pass laws to, I think there was one in 2007 requiring researchers to report results of human studies of experimental treatments. And then they tried to strengthen that in 2016, basically this like, you know, even if your drug doesn't come to market, like we need to have these studies and the results. Like, how is it even legal? It seems like you're bearing and it's almost falsification. Well, it is for sure, because you're also like if you're if you're talking about studies where you have multiple studies on, say, one drug that's an antidepressant.
And all you're doing is publishing the ones that have positive outcomes for that antidepressant. And you're just not publishing the ones that that showed no outcomes or maybe even harm. then yeah that should be illegal especially when you're talking about something like an antidepressant or in the biomedical field but it's certainly unethical for any field of science in particular just bury the stuff you don't like that doesn't support your conclusion it's a kind of a meta form of confirmation bias just putting aside the stuff that doesn't fit your hypothesis or your worldview and just promoting the stuff that does that's right boo I saw one way around this is the Lancet, you know, the very respected medical journal.
I think it's British. The Lancet has taken to accepting papers based on the study design and methodology and goals. So, when you first plan your study and you have it all together before you ever start, that's when you would apply to have your paper published in the Lancet. And that's when they decide whether it's a high quality enough study to publish or not. So, then they're locked into publishing your study, whether your outcome is negative or positive. And it has the knock-on effect of the Lancet basically being like, hey, this is a trash study. We would never publish this. Don't even bother.
So it's saving funds. And then the high-quality studies are the ones that are going to get published. And then also the positive outcomes and the negative outcomes get published regardless because they have no idea what the outcome's going to be because they accept the paper before the study's even been conducted. I saw another thing that said that a paper would be more likely to get published in The Lancet if it had cool illustrations. That's right. That never hurts. Everybody knows that. That's not unethical. Especially in color. Just put up a few of those New Yorker cartoons in there. Forget about it.
Everybody loves those. You got anything else? I got nothing else. You know, this is a little soapboxy, but this is something that we believe in. It's kind of like our episode on the scientific method a little bit. Mm-hmm. I liked it too. Yeah. Thanks for doing it with me, man. Thank you for doing it with me. Thank you for squeezing my lemon. Sure. If you want to know more about scientific bias, there's a lot, fortunately, a lot of sites and great articles dedicated to rooting that stuff out and to make you a smarter consumer of science. And so go check that out and learn more about it. And since I said learn more about it, it means it's time for Listener Mail.
You know, sometimes the listener mail dovetails quite nicely with the topic, and that was the case today, with our inclusion on the media bias list, which was pretty exciting. Yeah, what an honor. You know, there's something called the media bias. Is it called the media bias list? I believe so. And what it does is it takes news outlets and newspapers and TV and stuff like that, and it's a big chart. where they're ranked according to how biased they are, kind of up, down, left, and right. And they included podcasts this year. They did, and we were on the list, and it was really kind of cool. We had a bunch of people write in, and this is from Nicholas Beto.
He said, I found this post while I was scrolling through Facebook and waiting for the NFL season to start. Add Fonts Media. Is it Fonts or Fontes? We should know this. I'm not sure. One of the two. I'm going to say Fonts. It's a watchdog organization known for the media bias chart. They do a media bias chart where they rank every news outlet's political bias, and in the recent update, they included you guys. And wouldn't you know it, the most politically fair piece of media you can possibly consume and all the known universe's stuff you should know. That is so cool. You guys say you're liberal, but until I heard Chuck outright state it, I didn't even know.
Wow. Well, I think it slips through there some. well yeah we're we're certainly human beings and we have our own biases but we definitely try to keep them in check as we try to and i think it's just been confirmed because they're not just like listen to a couple of shows and we see these guys seem okay like they really listen and they really rank people um yeah they probably saw that too or perhaps they listened to the north korea episode where josh suggested wolf blitzer apply hot paper clips to his center thighs while writing a nice piece on Trump's Korean relations. Hilarious. Either way, thank you guys for your fairness and hilarity all these years.
You're both the best. That is from Nicholas Beddo. Thanks a lot, Nicholas. Thanks to everybody who wrote in to say that they saw that. We appreciate it. And it was neat to see ourselves right in the middle of the rainbow. Love being in the middle of that rainbow. I do too, Chuck. It's nice and warm and cozy in there, isn't it? Yes. Well, if you want to get in touch with us like Nicholas and the gang did, you can send us an email to stuffpodcasts at iheartradio.com. Stuff You Should Know is a production of iHeartRadio. For more podcasts from iHeartRadio, visit the iHeartRadio app, Apple Podcasts, or wherever you listen to your favorite shows.
Living with an autoimmune condition isn't easy, and every journey is different. That's why Season 5 of Untold Stories, Life with a Severe Autoimmune Condition from Ruby Studio and Argenix, shares powerful firsthand stories from people with conditions like MG and CIDP. Hosted by Martine Hackett, these conversations dive into what resilience really looks like through setbacks, breakthroughs, and finding strength in community. Listen on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Attention parents and grandparents. If you're looking for a gift that's more than just a toy, give them something that inspires confidence and adventure all year long.
Give them a Guardian bike, the easiest, safest, and number one kid's bike on the market. Yeah, with USA-made kid's-specific frames and patented safety technology, kids are learning to ride in just one day with no training wheels needed. It's why Guardian is America's favorite kid's bike and the New York Times and Wirecutter's top pick three years in a row. That's right. My daughter has a Guardian bike, and she loves it, and that thing was really easy to put together. And get this, this holiday season, Guardian is offering their biggest deal of the year, over 40% in savings on all bikes, plus $100 in free accessories.
Guardian bikes have become one of the most sought-after gifts of the season, and inventory is going fast, so don't wait. Join over a half a million families who've discovered the magic of Guardian. Visit guardianbikes.com to shop now. Banking with Capital One helps you keep more money in your wallet, with no fees or minimums on checking accounts and no overdraft fees. Just ask the Capital One Bank Guy. It's pretty much all he talks about, in a good way. He'd also tell you that this podcast is his favorite podcast, too. Thanks, Capital One Bank Guy. What's in your wallet? Terms apply. See CapitalOne.
com slash bank. Capital One N-A, member F-D-I-C. Did you know that over 70% of people file for divorce without hiring a lawyer? Hello Divorce is built for them. I'm Erin Levine. I spent 20 years as a divorce attorney and I built Hello Divorce to give you the support I wish more people had. With over 15,000 divorces, Hello Divorce is 90% cheaper, two-thirds faster, and the highest rated online divorce platform on Trustpilot. Find your way forward at hellodivorce.com slash iHeart. This is an iHeart podcast. Guaranteed human.