In times of uncertainty, we’re easily overwhelmed by too much information and tend to just “trust” the source, but these moments are when being critical of everything is the most important.
Humanity is in a weird time right now (3/28/2020) due to a pandemic (COVID-19) and all the news outlets are flooding our minds with an endless amount of information. This mountain of information comes in two flavors, fear or hope… Both flavors tend to be on extreme opposite ends, with one saying “It’s the end of the world!” and the other saying “We’ve found a cure!”.
This can be emotionally taxing for anyone, but especially for those without the “critical lens” to filter through all this noise (useless information), uncovering what is actually signal (meaningful information). I like many of you assume science is “sound” and have never really questioned the headlines, studies, or random facts that are thrown at me by media outlets. That blind faith in science and scientific studies is what I’m aiming to break this week.
This week’s wandering is about two things.
- Uncovering the weaknesses in scientific studies
- Discovering simple tools to filter through the noise
Honestly, after this week’s research, my perspective on science has completely changed… And my hope is that after reading this post yours will do the same.
Removing the mystery
Most science and scientific studies are complete mysteries to me, which I’ve accepted for most of my life… In the past, I would treat the claims from a scientist as truth, assuming that they must be right… They’re “experts”! Sadly, this is far from the truth.
Just because someone is an expert doesn’t mean they’re not a human. We, humans, struggle with rationality, bias, and many other things, which you’ll see impacts all of us, even scientists.
So… There are many branches of science, but for this post, we’re focusing on biology and that’s thanks to this pandemic.
Biology is hard to control when compared to other branches of science (e.g. physics). For example, in the CERN supercollider, we’re able to control the number, speed, and direction of the particles we’re slamming together, but when it comes to humans this kind of control is impossible. For a human example, let’s say we’re trying to figure out if eating more red meat increases the chances of someone getting cancer. It’s almost impossible to test strictly for the impact of red meat due to other random variables impacting the person’s health… For instance, a person that eats more red meat could smoke, eat lots of sugar, and never work out, while a person who eats less red meat does the exact opposite.
With that said, I think it’s safe to start this post off assuming that most scientific studies are flawed or are incorrectly interpreted by the media when shoved into our faces.
Finding the flaw
When articles flaunt new cures for a disease or increased deadly outbreaks, it’s easy to ride that emotional rollercoaster, but building a critical lens that questions that claim is important. Not just to find the truth, but to level out your own emotional psychology.
Here are some general things you should think about when reading news related to health stuff…
- Always ask yourself “Is this relative or absolute risk?” – In most articles, you’ll see a journalist talking about how there was some massive increase or decrease in a person’s health based on a behavior or drug they’ve taken (e.g. drug reduces the incidence of cancer by 50%). But this is most likely “relative”, which basically means they’re comparing the increase or decrease relative to the numbers around it… The below image explains this through an example.
Randomized controlled trials (the flawed Gold Standard)
- Another important question is if this study was done through a randomized controlled trial (RCT). This means the scientist separated two or more groups randomly, while not letting anyone know which group was actually treated. This type of study is seen as the gold standard because it’s our best defense against human biases. But even when the gold standard is used, it’s important to immediately turn skeptical because they’re flawed as well. A big flaw in biology that impacts RCTs is “confounding” variables. We’ll use the below image to explain this…
We’ve got three variables – Ice cream sales, murder rates, and temperature. If we observe ice cream sales and murder rates increase together, then we might think that if people eat more ice cream they’ll start killing more people. What’s missing here is the “confounding” (or hidden) variable – temperature, which impacts the increase in murders more than eating ice cream.
Statistical vs. practical significance (and P-values)
- A 2016 study of 1.6 million biomedical abstracts showed that 96% of these papers had “statistically significant” results. If there are so many “significant” results then why aren’t we making more breakthroughs in health?! This study perfectly packages one of the largest problems in the world of research… P-values. When a researcher labels something as significant they’re checking to see if a specific number is below a certain threshold (usually 5%). But just because that number is low enough it doesn’t mean that this study is “practically” significant and useful for the world. I recommend you watch this video to understand more about how this number is fundamentally hurting science.
There are many flaws in the world of scientific research, but we’re not going to cover them all… So if you’re interested in diving deeper go here, here, & here.
Tips for finding the “signal”
Now… I don’t want to think that I’m completely bashing science because there’s been a lot of good that has come out of this world. We need science more than most areas of study. I just want you (and me) to realize that it’s not perfect and there’s plenty of room for improvement.
Now that we’re both on the same page and realize scientific research is needed, how do we actually read this stuff without going to sleep?
Reading a research paper is nothing like reading a novel – This is a mistake I’ve made time and time again. 🙂
There are many different ways to read a research paper and it mainly depends on the kind of science (physics, chemistry, tech, biology, etc.), as well as why you’re reading it in the first place (expert vs. regular person). Instead of boring you with all the different approaches (here, here, here, and here), I’m going to highlight a few general tips.
- Before you begin – Before I even start skimming through a research paper, I look to see who the researchers are to find out who they’re associated with… This is important. You might find some researchers working for or on behalf of biased organizations with agendas. An obvious example could be an oil company funding renewable energy research.
- Skip the abstract – This is a surprisingly useful tip. When reading a research paper it’s easy to go straight to the abstract to see what the researchers concluded, but that immediately biases your opinion. To avoid this bias, I’ve found skipping the abstract and heading right into the introduction really helpful.
- Skim – Don’t kill yourself by trying to understand every little detail in a paper, unless it’s absolutely needed. I recommend skimming over a paper quickly (5 – 10 minutes) on your first pass… Meaning, read the abstract, look over all the sections and sub-sections, glance at the graphs, and read the conclusion. After this first pass if you generally know what’s going on and if it is valid then move on with your life, but if there’s something you didn’t understand or want to dig deeper into that’s when you dedicate an hour to a more detailed second pass through.
I’ll leave with these three for now, but as I mentioned if you’re interested in knowing more, head over to the links I’ve listed above.
Don’t fool yourself
As this whole pandemic continues to take over our lives and minds, it’s important that we have the tools to combat this constant barrage of news.
Remember, even though science is really all we have to combat against this world of complexity, it still has its flaws. Keep a “critical lens” for all the info you’re letting into your head!
I’ll leave you with a quote from one of my favorite people to walk this earth… Richard Feynman.
“The first principle is that you must not fool yourself — and you are the easiest person to fool.”
