What you will learn
- The origins, values, and history of Sci-Fit
- Our approach to finding truth
- Why we should be careful with arguments based on subjective experiences, anecdotes, and authorities
- Interesting concepts such as survivorship bias
Sci-Fit launched in late 2016, inspired and helped by Greg Nuckols of Strengtheory (now StrongerbyScience). It was created by myself, Adam Tzur, because I wanted to establish a home for critical thinkers who value evidence from scientific research.
Early on, I was joined by Brandon Roberts who supported the approach an wanted to be part of the team. Since then, Sci-Fit has grown, and more people have joined our projects as contributors, reviewers, co-authors, etc. We're always open to new talent and believe that everyone has something to contribute.
It is not always easy to find your way in the fitness and nutrition spheres. You will come across contradictory opinions and conflicting advice. Inevitably, you're lead into this position: you become increasingly confused by all the information. To solve this issue, you're either forced to pick one of several sources you trust, or you must live in uncertainty to what is true. You will most likely pick a source, such as a person, guru, or organization to guide you. The source most likely has an opposition (alternative sources of information that disagree with them). The two camps (and others) then fight over who is right. 1This is an oversimplification, but you've probably experienced or observed something similar
I was in this position, for a very long time. I lived in uncertainty to what is true in regards to fitness and nutrition. For every question, there were 1000s of opinions and it was impossible for me to tell which one was correct.
Oftentimes, the sources who disagreed were equally authoritative, i.e. two professors, or two elite bodybuilders/athletes.
I wanted a way to find truth, when in an informational stalemate. Let's start with the most commonly used ways:
Opinions, subjective experiences, and authorities
Anyone can make a claim or have an opinion. However, we all know that claims can be correct or incorrect.
For example, if someone says "I think [the ketogenic diet/intermittent fasting/other diet] is best because [it makes me feel good]", you have a claim followed by a reason. The reason supports the claim, but is it sufficient? Can we conclude from this statement that the given diet is superior? No, and here is why:
- A single subjective experience does not apply to the entire population, only to the individual (it is not representative)
- Subjective experiences are affected by various biases (read more below)
- Your experience might change from day to day, week to week, month to month. In that case, does absolute truth also change alongside your experience?
- Everything is specific. When we say that a diet is "best", we're in very vague territory, because it leads to the question: best for what? We must break down general claims into specific claims. For example, a diet can be superior or inferior to other diets for outcomes such as hypertrophy, fat loss, mood, adherence/sustainability, side effects, cognition, general health, diseases management, etc. The list is only limited by your imagination and what scientific evidence is currently available.
Here is another example of an argument:
"A high protein diet is superior for building muscle compared to a low protein diet, because Dr. Meat (PhD) said so".
The first part of this sentence is well supported in the literature, but we should be critical to how the overall argument is presented; Is the claim true because an authority made it? What if another authority disagrees?
Ultimately, this type of reasoning cannot be sustained. They lead us into scenarios where we either are forced to fully rely on our own experiences, emotions, or authorities, to find truth. And since these methods are highly fallible, I do not support this approach. Instead, I elect to use another system:
The Basis for Sci-Fit
Sci-Fit uses critical thinking, logical reasoning, and statistical analysis based on scientific data. This system allows us to conclude what works or not, on a higher level than individual experiences. With the scientific approach, you ideally collect data as objectively as possible 2There are cases where scientists have lied, cheated, and doctored data. This method is therefore not perfect or infallible. And with statistics, you see whether the data applies to larger populations, beyond the sample in your study. However, studies and statistical approaches can be flawed or biased. Therefore, we also critically analyze studies.
The three pillars
Our philosophy and approach is based on three main pillars:
- Scientific data as the brick and mortar: It is no secret that Sci-Fit runs on data. It is central from the moment a new idea is conceived until we publish the final text. Our conclusions are always based on what the data say as a whole.
- Independence: We will never accept funding from companies or organizations. However, this does not preclude that we might sell our own products such as e-books, in the future. The goal is that the site should be as independent from the industry as possible. We want to answer questions about fitness and nutrition without having someone looking over our shoulder, or prodding us in a certain direction.
- Transparency and Sharing: We want to show studies and share their data. This is why you find data tables in our articles. We also create study collections for every topic we write about. Most of our articles have plain language summaries at the top of our articles. This way, it is easier for you to understand our conclusions, even if you are not an expert in the field.
How we work
When we want to answer a question, we first start with the scientific literature. We search for and identify as many relevant studies as possible. From the studies, we extract the relevant data which is then systematized and cleaned. We then analyze the studies and run statistical analyses of the data, and present it in a myriad of ways. The final goal is to give you a simple answer based on complex data. This is challenging, because the human body is complex, and there is rarely a cookie-cutter solution. This is partly because of limitations in study methodology, but also because scientific conclusions are based in probabilities 3Probabilistic causation.
Why not anecdotes and personal experience?
In some circles, personal experience is considered more important than scientific evidence. No doubt, Sci-Fit agrees that a combination of scientific data and real-world experience is best. However, we still focus our primary analysis on hard numbers. This brings me to one of my favorite quotes of all time:
"Both anecdotal and research evidence have to be viewed critically. It’s given that research exists on a continuum of high- to low-quality. However, it’s still stronger on the whole than anecdotal evidence, which is precluded from any scientific rigor, and is thus subject to more validity threats than research will ever be."
- Alan Aragon
Without going into too much detail, I will lay out some of the problems we face by relying on anecdotes:
1 - The data is not objectively collected by an external person
Typically, when people discuss their experience with food, the gym, training programs, diets, etc. they do so with a basis in their subjective experience. Subjective experience is certainly important (especially for things like pain), but it is also highly susceptible to various biases (such as selective memory, false memory, egocentric bias, and many others)
2 - Are the strongest/fastest/biggest athletes also the best sources of information?
Should we listen to their advice based on their achievements? This begs the question to what caused their achievements in the first place. Was it superior knowledge, or was it amazing genetics combined with steroids? How many athletes tried the same training program/diet but failed (survivorship bias)?
"[Placebos] have neurobiological underpinnings and actual effects on the brain and body. They are not just response biases."
4 - Limited ability to establish cause-effect
Let's say you try a supplement and the supplement makes you feel good. Was the supplement the cause of the feeling, or was it something else?
Assuming we were able to control for the placebo effect, how do you know that the feeling is not due to random chance? Everything in your life varies from day to day, such as your level of tiredness, feelings of happiness, etc. Some days you naturally feel better merely as a result of these natural variations around the mean. Without a control condition, you can't know if you're experiencing normal variation or placebo. And even if you could control for randomness and placebo, how do you know whether the result from your personal experience can be extrapolated to larger populations (other people)?
5 - No extrapolation
Extrapolate: "to predict by projecting past experience or known data (...)"
Anecdotes cannot be extrapolated beyond the individual or group of individuals.
6 - Survivorship bias
When people discuss diets and training online, are their experiences representative? If you find 100 people that love [x diet or training program], does that imply that the program is good? Some might say yes, if a coach has 100s of people saying his training program is awesome, it must be awesome. But, it is not so simple.
Below, you see an edited image from Truby et al., 2006 where people tried various diets. The people up top in red circles, lost a lot of weight. You would probably find them in various communities telling people about their amazing progress and how good their diets were. Now look at the green circles. These people, who were on the same diets, lost almost no weight, or in some cases gained weight. These people would probably not have the same emotional experience with the diet. After all, losing 1 kg weight will probably not give you the same feeling as losing 25 kg.
Survivorship bias is when you only look at the people who "made it", and ignore all those who failed or made poor progress. This is one reason why we need statistics and sufficient sample sizes so we can examine how people respond as individuals and also on average.
Individual data points and experiences are still useful
As shown in the graph above. Some people respond very well to a diet or a training program, some do not. This is called inter-individual variation and it can be pretty substantial (i.e. some people lose lean body mass after a 12-weeks of strength training).
If people are different, then the standard statistical approach of comparing group means is not always useful for individuals. Indeed, scientific conclusions are based on averages and do not necessarily apply to individuals. This is why it's nice to have a coach that has one foot in science and the other in real-world application.
We must always exercise caution when evaluating claims based on personal experience (and the same applies to scientific data, because it has its own limitations). Our conclusion are firmly based in comprehensive literature review and data analysis, but we also critically evaluate the evidence. After all, nothing is perfect.
Do you want to learn more about fitness and nutrition?
Sign up for the newsletter.