What you will learn
- The origins, values, and history of Sci-Fit
- Our approach to finding truth
- Why we should be careful with arguments based on subjective experiences and anecdotes
- Interesting concepts such as survivorship bias
Brief history
Sci-Fit launched in late 2016, inspired and helped by Greg Nuckols of Strengtheory (now StrongerbyScience). It was created by myself, Adam Tzur, because I wanted to establish a home for critical thinkers who value evidence from scientific research.
Early on, I was joined by Brandon Roberts who supported the approach an wanted to be part of the team. Since then, Sci-Fit has grown, and more people have joined our projects as contributors, reviewers, co-authors, etc. We're always open to new talent and believe that everyone has something to contribute.
The Basis for Sci-Fit
Sci-Fit uses critical thinking, logical reasoning, and statistical analysis of scientific data. This system allows us to conclude what works or not, on a higher level than individual experiences. With the scientific approach, you collect data as objectively as possible. And with statistics, you see whether the data applies to larger populations, beyond the sample in your study. However, studies and statistical approaches can be flawed or biased. Therefore, we also critically analyze studies.
The three pillars
Our philosophy and approach is based on three main pillars:
- Scientific data as the brick and mortar: It is no secret that Sci-Fit runs on data. It is central from the moment a new idea is conceived until we publish the final text. Our conclusions are always based on what the data say as a whole.
- Independence: We will never accept funding from companies or organizations. However, this does not preclude that we might sell our own products such as e-books, in the future. The goal is that the site should be as independent from the industry as possible. We want to answer questions about fitness and nutrition without having someone looking over our shoulder, or prodding us in a certain direction.
- Transparency and Sharing: We want to show studies and share their data. This is why you find data tables in our articles. We also create study collections for every topic we write about. Most of our articles have plain language summaries at the top of our articles. This way, it is easier for you to understand our conclusions, even if you are not an expert in the field.
How we work
When we want to answer a question, we first start with the scientific literature. We search for and identify as many relevant studies as possible. From the studies, we extract the relevant data which is then systematized and cleaned. We then analyze the studies and run statistical analyses of the data, and present it in a myriad of ways. The final goal is to give you a simple answer based on complex data. This is challenging, because the human body is complex, and there is rarely a cookie-cutter solution. This is partly because of limitations in study methodology, but also because scientific conclusions are based in probabilities 1"Probabilistic causation".
Why not anecdotes and personal experience?
In some circles, personal experience is considered more important than scientific evidence. No doubt, Sci-Fit agrees that a combination of scientific data and real-world experience is best. However, we focus our primary analysis on data from studies.
Here are some of the problems we face by relying on anecdotes:
1 - The data is not objectively collected by an external person
Typically, when people discuss their experience with food, the gym, training programs, diets, etc. they do so with a basis in their subjective experience. Subjective experience is certainly important (especially for things like pain), but it is also highly susceptible to various biases (such as selective memory, false memory, egocentric bias, and many others)
2 - Are the strongest, fastest or biggest athletes also the best sources of information?
Should we listen to their advice based on their achievements? This begs the question to what caused their achievements in the first place. Was it superior knowledge, was it amazing genetics, or perhaps steroids? How many athletes tried the same training program and diet but failed? Only focusing on those who made it past some selection process is known as survivorship bias.
3 - The placebo effect. 1, 2, 3, 4, 5
"[Placebos] have neurobiological underpinnings and actual effects on the brain and body. They are not just response biases."
4 - Limited ability to establish cause-effect
Let's say you try a supplement and the supplement makes you feel good. Was the supplement the cause of the feeling, or was it something else?
Assuming we were able to control for the placebo effect, how do you know that the feeling is not due to random chance? Everything in your life varies from day to day, such as your level of tiredness, feelings of happiness, etc. Some days you naturally feel better merely as a result of these natural variations around the mean. Without a control condition, you cannot know if you're experiencing normal variation or placebo. And even if you could control for randomness and placebo, how do you know whether the result from your personal experience can be extrapolated to larger populations (other people)?
5 - Extrapolation
Extrapolate: "to predict by projecting past experience or known data (...)"
Anecdotes cannot be extrapolated beyond the individual or group of individuals.
6 - Survivorship bias
When people discuss diets and training online, are their experiences representative? If you find 100 people that love [x diet or training program], does that imply that the program is good? Some might say yes, if a coach has 100s of people saying his training program is awesome, it must be awesome. But, it is not so simple.
Below, you see an edited image from Truby et al., 2006 where people tried various diets. The people up top in red circles, lost a lot of weight. You would probably find them in various communities telling people about their amazing progress and how good their diets were. Now look at the green circles. These people, who were on the same diets, lost almost no weight, or in some cases gained weight. These people would probably not have the same emotional experience with the diet. After all, losing 1 kg weight will probably not give you the same feeling as losing 25 kg.
Survivorship bias is when you only look at the people who "made it", and ignore all those who failed or made poor progress. This is one reason why we need statistics and sufficient sample sizes so we can examine how people respond as individuals and also on average.
Individual data points and experiences are still useful
As shown in the graph above. Some people respond very well to a diet or a training program, some do not. This is called inter-individual variation and it can be pretty substantial (i.e. some people lose lean body mass after a 12-weeks of strength training).
If people are different, then the standard statistical approach of comparing group means is not always useful for individuals. Indeed, scientific conclusions are based on averages and do not necessarily apply to individuals. This is why it's nice to have a coach that has one foot in science and the other in real-world application.
Conclusion
We must always exercise caution when evaluating claims based on personal experience (and the same applies to scientific data, because it has its own limitations). Our conclusion are firmly based in comprehensive literature review and data analysis, but we also critically evaluate the evidence.
Do you want to learn more about fitness and nutrition?
Sign up for the newsletter.