Written by
Adam Tzur, Anthony Roberts and Brandon Roberts.

Editing and contributions by
Alex Leaf (Examine) and Brad Dieter.

Published: 16.01.2018 

Plain Language Summary

By reading this article, you will:

1. Understand what happens behind the scenes in keto science.
2. Get greater insight into how studies get funded.
3. Learn to critically question research without relying on black/white thinking.

What we did, and why

In part 1 of our ketogenic diet series, we collected and analyzed a large amount of ketogenic diet studies. On closer inspection, we discovered that many of the central keto researchers had direct connections to the keto industry. We also discovered that their conflicts of interest are not always disclosed.

Since transparency is very important for scientific integrity, we decided to take a closer look at the keto scientists, their books, companies, products, and funding sources.

This is the first analysis of its kind.

Funding sources and conflicts of interest

A conflict of interest (COI) means that a scientist may have something to gain by making a claim or by conducting a study. A common conflict of interest is who the scientist gets paid by (funding source), but it can be other things too.

Many people question research that has been funded by large organizations or industries (i.e. “Big pharma”, “Big dairy”, etc.). You’ve probably heard people say that industry funded research is biased (funding bias) and can’t be trusted. This claim has some scientific backing, but it varies from industry to industry.

Still, a funding source or conflict of interest does not automatically imply that the researchers are biased. This means that we can’t discount a ketogenic diet study just because the scientists are affiliated with keto companies. If we want to figure out if a ketogenic diet study is biased, we have to critically engage with:

• The study methodology
• How the study connects to the rest of the literature (is it an outlier?)
• The author backgrounds (business interests, personal opinions, values)
• Funding sources
• How the data was analyzed and interpreted

Only after analyzing these points can we make a decision about bias.

Central keto companies

There are two companies that connect to several of the biggest keto researchers. They are Virta Health and Prüvit. They sell products, services, employ keto scientists, and fund research.

Keto scientists and their COIs

• Researchers from all sides of the keto debate have COIs such as books, products they sell, company affiliations, industry funding, and more. This includes researchers who are not pro-keto.
• On the pro-keto side, many studies are conducted by central groups of keto scientists. Many of these researchers have industry connections and business interests such as selling keto books, keto supplements, and consulting services). In many cases, these conflicts of interest are not disclosed.

Central Keto Companies

Below you can see the connections between keto scientists and companies. We’ve only included companies with direct links to keto scientists.

Central keto companies

Virta Health

Owned by: Phinney, Volek, and Inkinen.

Associated researchers: Phinney, Volek, Wood

Virta Health launched in early 2017 with a mission to reverse type 2 diabetes. Virta Health was founded by Sami Inkinen, Dr. Stephen Phinney, and Dr. Jeff Volek using ~$37 million possibly invested by Inkinen himself after his experimentation with the ketogenic diet to treat his self-diagnosed prediabetes. The goal of Virta is to use personalized nutrition, coaching, and remote monitoring to reverse type 2 diabetes without medication.

Quote from CEO Sami Inkinen:

“Virta draws on the latest medical and nutrition science and advances in technology to address the cause of Type 2 diabetes, and deliver on-demand, personalized treatment and expert support on a near real-time basis.”

Virta Health has already collaborated with Indiana University to run one clinical trial using a ketogenic diet (McKenzie et al., 2017). They are also paired with Purdue University to offer their services to employees with type 2 diabetes. To see an update on their recent activities click here.

Prüvit

Owned by: Underwood, Farding, Lacore

Associated researchers: Wilson, Lowery, D'Agostino,

Sells: Ketone supplements. The flagship product is Keto//OS (Ketone Operating System).

According to the Ketogenic supplement reviews, Pruvit launched in 2012. They are located in Melissa, Texas where they filed as a domestic for-profit corporation in 2014. They are not accredited by the Better Business Bureau and have an F rating, although there are only a few reviews.

The Prüvit statement of policies and procedures can be found here.

Their compensation plan can be found is here.

Prüvit claim that they are doing keto research.

They recruit people as promoters of their products, similar to multilevel marketing:

“The Pruvit Compensation Plan is an exciting opportunity that rewards you for selling products and services and for sponsoring other participants who do the same. Although the opportunity is unlimited, individual results will vary depending on commitment levels and sales skills of each participant.” - Pruvit income disclaimer

Connection between Pruvit and keto researchers:

Noteworthy keto websites

Ketogenic.com

Owned by: Not stated (identity hidden during website registration)

Associated researchers: Jacob Wilson, Ryan Lowery, and Rachel Gregory

Business affiliations: Markets The Ketogenic Bible

My Keto Recipes

Owned by: Austin Miller (Independent Pruver)

Associated researchers: Ryan Lowery, Jacob Wilson

Business affiliations: Affiliated with Pruvit

Keto Scientists and their COIs

How we chose which scientists to investigate

Choosing scientists is a sensitive topic, because it is easy to single out specific scientists. On one hand, we can choose to only look at pro-keto researchers, in which case the analysis becomes biased because we are only critical of one side of the debate. On the other hand, if we include all scientists who have ever done keto research, we will have over a hundred people to investigate. The analysis would thus become muddied, unfocused, and filled with noise.

We’ve chosen a different approach - we’ve looked at the most prominent and/or outspoken authors from both sides of the keto debate. By “keto debate”, we are referring to the discussions on how ketogenic diets compare to other diets for losing body fat, building muscle, improving health, etc. We’ve also chosen researchers that have direct financial links to the industry, such as selling books about keto or being affiliated with keto companies.

There isn’t a sizable camp of “anti-keto researchers”, but there is a camp that favours the calories-in-calories-out model rather than the proposed metabolic advantage of keto.

However, most CICO scientists have not done keto research. This has lead to us looking into more “pro-keto” scientists than “pro-CICO” scientists. Why? Because there is a natural imbalance in the literature - there are many more outspoken keto promoting scientists, who have also done keto research. We may do a COI analysis for the CICO literature in the future.

If you feel like we’ve missed central pro-keto or pro-CICO researchers in our analysis, please send us an e-mail. It is required that the researcher in question has done keto related research and has made statements about keto or CICO.

Overview of keto scientists

Note that the descriptions below, such as “promotes keto”, “pro-keto” “pro-CICO” are not meant to be judgements, but descriptors. Pro-keto implies that an author is for the diet, but does not necessarily actively promote it.

Clustering of pro-keto authors

Clustering happens when several researchers form groups. These groups will tend to work on a specific topic (in this case, the ketogenic diet and ketosis). It’s possible that clustering could increase bias (Décombaz et al., 2003), because the researchers typically have a shared set of opinions.

In the context of this article, the researchers form clusters around what they believe in. You’d probably not see a CICO scientist such as Kevin Hall form a cluster with pro-keto scientists such as Wilson and Volek.

Birds of the same feather flock together.

One reason for why clustering could matter, is allegiance bias. This is when a scientist believes very strongly in the treatment he is researching (Munder et al, 2013; Dragioti et al., 2015). Several researchers are looking into the best way of fixing allegiance bias. Some say that we should “balance” allegiances (Wilson et al., 2011). In our context, this means including both pro-keto and pro-CICO scientists as authors on the same study.

Top 5 conflicts of interest

Prominent keto researchers typically have one or more of the following conflicts of interest:

• Keto business interests
• Selling keto books
• Working for keto companies
• Providing consulting services for a low-carb lifestyle
• Having studies funded by keto companies

In addition, scientists may:

• Not disclose conflicts of interest
• Mostly work with other pro-keto authors (clustering)

All of the above may suggest bias, but does not guarantee it.

Now, let's say there are two scientists.

Scientist 1 has done a keto study with lab materials sponsored by a keto company. He discloses the source of the materials.

Scientist 2 has several conflicts of interest, he mostly works in clusters, and only discloses some conflicts of interest.

Both scientists may be “sponsored by the industry”, but the two cases are clearly different. This is why nuance is so important.

Our conclusion is that we have more reason to be critical of the latter scientist, than the former. However, we would still need to analyze their studies to make a well-informed judgement. Jump down to Do Conflicts of Interest Imply Bias? for a deeper analysis.

Do Funding Sources Matter?

When industry-funded studies are published, people may say that they are biased. By this reasoning, only government or university funded studies are unbiased. Yet, is this true?

As we all know, scientific research is not free. Conducting an 8 week study with 30 participants is very costly, because you require not only a lab, but also equipment, supplies, tools, subjects, and scientists. The study must be planned, performed, and the data collected must then be analyzed using statistics software and visualized using graphics software. Finally, the study must be written, reviewed by all authors, peer-reviewed and edited before it can even be accepted for publication in a scientific journal. Most scientific journals also require large fees to publish.  

These things take time, effort, and are not cheap. Scientists must find ways to pay for their expenses.

How scientists get studies funded

Grant writing

Researchers can write applications to get funded by the government. These are called grants. Federally funded organizations such as the National Institute of Health (NIH) and National Science Foundation (NSF) have grant application cycles throughout the year.

The chance of getting a grant funded by the NIH/NSF is very low. The typical budget for a large research grant (R01) is $250,000 a year for up to 5 years. After accounting for salaries and cost of supplies or reagents, this leaves little money for studies outside of those in the submitted grant application. This often leads to a need for industry funding to supplement studies that the NIH or NSF do not fund. Furthermore, it is very difficult to get a grant funded to study exercise or nutrition in healthy people.

Industry funding

Companies can provide grants, tools, equipment, a lab, or materials (i.e. products) for a study.

Typically, industry funding has less peer-review ¹This means a panel of independent scientists reviews the funding application requirements than funding from the National Institute of Health (NIH) or National Science Foundation (NSF). However, this is not always the case.

Industry funding can result in very large grants or funding opportunities that propel research forward in cases where NIH funding is not enough. This is especially true for small groups of researchers.

Internal University funding

Universities can fund their own research. They usually fund small amounts (<200k) where the NIH/NSF give out larger grants. University funding sources can come from start up packages for faculty, “institute” grants, or other internal sources.

Self funding

In some cases where expensive equipment is not needed, scientists can fund their own studies. For example, they may be doing a case study and they already have free access to the equipment they need to use, such as a DXA machine.

Difficulties in getting funding

Getting studies funded without the industry providing some type of compensation can be difficult. For example, it is not uncommon for supplement companies to provide protein for studies (Antonio et al., 2016). Even some coaching services fund research. There are also large organizations who partner with companies to provide funds. These are all well-established and do not invalidate a study.

Funding bias

Funding bias or “funding effect” is the tendency of research outcomes to support
the best interests of the study sponsor (Krimsky, 2005. Krimsky, 2012; Lexchin, 2012). There have been multiple analysis to determine if specific fields have funding bias. More recently, this has become an issue in the nutrition field (Lucas, 2015).

Before we start our dive into the conflicts of interest, we have to answer some questions:

• Does funding from a company imply that the results of a study will be biased, or that a researcher is biased?
• Should a study be discarded on the basis of funding alone?
• Is a COI, such as a book or product, enough to discard the claims of a person or a study?

Here are arguments for both sides of the issue:

Why industry funding may not affect the results of a study

• The methods section of a study is more important: How rigorous was the study?
  • Most registered clinical trials require a statistical analysis plan to be submitted before the study begins. This can prevent data torturing [footnote explanation]. You can find most clinical trials at Clinicaltrials.gov which is a database of private and publicly funded clinical trials.
• A researcher risks his or her entire career by forging data/lying so that a study conforms to the expectations of a company or funder. If it is later discovered that the researchers forged data, he or she might lose their job, PhD, and reputation. There are many cases where this has in fact happened (see Retraction Watch, The Scientist 2016, The Scientist 2015)
• Many studies have multiple funding sources, including state grants and self-funding by large universities. Multiple funders could reduce the chance that a single funder influences the outcome.
• Several of the most important and influential nutrition researchers have performed research funded by the industry. Would you dismiss for example, Stuart Phillips, one of the most important protein researchers, because of prior funding?
• Following up on the point above, if you claim that a researcher is biased due to industry funding, the onus is on you to provide evidence that this is the case. Merely dismissing a person’s hard work due to a potential funding bias, is not intellectually or methodologically rigorous.
• Several researchers and journals report that funders have no hand in the design of the study or in the presentation of the results. Lying about this puts a scientists career at risk.
• Many journals require Conflict of Interest (COI) to be disclosed.

“Transparency and objectivity are essential in scientific research and the peer review process” - Elsevier

Why industry funding may affect the results of a study

• Conflicts of interest and funding sources are often not disclosed in scientific studies and books (Jagsi et al., 2009; Roseman et al., 2011; Piper et al., 2015; Rasmussen et al., 2015; Shnier et al., 2016; Resnik et al., 2017). What is the motivation for not disclosing COIs, especially when journals require it?
• Several reviews suggest that funding bias might influence study results (Bekelman et al., 2003; Lexchin et al., 2003; Huss et al., 2007; Jagsi et al., 2009; Lundh et al., 2012; Lexchin, 2012; Flacco et al., 2015; Schillinger et al., 2016; Lundh et al., 2017). Though some disagree (Buchkowsky and Jewesson, 2004; Mugambi et al., 2013; Naci et al., 2014; Chartres et al., 2016). It might depend on which industry and field we are discussing (some fields might be at greater risk of funding bias). It seems like pharmacology might be such a field (Buchkowsky and Jewesson, 2004; Lexchin, 2012; Lundh et al., 2017).
• There are studies undertaken on behalf of companies that produce, market, and sell the products being tested. A dead giveaway is that the study has 0 journal citations even if it’s several years old. Perhaps it is only cited by the same authors. Examples: Oben et al. (2008) followed by Anderson (2013). In this case, Oben's work has been cited primarily in class action lawsuits against companies selling the tested product (1, 2)."
  
• If the primary paper (the first paper reporting the findings of the study) has a statistical analysis plan that is different from the initially submitted one it is likely that there may have been “p-hunting”..

“Pharmaceutical companies fund the vast majority of the clinical research that is undertaken on medications but face a conflict of interest between producing good science and results that will enhance the sales of their products.” - Lexchin, 2012

Do Conflicts of Interests Imply Bias?

According to Elsevier, a conflict of interest is “When an investigator, author, editor, or reviewer has a financial/personal interest or belief that could affect his/her objectivity, or inappropriately influence his/her actions”. This means that there are both financial and non-financial conflicts of interest (Marcovitch, et al., 2010)

As we have discussed in the section above, we can make arguments for why industry funding matters, and why not. If we want to find out whether a conflict of interest matters, we have to be rigorous and not only analyze the study in question, but also the background of the researchers.

Factors that increase risk of bias

There are several situations where scientists are more likely to be biased. Here are some things to look out for.

Undisclosed conflicts of interest

“Everyone has competing interests; financial or private, or both. The main problem with competing interests is nondisclosure [5]. As with all competing interests, it is not possible to reliably judge our own biases. Instead, declaring them allows others to make informed judgments about whether the competing interests are relevant or not.” - PLoS Medicine, 2008

“Undeclared financial conflicts may seriously undermine the credibility of the journal, the authors, and the science itself.” - Elsevier

There isn't anything wrong in having competing interests. They may be near universal in medicine. Most doctors have had their lunch bought for them by a pharmaceutical company and write their prescriptions with pens provided by the industry. Many have been flown to very agreeable locations by the companies to listen to speakers, staying—at the companies' expense—at the best hotels. The problem with conflicts of interest is not declaring them. - Smith, 2002

Many conflicts of interest

This is when researcher(s) have multiple COIs. For example, they can be working for a company, sell books, promote products, provide consulting services, and more.

Selective data presentation and statistical analysis

Scientific data is typically collected and presented in numbers. Numbers may seem objective, but they can be manipulated to tell any story you would like. This is called data torturing (Mills, 1993).

“If you torture the data long enough, it will confess to anything” - Ronald Coase

Scientists can selectively choose, interpret, and present data (Mills, 1993). This is typically done in the conclusion, discussion, or the abstract of a study. This involves hiding or ignoring data that is “unwanted” and focusing on the data that supports the researcher’s conclusion.

“This sort of ‘within-study publication bias’  is usually known as outcome reporting bias or selective reporting bias, and may be one of the most substantial biases affecting results from individual studies (Chan 2005).” - Cochrane

This is why good data reporting and transparency is crucial. It’s also important that scientists follow the original statistical analysis plan submitted with the registration of the trial.

Poor study design

Researchers can design a study with a methodology that favors a desired outcome.

A typical example in the keto literature is to have the keto group eat more protein than control.

Protein improves satiety, helps build muscle, and slightly increases the Thermic Effect of Food (energy expenditure). Thus, it is easier to improve body composition on a high protein diet.

Ignoring alternative hypotheses

This occurs when researchers do not critically reflect on their claims, opinions, or products.

I.e. only presenting data supporting his or her conclusion.

Clustering

Clustering is when scientists work together over multiple projects. (Example: Décombaz et al., 2003).

Direct links to companies

Researchers directly working for a company with their own lab, only doing research on their product. Especially if they are not affiliated with a university.

Factors that decrease risk of bias

Trial registration

“Trial registration is the practice of publishing the methodology of clinical trials before they begin. This reduces problems from publication bias and selective reporting (such as choosing the methodology after the results are known)” - Wikipedia

Studies can be registered at Clinicaltrials.gov

Disclosure of conflicts of interest

All relevant COIs and funding sources are disclosed. This shows honesty and transparency, which are important values.

Solid data reporting and interpretation

Solid data reporting (i.e. reporting objective values, all relevant data shown). Being fair and balanced when writing the abstract and conclusions.

It is crucial to note that data can be gathered, interpreted, analyzed, and presented in many different ways. Minute changes in data analysis can completely change how it is interpreted.

I.e. doing significance tests for percentage changes rather than absolute values. Reporting performance per kg body weight, so that a decrease in bw means an increase in relative performance.

Study methodology

Solid study methodology that does not favor one group over the other. I.e. proper randomization, representativeness, sufficient study power, and more.

Independence of trial

Those that fund a study should not play a role in designing, analyzing or publishing the research. Some companies have their own labs where they actively design their own trials and study methodologies.

Should scientists always disclose their conflicts of interest?

A recent viewpoint was published discussing disclosure in nutrition research (Ioannidis & Trepanowski, 2017). The main idea was that authors should disclose all types of opinions and conflicts of interest (financial and nonfinancial). It is important for the health and fitness field to disclose conflicts of interests in books and other content since it can greatly influence a large number of people who are unaware of how to interpret the research themselves.

Read Quote

Further Reading

All of the resources below are free full texts, and are not hidden behind a paywall.

• Conflict of Interest in Science Communication: More than a Financial Issue (Marcovitch, et al., 2010)
• Making progress with competing interests (Smith, 2002)
• Conflict of Interest Factsheet (Elsevier, 2015)

Our Conflicts of Interest

This article was written and published without sponsorship from any company, individual, or organization. The Science of Fitness (SCI-FIT) receives no compensation or funding from any source, and is not affiliated with any other companies or organizations.

• Adam Tzur is the head of SCI-FIT. No COIs to report.
• Anthony Roberts is a journalist who writes articles on medium.com
• Brandon Roberts is a scientist and physique/strength coach. He works for The Strength Guys.