Critical analysis of Wilson et al’s. 2017 keto study

 

Summary


  • Wilson et al. just published a study where 25 trained lifters were divided into two groups: a keto group and a western diet group. Both groups did 9 weeks of strength training and results for hypertrophy, LBM, fat mass, etc. were collected.
  • As you will see below, the results were quite extraordinary for the keto group. Many people have now called the authors out for publishing unlikely results with multiple unreported conflicts of interests.
  • I’ve analyzed the study in-depth, and I conclude that there is a high risk of bias. The methodology is poor, and the results are highly unlikely to occur in trained lifters, in a caloric deficit, with a ketogenic diet.
  • Scroll down to the results to see tables and figures

If you find any inaccuracies or misrepresentations in this article, please contact me at adam@sci-fit.net or on Facebook.

 

Basic study facts


Title: “The Effects of Ketogenic Dieting on Body Composition, Strength, Power, and Hormonal Profiles in Resistance Training Males”

Here is a link to the study

Sign up for the newsletter and stay updated when new articles, videos, and e-books are published on SCI-FIT!

 

The study: facts and information


Hypothesis

“We hypothesized that the KD would decrease body fat to a greater extent than a WD group, while maintaining skeletal muscle hypertrophy, strength, and power.”

Methods

Subjects

There is little data on the subjects. The authors state that 30 subjects joined, but 5 dropped out due to various reasons. Participants needed to have a 1.5x bodyweight squat if they wanted to be included in the study.

“(…) subjects had an average squat of 1.56 ± 0.14 times their body weight”

Measurements

They used DXA for LBM and ultrasound for body composition measurements. I go into more detail on this further down in the article.

  • DXA: “The coefficient of variation (12) between body composition assessments was 1.5%.”
  • Ultrasound; “The coefficient of variation between muscle thickness assessments was 3.4%.”

Resistance training program

“All subjects were instructed to not perform any additional resistance training or endurance training throughout the duration of the study. For all criterion lifts, subjects were given a target repetition range based on a percentage of their 1RM”

Giving subjects a target repetition range based on 1RM testing is not ideal, because muscle endurance varies a lot from person to person (Dankel et al., 2016)

“Following the two-week diet adaptation period, all participants performed a 7-week high volume resistance training protocol followed by a 2 week taper”

Diet

“Carbohydrates were reintroduced to the subjects in the KD treatment from week 10 to week 11”.

Statistical analyses

  • Data are presented as mean±SD.
  • Confidence intervals not shown.
  • 2-way ANOVA with repeated measures (time and group interactions).
  • Post-hoc with Tukey’s adjustment
  • Data on LBM and fat mass were only presented in graphs, not as absolute numerical values

 

Results


Body weight

How does the keto group, on average, gain 5 kg of total body mass in one week (10->11)? They consumed the same amount of calories as they did in previous weeks.

Fat mass

How does the keto group, on average, gain 2 kg of fat mass in one week (10->11)? They consumed the same amount of calories as they did in previous weeks.

Lean Body Mass (LBM)

LBM was not reported in absolute numbers, but as percentages. I had to look at the graphs and try to estimate absolute LBM values. Standard deviation is not reported, only displayed. The graph below is an estimation based on my reading of the graph! After I read the baseline LBM from the graph, I used the reported percent changes to estimate absolute LBM changes.

Muscle thickness

Bench press strength

Keto group added ~9 lbs to their bench press from week 1 to 10.

Keto group added ~14 lbs to their bench press from week 10 to 11. (!)

WD added ~14.5 lbs to their bench press from week 1 to 10.

WD added ~2 lbs to their bench press from week 10 to 11.

 

Are the gains realistic?
Comparing results to previous research


Deficit gaining

The data presented in this study suggests that both keto and WD groups gained muscle mass and strength in a caloric deficit. I’ve previously reviewed the deficit gaining literature, and concluded that it is possible to gain LBM and strength in a deficit. However, the keto group in this study lost 4 kg of fat mass while gaining ~1.5 kg LBM and increasing their muscle thickness. This is very strange. When your glycogen stores deplete (as they do with a very low carb diet), you will “lose” LBM.

From my article on detraining:

When muscle glycogen stores shrink during detraining, we “lose” muscle mass, or more accurately, the water contents of our muscles decreases (Costill et al., 1985). On the flipside, you could even “trick” body estimates by going on a high carb diet to glycogen load your muscles (Nygren et al., 2001; Rouillier et al., 2015; Bone et al., 2016). Eating low-carb or going on a cut can lower your glycogen and water stores. This affects “wet” muscle mass (Hulmi et al., 2016)

This is why we have to take the atrophy and hypertrophy studies with a grain of salt. When beginners start exercising, their muscle glycogen stores will grow quickly and they will retain more water, as shown below:

 


What 16 weeks of strength training does for total body water in beginners.
Figure by
Ribero et al., 2014

It is very unlikely that trained lifters will have “glycogen/water gains” since these are some of the first adaptations the body makes.

Keto studies

I will have to do a complete review of the keto literature, but generally speaking, keto isn’t known for it’s ability to build mass quickly.

[Disclaimer: I have not reviewed the studies below and cannot speak to their accuracy or quality]

From Tinsley and Willoughby, 2016:

“Jabekk et al. (2010) reported no change in fat-free mass in subjects following a resistance training program in combination with a ketogenic diet, and Wood et al. (2012) reported a decrease.”

From Meirelles and Gomes, 2016:

“Twenty-one volunteers engaged in an eight-week progressive RT program three times per week were assigned to a CRD [< 30 g carbohydrate; n = 12] or a CONV [30% energy deficit n=9].“

(…)

“In conclusion, hypoenergetic diets combined with RT led to significant increases in muscle strength and were capable of maintaining muscle thicknesses in the upper and lower limbs of overweight and obese participants, regardless of the carbohydrate content of the diets.“

From Paoli et al., 2012:

“No significant differences were detected between VLCKD and WD in all strength tests. Significant differences were found in body weight and body composition: after VLCKD there was a decrease in body weight (from 69.6 ± 7.3 Kg to 68.0 ± 7.5 Kg) and fat mass (from 5.3 ± 1.3 Kg to 3.4 ± 0.8 Kg p < 0.001) with a non-significant increase in muscle mass.”

It is already hard to increase LBM in trained lifters, so when this study shows a LBM gain of 4.5 kg in the keto group after 7+2 weeks of RT, I’m a bit skeptical. Even the 1.5 kg LBM increase at week 10 is impressive. So are the gains for WD!

Remember, the keto group:

  1. was in a caloric deficit
  2. was on a ketogenic diet
  3. had trained lifters

All these factors restrict how much gains you can make. I’m surprised when studies on trained lifters even get a statistically significant result! Especially if they have low statistical power.

 

Conflicts of interest


I searched the study paper for “coi”, “interest”, “conflict”, “fund”, and “funding”. I found no mention of funding or conflicts or interests.

This alone does not mean that we should automatically discard the results of the study, as I’ve written about in this article! But, we should be aware of these COIs.

Shout-out to Anthony Roberts for letting me know about D’Agostino and Volek!

 

Problems and errors in the study


Lack of anthropometric data

At the start of studies, it is customary to place a table with data about height, weight, age, and so on. This is not done in this study. They only tell you the age of the participants in the results section, late in the paper. And their weight is only shown in the results table. Height is not reported!

This shows that the researchers do not follow the conventions of the literature. And why would you not?

Lack of absolute LBM and fat mass values

LBM/fat mass is not reported in absolute numbers, but as percent changes. I had to look at the graphs and try to estimate absolute LBM/FM values. Why didn’t they just put this in Table 3 with the rest of the data?

How long was the resistance training period?

  • Claim #1: “both groups participated in an 8 week supervised, periodized resistance training.”
  • Claim #2: “all participants performed a 7-week high volume resistance training protocol followed by a 2 week taper

So, which one is it? Was it 8 or 7+2 weeks? According to the volume loads in Figure 2 in the study, it was 7 weeks of high volume and 2 weeks of tapering.

Baseline strength, inclusion criteria, and standard deviations

“25 subjects had an average squat of 1.56 ± 0.14 times their body weight”

The numbers here are a little strange. The standard deviation is small. Furthermore, the SD goes below the minimum required level for squat strength. Per the inclusion criteria, it is at least 1.5x BW. Given a normal distribution, several subjects would fall below 1,5x. Thanks to Jorn Trommelen for pointing this out. Thanks to Greg Nuckols for running the statistical test to check the probabilities.

1.56 – 0.14 = 1.42

After a visual inspection of boxplots in order to identify any outliers, a normality test (i.e. Shapiro Wilk) confirmed the normality of the data

Mean age and training experience

The participants were this old, in years: “KD = 23.5 ± 4.5 vs. WD = 21.3 ± 3.7”

The participants also had an average of 5.5 years resistance training experience.

This suggests that, on average, the subjects lifted weights since they were ~18 years. This is kind of impressive, given everything that happens during these transition years. It’s not impossible, but it is impressive that the researchers originally found so many people at this age with this type of experience.

Citations

In the following sentence about DXA, Wilson et al., link to a study:

“The coefficient of variation (12) between body composition assessments was 1.5%.”

Citation (12) is entitled: “Effect of a Geriatric Consultation Team on Functional Status of Elderly Hospitalized Patients”.

I haven’t been able to figure out how it connects to DXA CV. Looks like they just linked a random study.

Weird data reporting

  • No CI
  • Graphs are poorly labeled
  • No absolute values reported for LBM and fat mass

25 or 26 participants?

A poster-presentation from the study was published in 2014. In the presentation, it is stated that 26 participants were part of the study. However, the 2017 publication states that 25 participants were in the final analysis.

Thanks to Philip Bloem and Jorn Trommelen for pointing this out!

Bias in abstract

The abstract is formulated in such a way that it selectively interprets the data and favors keto.

 

Can we trust the results?


Accuracy of DXA and ultrasound

I’m not going to go into a whole discussion of the accuracy of these body comp measurement tools. Sufficient to say, all measurement tools (even MRI) are affected by carb intake. Muscle glycogen loading and depletion affect the results (Nygren et al., 2001; Rouillier et al., 2015; Bone et al., 2016). Indeed, you can “trick” DXA by glycogen loading before measurements. Like they did in this study from week 10 to 11.

If you want to learn more about glycogen and muscle mass, check out my article:
The Science of Detraining: How long you can take a break from the gym before you lose muscle mass, strength, and endurance

You should also check out James Krieger’s articles on DXA, BIA, etc. Here’s one of his articles from weightology: Cheat Your Body Fat Test

 

Huge LBM and muscle thickness gains

As discussed under “Are the gains realistic?”, lifters with an average of 5.5 years of lifting experience generally don’t make quick gains in muscle or strength. Especially if they are on a low-carb diet (low glycogen) and are eating at a deficit.

Reputation

Wilson et al., have a reputation among some experts for publishing results that are extraordinary and unusual. And they don’t disclose COI.

Check out this letter to the editor by Stuart M. Phillips, Alan A. Aragon, Shawn M. Arent, Ben Esgro, D. Lee Hamilton, Eric R. Helms, Menno Henselmans, Jeremy P. Loenneke, Kevser Ermin, Layne Norton, Michael Ormsbee, Brad J. Schoenfeld, Matthew Vukovich, Colin Wilborn, and Darryn Willoughby.

These top authors are highly skeptical of Wilson et al’s research.

Conflict of interest

Multiple important COIs for Wilson et al. were not mentioned in the paper! Why not?

Reporting COI should be standard in 2017 (!).

I’m very disappointed with the National Strength and Conditioning Association and the Journal of Strength and Conditioning Research for not making the authors report their COIs.

 

Should we reject the paper?


First, I want to state that we should reject or accept a paper based on its quality. This means we must read beyond the abstract and conflicts of interest. We must dig into the meat and bones of the paper. The more errors and strange data, the more likely it is that something is off.

My conclusion is that we should reject the paper, for the following reasons:

  • There are too many errors and problems with the study, the data, the reporting, interpretation…
  • The researchers do not follow the conventions of the literature
  • Strange study design favors the keto group
  • The results are highly unlikely, when compared to previous research
  • The abstract only reports some results and it is biased towards the keto group
  • Potential conflicts of interest which are not disclosed
  • The authors already have a reputation for publishing unbelieveable results

Is there anything we can conclude from this study?

If we ignore the mess that is weeks 10 to 11, the study suggests that ketogenic diets might be worse for LBM and strength. This is speculative, and it’s hard to trust data coming from these authors at this point.

 

Limitations


It should be said that I’ve been very nit-picky in my analysis of this study. Very few studies are so good that they would be able to survive such scrutiny. In fact, it is said that we should be somewhat lenient when criticizing studies, because researchers are often constrained by financial resources and other limitations.

Furthermore, it can be argued that the carb refeed was necessary so that both groups had equal amounts of glycogen when the testing was done. This would give us fairer results, than comparing glycogen depleted subjects to subjects with normal glycogen levels. However, there are some issues.

  1. A one-week carb refeed might lead to glycogen supercompensation and excessive water retention. DXA is highly sensitive to changes in water weight, and it is generally not recommended to make large dietary changes right before a measurement. To fix this, the study would have needed either a 4C measurement analysis that could account for water weight, or the carb refeed would have had to occur over a longer period of time (i.e. 2-3 weeks). Since they didn’t use 4C to check water weight, I don’t think they should have done a refeed. This is because LBM and muscle thickness measurements are strongly confounded.
  2. Is the refeed truly justified in terms of external validity / ecological validity? From the abstract, it is clear that the authors are making claims about external validity (the usefulness of keto for lifters). If they just did carb refeed to check how DXA and ultrasound measurements are affected, then it wouldn’t be much of an issue. There is, however, a problem when they interpret the findings the way they do.

This study doesn’t have one or two small issues; there are multiple, large issues and the authors already have a poor reputation for possibly falsifying data. A nit-picky critical analysis is therefore justified.