How NOT to do science: very low carbohydrate diets and Type 1 diabetes

Every now and then a piece of research is published that's just not very good quality. Despite the (presumably) best efforts of editors, peer reviewers, and the editorial boards of major journals, not all pieces of research are as rigorous as others.

And then sometimes research is published that is so egregiously poor that it's an embarrassment to the journal that published it, the peer reviewers who gave the manuscript the green light, and the major news media who cover it.

Today's article is one such and has been published in the June issue of Pediatrics (https://doi.org/10.1542/peds.2017-3349).

The authors surveyed a group of people who had been diagnosed with Type 1 diabetes and followed a very-low-carbohydrate diet (VLC diet), defined as less than 50 grams of carbohydrate per day. The authors found that those who followed that VLC diet had "measures of glycemic control in the near-normal range, low rates of hypoglycemia and other adverse events, and generally high levels of satisfaction with health and diabetes control." They go on to state that "these findings are without precedent among people with [Type 1 diabetes]."

Sounds like a miracle cure, doesn't it? There are just a few problems with the way the authors gathered the evidence, however, starting with who they chose to survey.

First, the participants in the survey were all members of a Facebook group dedicated to following a Very Low Carbohydrate Diet. That would seem reasonable enough, if they weren't dedicated to following the diet plan published in a book written by one of the authors of this article. Did the participants know that the author of their chosen diet was involved in the research? It doesn't matter if they did or not: the author should not have been involved in researching his own book. After all, he has a vested interest in finding that his diet works, doesn't he?

Second, I will give the authors credit for attempting to confirm the health information provided to them by the participants: they requested permission to contact the participants' physicians to obtain medical information regarding the participants' glycemic control as well as other health information. The problem is that the authors contacted 182 providers and received records from just 97 of them: 53%. Another 238 participants promised to provide written medical records from their provider, but only 101 actually did so. Essentially the authors took the word of fully half of the participants regarding their health outcomes and even their diagnosis of Type 1 diabetes.

The authors also surveyed the participants regarding their carbohydrate intake. Ordinarily this information would be obtained using detailed dietary questionnaires - ones that ask what foods people eat - so that the authors could better assess the actual amount of carbohydrates consumed. Instead, the authors asked the following questions (I downloaded the survey they used):

6. When did you start on a low carbohydrate diet?
How long have you been on the low carbohydrate diet? (select one)
* 12 months or more
* Less than 12 months
Month you started the low carbohydrate diet:
Jan-Dec (dropdown)
Year you started on a low carbohydrate diet: 1950-2016 (dropdown)
Have you followed the low carbohydrate diet in the past week? (select yes or no)
* yes
* no

7. Do you keep count of your carbohydrate intake? (select yes or no)
* yes
* no
If yes:
a. How many grams of carbohydrate do you plan to eat every day?
* _____ g (free text)
In the past week, how many days were you able to meet this goal?
* 0 days-7 days/don't know (dropdown)
Do you plan to eat a certain amount of carbohydrate for each individual meal? (select yes or no)
* yes
* no
If yes:
How many grams of carbohydrate do you plan to eat at each meal?
* breakfast: ___ g, lunch: ___ g, dinner: ___ g, typical snack: ___ g, all snacks combined: ___ g (free text)

Despite the fact that the participants were members of a low-carbohydrate-diet Facebook page and were thus likely to want to represent their carbohydrate intake in the most positive light, and despite the fact that people do not generally recall their food intake accurately unless they keep detailed and descriptive food diaries, the authors simply took the participants' word for how many carbohydrates they consumed in a day. Even after using a food-frequency questionnaire, most quality research attempts to validate the questionnaire they use by having a subgroup of the participants work with dietitians or other experts to evaluate their actual intake. None of that was attempted here.

So here we have a researcher involved in research on his own published book (a conflict of interest), using unreliable clinical data and using unreliable dietary data.

As a friend described it: "That's not research, that's a Buzzfeed quiz."

Pediatrics should be ashamed of itself for publishing something that doesn't come anywhere close to quality research.

What this means for you

It's unfortunate that even the venerable New York Times apparently failed to actually read the article and apply any sort of critical thinking to this research. As usual, if it sounds too good to be true, it usually isn't true.

First posted: May 16, 2018