Medicine Methodology/Interpretation

Arguably making way too much of insufficient numbers— an instructive example from medicine

© 2016 Peter Free

An exaggerated health and age claim, unwarranted by the study’s actual numbers

From the University of Edinburgh:

Older people in Japan are more resistant to the impacts of disasters on their health than younger generations, a study suggests.

Research into the aftermath of the Fukushima earthquake, tsunami and subsequent nuclear meltdown [see here] found that the oldest were least likely to experience a deterioration of existing chronic conditions.

The study also reveals that the health of people living in the countryside was more resilient than that of urban dwellers following the triple disaster of 2011.

The findings are in contrast to previous studies that suggested that young, city-dwellers would be less susceptible to ill-health in the aftermath of a major disruptive event.

© 2016 University of Edinburgh, Elderly Japanese most resilient in wake of triple disaster, ed.ac.uk (19 July 2016) (extracts)

Looking at the source study itself, we see that this claim is arguably nonsense:

A retrospective cohort study was undertaken with 404 patients with diabetes at a public hospital in Minamisoma City, Fukushima Prefecture. Glycated haemoglobin (HbA1c) levels were measured in 2010, 2011 and 2012 to capture changes in glycaemic control postdisaster.

Age, sex, urban/rural residency, evacuation status and medication use were also assessed.

There was an overall deterioration in glycaemic control after the disaster, with the mean HbA1c rising from 6.77% in 2010 to 6.90% in 2012 . . . .

Rural residency was associated with a lower likelihood of deteriorating control . . . OR [odds ratio] 0.34 . . . compared with urban residency.

Older age . . . OR 0.95 . . . was also slightly protective against increased HbA1c. Evacuation and sex were not significant predictors.

Patients with diabetes who were affected by Japan’s triple disaster experienced a deterioration in their glycaemic control following the disasters.

The extent of this deterioration was mediated by sociodemographic factors, with rural residence and older age protective against the effects of the disaster on glycaemic control.

© 2016 Claire Leppold, Masaharu Tsubokura, Akihiko Ozaki, Shuhei Nomura, Yuki Shimada, Tomohiro Morita, Sae Ochi, Tetsuya Tanimoto, Masahiro Kami, Yukio Kanazawa, Tomoyoshi Oikawa, and Sarah Hill, Sociodemographic patterning of long-term diabetes mellitus control following Japan’s 3.11 triple disaster: a retrospective cohort study, BMJ Open 6:e011455, DOI: 10.1136/bmjopen-2016-011455 (07 July 2016) (at Abstract) (paragraphs split)

First, from a medical perspective, an increase in HbA1c from 6.77 to 6.90 percent is trivial

Someone is not going to keel over because they went from that starting to ending point.

This vacuity in medical magnitude means that the study was measuring something that turned out to be close to meaningless — and then made too grand claims from it.

Second, the claim that older people coped better with Fukushima is quantitatively questionable. Since when does a 5 percent reduction in risk justify a blanket claim of the “protective” type the paper puts forward? Especially in such a small sample of patients.

In the presence of an (apparently imaginary) previous study showing dramatically poorer coping by older people — as compared to younger ones (in the face of a similar disaster) — the 0.95 odds ratio would look really good. But no such previous findings are presented. As it stands, some of the oldsters outperformed some comparative youngsters by an alleged 5 percent. “BFD” in kids’ talk.

Admittedly, the 66 percent reduction in risk that allegedly showed up for those living in rural areas is interesting. But it too suffers from a lack of context. Does such a finding mirror other studies that were conducted for other medical conditions in other disasters? Is the assumption in this study that people who were evacuated from rural areas to cities suffered added stress? If so, why would this be surprising? Virtually nobody likes being uprooted because a group of reckless jerks liberally sprinkled deadly radiation around.

Last, the study only covered 3 years. Surely Fukushima, which remains ongoing in its radiation effects, continues to impact older people’s lives. Why was the research not extended? Why do we assume that 3 years of a trivially elevated HbA1c is meaningful in some medical way?

Then there is this final tidbit, too casually thrown in toward the end of the paper

The “oh, by the way” caveat:

A key limitation of our study was the considerable proportion of the study cohort lost to follow-up (20.8%).

© 2016 Claire Leppold, Masaharu Tsubokura, Akihiko Ozaki, Shuhei Nomura, Yuki Shimada, Tomohiro Morita, Sae Ochi, Tetsuya Tanimoto, Masahiro Kami, Yukio Kanazawa, Tomoyoshi Oikawa, and Sarah Hill, Sociodemographic patterning of long-term diabetes mellitus control following Japan’s 3.11 triple disaster: a retrospective cohort study, BMJ Open 6:e011455, DOI: 10.1136/bmjopen-2016-011455 (07 July 2016) (at 1st paragraph under section titled, Strengths and limitations of this study)

This means that the initial sample size was reduced from an already statistically marginal 404 patients to 320. These numbers are not sufficient to meaningful penetrate demographically, geographically, and medically complicated phenomena.

The moral? — The claim about older folks with health chronic conditions — coping meaningfully better with disaster than younger ones — was not justified

By either the study’s methodology or its numbers.

Which is not to say that the study is of no value. But simply to point out that its claims are pretty arguably too exaggerated at this point.