Full disclosure: We consult professionally in the context of personal-injury litigation. One of us (SMD) has served on more than one occasion as an expert witness opposite Dr Audrius Plioplys.
On 19 June 2013, Dr Tom Frieden, Director of the Centers for Disease Control and Prevention (CDC), delivered the Keynote Address at the 46th Annual Society for Epidemiologic Research (SER) Meeting in Boston, Massachusetts. He began with a quip (paraphrasing): an epidemiologist is someone who loses sleep worrying about denominators. His point was well understood by his audience, including yours truly. Mortality rates, survival probabilities, incidence rates, incidence rate ratios, hazard ratios, odds ratios, and standardized mortality ratios are but a few of the commonly calculated epidemiologic measures that involve dividing one number (the numerator) by another (the denominator). Getting these calculations right can be critical, and when gotten wrong, the consequences can be embarrassing, or worse.
In this three part post, we will be taking a hard look at errors in calculations of survival probabilities in a study by Plioplys et al. 1998.1 As we shall see, the errors in this study were at their core errors in determining correct numerators and denominators in calculations of survival probabilities.
In Plioplys et al. 1998,1 the authors write:
"This study was done to determine survival rates in subpopulations of severely neurologically disabled children who reside in pediatric skilled nursing facilities and to compare these survival rates with those in previously published studies."
"Our results show substantially better survival rates than those previously reported."
They then conjecture as to the reasons for this apparent better survival:
"These improved results are most likely related to much more intense medical management of severely disabled children in skilled nursing facilities than at home or in other residential settings."
This sounds promising, and would seem to suggest that with "more intense medical management", children with severe neurological disabilities might live considerably longer, on average, than has been reported in a number of care settings around the world. There are a number of things amiss here, however, and in the end, few serious conclusions can be drawn from the Plioplys et al. study, except that the authors carried out flawed analyses rendering all estimated probabilities of survival erroneously high.
Before getting to the crux of the errors in Plioplys et al., however, we shall, in the first of this three part series, examine two earlier studies of the survival of neurologically-impaired children. These prior studies are the focus of much discussion and comparison in the Plioplys et al. study, and, although you would not know it from reading that study, these earlier studies contained significant errors themselves, rendering their estimated probabilities of survival erroneously low.
The first of these previous studies appeared in the New England Journal of Medicine in 1990.2 In this study by Eyman et al., survival probabilities, mortality rates, and life expectancies of children with neurological disabilities were reported, stratified by level of disability. Unfortunately, mortality rates were calculated erroneously. In a given period of time, a mortality rate is determined by counting deaths and exposure time (in person-years) during the period, and dividing (deaths ÷ exposure time). In Eyman et al. 1990, the authors counted deaths observed over a three year period but exposure time over a one year period. To repeat, deaths were counted over three years, while person-years of exposure were counted over a one year period. That's not right. They got the denominator very wrong. The result was that mortality rates were about three times higher than they should have been. This was true in each age category, and in each stratum of children considered (strata were formed according to level of disability). The unreasonably high mortality rates had the obvious ripple effect of biasing survival probabilities and life expectancies downward. Eventually the data were re-analyzed, and corrected results posted on the web site of Strauss & Shavelle, Inc. (the Life Expectancy Project) [http://www.lifeexpectancy.com]. This organization, or a prior iteration of it, was directed by Drs Eyman and Grossman at the time of their 1990 study2, and was taken over by Drs David Strauss and Robert Shavelle sometime in the early 1990s. (The Life Expectancy Project is now a registered trademark of Strauss & Shavelle, Inc.)
The second earlier study was a 1993 study, again by Eyman et al.3 This study similarly reported the long-term survival probabilities and mortality rates of persons with varying levels of severity of disability. The error in this study was more subtle than in Eyman et al. 1990,2 and it has been described in a letter to the editors of the journal in question.4 Briefly, children whose level of functioning changed significantly during the period of the study were excluded from further analyses. Because children who died young were less likely to have changed, they were preferentially included in the analyses. This, to repeat, is a somewhat more subtle error than that in the 1990 study,2 and in fact the authors have argued in the past that this was not an error at all, but simply a choice in the analysis: they were interested in knowing the survival probabilities of children who did not improve (or decline) in function.4 It is impossible, however, to identify those persons a priori, and impossible to know that those who died young, had they survived longer, might in fact not have been among them. Such an analysis is impossible to perform without introducing the negative bias described above. It is an odd choice at best, and one that leads to an analysis that is ultimately of little interest or utility.
The foregoing studies2,3 contained errors, but they were nevertheless groundbreaking in demonstrating (unequivocally, in spite of the errors) that mortality rates of children with neurological disabilities vary according to level of disability: the greater the deficits (especially in gross motor functioning and feeding ability) the higher the mortality rates. This has borne out in myriad subsequent studies from around the world, many of which used standard and accepted methods of analyses.
Now back to the 1998 study by Plioplys et al.1 The first thing about this study that one might quibble with is its focus on comparing its results with those in Eyman et al. 19902 or Eyman et al. 19933 without noting the errors in the earlier studies. Given that the survival probability estimates in Eyman et al.2,3 were biased downward, in some cases severely so, the most obvious explanation as to why the results in Plioplys et al.1 might have shown better survival ought to have been the errors in the prior studies. (This may seem to assume the analyses in Plioplys et al. were done correctly; they were not.) Of course, to be fair, it is possible that Plioplys et al. were not aware of the errors in either of the Emyan et al. studies. Dr Plioplys has since become aware of those errors and has written about them in a later article and a textbook chapter - we will address other issues in those in future blogs.
The analyses of survival probabilities reported by Plioplys et al. 19981 were systematically biased upward. But even supposing for the moment Plioplys et al. had done everything right, finding better survival than reported in Eyman et al. 1990 or 1993 should have led the authors to first note the very biased results in each of those previous studies. Instead, Plyoplys et al. suggest that, "There may be several reasons why our survival rates were better than those of Eyman et al. [referencing the 1993 study3]," hypothesizing that these reasons may have included: the later years of the Plioplys et al. study, with possible corresponding advances in medical care and attitudes toward the developmentally disabled; possible diversity and discrepant levels of quality of care across the 21 California Regional Centers from which the data used by Eyman et al. were derived; and the fact that all of the subjects in the Plyioplys et al. study were cared for in skilled nursing facilities, with 'round-the-clock access to registered nurses for the provision of needed medical care.
Of course, any of those factors could, hypothetically, have an impact on survival, mortality rates, and life expectancy. In fact, there is published evidence suggesting that children cared for in larger facilities in California (including the Regional Centers, Medical Facilities, etc.) have lower mortality rates (after controlling for important factors of independent mobility and feeding) than those of children cared for in their own homes or in smaller group homes.5-7 This may in part speak to the notion that "'round-the-clock care", including registered nursing care, may have some impact by comparison to settings where those factors are not universally present. But to focus on these possible explanations while ignoring the obvious issue of the biased analyses in Eyman 1990 and 19932,3 misses a very important point. Compounding this omission are the errors made by Plioplys et al. in their study,1 which resulted in a systematic bias in the opposite direction to that in Eyman et al.2,3 Ultimately these errors are the most likely explanation for the reported survival probabilities in Plioplys et al. 1998 appearing to be better than previously published results in Eyman et al.2,3 or elsewhere before or since Plioplys et al. 1998.1 We will elaborate on the errors in Plioplys et al. 1998 in Parts II and III of this blog.
- Plioplys AV, Kasnicka I, Lewis S, Moller D. Survival rates among children with severe neurologic disabilities. South Med J. 1998 Feb;91(2):161-72.
- Eyman RK, Grossman HJ, Chaney RH, Call TL. The life expectancy of profoundly handicapped people with mental retardation. N Engl J Med. 1990 Aug 30;323(9):584-9.
- Eyman RK, Olmstead CE, Grossman HJ, Call TL. Mortality and the acquisition of basic skills by children and adults with severe disabilities. Am J Dis Child. 1993 Feb;147(2):216-22.
- Grossman HJ, Eyman RK (with reply from Strauss DJ, Shavelle RM). Survival estimates of severely disabled children. Pediatr Neurol. 1998 Sep;19(3):243-4.
- Strauss D, Eyman RK, Grossman HJ. Predictors of mortality in children with severe mental retardation: the effect of placement. Am J Public Health. 1996 Oct;86(10):1422-9.
- Strauss D, Anderson TW, Shavelle R, Sheridan F, Trenkle S. Causes of death of persons with developmental disabilities: comparison of institutional and community residents. Ment Retard. 1998 Oct;36(5):386-91.
- Shavelle R, Strauss D. Mortality of persons with developmental disabilities after transfer into community care: a 1996 update. Am J Ment Retard. 1999 Mar;104(2):143-7.