Dr. Kay Jones’ analysis of the Environmental Protection Agency’s (EPA) annual mortality estimates from fine particles has resulted in the discovery of several problems. His study demonstrates that the EPA’s dependence on a key but highly flawed study, as well as its questionable use of this study’s data, has led it to exaggerate the health risk from PM2.5. Dr. Jones concludes that promulgation of the proposed standard is scientifically unwarranted.
A statistical error recently prompted the EPA to lower its estimate of the number of annual deaths from fine particulate matter (PM2.5) in the atmosphere for the third time, from 20,000 to 15,000 (see graph on next page). According to former Carter administration environmental advisor Dr. Kay Jones, who is credited with catching the statistical error, even this “correction” fails to show the true implications of the statistical error.1 In fact, the implications, revealed in a recent Citizens for a Sound Economy (CSE) Foundation report by Jones, suggest that the underlying science does not support the standard proposed by the agency, and that the EPA’s annual estimated mortality from PM2.5 should have been lowered to less than 1,000.2
Mean-median error. The EPA used mislabeled air quality data from an American Cancer Society (ACS) study to underpin its new standard.3 Air quality data in the ACS study was used by the EPA to determine a threshold — the level of PM2.5 exposure where mortality effects appeared noticeable. The agency concluded that these effects began at 15 micrograms per cubic meter (g/m3), where it then set the annual standard. But the data used came from a chart that was labeled as mean PM2.5 exposure levels, when it actually contained median data. Median exposure levels are not appropriate for use in standard setting, since they do not relate to what people breathe. Jones concluded the mean exposure level of PM was at least 25 percent above the EPA’s median level. As a result, the level of PM2.5 at which the EPA claimed mortality was noticeable in the key ACS study was not the reported 15 micrograms per cubic meter (g/m3), but 18 g/m3 or greater. This should have led the agency to increase the proposed level of the standard, which would also decrease the number of people exposed to PM levels beyond the EPA’s stated margin of safety. However, any PM2.5 standard would still remain unjustified in light of the inadequate mortality data in the ACS study. The agency responded to this error by decreasing the estimated annual mortality by 5,000 people, although it did not increase the level of the standard.
Phantom Benefits. The new mortality estimate, however, fails to consider yet another problem in the EPA’s calculation of annual mortality.4 The EPA claims reductions in annual mortality that cannot be attributed to the proposed standard. In the proposed rule, the agency stated the new standard provided an adequate margin of safety, meaning it believed exposure to levels of PM2.5 below the 15 g/m3 standard posed no significant health risk.5 But Jones found that the EPA’s mortality estimates wrongly included deaths from PM2.5 exposure below the 15 g/m3 standard. After recalculating annual mortality, without predicted deaths from people exposed to PM2.5 levels below the corrected 18.75 g/m3 standard, Jones estimated the number of lives at risk to be fewer than 1,000.
EPA’s Response. The EPA has rejected Dr. Jones’ analysis without explanation and has yet to specifically rebut the problems detailed in the CSE Foundation study. The agency has only said that the premises behind his recalculated mortality estimates were invalid stating, “EPA scientists believe that Dr. Jones’ critique of this particular study is inaccurate.”6 The agency is currently standing by its estimate of 15,000 annual premature deaths from PM2.5.
The EPA’s failed and flawed analysis of the ACS study. The agency’s mortality estimates must be further discounted due to its reliance on a controversial ACS study.7 The agency used one important statistic from this study, called the “relative risk,” in its calculation of the annual mortality from PM2.5. Relative risk is the statistic epidemiological studies produce to quantify the level of risk from a given substance. Jones found several problems with that study, which lead to an overstated relative risk associated with PM2.5 exposure. He found problems with the data in the study severe enough to disqualify it from further use in the standard setting process, but the agency never caught these problems. In discussions with EPA staff, Dr. Jones discovered that the ACS study never received the type of scrutiny it should have received, especially since the agency relied so heavily on its results. Apparently, EPA scientists found the ACS study’s appearance in the American Lung Association Journal enough to validate its results.
Jones, along with Michael Gough and Peter VanDornen of the Cato Institute, tried to reproduce the ACS study’s findings to assess the accuracy of the risk it reported. In their analysis, they found the study’s data was plagued by inaccurate and incomplete air quality and mortality data, which the EPA staff failed to note. Jones et al. attempted to obtain the omitted and inaccurate data to make the appropriate corrections, but they were denied access by the ACS study’s authors. Jones et al. proceeded to evaluate additional data using information from readily available EPA and Centers for Disease Control databases. Unfortunately, until the underlying data from the ACS study is publicly released, interested scientists will not be able to tell for sure the extent of the study’s inaccuracy. The following problems discovered by Jones et al., however, are an indication that the EPA must reconsider the study:
The ACS study did not include data from several major metropolitan areas. The data in the ACS study was skewed, because it did not truly represent national exposure to PM2.5. In fact, Jones et al. found that available data from major cities, such as New Orleans, Pasadena, New York, Pittsburgh, St. Louis, and San Antonio, were omitted from the study without explanation. Jones et al. attempted to obtain the missing data from the ACS study’s authors, but were refused release of the data. Instead, they attempted to reproduce the study using more recent and readily available data for 24 cities, which included data from six cities included in the ACS study. The association between PM2.5 and mortality was not statistically significant.
Faulty data from one city resulted in a further overestimation of the risk from PM2.5. The PM2.5 exposure data from one city, Ashland, Kentucky, was radically different than other cities in the study. EPA staffers should have viewed this city’s measured PM levels with skepticism, since this single data point was pivotal to the EPA’s interpretation of the data above the proposed standard. Jones et al. found the PM levels in Ashland suspiciously high and attempted to reconcile the exposure levels reported by ACS with another database. They found that the PM2.5 data for Ashland should have been corrected or thrown out.
The ACS study concluded that non-smokers are more at risk from PM2.5 exposure than smokers, which suggests the health risk from PM2.5 was inflated by inaccurate mortality data. This conclusion should have raised red flags with EPA staff, since smokers are much more susceptible to heart and lung disease. It is highly unlikely that more non-smokers died from PM exposure than smokers. This conclusion suggests that the ACS study was counting smokers as non-smokers. The cause of this spurious conclusion is likely self-reporting error. Instead of using data collection specialists, the ACS study estimated mortality using health history data collected from friends and relatives of the deceased who were unlikely to have accurately reported smoking history. Further, the ACS study reported that 22 percent of the people who participated in the study were smokers, which is much lower than the 32 percent of Americans who were active smokers at that time, according to the Center for Health Statistics.
The errors found by Jones et al. raise serious concerns about the problems with the EPA’s estimates of annual mortality reductions that would result from its proposed standards. These problems only add to the existing data gaps and uncertainties surrounding the EPA’s PM2.5 rule. Once these problems are considered, the validity of the new standard as a whole must be questioned. Some of the more fundamental problems include:
The EPA is relying on a statistical association between PM exposure and mortality that is so weak that many consider it insignificant;
The EPA has not identified a biological mechanism to show why PM2.5 is harmful;
Sufficient PM2.5 exposure data has yet to be collected. Without a nationwide monitoring network, there is not enough air quality data to answer the numerous questions about the health effects of PM2.5 exposure.
Conclusion. The questions raised by Jones et al. about the ACS study and the EPA’s analysis will not be fully answered without a substantial re-analysis of the underlying data, which the ACS study’s authors refuse to release. Now that only a few weeks remain before the standard is finalized, the EPA must halt the rulemaking process to allow questions about the underlying science to be answered. The unwillingness of the study’s authors to allow access to basic data is troubling; such secrecy is incompatible with the modern scientific process. Worse yet, this may be evidence that the political process of administrative rulemaking has made real science within the EPA difficult, if not impossible.
Ultimately, if the EPA is to prove the scientific validity of the PM2.5 standard, it must see that the data behind the ACS study is publicly released for further re-analysis. Only through an open, scientific debate between critics and supporters of the new standard will the public, which will pay billions of dollars for the new standard, be convinced that a genuine health risk from PM2.5 exists.
1 Dr. Jones is the head of Zephyr Consulting, a Seattle-based environmental consulting firm. Dr. Jones served as senior advisor on air quality at the President’s Council on Environmental Quality (1975-79) and also as a senior technical advisor and research manager for the Environmental Protection Agency. He has written the air quality status and trend chapters for eight of the last fourteen Presidential CEQ Environmental Annual Reports to Congress. Dr. Jones is an adjunct scholar at Citizens for a Sound Economy Foundation.
2 Kay Jones, Michael Gough, and Peter Van Doren, “Is EPA Misleading the Public About the Health Risks From PM 2.5, An Analysis of the Science Behind EPA’s PM 2.5 Standard, Addendum,” Citizens for a Sound Economy Foundation, May 12, 1997.
3 C. A. Pope, III, M. J. Thun, M. M. Namboodin, D. W. Dockery, and J. S. Evans, “Particulate Air Pollution as a Predictor of Mortality in a Prospective Study of U.S. Adults,” American Journal of Respiratory Critical Care Medicine, Vol. 151, 1995.
4 The EPA issued a press release on April 2, 1997, acknowledging the error and provided a revised estimate of benefits that reduced the number of lives saved annually from 20,000 down to 15,000.
5 EPA, Federal Register, Vol. 61, No. 241, December 13, 1996, p. 65660.
6 Katherine Rizzo, “Scientist raises new questions about deaths from dirty air,” Associated Press, May 12, 1997.
7 C. A. Pope, III, M. J. Thun, M. M. Namboodin, D. W. Dockery, and J. S. Evans, “Particulate Air Pollution as a Predictor of Mortality in a Prospective Study of U.S. Adults,” American Journal of Respiratory Critical Care Medicine, Vol. 151, 1995.