How to Fix Our Broken Relationship With COVID Math

Four rules to improve reporting about risk.

Throughout the pandemic, Americans have grappled with, and largely failed to make sense of, COVID-19 statistics. One major reason for this failure is that the public has found itself at the mercy of commentators who simultaneously report and interpret the math for them. Too often, these interpretations are skewed to support a narrative that resonates with their audiences, either painting a drastic scenario about the risks (school is dangerous for children!) or one that minimizes these same risks (COVID-19 is just another flu!).

It is essential that we use better, more thoughtful COVID-19 math so we can get an accurate idea of the real risks of COVID-19, and of the potential downsides of interventions. Nearly two years into this pandemic, we are still estimating risk as though it were March 2020. We are failing to acknowledge that we have a sizable amount of data telling us what the actual risks are and who is least and most at risk—if we would just do the math. For example, quarantine policies have removed thousands of “exposed” children and staff from school, even though very few—63 out of 30,000 quarantined, in recent data from Los Angeles United School District—subsequently tested positive. This is not a good way to balance harms and benefits.

An underlying cause of confusion is the fact that local data—essential for decision-making around COVID-19 risks—are not easily accessible or understandable to the public, including the media. If hospitals are full in Missouri, that does not mean the health system in Massachusetts is teetering on the brink of collapse.

In Oregon, for example, it is difficult, if not impossible (depending on what you are looking for) to access data from weeks prior on the Oregon Health Authority website. Important data are not updated in a timely fashion and, in the case of hospitalizations, are extremely difficult to interpret. Meanwhile, not enough states release data showing who is hospitalized in terms of age and vaccination status. Age, as we show below, is the most important influence on COVID-19 risk, both as a stand-alone risk factor and as it compounds other risk factors.

The problem of inadequate and inaccessible local data could be remedied relatively easily. Every state should promptly release hospitalization and mortality data—the two most important outcomes when it comes to COVID-19—in the form of simple tables and graphs. These should be stratified by vaccination status, age, race and ethnicity, local employment categories, and important comorbidities such as obesity, diabetes status, and heart disease.

Data presented in an easy-to-use way, so people can see precisely which risk category they and their loved ones fit into, would be enormously useful. Getting an accurate sense of the risk involved in normal activities such as going to school, a doctor’s office, or dinner at a friend’s house would allow individuals to more promptly return to doing things that are important to them at a level of risk they are comfortable with.

The media also play a crucial role in informing Americans’ perceptions of COVID-19 risk. A recent article in The Washington Post provides an excellent example of Americans’ statistical confusion. The article linked to, but did not adequately discuss, data on the very low risk of a serious COVID-19 outcome for unvaccinated children (who, the story fails to clarify, have, on average, a similar risk of severe disease and a lower risk of death from COVID-19 than their vaccinated parents). Reporters often fail to integrate and contextualize pertinent data even when they are available.

With this in mind, we have devised four rules to help identify and prevent common mistakes in reporting about COVID risks. We hope these guidelines can provide an important toolkit for improving everyone’s COVID math.

1. One size fits none: Don't frame risk in terms of a generic person. 

Specific demographic factors such as race and ethnicity, obesity, diabetes, and other comorbidities make an enormous difference in determining risk of a bad COVID-19 outcome. But the single most important risk factor, age, is often relegated to a few short lines in too much news coverage. Downplaying the huge decreased risk of mortality in children compared with mortality risk in the elderly is simply not factual reporting.

It can be tempting to frame risk using a generic person who is ageless and has no particular health status—but this makes accurate discussion of risk impossible. As The New York Times reported, in response to the question, “What are the chances somebody with COVID-19 must be hospitalized?” 41% of Democrats and 28% of Republicans surveyed answered that the risk was “over 50%.” Not only did answers differ widely by political affiliation, but a large swath of respondents was not even close to a correct answer.

Part of the misunderstanding on display here is that none of us identify as a generic “somebody.” The correct answer to this question depends greatly on age, and other risk factors, of the individual. The actual risk of COVID-19 hospitalization for a child under 18 who has COVID-19 is less than 0.2% per infection, or less than 1% per diagnosed case by current estimates; but it is 23% for an adult 65 or older, and even higher for those over 65 with comorbidities. Respondents to The New York Times poll entirely missed this distinction.

2. Place risk assessment in the context of other risks. 

This rule has two parts. The first is that discussions of COVID-19 risk need context. A number such as 2/100,000—the approximate current hospitalization rate for children in Oregon due to COVID-19—means nothing to most people. Road accidents and drowning kill many more children between 4 and 14 than COVID-19, yet we still let them get in cars and go to pools. Another example is that children are around 20 times more likely to contract COVID outside of school than in school. The media need to put COVID-19 numbers into perspective instead of quoting a stand-alone number that readers cannot adequately assess compared to other risks they are willing to accept.

Second, it’s important to remember that policies designed to mitigate against COVID often cause other risks to increase. Depression, obesity, eating disorders, opioid overdoses, and suicidality have all gone up markedly during the pandemic. The CDC’s own research showed recently that children who were already struggling with weight gain more than doubled their weight increase during the pandemic. The report pointed to school and sports closures as an important cause. This is one of many tragic consequences of our failure to take into account all harms, not just those caused by COVID-19, during this pandemic.

3. Report raw numbers instead of percentages. 

We often read that cases in children now make up over 20% of new infections. But news outlets often fail to point out that, as vaccination rates climb in older age groups and new infections fall in those groups, new cases in children will logically make up a greater proportion of all new infections.

Similarly, although the number of cases has gone up due to the increased transmissibility of the Delta variant, hospitalization rates per infection in children, an indicator of the severity of disease, have not. Using raw numbers instead of relative percentages when describing rises and declines in COVID-19 data is essential, as is correcting for proportions in the population of different age groups by using numbers per 100,000 of population. Presenting COVID-19 data without taking these two steps can badly distort reporting on surges—or declines, for that matter—and make them look more dramatic than they are. It obscures important information, such as the fact that Delta is not significantly more harmful to children than previous variants.

4.  Break down data by age category to avoid confounding variables.

While our first rule was to stop framing risk in general terms, our final rule relates to a specific problem that occurs when data are not broken down by risk factors.

Since May, when the Delta variant spread worldwide, much attention has been paid to the question of whether vaccines work against this new variant as well as they did against earlier variants. The data show that vaccine protection has certainly decreased, at least against mild or asymptomatic infection. But the degree to which vaccines still offer protection to the general population from severe disease requiring hospitalization is being underestimated—or at least, the full complexity of the picture risks being obscured—largely due to a statistical phenomenon called Simpson’s paradox.

Simpson’s paradox refers to a situation where, when you look at a single group of data, you draw one conclusion, but when you look at the same data after separating it into subgroups, you end up drawing the opposite or a different conclusion. In this case, Simpson’s paradox is making our vaccines appear less effective overall than they are in each individual age group.

This is because the vaccinated group has a lot of older people in it who are at high risk for hospitalization if they fall ill, and the unvaccinated group has a lot of younger people in it who are at low risk for hospitalization. This “lurking” variable of a much higher vaccination rate among the highest-risk group distorts the apparent protectiveness of vaccination against hospitalization.

When we separate the vaccinated and unvaccinated by age, this distortion is effectively removed. Take Oregon, where data relating to vaccination status of hospitalized people recently became available statewide. Oregon has the 11th highest proportion of older people in the U.S. As in every state, older people make up a high proportion of its vaccinated cohort. In addition, younger, healthier people make up a large share of the unvaccinated cohort. 

Table 1 (below) shows how immensely protective vaccination is against hospitalization with COVID-19, including against the Delta variant, when you stratify by age group. Using a dataset that lumps Oregon’s data across ages, if you are unvaccinated, you have a 7.8 times greater chance of being hospitalized with COVID-19 than if you are vaccinated. This is still a high level of protection, but 7.8 times is significantly lower than the protection level that vaccination offers in every individual age group under 60.

In 20-59 year-olds, vaccination is protective for hospitalization by a factor of 28. Breaking that group down further, vaccination protects people aged 30-39 by a factor of 38. If you are over 60, by contrast, vaccination protects from hospitalization by a factor of 8.0. This last number is much smaller than in younger groups, but still higher than the total effectiveness when you lump all ages together.

Table 1

---

Math—specifically, accurate and transparent epidemiological analyses—should have been a major ally in minimizing overall harm from COVID-19. Sadly, it was not.

The fact that the vast majority of children have multiple greater threats to their health and well-being than COVID-19, and that those threats to health worsened as we tried to shelter children from any risk of COVID, should have been quickly integrated into our public health plans. Instead, school and sports closure policies in many cases aggravated these threats, while failing to have any major impact on controlling COVID-19 spread. The fact that vaccines are incredibly effective—way more so than is often reported—at preventing hospitalizations and severe disease should have been constantly and consistently emphasized and re-emphasized.

Research has shown that people will make rational decisions when they have the right information. Cutting through the barrage of misinformation around vaccination, and understanding clearly who is and is not at high risk from COVID-19, will remain difficult unless we can do a better job at helping people get a handle on COVID math.

Leslie Bienen is a veterinarian, scientific editor, and professor of public health at the OHSU-Portland State School of Public Health.

Tracy Beth Høeg is a private practice physician in Northern California with a PhD in epidemiology and public health researching COVID-19-related topics in children.

Eric Happel is director of global strategy, capabilities, and excellence for Nike value stores.