Why experts are questioning two hyped antibody studies in coronavirus hotspots

Two studies that estimate people have contracted Covid-19 in California hotspots at dozens of times the rate that current counts indicate have made headlines across the US. But the studies have sparked a methodological dustup in the science world, raising questions about the statistical calculations and the real dangers of coronavirus.

The studies were published in the past week by Stanford University and the University of Southern California and they estimated how many people in Santa Clara and Los Angeles counties have contracted Covid-19 by examining the prevalence of antibodies in small samples of volunteers.

They were the first in what is expected to be a burgeoning field of research as governments around the world explore when to ease stay-at-home orders. But their release – as preliminary drafts, without going through a rigorous peer-review process – highlights the challenges of a complex field of research pressed to quickly provide information on a virus that is still not fully understood.

The studies

The Stanford researchers in Santa Clara county – which new findings indicate saw the first recorded coronavirus deaths in the US in February – estimate based on tests of 3,300 people that as much as 4.16% of the county’s population, as many as 81,000 people, had already contracted Covid-19 by early April, many without knowing it. At the high end, that would mean the case count in the county at the time of the study was 85 times higher than the recorded case count, which was only 956 cases.

5:58

The USC study in Los Angeles county has not been fully made available online yet. But according to the team’s press spokesperson, the researchers estimate based on 863 tests that as many as 5.6% of adults in the county, or 442,000 people, had already had the virus by early April. The number of confirmed Covid-19 cases in the county at the time was only 7,994.

If the estimates of both studies were proven accurate, this would mean coronavirus is not only more common, but also less deadly than scientists currently believe.

The controversies

The studies’ overall findings appeared to align with what scientists had long known: that Covid-19 cases are undercounted both because a shortage of tests means only the sickest patients are tested and because some of those infected never show symptoms.

But they also drew immediate controversy. Protesters advocating for the end of stay-at-home orders latched on to them to argue that the virus is only about as deadly as the common flu and that it is time for the US to reopen for business. Some critics questioned comments by one of the authors of the Santa Clara study, the Stanford medicine and epidemiology professor John Ioannidis, who used the results to promote the controversial view that the coronavirus is “not the apocalyptic problem we thought” and that societal lockdowns were an expensive and potentially deadly overreaction.

Well-known scientists expressed concerns over the types of error rates of tests used, the methodology of the Stanford analysis, and the lack of available details about the research in LA.

The studies, as the medical writer Lisa Krieger aptly noted in the San Jose Mercury News, had academics debating “sampling methods, false positives and Bayesian inferences with a furor reminiscent of the banning of @BabyYodaBaby”.

Both studies used an antibody test made by Premier Biotech company that has not been approved by the FDA and comes with an acknowledgment that it can record false positives.

Hundreds of antibody tests have emerged on the world market in recent weeks, including some that promise a result from a finger prick in just hours, an executive from the diagnostics and pharmaceutical company Roche told Reuters on Tuesday. None of them currently have FDA approval and some of them are “a disaster”, the Roche CEO, Severin Schwan, said.

Scientists also warn it hasn’t yet been definitively proven that getting Covid-19 once protects people from contracting it again. “We do not know if antibodies protect you and for how long,” Arthur Reingold, an epidemiology professor at UC Berkeley, told the Guardian last week.

Then there are concerns about the Stanford study’s sample and statistical analysis. The biggest criticism was that it estimated cases for the whole county’s population based on detecting only 50 positives out of 3,300 people sampled. And since the tests had a false positive rate in one assessment of two out of 371, critics argued all the Covid-19 cases detected by the tests in Santa Clara could conceivably have been false positives.

“I think the authors of the above-linked paper owe us all an apology,” wrote Andrew Gelman, director of the applied statistics center at Columbia University, who has written numerous books on teaching statistical methods. “We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error.”

The prominent Washington state genetics researcher Trevor Bedford said on Twitter he was glad to see antibody studies emerging but was “skeptical” of the high results. The author and biotech investor Peter Kolchinsky tweeted that the “flaws with this study could trick you into thinking that getting shot in the head has a low chance of killing you”.

The study was also criticized for recruiting its volunteers on Facebook, a method some critics charged could have induced some to participate in the study because they had had symptoms but were unable to get tested. Researchers say they attempted to screen for this by collecting information from participants on any recent symptoms, such as coughing or fever.

With just the summary available, the LA study has received less scrutiny. The study was small, successfully conducting just 846 tests. Participants were selected by using a balanced selection of candidates obtained from a marketing firm.

But experts said it was difficult to evaluate the findings without more information, urging the USC team this week to quickly make available a preprint – an early version of a study awaiting peer review. One scientist even started a hashtag: #whereisthepreprint.

Both the Stanford University team and the researchers at USC declined to respond to a request for comment.

Prevalence in society remains low

Dr George Rutherford, an epidemiology professor at the University of California, San Francisco (UCSF), told the Guardian he was not surprised to see that more people may have had the virus than official case counts are recording, but said the estimates in the studies may be high. “They may have overinterpreted their data,” Rutherford said.

In any case, Rutherford cautioned, even if the studies showed a higher number of cases than previously thought, “prevalence in society is still low”. With only a small percentage of people having had Covid-19, the virus could still easily spread widely and safety precautions remained important, he added.

“We’ve saved literally tens of thousands of lives in the Bay Area by acting early and doing what we did,” Rutherford said.

USCF researchers now hope to learn more from a study launched in northern California this week that aims to test all community members in the small, secluded town of Bolinas. Participants there will receive a PCR test, which looks for the presence of a viral infection, and an antibody test. A similar testing program is under way in San Francisco’s Mission District.