Startling coronavirus antibody study...

Experts question results of startling Santa Clara coronavirus antibody study

By Eric Ting

The first large-scale COVID-19 antibody test in Santa Clara County suggests that the number of infections in the county has been underreported by a factor of 50 to 85, a startling result that has a number of implications.

In early April, Stanford University researchers conducted an antibody test of 3,300 residents recruited by Facebook ads, with participants selected based on their age, race, gender and zip code to put together a sample that was representative of the county's population. Researchers found a raw, unadjusted antibody prevalence of 1.5 percent, which was scaled up to 2.5-4.2 percent when adjusting for population and test performance characteristics.

If the numbers in the non-peer-reviewed study hold, that means 48,000 to 81,000 Santa Clara County residents have already been infected, and the true hospitalization and mortality rates are much lower than current estimates. The Stanford researchers projected a true mortality rate in the county between .12 and .20 based off their estimates and the lag time between infection and death.

The study results were released Friday, and a number of outside researchers have had time to review the preliminary study and its methodology. While all agree with the general takeaway that infections are vastly underreported, most believe the underreporting is not off by a factor as high as 50 to 85.

Dr. George Rutherford, an epidemiologist at UCSF, highlighted the fact that the antibody test the researchers used was not FDA-approved, as very few antibody tests have received approval to this point. The Stanford researchers acknowledged as much in the study and used test performance weights to scale results, but Rutherford was skeptical of these weights, as well as the population weights the researchers used. Instead, Rutherford believes we should just look at the raw antibody prevalence percentage of 1.5 percent.

"At end of day, the percent positive for antibodies was 1.5 percent," he said. "I don’t know what to make of the original sample, I don't know what to make of their adjustments for laboratory tests or the general population weight. I walk away thinking they found 1.5 percent of people have antibodies. They're smart as whips but felt crushed to get this out quickly, which is understandable."

Rutherford added that a 1.5 percent antibody prevalence is in line with what he would have expected.

"In the medical community, the thought is that one percent of people have been exposed in the Bay Area, and it's a little higher in Santa Clara County," he said.

Rutherford was not alone in questioning the validity of the weights the researchers used. Dr. Natalie E. Dean, a professor of biostatistics at the University of Florida, tweeted that she had concerns with the adjustments made for clustering (some participants brought children and other household members with them to the test) and test characteristics (researchers assume no false positives but some false negatives).

"Having had experience with these types of weighted surveys, I am always a little skeptical when the weighted result is very different from the unweighted result," she wrote in a Twitter thread. "Here, nearly double. This can be due to a few highly influential observations. Weights can be wonky."

When using the unweighted antibody prevalence of 1.5 percent, that translates to a total of 28,920 residents in the county that have been infected. Since the county had reported just under 1,000 cases at the time the study was conducted, the unweighted antibody number suggests infections are underreported by a factor of 30, and not the factor of 50 to 85 the weighted figures suggest.

Underreporting by a factor of 30 is still significant, as it once again substantially lowers the mortality rate. When using the researchers' estimates for deaths through April 22 (100) and the unweighted antibody figures, the true mortality rate in Santa Clara County becomes .35 — a figure almost identical to the "true" mortality rate calculated following antibody tests in a hard-hit German town. A .35 mortality rate is still nothing to sniff at, however, as it is three-to-four times deadlier than the seasonal flu (with a mortality rate of .1) and can be significantly higher for elderly residents and individuals with underlying conditions.

However, many question if the 1.5 percent prevalence of antibodies was the result of a sampling bias that led to a disproportionate number of individuals with COVID-19 symptoms participating in the study.

In a peer review of the antibody study published to Medium, former Stanford lecturer Balaji Srinivasan, who specializes in statistics and computational biology, argued the selection methods could have overstated the prevalence of the disease in the county.

"What if their group of participants was enriched for positives relative to the general population?" He writes. "What if their participants had a much higher rate of COVID-19 than normal? As a reductio ad absurdum, if you went to a hospital and tested all recovering patients for COVID-19 antibodies, you’d probably get a very high percentage of positives. You wouldn’t be able to generalize from that to say that much of the general population had already gotten the illness."

He cites the region's limited testing as something that could drive individuals who likely had the virus to use the antibody test as a way to confirm their suspicions. If that's the case, the sample is not truly random or representative of the entire county.

"After all, in the Bay Area in early April, it was really hard to get a test for people with mild symptoms or exposure," he notes. "So people who thought they were exposed or symptomatic may have signed up for the study to get access to a free COVID-19 test they could get no other way. We only need ~50 out of 3,330 to exhibit this behavior."

In other words, a factor of 30 may also be too high.

It is well-known the United States is underreporting the number of total infections, but there are wildly different estimations for what the factor of underreporting actually is.

Trevor Bedford, an epidemiologist at Fred Hutchinson Cancer Research Center in Seattle and one of the leading voices on the United States coronavirus outbreak, estimates the country is only finding between one in 10 or one in 20 infections, which is lower than the unadjusted Santa Clara figures would suggest. Meanwhile, Dr. Scott Morrow, the San Mateo County public health officer, guessed last week his county is underreporting infections by a factor of 21 to 35.

Whether we're off by a factor of 10, 20, 30, or 50 has massive implications for hospital surge planning as well as policy related to stay-at-home orders and social distancing. Further large-scale antibody tests should help provide clarity on the true factor of underreporting.