As FDA tightens reins on COVID-19 antibody testing, Roche unveils test specs via EUA
BioCentury is providing this story for free given the urgent need for information about the COVID-19 crisis. For more analysis, sign up for our daily email.
FDA’s reversal of its hands-off policy on antibody tests for the novel coronavirus, coupled with its authorization of a highly anticipated test from Roche, is raising the bar for COVID-19 serology. Yet accessing enough patient samples to minimize uncertainty in test sensitivity remains a challenge, even for the Swiss diagnostic company.
The agency’s updated guidance on COVID-19 tests, published Monday, forces commercial manufacturers to seek Emergency Use Authorization (EUA) for tests detecting patient antibodies against SARS-CoV-2.
These serological tests were previously allowed on the market without regulatory review, so long as test developers conducted their own validation studies, notified FDA, and provided certain disclaimers.
Commercial manufacturers are now expected to submit EUA requests containing that validation data within 10 business days from the date they notified FDA of their validation testing, or from the May 4 release of the new policy, whichever is later. High-complexity laboratories can still market laboratory-developed tests (LDTs) according to the original guidelines, but are encouraged to apply for EUA.
The move signals FDA’s acknowledgement that the risks of poorly performing serological tests are substantial enough to warrant regulatory scrutiny, even if the tests are not used to make clinical diagnoses, and are needed urgently.
“The careful balancing of risks and benefits has shifted from where it was in mid-March.”
Leaving individuals and institutions to differentiate between good and bad antibody tests was arguably an abdication of FDA’s responsibility. Decisions based on faulty serological tests could have catastrophic consequences, such as giving a false sense of security that leads vulnerable people to expose themselves to infection (see “FDA is Wrong on Self-Validation of COVID-19 Tests”).
The updated guidance contains templates to help serological test developers apply for EUA. These include recommended performance thresholds of 90% sensitivity and 95% specificity, indicating FDA has tightened its standards since it began granting EUAs for serological tests April 1 (see “Good Test Hunting”).
Roche’s EUA, granted Saturday, marked the pharma’s first reveal of its COVID-19 serological test’s performance. At 99.8% specificity and 100% sensitivity for samples collected at least 14 days after COVID-19 diagnosis via a molecular test, the pharma’s high-throughput antibody assay appears to outperform previously authorized tests.
Yet despite having more validation samples than any other serological test manufacturer with EUA, only about 1% of those came from COVID-19 patients, reflecting the ongoing need for widespread COVID-19 sample banking to thoroughly validate these tests (“Wild West of Antibody Tests”).
Roche expects to deliver “high double-digit millions of tests per month” (see “Roche to Scale Up Serology”).
Figure: Serological test performance
A COVID-19 antibody test from Roche (SIX:ROG; OTCQX:RHHBY) is the sixth serological test granted Emergency Use Authorization (EUA) by FDA to meet the agency's new sensitivity and specificity thresholds in internal validation studies. At least five other antibody tests given EUA do not meet these standards (not shown).
FDA is looking for at least 90% sensitivity/positive predictive agreement (PPA) and 95% specificity/negative predictive agreement (NPA). PPA and NPA measure agreement with a molecular diagnostic for COVID-19; error bars represent the 95% confidence interval for each metric.
The updated guidance was the second time FDA laid out criteria of overall 90% sensitivity and 95% specificity for serological tests, despite the agency having already approved tests that fall outside these bounds (see “Comparing COVID-19 Antibody Tests”).
But unlike FDA’s April 28 Letter of Authorization, which uses these values as strict cutoffs for other agencies to grant tests EUA under an “umbrella pathway,” the guidance gives FDA leeway to approve tests that meet lower cutoffs, depending on the number of samples studied, and the bounds of the 95% confidence intervals.
The guidance also specifies a minimum number of samples on which to gauge performance: at least 30 SARS-CoV-2 antibody-positive samples, and 75 antibody-negative samples from patients confirmed to not be infected with the virus.
Because Roche validated its test using 5,272 antibody-negative samples the pharma has tightly pinned down specificity -- the proportion of negative cases the test correctly identifies -- reporting a 95% confidence interval of 99.6-99.9%.
But the test’s sensitivity -- the proportion of positive cases it finds -- is more uncertain. The 100% sensitivity metric Roche reports is based on samples from 29 patients, resulting in a wider 95% confidence interval of 88.1-100%.
The pharma tested samples from a total of 69 symptomatic patients with confirmed SARS‑CoV‑2 infection; the other 40 patients had samples collected at earlier time points during infection. Antibodies are thought to arise 10-14 days after infection, peak at about day 28, and remain in the circulation for months afterward; in general, serological tests perform better when samples are collected at least two weeks after molecular diagnosis or symptom onset.
Among serological test developers with EUA, the only two to report test performance based on samples from more than 100 patients are Cellex Inc. and Autobio Diagnostics Co. Ltd., which both have headquarters in China, where the pandemic first took hold, and used samples collected from Chinese patients.
Strict controls on how foreign parties use biological samples and data from Chinese individuals has limited the availability of those resources to others around the world (see “Border Security for Chinese Genomes”).
U-turn for safety
According to FDA, its original serological test policy was not a mistake. Rather, the agency changed course because “the careful balancing of risks and benefits has shifted from where it was in mid-March,” said FDA’s Anand Shah and Jeffrey Shuren in a letter explaining the move.
Shah is deputy commissioner for medical and scientific affairs, and Shurn is director of FDA’s Center for Devices and Radiological Health (CDRH).
They said the regulatory flexibility granted by the March 16 guidance was necessary at the time, “given the nature of the public health emergency,” and that “early availability of serology tests has helped generate important information that can inform” their future use.
The reversal was partly due to “unscrupulous actors” who marketed fraudulent kits or made false claims about tests.
But another driver was FDA becoming aware that tests on the market were performing poorly, based on an independent test validation initiative led by NIH’s National Cancer Institute (see “NCI Takes Lead on Validation”).
The statement is at least the third to sound the alarm on the performance of marketed serological tests; University of Oxford Regius Professor of Medicine John Bell and Roche CEO Severin Schwan have both reported that marketed tests have fared poorly in independent validation experiments conducted by their teams (see “Assays Don’t Meet Criteria”).
Shah and Shuren said NCI has shared validation data from 13 test kits with FDA, and the agency will make the NCI results available “once FDA has reviewed and determined if any further actions are appropriate for those test kits prior to publication.”
High test performance is especially critical when case prevalence is low. The estimate for COVID-19 exposure in the U.S. is about 5% of the population.
Under those conditions, a positive result from a test with 95% sensitivity and 95% specificity can only be trusted half of the time; U.S. officials have suggested that using two independent tests sequentially could improve prediction accuracy (see “Sequential Testing as Route to Minimize False Positives”).
Further analysis of the coronavirus crisis can be found at https://www.biocentury.com/coronavirus.