Navigating the Wild West of COVID-19 antibody tests
BioCentury is providing this story for free given the urgent need for information about the COVID-19 crisis. For more analysis, sign up for our daily email.
The expectation that widespread COVID-19 antibody testing will be a cornerstone for reopening society has prompted a speedy, hands-off regulatory approach. But reports of poor performance by marketed tests suggest moving too quickly could result in individual and public health decisions that are based on unreliable results.
Dodging that bullet will require centralized resources, including broad access to the large numbers of patient samples needed to stringently validate the assays.
Tests that measure antibodies against the virus, known as serological tests, identify who has already been exposed and theoretically become immune.
Unlike molecular diagnostics that measure viral load, serological tests can’t be used to diagnose COVID-19 or determine whether people are at risk of infecting others or advancing to severe disease (see “Limits of Detection for the New Coronavirus”).
Instead, the assays are being positioned as a way to identify people who can safely return to work, especially in front-line healthcare settings; determine how and where COVID-19 is spreading, which could guide resource allocation decisions; and identify potential donors of convalescent sera.
FDA’s March 16th guidance on COVID-19 diagnostics opened the runway for serological tests to launch in the U.S. without FDA review under certain conditions. Such tests had already been made available by Chinese, Korean, Singaporean and European regulators weeks before.
“That’s really the impetus for why you’re seeing a flood of these tests coming into the market,” said Troy Hopps, point-of-care diagnostic leader at Becton Dickinson and Co. (NYSE:BDX), which co-launched a rapid serology test with BioMedomics Inc. on March 31.
FDA’s rationale for not requiring regulatory review is that the tests can’t be used to diagnose disease, are less complex than molecular diagnostics, and are needed as soon as possible.
“All the tools they would normally apply to qualifying and proving assays, they don’t have those tools available at the moment, there is no time,” said Andrew Levin, CEO and CSO of Kephera Diagnostics LLC.
Yet the risks posed by insufficiently sensitive and specific serological tests are arguably as serious as those posed by faulty diagnostics. In particular, false positives could lead people to resume normal activities while they are still vulnerable to COVID-19.
“Until you have those blood samples, you're shooting in the dark.”
Worryingly, reports from Europe suggest marketed serology tests are not accurate enough to guide decision-making, especially rapid dipstick tests that give results in 15-30 minutes, as these lack a signal amplification step used in the other major type of serology test: high-throughput enzyme-linked immunosorbent assays (ELISAs).
In a blog post Sunday, Regius Professor of Medicine Sir John Bell said a University of Oxford campaign to validate commercially available tests showed unacceptable rates of false positives and negatives. “None of the tests we have validated would meet the criteria for a good test as agreed with the MHRA (Medicines & Healthcare products Regulatory Agency),” he said, adding that Spanish authorities had returned kits due to poor test performance (see “Hurdle in COVID-19 Serology Testing”).
With little pre-market gate keeping by regulators, governments are now ramping up validation of commercialized assays.
According to Bell, the U.K. has gathered the tools and patient samples it needs to find an optimal test.
And this week, the U.S. announced NIH’s National Cancer Institute would spearhead a collaboration to validate serological tests, drawing on NCI’s expertise in monitoring antibody responses to vaccines against the oncogenic virus HPV (see “NCI Takes Lead on Serological Test Validation”).
Levin said the extent to which companies can validate their own assays is limited by the scarcity of banked COVID-19 patient blood samples, and the lack of centralized validation panels. “It’s been very difficult to get some kind of reading on how well these serological tests are working without having an adequate base of patient samples to work with.”
Making up for lost time
The tension between the dueling needs for urgency and control of COVID-19 testing is writ large in U.S. policy.
The Trump Administration has been playing catch-up after dissolving rules that restricted COVID-19 testing to public health labs using CDC kits later found to be hampered by flaws (see “Coronavirus Crisis Calls for Competence”).
A turning point was FDA’s Feb. 29 policy allowing high complexity, CLIA-certified laboratories to immediately begin testing for COVID-19 while pursuing Emergency Use Authorization (EUA); the agency may compel companies to remove tests from the market if it finds issues down the line.
FDA opened the door wider on March 16, saying labs could seek authorization for COVID-19 testing from state governments instead of via an EUA from FDA.
“Some firms are falsely claiming that their serological tests are FDA approved.”
Of the 32 tests now authorized by FDA, most diagnose active infections by detecting viral genomic sequences using reverse-transcription polymerase chain reaction (RT-PCR) or non-thermocycling probes.
Reports of negative diagnoses in people with COVID-19 symptoms and other conflicting results have created political pressure for more transparency about the performance of COVID-19 tests with EUA.
In an April 9 letter to FDA Commissioner Stephen Hahn, U.S. Representatives Lloyd Doggett (D-Texas) and Rosa DeLauro (D-Conn.) called on FDA to publicly release updated data from test manufacturers and other sources about the sensitivity and specificity of COVID-19 tests with EUA, a list of actions FDA has taken in response to the data, and a list of laboratories that failed to provide follow-up data to FDA before the 15 business day deadline.
While Cellex Inc. is the only company to receive EUA for a serological test against COVID-19, 90 test developers have taken advantage of the opportunity to launch COVID-19 serological assays without any agency review.
The number of unreviewed antibody tests is poised to grow even more quickly; in a briefing last week, White House Coronavirus Task Force Response Coordinator Deborah Birx urged universities to develop their own.
In its March guidance, FDA said it “does not intend to object to the distribution and use of serology tests to identify antibodies to SARS-CoV-2,” so long as the developers have validated the tests and included language saying the tests have not been reviewed by FDA and should not be used “as the sole basis to diagnose or exclude SARS-CoV-2 infection or to inform infection status.”
The agency has since had to issue a warning to bad actors.
In an April 7 statement, Hahn said that “some firms are falsely claiming that their serological tests are FDA approved or authorized, or falsely claiming that they can diagnose COVID-19. The FDA will take appropriate action against firms making false claims or marketing tests that are not accurate and reliable.”
The Trump Administration unveiled the NCI-led serology test validation program in a press conference Monday.
The timing to determine what constitutes an acceptable test is tight; U.S. assistant secretary for health and head of the public health service Brett Giroir said the U.S. anticipates it will have millions of tests available on the market by May.
Getting results that stick
Though immunoassays that screen patient blood for antiviral antibodies are a well-established diagnostic modality, getting their sensitivity and specificity right is less straightforward than for tests designed to detect viruses via genetic sequencing (see “COVID-19 Diagnostic Tech Tableau”).
Serological tests expose patient blood to target antigens on immobilized surfaces or nanoparticles, with the goal of trapping antibodies against the antigens. The trapped antibodies are detected by applying mAbs that recognize the Fc portion of patients’ IgM and IgG antibodies.
It is thought that detecting only IgM means an infection is still in progress, while detecting both IgM and IgG indicates a patient is either in the late stage of an infection, or has already cleared the virus. The antibodies are thought to arise 10-14 days after infection, peak at about day 28, and remain in the circulation for months afterward.
One challenge is choosing which viral antigens to display, and how to display them, in order to capture antibody responses from as many patients as possible and avoid false negatives.
“All the tools they would normally apply to qualifying and proving assays, they don't have those tools available at the moment.”
For example, some companies choose to display full-length, recombinantly expressed versions of viral proteins like the SARS-CoV-2 spike protein; others like Kephera are displaying chemically synthesized versions of specific viral protein epitopes.
Unlike tests that detect viral genomes, whose sensitivity can be assessed via analytical limit of detection (LoD) experiments, the sensitivity of serological tests can only be evaluated using patient samples.
Another challenge is making the tests specific enough that they don’t flag immune responses to irrelevant viruses, leading to false positives.
Variability in human immune responses, coupled with limited understanding of what SARS-CoV-2 epitopes are most relevant at different stages of the infection, makes it critical for test developers to access large numbers of patient samples in order to understand their accuracy.
“Until you have those blood samples, you’re shooting in the dark,” said Levin. He said companies typically need 1 mL patient samples early in test development, and only a drop or two per patient later on.
Acing the test
To establish confidence in serological tests, authorities should coordinate widespread banking of patient samples, and develop standardized validation panels that can be used to compare across assays.
Bell said validating tests requires positive control sera from recovered patients collected approximately 28 days after they became infected, as well as negative control samples from people who donated before the epidemic to screen for cross-reactivity with other coronaviruses. The Oxford team is also running tests on sera from a varied range of patients “to determine whether the tests can identify both low and high levels of antibodies,” he wrote.
Although confirmed cases in the U.S. now number in the hundreds of thousands, the country’s fragmented healthcare system and lack of banking directives from authorities means access to COVID-19 patient samples has been limited, said Levin.
He said test developers in China have had abundant access to local patient samples, but strict controls on how foreign parties use biological samples and data from Chinese individuals has kept those resources unavailable to others around the world (see “Border Security for Chinese Genomes”).
In a virtual advisory committee Thursday, Deputy Director Doug Lowy said NCI has asked hospitals in its network of cancer centers to contribute COVID-19 patient samples to the test validation initiative.
Levin said reference panels of COVID-19 patient sera would provide “public health agencies, FDA and users a way to compare tests on an even playing field.” Ideally these panels would consist of “several dozen to 100 samples” and simulate the diversity of patients in the field via samples corresponding to different stages and severities of diseases, he said. FDA and other agencies have created similar panels for other infectious diseases including HIV.
Regulators have not set cutoffs for what performance metrics COVID-19 serological tests should hit, and on how many samples.
Hopps said BioMedomics evaluated its test on over 500 samples, and has determined its sensitivity and specificity are “in the 90% range.”
Cellex’s FDA-authorized test has a sensitivity of 93.8% and a specificity is 95.6%; it was tested at two Chinese hospitals in a total of 128 and 250 patients that tested positive and negative for COVID-19 via RT-PCR, respectively. The company did not return requests for comment.
Levin thinks 100 confirmed positives and several hundred confirmed negatives are needed to robustly evaluate a test, and pointed to HIV tests as a benchmark.
“For HIV, for instance, serological test sensitivities are typically in the 98-100% range, and specificities well above 99%,” said Levin. “COVID-19 is also a life-threatening disease, so I would expect FDA and other public health organizations will want to see performance values for tests improve and approach those ranges.”
He noted the performance metrics companies initially report are not set in stone. “Given the paucity of available COVID-19 samples, the values being claimed for sensitivity and specificity for some tests may see changes when larger numbers are eventually tested.”
Further analysis of the coronavirus crisis can be found at https://www.biocentury.com/coronavirus.