Roche, Abbott say sample collection differences hampered COVID-19 sensitivity in independent studies
Companies say the sensitivity of their COVID-19 tests have a lot to do with when samples are collected
Roche and Abbott each responded to independent reports that their COVID-19 tests are less sensitive than claimed with additional validation data supporting stronger performance and statements attributing discrepancies to sampling protocols.
Both companies pointed to sample collection time as a key variable, highlighting how emerging understanding of viral and immune kinetics for COVID-19 is shaping the way tests are evaluated and used in real time.
A Tuesday announcement from Roche (SIX:ROG; OTCQX:RHHBY) that its COVID-19 serological test -- an immunoassay that captures past exposure to the virus by detecting antibodies against SARS-CoV-2 -- went live at more than 20 sites around the U.S. came on the heels of a Public Health England report indicating its sensitivity is lower than the pharma claimed. The agency published a similar finding for an Abbott serological assay.
Abbott Laboratories (NYSE:ABT) also faced scrutiny after a May 12 bioRxiv preprint from New York University said its ID NOW molecular test -- a rapid probe-based assay that diagnoses active infection by detecting the SARS-CoV-2 virus itself -- missed a large fraction of positive samples caught by another rapid test from the Cepheid unit of Danaher Corp. (see “Re-evaluation of Rapid COVID-19 Tests”).
In each case, the companies said the independent groups did not use the tests as intended.
For Roche’s serological test, blood samples were collected during time windows set by the wrong starting point; for Abbott’s molecular diagnostic, the studies were done on banked nasopharyngeal swabs, rather than at the point of care. Abbott also said said its test performs better when run early after symptom onset.
The consequences of false negatives due to low sensitivity, and the time windows where those are most likely to occur, are different for each type of COVID-19 test.
In each case, the companies said the independent groups did not use the tests as intended.
For antibody tests, the bigger safety risk comes from false positives due to low specificity, which can give individuals a false sense of security that they have built up antibodies while they are still susceptible to infection. But high rates of false negatives also pose risks when used to guide personal and public health decisions.
Antibodies are thought to arise 10-14 days after infection, peak at about day 28 and remain in the circulation for months; false negatives are most likely to arise when samples are collected too early.
For molecular diagnostics, a false negative poses more danger because it could lead an infected person to spread the virus to others or forgo care. Virus levels are thought to peak at symptom onset and decline thereafter, meaning the likelihood of false negatives is increased by waiting too long to test.
The discrepancies also highlight the ongoing need for broad access to the large numbers of COVID-19 patient samples required for confidence in validation studies, particularly for serological tests. Despite ballooning numbers of COVID-19 cases, access to patient blood samples in the U.S. has been limited by a lack of centralized banking protocols (see “Wild West of COVID-19 Antibody Tests”).
Antibodies in good time
Roche said May 19 that Laboratory Corp. of America Holdings (NYSE:LH) and other commercial manufacturers and hospitals in the U.S. had begun performing its Elecsys Anti-SARS-CoV-2 antibody test. The pharma said that in the coming weeks, it plans to expand to more than 200 commercial and hospital lab sites with the ability to perform millions of tests per week.
When Roche’s COVID-19 serological test received FDA Emergency Use Authorization (EUA) May 2, the pharma said it had 99.8% specificity and 100% sensitivity for samples collected at least 14 days after COVID-19 diagnosis via reverse transcription polymerase chain reaction (RT-PCR), edging out other tests with EUA.
But while Roche validated its test on more than 5,000 negative control samples, giving a high level of confidence about its specificity, the company’s sensitivity assessments were based on samples from 29 patients, and had a 95% confidence interval of 88.1-100% (see “Roche Unveils Test Specs”).
Public Health England’s evaluation of Roche’s test, published Monday, found its specificity to be 100%, in agreement with the company’s assessment. However, the agency found the test had an overall sensitivity of 83.9% across all sample collection time points studied, and a sensitivity of 87% at ≥ 14 days after symptom onset, below the lower bound of Roche’s reported 95% confidence interval for a similar time period.
The agency’s evaluation was based on serum samples from 472 negative controls and 93 convalescent patients, 77 of them collected on or after day 14 post-symptoms -- more than twice as many patients as Roche used to internally validate its test for its recommended time window.
Roche spokesperson Patrick Barth told BioCentury that postlaunch, the pharma evaluated its assay on another 156 patient samples taken at least 14 days after confirmation of infection with RT-PCR, out of which 155 (99.4%) were correctly identified.
He attributed the discrepancy between Public Health England and Roche’s analyses to their different sample collection starting points: while Public Health England used the time after onset of symptoms as a starting point, Roche used time after a confirmed RT-PCR test.
Given that most COVID-19 RT-PCR tests are performed after symptoms appear, using RT-PCR confirmation as a starting point will generally capture patients who have had more time to build up an antibody response. BioCentury’s analysis of COVID-19 serological tests with EUA in early May showed all seven manufacturers that had disclosed performance data at that time reported the sample collection window in terms of time since symptom onset (see “Good Test Hunting”).
Public Health England also published an evaluation of an FDA authorized test from Abbott, which the company said had a specificity of 99.6% based on 1070 negative controls, and a sensitivity of 96.8% based on 122 samples from 31 COVID-19 patients collected at least 14 days after symptom onset.
Similarly to its report on Roche’s test, Public Health England’s study -- based on 96 samples from convalescent patients and 759 negative controls -- was concordant with Abbott’s for specificity, but reported lower sensitivity: overall sensitivity was 92.7%, and sensitivity for 82 samples collected ≥14 days after symptom onset was 93.9%.
ID NOW, not later
Abbott’s new data for its ID NOW molecular test highlights how the limit of detection (LoD) data companies submit when seeking EUA for COVID-19 diagnostics provides only a partial measure of its sensitivity, showcasing the test’s intrinsic performance in the lab but not its robustness in the real world.
LoD is defined as the lowest amount of the target that the test can detect, at least 95% of the time. A lower LoD indicates a more sensitive test; Abbott reported an LoD of 125 copies/mL, while Cepheid’s test, which caught more infections, had an LoD of 250 copies/mL.
On Thursday, Abbott released clinical data on ID NOW’s positive percent agreement (PPA) and negative percent agreement (NPA) with lab-based reverse transcription polymerase chain reaction (RT-PCR) tests.
PPA and NPA represent sensitivity and specificity, respectively, when there is no single reference test to act as a gold standard, as is the case for COVID-19; because lab-based RT-PCR is a more well-established modality than Abbott’s rapid probe-based system, the results of FDA authorized RT-PCR tests served as a stand-in for ground truth.
Abbott reported its test had ≥94.7% PPA/sensitivity and ≥98.6% NPA/specificity when compared with a CDC RT-PCR test in an urgent care center study of 225 subjects, 29 of which were positive. The subjects were tested a mean of 4.1 days after symptom onset, and 90% were tested within 7 days after symptom onset, which Abbott said is typically when patients show up for care.
The results were an interim analysis of an ongoing study examining the test’s performance at clinics in New Jersey, Tennessee, Louisiana, Texas and South Carolina.
Abbott also announced results from separate studies it funded at the Everett Clinic in Washington, and in hospitalized and nursing home patients tested with late symptom onset.
The Everett study showed a sensitivity/PPA of 91.3% and specificity/NPA of 100% in a study of 763 symptomatic and 192 asymptomatic individuals, while the study of hospitalized and nursing home patients showed ≥83.3% sensitivity/PPA and ≥96.5% specificity/NPA in an undisclosed number of individuals.
Abbott attributed the lower sensitivity in these two studies to the later time of sample collection and testing.
Patients were tested a mean of 9.7 days and 12.1 days post-symptom onset in the the Everett and hospitalized/nursing home patient studies, respectively.