It is no secret that industry scientists often cannot reproduce research published by others, but approaches to solving the problem have been scarce and usually involve a would-be investor footing the bill for additional studies. The newly launched Reproducibility Initiative has a different approach and looks to put the onus on the originating academics to pay for confirmatory experiments. Precise details regarding funding, standards of reproducibility and potential conflicts of interest have yet to be worked out.

Earlier this month, research service provider Science Exchange partnered with publisher PLOS and the open-access data repository figshare to launch the initiative, which lets researchers apply to Science Exchange to have their studies replicated by a CRO or a university lab facility. When the replication study is completed, the researchers have the option of uploading the new results to figshare and publishing them in the online journal PLoS ONE, which will contain a link to the publication in which the original experiments appeared.

figshare is part of the digital science division of Macmillan Publishing, which is the parent company of Nature Publishing Group.

Researchers also will receive a certificate indicating their results were successfully replicated.

The researchers are responsible for all costs associated with the validation study, in addition to a 5% transaction fee paid to Science Exchange for handling communication, data transfer and payment between the researchers and the CRO.

Whether funding agencies would welcome such a mechanism or would rather up the ante on the quality of the initial research remains to be seen.

"Ultimately, we would like to see funding agencies step in and help fund this initiative," Science Exchange cofounder and CEO Elizabeth Iorns told SciBX. "But in the meantime, we think we have incentives in place to encourage researchers to fund the validation work themselves. First of all, the researchers get an additional publication in PLoS ONE. Secondly, they receive a certificate that provides proof of third-party validation to potential licensing partners and investors," she said.

"Finally, we estimate the validation study could cost as low as 10% of the total cost of a lab's research budget for the findings in a particular paper. That's because the goal is not to replicate all of the preliminary and exploratory work that is part of most research programs. Rather, the goal is to replicate only those few key published experiments that underlie the translational potential of a particular set of findings," concluded Iorns.

The scientific advisory board (SAB) of Science Exchange will calculate the total cost for a validation study based on which experiments require replication and which of the more than 1,000 CROs and university lab facilities in the Science Exchange network is best equipped and qualified to carry out the study.

The 10 individuals on the SAB are Iorns; Bruce Booth, partner at Atlas Venture; Lee Ellis, professor of surgery at The University of Texas MD Anderson Cancer Center; John Ioannidis, professor of medicine at the Stanford University School of Medicine; Bernard Munos, founder of the InnoThink Center for Research in Biomedical Innovation; Brian Nosek, professor of psychology at the University of Virginia; Heather Piwowar, postdoctoral researcher at Duke University; George Robertson, professor of psychiatry at Dalhousie University; G. Sitta Sittampalam, professor of pharmacology and toxicology at The University of Kansas Medical Center; and Victoria Stodden, professor of statistics at Columbia University.

If the original researchers agree to the cost proposed by the SAB, they are then allowed to communicate with the CRO only through Science Exchange, which acts as an intermediary throughout the process. That arrangement ensures that the researchers cannot influence the outcome of the confirmatory study.

Once the validation study is completed, the results are presented to the original researchers and to the SAB. Finally, the SAB determines whether or not the original experiments were successfully replicated.

If the experiments are not replicated, the researchers can appeal to have a second CRO attempt to replicate them-again at the researchers' cost. A second failure to replicate is considered strong proof that the original experiments are not reproducible. At that point, it is entirely up to the original researchers whether to disclose the failure to reproduce and/or to retract the original publication. The initiative has no authority to demand a retraction based on a failure to replicate.

Iorns did not comment on whether there are any concerns that conflicts of interest and ethical issues may arise as a result of SAB members being privy to confidential information on papers that do not pass the reproducibility test and for which the authors decide not to publish negative results.

The entire validation study is carried out under a confidentiality agreement, and the researchers are not obligated to make the new data public. However, Science Exchange encourages them to publish the results in the PLoS ONE Reproducibility Collection and to post them on figshare.

The purpose of the collection is to provide a venue for the publication of studies that reproduce published work, either confirming or refuting the original result, said PLoS ONE associate editor Elizabeth Silva.

"We are one of only a handful of publications that actively encourage these types of submissions; traditionally they have been extremely difficult to get published, mainly because journals don't feel they are novel or exciting enough," she said.

The Reproducibility Initiative will accept 40-50 studies initially for validation. The SAB "is looking for studies with clear translational potential that have, for example, identified a putative cancer target. Ideally, the studies will have used standard models and methods that should be straightforward to replicate in a CRO, such as mouse xenografts or cell culture lines and methods such as an ELISA, mass spec or genomic analysis," said Iorns.

Science Exchange plans to publish a meta-analysis of all the studies that looks for trends of reproducibility and non-reproducibility across various experiments, said Iorns. Researchers can choose to have their names withheld from that analysis.

Deciding whether or not a paper has been successfully replicated could turn into a more complex discussion if the results of the reproducibility study are mixed-how the SAB will handle such situations remains to be determined.

Data obtained from those first studies "will give us a sense in real life how 'non-reproducible' manifests itself and in how many different versions and degrees of severity," said Ioannidis. "Any discrepancies between the initial experiments and the reproducibility ones will be carefully scrutinized to find out what happened and why."

The venture side

The Reproducibility Initiative could help entrepreneurs and early stage investors by saving time and money that can be used to move the technology forward rather than to do third-party validation, Daphne Zohar told SciBX.

"A technology that already has third-party validation data would be more attractive to investors than one that doesn't," she said.

Zohar is founder and managing partner of PureTech Ventures, a VC that specializes in funding early stage life sciences research. PureTech typically has key experiments repeated by third parties, using CROs and sometimes academic labs, said Zohar.

In March, a group at Amgen Inc. claimed the findings of only 6 of 53 (11%) papers deemed landmark studies by the hematology and oncology research communities were able to be confirmed by industry scientists.1 Similarly, in 2011, a team from Bayer AG's Bayer HealthCare subsidiary reported that only 25% of published preclinical studies could be validated.2

In 2011, Booth told SciBX that the "unspoken rule" among early stage VCs is that at least 50% of published studies cannot be repeated by an industrial lab. As a result, Atlas insists on external validation studies of a new company's basic science as a precondition to further investment.3

Fulmer, T. SciBX 5(34); doi:10.1038/scibx.2012.888
Published online Aug. 30, 2012

REFERENCES

1.   Begley, C.G. & Ellis, L.M. Nature 483, 531-533 (2012)

2.   Prinz, F. et al. Nat. Rev. Drug Discov. 10, 712 (2011)

3.   Osherovich, L. SciBX 4(15); doi:10.1038/scibx.2011.416

COMPANIES AND INSTITUTIONS MENTIONED

      Amgen Inc. (NASDAQ:AMGN), Thousand Oaks, Calif.

      Atlas Venture, Boston, Mass.

      Bayer AG (Xetra:BAYN), Leverkusen, Germany

      Columbia University, New York, N.Y.

      Dalhousie University, Halifax, Nova Scotia, Canada

      Duke University, Durham, N.C.

      figshare London, U.K.

      InnoThink Center for Research in Biomedical Innovation, Indianapolis, Ind.

      PLOS, San Francisco, Calif.

      PureTech Ventures, Boston, Mass.

      Science Exchange, Palo Alto, Calif.

      Stanford University School of Medicine, Palo Alto, Calif.

      The University of Kansas Medical Center, Kansas City, Kan.

      University of Virginia, Charlottesville, Va.

      The University of Texas MD Anderson Cancer Center, Houston, Texas