The Global Biological Standards Institute has determined that material and procedural standards are the key battleground for improving the reproducibility of preclinical studies-an area of growing concern among funding agencies and industry stakeholders.1-4 This year the institute will form task forces that will begin developing standards in research areas in which the institute deems they are most needed, such as human cancer cell lines, antibody reagents and next-generation genome sequencing.

GBSI was founded in 2012 to catalyze the development and use of biological standards to enhance the reproducibility of basic and translational life sciences research. The organization's scientific advisory council (SAC) includes representatives from academia, industry, publishers and the NIH's National Center for Advancing Translational Sciences (NCATS).

Other groups are grappling with the reproducibility problem, albeit in a different way than GBSI's pursuit of standards. For example, in 2012, research service provider Science Exchange, publisher PLOS and the open-access data repository figshare launched the Reproducibility Initiative.5

The goal of the initiative is to enable academic researchers to have their studies replicated by a CRO or university lab facility. The entire validation study is carried out under a confidentiality agreement. The researchers are not obligated to make the new data public, although they are encouraged to publish the results in the PLoS One Reproducibility Collection and to post them on figshare.

Last month, GBSI released its first white paper, The case for standards in life sciences research: seizing opportunities at a time of critical need.6 Based on input from nearly 60 stakeholders, the paper addressed the quality of research methodologies, identified areas of concern and recommended the use of standards to improve the reproducibility of preclinical research.

President Leonard Freedman said that each GBSI task force will ideally include key opinion leaders and stakeholders from academia, industry, the NIH, the NCATS, the National Institute of Standards and Technology and the FDA, who will focus on developing standards for one class of research reagents, materials or procedures.

Freedman previously was vice dean for research and a professor of biochemistry and molecular biology at the Jefferson Medical College at Thomas Jefferson University. Prior to that he was VP of women's health and musculoskeletal therapies at Wyeth (now part of Pfizer Inc.) and executive director and head of the Department of Endocrinology at Merck & Co. Inc.

On Jan. 22, the GBSI SAC was to have met for the first time to prioritize research areas for the new task forces and identify key opinion leaders to include in them.

Key areas of interest to GBSI include authentication of human cancer cell lines, validation of antibody reagents and standardization of next-generation genome sequencing technologies, said Freedman.

"We want to follow up on ATCC's work on the human cell line authentication standard," which can confirm the origin of a human cell line but not identify its tissue type or cancer status, he said. An authentication standard for cancer cell lines "has obvious implications for drug screening and development and would complement what's already been done by ATCC."

Freedman is CSO of the ATCC, a not-for-profit organization that characterizes cell lines, microorganisms and other biological research materials, develops and evaluates techniques for validating those materials, preserves them and distributes the materials to the research community.

The organization published a human cell line authentication standard in 2011 after multiple studies suggested that up to 30% of cell lines used in research were misidentified.7

For antibodies used in research and diagnostics, Freedman said, "the key questions here are: what techniques should we use to standardize them? How do we cross-compare readouts from different antibodies against the same target? Everyone I talk to brings up this need because there are very few standards for the generation and use of antibodies" for those purposes.

A third area of potential interest to GBSI-and the one that could be the most challenging-is standards for next-generation genome sequencing. "As the technology and procedures for sequencing the human genome get cheaper, all kinds of questions get raised about standards, such as the type of nucleic acid material used as controls and the lack of uniformity around instrumentation, informatics and software," Freedman said. "This area presents a tremendous opportunity for us, but it's also daunting because it's so vast."

Calling for consensus

In conjunction with the release of the white paper, GBSI convened a panel in Washington, D.C., to discuss the development of standards that will improve the reproducibility of preclinical research. The take-home message was: follow the decades-old lead of clinical research, in which established processes for developing standards by consensus have changed the landscape.

The four panelists proposed specific steps that research institutions, funding agencies and journals could take to encourage researchers to adopt the standards once they have been developed.

"Clinical research began addressing the problem of reproducibility decades ago, and standards are now built into the system, but we don't have anything comparable in the preclinical arena," said panelist
C. Glenn Begley. "Things that were acceptable in clinical trials in the 1970s aren't acceptable any longer."

As examples of standards now established in clinical research, Begley cited the use of control arms, double-blind experiments and all patients in the data analysis.

Begley is CSO and SVP of TetraLogic Pharmaceuticals Corp. and a member of GBSI's SAC. He previously was a VP and global head of hematology/oncology research at Amgen Inc., where he coauthored a Nature commentary stating that he and Amgen scientists had been able to replicate only 6 out of 53 'landmark' preclinical studies in the literature.1

Panelist Mary Lou Gantzer, immediate past president of the Clinical and Laboratory Standards Institute (CLSI), said that it will be important to ensure that standards for preclinical research introduce greater conformity, control and reproducibility without stifling innovation.

CLSI is a not-for-profit organization that develops consensus standards for clinical research with input from academia, industry, government and healthcare professionals.

Because scientific research often requires the development of new methods, "the process of developing and setting a consensus standard has to be dynamic," said panel member Yvonne Reid, manager and scientist at ATCC. "You have to think about this when you put a standard in place, so that newer, better methods can emerge" as scientific knowledge and technology advance.

As an example, Reid said that most of the data used to establish the cell line authentication standard were based on eight genetic markers. "Now it is based on 18 markers," she said.

Promoting adoption

Although GBSI's task forces will bring stakeholders together to develop consensus standards, the organization will not be involved in implementing those standards, Freedman said. Instead, that responsibility should fall to funding agencies, academic institutions and journals.

Begley said that funding agencies should take the lead in insisting on standards because they have the greatest influence over researchers. "Any grant would be contingent on the researcher's previous and ongoing use of standards and proper procedures. If the researcher doesn't use them, the agency shouldn't renew the grant or should think twice about funding that person again," he said.

Panelist and GBSI SAC member William Bentley added that funding agencies "could also provide more financial support for researchers interested in translating their work to industry," and academic institutions could also do more to educate researchers about translational science.

"Researchers who want to see their results translated to commercial settings would adhere to consensus standards because they would have a vested interest in making sure companies can replicate those findings," added Bentley, who is chair of bioengineering at the University of Maryland, College Park.

Academic institutions also could help to implement standards by having a researcher's findings independently validated before applying for a patent on them, Begley said.

"Most institutions spend hundreds of thousands of dollars on filing patent applications, many of which are not going to stand the test of time" because the findings are not reproducible, he said.

If the institution validated the findings first, "this would have two immediate consequences: there would be a decrease in patents with little value, and the reputation of the institution would increase in the eyes of venture capitalists. This process could conceivably become self-sustaining for the institutions, with the savings from patent applications helping to pay for data validation," he said.

The panelists agreed that journals also could help implement consensus standards by publishing only studies that follow them, but Reid noted that "journals are sometimes still a little nervous about requiring adherence to a standard across all studies."

For example, she said, "if a scientist is using a human cell line as a positive control in an experiment, then it is just a tool, and he might not care whether the cell line" has been properly authenticated according to the ATCC-published standard. "But if that cell line is used as a model for a tumor type, then the identity of the cell is important, and misidentification can be a problem. Journals have difficulty in deciding when to insist on authentication of a cell line" because they perceive that a potential misidentification may not always affect the results.

"I take a harder stand and say that if the cell line has been misidentified-regardless of how it's used-then that's a problem" because it may affect the reproducibility of that study and enable the use of the misidentified cell line in future studies, Reid said.

Back to basics

In addition to publishing only studies that adhere to established consensus standards, panelists said that journals could take several other steps to address the problem of irreproducibility-such as requiring researchers to provide extensive details on their methods and materials.

Begley cited the need for detailed information about methodologies and reagents in papers. This would avoid the confusion that can arise when authors cite an earlier paper that describes a method instead of spelling out exactly what they did in the new study, he said. "The researchers might have introduced small changes or differences that need to be reported in the new paper."

Begley also suggested that journals publish only studies that use basic scientific and procedural standards, such as blinded experiments, proper positive and negative controls and independently validated reagents.

Bentley agreed. "All of these are factors that enable reproducibility, and it is easy for the journal's reviewers to see whether or not the study used these procedures," he said. "Journals require some of these procedures but not all of them-and the requirement is not universal" across all journals.

Begley and Reid noted that these basic standards could be implemented quickly, without the need for prolonged discussion among stakeholders.

"I think the adoption of these basic standards would happen overnight if funding agencies and journals said, 'This is how it has to be done'," Begley said.

Haas, M.J. SciBX 7(3); doi:10.1038/scibx.2014.76
Published online Jan. 23, 2014


1.   Begley, C.G. & Ellis, L.M. Nature 483, 531-533 (2012)

2.   Bissell, M. Nature 503, 333-334 (2013)

3.   Freedman, L. Nature 504, 376 (2013)

4.   Haibe-Kains, B. et al. Nature 504, 389-393 (2013)

5.   Fulmer, T. SciBX 5(34); doi:10.1038/scibx.2012.888

6.   Global Biological Standards Institute. The case for standards in life science research: seizing opportunities at a time of critical need. (December 2013)

7.   Capes-Davis, A. et al. Int. J. Cancer 127, 1-8 (2010)


      Amgen Inc. (NASDAQ:AMGN), Thousand Oaks, Calif.

      ATCC, Manassas, Va.

      Clinical and Laboratory Standards Institute, Wayne, Pa.

      figshare, London, U.K.

      Food and Drug Administration, Silver Spring, Md.

      Global Biological Standards Institute, Washington, D.C.

      Jefferson Medical College at Thomas Jefferson University, Philadelphia, Pa.

      Merck & Co. Inc. (NYSE:MRK), Whitehouse Station, N.J.

      National Center for Advancing Translational Sciences, Bethesda, Md.

      National Institute of Standards and Technology, Gaithersburg, Md.

      National Institutes of Health, Bethesda, Md.

      Pfizer Inc. (NYSE:PFE), New York, N.Y.

      PLOS, San Francisco, Calif.

      Science Exchange, Palo Alto, Calif.

      TetraLogic Pharmaceuticals Corp., Malvern, Pa.

      University of Maryland, College Park, Md.