Replicable research results & public accountability

By Dr Peter Morand

Last year, in collaboration with the Association of Universities and Colleges of Canada (AUCC), the three granting councils consulted with their stakeholders to review mechanisms and policies to ensure "that the research they fund be carried out in accordance with high standards of ethics and integrity, and that the grant funds be spent in compliance with certain procedures and requirements."

This resulted in the launch last December of the Tri-Agency Framework: Responsible Conduct of Research, an updated document that outlines the three federal funding agencies' policies and expectations related to applying for and managing agency funds, research performance and publication of results. As well, researchers' host institutions must follow a prescribed process whenever there is an indication of a breach of agency policy. All this is commendable in the context of the large sums of public money involved. However there is an elephant in the room that cannot continue to be ignored.

Fraud and plagiarism clearly are not to be tolerated. But what about the increasing uneasiness concerning replication, the gold standard in scientific research of independently confirming the results and conclusions of published studies?

Within the research community there is much frustration about the increasing number of retracted papers and the time and effort wasted on trying to reproduce published research results because of the reluctance of some authors to disclose full experimental details. After much-publicized instances where policy-making has been undermined by unreliable scientific data (e.g. climate change, vaccines/autism), the public needs to be reassured that investment in scientific research produces replicable results.

This unease, in contrast with the implicit trust in findings published in respected peer-reviewed journals, is what undoubtedly prompted Bruce Alberts, the editor of Science, to devote an entire issue (Dec 2/11) to replication of scientific research. Contributors wrote about the merits of lab-based studies as opposed to field-based studies and about the interpretation of the flood of data now available by using sophisticated computing capabilities, as well as about some of the more obvious reasons for failing to reproduce published research results. However, it is the reproducibility of research results in preclinical studies that has been attracting unwanted media attention.

Not surprisingly, replication of preclinical results has been a longstanding concern of investors and life sciences companies when making decisions to commit vast resources of time, money and people to proceed to clinical trials in the development of preventive and therapeutic products.

A recent article in the Proceedings of the National Academy of Sciences (Oct/12) confirms the deterioration in the reliability of peer-reviewed scientific publications. In light of the epidemic growth of retracted papers, the authors carried out a detailed review of 2,047 biomedical research papers indexed as retracted in PubMed on May 3, 2012 and found that of these retractions, 21.3% were attributable to errors and 67.4% to misconduct (fraud, 43.4%, duplicate publication, 14.2%, plagiarism, 9.8%).

The authors report an approximately 10-fold increase in the percentage of retracted articles due to fraud since 1975 and further comment that retractions "exhibit distinctive temporal and geographic patterns that may reveal underlying causes".

Through lack of diligence, gullible optimism, or both, many high profile biopharmaceutical companies (e.g. Amgen, Pfizer) have been on the hook for initiating expensive projects that were doomed to failure because reported experimental results could not be replicated. Last year, Germany's Bayer published the results of a study documenting the termination of over 60% of its early drug target projects primarily because Bayer researchers were unable to validate claims made in scientific journal articles.

Over the past several months things have come to a head about what one Wall Street Journal reporter refers to as "one of medicine's dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can't be reproduced." Earlier this year commentary in the journal Nature by Glenn Begley, former VP hematology and oncology research at Amgen, added further weight. The article reports that Amgen scientists were able to reproduce results in only 11 of 53 published papers.

Is it any wonder that it's next to impossible to attract VC into projects before the onset of clinical trials? For example, Atlas Ventures, a US-based VC firm in the biotech sector, now enlists the help of independent labs to validate experimental data for projects being considered for funding. Bruce Booth, a partner in Atlas' Life Sciences group, claims that experimental results can be reproduced in only about half the cases.

Many contributing factors have been postulated to account for this disturbing trend in scientific publications and these include:

* competitive pressure to publish more breakthrough papers (even by top-tier journals) due to the escalation in the number of research journals;

* intricacy of experiments (accumulation of unintended errors due to many variables, differences in materials/equipment;

* a penchant for positive results by researchers and investors alike (winner take all syndrome);

* selective publication of data (with the finger often pointing to drug companies);

* instances of blatant competition rather than collaboration among some laboratories; and,

* pressure to publish to enhance one's career and generate more grants.

The erosion of public trust in published research results requires the immediate attention of the scientific community, research institutions and funding agencies. It is not enough to make recommendations or to have guidelines on ethics and procedures for dealing with breaches of this trust after the fact. Elizabeth Iorns, co-founder of Science Exchange in Palo Alto, emphasizes that immediate action is required and has suggested the introduction of mechanisms whereby added recognition be given to reproducible research results.

This is a pressing global issue in which Canada's federal funding agencies, in consultation with the research community, universities and the private sector, could take the lead in bringing creative and viable solutions to the table.

Peter Morand is former dean of science and Engineering, University of Ottawa, past president, NSERC and past president & CEO, Canadian Science & Technology Growth Fund. petermorand@rogers.com