Indicators should inform, rather than replace, expert judgment in science assessment

By Dr Rita Colwell and Dr Max Blouw

Canadians expect their governments will invest public funds wisely, by acting with accountability and transparency. Today, public research funders are working hard to meet these expectations. The value of investing in research is clear: scientific advances provide economic and social benefits that help propel the prosperity of the country.

Finding the right balance between continuing to foster areas of research that have borne historical achievement, versus allocating resources to new areas of exploration — a much more risk-prone tactic — has always been difficult.

As a major Canadian research funder, the Natural Sciences and Engineering Research Council (NSERC) strives to be at the leading edge for research funding practices. They are responsible for allocating approximately $1 billion a year on scientific research. Over one-third of that goes directly to support discovery-based research through the flagship Discovery Grants Program. Therefore, NSERC, like many research funding organizations around the world, are looking for the best information and evidence that will inform the decision-making process for research funding allocation.

In 2010 NSERC requested the Council of Canadian Academies conduct an expert assessment that would examine research funding mechanisms, resource allocation evaluation and research investment. The Council assembled an expert panel to conduct this in-depth assessment, of which we were members. Our panel was composed of 16 experts from across Canada and from as far away as Australia. Over approximately 18 months, we came to a number of evidence-based conclusions in the report, Informing Research Choices: Indicators and Judgment. The report was released by the Council in early July and is available at www.scienceadvice.ca.

In, Informing Research Choices, we consider the value of the different types of science indicators in helping research funders arrive at the best decisions. Our focus was science performance at the national level of research fields in the natural sciences and engineering, and the indicators and methodologies most relevant to discovery research. The report examines both quantitative indicators such as funding measures, scientific publication counts, citations and webometrics; as well as deliberative processes like the opinion of researchers and experts in the field.

"Although many types of quantitative indicators can be reliable and informative in science assessments at the national field level, these indicators should not be used to support research funding allocation without expert judgment. The body of evidence now available recognizes that the most promising strategies rely on a balanced use of quantitative indicators and expert judgment. A review of recent experiences in selected countries and research funding organizations globally lends further support to this conclusion."—
   Informing Research Choices: Indicators and Judgment

How the two interact with each other, and stand alone, is at the core of the panel's findings. After examining the available evidence, the expert panel concluded that quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation.

The surprise for the panel was that there was so much evidence and lessons learned on the issue of expert judgment coupled with indicators that are quantitative, that what seems to be entirely intuitive, perhaps just common sense, is well grounded in experience and practice. This is an important conclusion of the panel because it gives an authenticity and credibility to all our findings.

Based on a review of the existing evidence and our examination of international practices we know that there is no one-size-fits all solution or formula for evaluating science and effectively allocating funds in the natural sciences and engineering disciplines. The individual circumstances of the assessment and the research funding context need be considered. Decisions must reflect goals (in terms of desired output or results) and the ultimate objectives of the funding program and/or the organization. For example, funding decisions by a granting council such as NSERC would need to consider the objectives of its funding program and also the overarching strategy of the government.

Guiding principles

An appropriate element within our report is the articulation of four guiding principles that can support research funders as they consider how best to allocate funds. Understanding context, as noted above, is one of these principles. The others include: do no harm; transparency is critical, and expert judgment remains invaluable. These principles and the expert panel's findings are relevant not only for NSERC but for research funders around the world.

In conclusion, we believe the expert panel has developed an in-depth assessment regarding science performance indicators and global practices that will inform funding allocation for years to come. We hope this report will serve as an important tool for NSERC as they consider the appropriate methodology for allocating funds from the Discovery Grants Program. We commend NSERC for their efforts to modernize and ensure processes for determining research funds are at the leading edge.

Dr. Rita Colwell is a Distinguished University Professor at the University of Maryland, College Park and at John Hopkins University, Bloomberg School of Public Health. Dr. Max Blouw is the President and Vice-Chancellor of Wilfrid Laurier University in Waterloo ON.