Expert panel says NSERC needs indicators and expert advice for decision-making

Science indicators are useful in making decisions for research funding in the natural sciences and engineering but must be used in combination with deliberative methods such as expert or peer review. That's the conclusion of an expert panel struck by the Council of Canadian Academies (CCA) to assist the Natural Sciences and Engineering Research Council (NSERC) in allocating resources through its flagship Discovery Grants Program (DGP).

The Expert Panel on Science Performance and Research Funding issued its report — Informing Research Choices: Indicators and Judgment — earlier this month in response to NSERC's request to determine what performance indicators and related best practices were best suited to support discovery research.

In recent years, NSERC has made significant changes to the way it allocates grant funding through its DGP (R$, April 30/09, April 23/10 & June 4/12). While the CCA report focuses exclusively on science performance at the national level and does not make recommendations, its observations are meant to inform future NSERC decision-making.

"New indicators and an emerging "science of science policy" can potentially improve the overall effectiveness and transparency of how funding agencies allocate resources and monitor the performance of their research investments." — Expert Panel on Science Performance & Research Funding

The 16-member expert panel was chaired by Dr Rita Colwell, a professor at the Univ of Maryland at College Park and Johns Hopkins Univ's Bloomberg School of Public Health, and a former director of the National Science Foundation. (A column by Colwell and expert panel colleague Dr Max Blouw, president and vice chancellor of Wilfred Laurier Univ, will appear in the next issue of RE$EARCH MONEY).

The panel met four times in 2011 to review a wide range of evidence from published studies and examine the science assessment practices of 10 countries in detail.

"The body of evidence ... recognizes that the most promising strategies rely on a balanced use of quantitative indicators and expert judgment," states the report. ""There are many ways in which science indicators can be informative and useful in aiding research funding allocation decisions ... These indicators, however, should always be accompanied by expert judgment — both to ensure that indicator-based information is correctly interpreted and that research funders carefully consider the appropriate funding response to any information they may convey.

Human expertise and judgment are particularly valuable, states the report, when funding decisions are made within the context of "the overarching federal S&T strategy as well as the mandate of NSERC and the specific objectives of its programs".

The panel concluded that decisions must be made using sets of indicators for assessing research quality, research trends and research capacity. It developed a taxonomy for each assessment type and assessed the validity of these indicators with respect to the objective of each assessment.

R$