Identifying and rewarding research excellence needs to go beyond traditional academic peer-review metrics such as number of publications and citations, say reviewers at the national Killam Awards program.

Peer-review processes underpin nearly all major academic assessments, from publications and funding to promotions and awards.

But these processes have well-documented flaws and inequities, including bias that favours researchers who are men over those who are women, members of the National Killam Program Selection Committee said during a Canadian Science and Policy Centre webinar.

Other flaws of traditional peer review include:

• a lack of concordance between reviewers.
• over-reliance on quantitative scores with small numerical margins defining top candidates.
• variation in the weight given to different application components.
• recurrent patterns of cognitive biases such as familiarity.
• biases that undervalue applications from new researchers, researchers at smaller institutions, women and some groups of racialized people, regardless of the objective merits of the application, proposal or nomination.

The assessment approach used by the Killam Awards program offers a model for incorporating more holistic processes and equally important factors into evaluating research, while maintaining fairness and excellence, panelists said during the webinar.

Dorothy J. Killam established the Killam Trusts and foundational award programs in 1965, in memory of her husband, Izaak Walton Killam.

The national Killam Program is guided by the insight that Dorothy Killam imparted, said panel moderator Amélie Quesnel-Vallée (photo at right), chair of the National Killam Program Selection Committee. She is also inaugural chair and professor in the Department of Equity, Ethics (Faculty of Medicine and Health Sciences) and a professor in the Department of Sociology (Faculty of Arts) at McGill University.

Killam said in her will that the purpose of the Killam Awards is to help build Canada’s future by encouraging advanced study and should contribute to the advancement of learning or to distinction in a profession, Quesnel-Vallée said.

Killam also cited the importance of inclusion and that no person should be qualified or disqualified as a Killam scholar on account of his or her race or religious opinions.

“The Killam Program strives to uphold these principles through all their practices and notably through their selection processes,” Quesnel-Vallée added.

The Killam Trusts comprise a multi-hundred-million dollar endowment that has provided over $1 billion in total funding since inception. The Killam Prizes, for example, are worth $100,000 each, and five are awarded every year nationally across health sciences, natural sciences, engineering, social sciences and humanities.

The national Killam Program is administered by the National Killam Program Office at the National Research Council of Canada (NRC).

During the first year of the NRC’s administration of the program, Maydianne Andrade (photo at right), then chair of the National Killam Program Selection Committee, co-led, with the National Killam Program Office, and implemented an initiative to renew and revise a set of guidelines for the program that “clearly expressed the goal of an inclusive and holistic review” for assessing award applicants. Andrade is dean and professor of the Faculty of Science at York University.

Assessment is now built around three sequential stages, “each with specific mechanisms that are designed to interrupt the points where the literature says that bias most commonly enters into peer review,” she said.

 The Killam Program assessment process

At stage one – determining the submission pool of award applicants – every file receives two independent expert peer reviews, which are common in most processes. Then the National Killam Program Office creates ranking groups based on the external peer review scores.

The office then shares both the scores and the reviewer narratives with non-voting chair of the National Killam Program Selection Committee, who reviews all the files. The chair gives each of the files a categorical ranking – typically without looking at the peer-review scores, except when a “deep read” is needed to understand why peer reviewers scored the way they did.

Adrade said a “distinctive part” of the evaluation process is that the National Killam Program Office will flag all applicant submissions where there are significant peer-review score differentials. This can include any case where two expert reviewers rated the same file very differently, where the reviewer narrative and the scores don’t seem to match, or the reviewer narratives are so short that it’s hard to justify the scores.

“High [score] differentials lead to lower average scores, and in some studies, in particular one that looked at CIHR (Canadian Institutes of Health Research) data, they're more common in files for women,” Adrade noted.

If a low score appears to be poorly justified, the chair of the National Killam Program Selection Committee can have it removed from the ranking process, and that rank is then recalculated.

The National Killam Program Office then produces an estimate of which files should be in the consideration pool that move on for further assessment by the selection committee.

Before the selection committee has seen any submissions, it reviews aggregated representation data that the National Killam Program Office has compiled into visualizations.

The selection committee then compares the proposed consideration pool with the original set of candidates to look for gaps, such as where representation has dropped off. If the committee identifies a gap, then the program office will again review those files to see if anything was missed that could close the gap.

All this is done “with no knowledge of the self-identification of any one candidate,” Adrade noted.

Stage three is the assessment by the selection committee of the consideration pool itself. The peer-reviewed scores aren’t shared with the committee – only the narratives received from external peer reviewers.

“Each candidate is evaluated by two committee members using the same rubric as the external peer reviewers,” Adrade said. “That same rubric that is available publicly for all the candidates to use when they're putting together their applications.”
Committee members then have six to eight weeks to complete their evaluations. In the selection meetings, every submission gets equal airtime and no candidate can be removed without consideration through a thorough discussion. This process is overseen by the non-voting chair, and together with the program office, the chair provides summaries of what they’re hearing to guide the selection committee towards consensus.

“So what you're looking at is a process that's designed from the ground up to preserve expert judgment from peer reviewers, while systematically blocking the channels through which bias typically flows,” Adrade said.

How effective is the Killam adjudication process in improving fairness and equity?

The Killam Program traced candidates from submitted application through to receiving an award over three competition cycles, from 2023 to 2025.

In the submission stage, the Killam Program’s evaluation resulted in 59 percent women being in the consideration pool and 41 percent men.

In terms of awards, the Dorothy Killam Fellowships went 62 percent to women and 38 percent men. “So the [evaluation] process not only helped parity, it slightly exceeded it,” Quesnel-Vallée said.

She pointed out that an average 44.7 percent of fulltime teaching staff at Canadian universities are women, but there are greater imbalances in the higher ranks of faculty.

“So the submission pool itself already reflects a pipeline that over-represents women relative to the eligible population [at Canadian universities], and the adjudication process honors that throughout,” she said.

Adrade said for the Dororthy Killam Fellowship consideration pool, 59 percent of women and 41 percent of men got received high peer-review scores of 12 through 16 or above.

“This mirrors the submission pool, but as the threshold rises towards 18 (for a peer-review score), you see some fluctuation, and at the highest score, 100 percent of the candidates are men,” she said. That mirrors what is seen in the literature for peer-review scores – reflecting a bias in the peer review process.

But when the Killam Program Selection Committee evaluates candidates using only their narratives, with no access to peer-review scores, “they’re not replicating whatever skew may be present in the external review process,” Adrade said.

The gender distribution then actually shifts toward women at the higher score thresholds. At a score of 17, 68 percent in the consideration pool are women. At a score of 18, it’s evenly divided – 50 percent women and 50 percent men.

“The anonymization of scores before committee deliberation, the rubric, the deep read function, these are not just procedural niceties,” Adrade noted. “They're creating measurably different outcomes than what you would get from a process that simply passes external [peer] review scores downstream.”

“We really think this represents a scalable model, not specific to the Killam Program, but applicable to any high-stakes national or institutional adjudication process that is seeking to align its practices with its stated values about research excellence and equity, while preserving the process but also privacy of the candidates who provided their self-identification data,” she said.

“The challenge is that even if all of us use the same understanding or definition of research excellence, we're not applying the same interpretation or same understanding to it,” said Killam Selection Committee member Carl James (photo at left), Jean Augustine Chair in Education, Community & Diaspora in the Faculty of Education at York University.

But the Killam Program seeks to define or look at research excellence “in ways that enable access and opportunities and engages in a more holistic understanding of the research process and what people are trying to do with the work that they are presenting,” he said.

The interdisciplinary National Killam Program Review Committee looks at the dynamic interplay between the context and the assessment, he added. Committee members are able to “talk between ourselves, knowing that we understand what's being presented to us with our respective ideas and so forth. We also bring the understanding of societal context, the political, the social, the cultural.”

Given that change is an inevitability, committee members “are going to constantly be paying attention to how we're interpreting excellence, how excellence in the current context might be different from and thought through differently than in another context,” James said.

How research excellence is assessed affects what research gets funded and rewarded and its impact

The Killam Program assessment process incorporates the principles of adjudication set out in the San Francisco Declaration on Research Assessment (DORA).

Adrade said the DORA principles offer “a holistic assessment for thinking about impact more than just what academics have traditionally called excellence, which doesn't always talk about how things get into public use, how our publicly funded research can assist society, nor things like multiplier effects through mentorship, through how we train the next generation of scientists.”

“Really, when you come right down to it, the ways in which we frame our thoughts about assessment based on traditions, academic traditions of what is counted and what is rewarded, those are biases,” she said.

Yet how research is assessed affects where researchers put their efforts, and what research gets funded and rewarded, she pointed out. “So it shapes not just the research enterprise, but our ability to have the positive impact that I think most of us want to have on society.”

“We have to start acknowledging just how much the biases that are built in to our [peer-review] systems can influence the research that is actually done,” she said.

Rather than having the risk of biases leaning toward, for example, how many papers researchers publish and the impact factor of those papers, a holistic assessment considers factors such as whether researchers take their work into the public sphere, how many graduate students they supervise, whether they’re able to become successful whether doing fundamental research or working in an applied field.

Instead of often defaulting toward the ease of review “by counting rather than evaluating,” Adrade said, “the thing that becomes very clear when we work in a process like Killam is that it is essential that we have that more holistic approach to understanding people's impact. What did they want to accomplish and then how did they accomplish it?”

Adrade said the DORA principles, the Killam Program adjudication process, the initiatives that the Tri-Council funding agencies are implementing – such as requiring a narrative-driven CV from researchers – “can change the scholarly enterprise and tie us more closely, I think, to the goals that Mrs. Dorothy Killam talked about: Advanced study, the whole person, and how we contribute to society.”

Killam Program Selection Committee member Bilkis Vissandjée (photo at left), professor and researcher in the School of Nursing, Centre de recherche en santé publique at Université de Montréal, urged those assessing research to think about the strategies to be considered in the adoption and implementation of principles for more holistic assessment of research.

“In particular, by the need to better value in response to what we must consider – interdisciplinary research, more applied, co-construction of research, and also the opportunities that may take time, but which are essential with the communities concerned,” she said. Evaluation of researchers and their research must take into account the societal impact, the mentoring, the strengthening of the capacities, “the integration of the knowledge that we call experiential, of the autochthonic knowledge that is derived from the practice,” Vissandjée said.

Barriers to implementing a more holistic approach

However, there are several structural and cultural barriers to a more holistic approach to assessing research and broadening definitions of research excellence, panelists said. They include institutional resistance to change, lack of clear guidelines, and ensuring fairness and consistency throughout the process.

Killam Program Selection Committee member William Whelan (photo at right), professor in the Department of Physics at the University of Prince Edward Island, and professor in the Department of Biomedical Sciences at Atlantic Veterinary College, said it's important to leverage faculty expertise in identifying the specific ways in which bias can manifest in peer review, to engage with faculty or institutional and local experts in evaluating current practices and approaches for the communications of awards and grants.

Training reviewers in the design of rubrics for assessment is also necessary, he said. Also, “it’s critically important to publish your evaluation rubric [and] make it easily accessible to candidates, to nominators, to peer reviewers.”

The Killam Program’s evaluation rubric is published online, he noted. For example, the frequently asked questions for the Dorothy Killam Fellowships explicitly describe what is expected of researchers in a knowledge mobilization plan, for sharing knowledge beyond the academic community.

The language in the Killam Program’s rubric is that the nomination package should clearly demonstrates the contribution of the candidate's ideas and research results to Canadian society, including the nation's intellectual and or cultural life, Whelan said. “So that's our guiding light in terms of impact.”

Whelan said his experience as a member for two years of the National Killam Program Selection Committee “has had a profound impact on my approach to [research] assessment and the way I view assessment.”

“I've become more aware of the important features of a fair assessment process, the importance of a clear rubric that's communicated to everyone associated with the file, including the reviewers,” he said.

Committee members are regularly invited to share concerns at any point if they feel something in the process or the discussion is deviating from its intended purpose, Whelan said. “I think it's absolutely fantastic, because what it does is it helps to establish a cohesive and a collaborative environment among the panelists.”

Establishing trust and empowerment “in all aspects” among committee members is essential, Adrade said. She noted that selection committee chairs are non-voting, so they’re viewed by committee members as guarding or ensuring the holistic research evaluation process rather than trying to influence it.

The committee also regularly refreshes itself by bringing on new interdisciplinary members, while also retaining members with institutional knowledge.

“The new folks will come in and question every aspect of the process. The people who have been there longer may sometimes need reminding about the specifics of why they're assessing particular ways, and together they work really well,” Adrade said.

The whole point of using a more holistic approach to evaluate research “is that there are many, many ways our research can have impact or be excellent,” she said. “It may be through impacts on scholarly fields. It may be through impacts on communities. It may be when your work goes into a patent.”

A holistic research assessment process does consider traditional peer review parameters, such as publications and citations.

But, Adrade noted, a fair and equitable holistic also asks: What are you trying to accomplish that will change things? Why will things be different when you leave than when you got here?

“You can build an adjudication system that preserves the value of expert judgment while substantially reducing the channels through which bias flows,” Quesnel-Vallée said.

“And when you do, when you look at the data with that honesty and act on what they show, you don't sacrifice excellence. You find more of it.”

Editor’s note: In addition to the Killam Prizes, the Killam Trusts also includes:

• Dorothy Killam Fellowships (80,000 per year for two years).
• Killam NRC Paul Corkum Fellowships ($100,000 per year as a grant plus up to $50,000 for travel/dissemination.
• Killam Doctoral/Memorial Scholarships: vary by university, typically ranging from $36,000 to $45,000 per year.
• Killam Postdoctoral Fellowships: valued around $50,000 to $60,000 per year.
• Killam Annual Professors/Faculty Research Prizes: local university faculty awards typically range from $5,000 to $10,000 cash prizes.

R$