Evaluation Report

Prepared for: Social Sciences and Humanities Research Council of Canada (SSHRC)

Prairie Research Associates
500-363 Broadway
Winnipeg, MB, R3C 3N9
Phone: 204.987.2030 Fax: 204.989.2454
Toll-free (English): 1-888-877-6744
Toll-free (French): 1-866-422-8468
Email: admin@pra.ca

PDF version (2.44 MB)

---

Table of Contents

• Abbreviations
• Executive summary
  • Key findings
  • Recommendations
• 1. Evaluation scope and methodology
  • 1.1 Evaluation questions
• 2. Canada Research Chairs Program
• 3. Continued relevance of the CRCP
• 4. Fostering research excellence and capacity
  • 4.1 Research productivity
  • 4.2 Collaborations
  • 4.3 Cross-disciplinary research
  • 4.4 Students
  • 4.5 COVID-19
• 5. Attracting and retaining a diverse cadre of excellent researchers
• 6. CRCP’s EDI requirements
  • 6.1 Institutional EDI Action Plans
  • 6.2 Equity targets
• 7. Efficiency of program design and delivery
  • 7.1 Status and prestige of the CRC award
  • 7.2 CFI funding
  • 7.3 Tier 2 stipend
  • 7.4 Length of the CRC awards
  • 7.5 Reallocation
  • 7.6 Value and use of the CRC award
  • 7.7 Institutional support
  • 7.8 Nomination and review processes
  • 7.9 Program expenditures
• 8. Conclusions and recommendations
  • 8.1 Conclusions
  • 8.2 Recommendations
• Appendix A – Methodology
• Appendix B – CRCP Logic Model

---

Abbreviations

CERCs	Canada Excellence Research Chairs
CFI	Canada Foundation for Innovation
CFREF	Canada First Research Excellence Fund
CIHR	Canadian Institutes of Health Research
CoR	College of Reviewers
CRC	Canada Research Chairs
CRCP	Canada Research Chairs Program
DORA	San Francisco Declaration on Research Assessment
ECR	early career researcher
EDI	equity, diversity and inclusion
HQP	highly qualified personnel
JELF	John R. Evans Leaders Fund
IAC	Interdisciplinary Adjudication Committee
LGBTQ2S+	Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Two-Spirit
NSERC	Natural Sciences and Engineering Research Council of Canada
SSHRC	Social Sciences and Humanities Research Council
TIPS	Tri-agency Institutional Programs Secretariat

---

Executive summary

This report presents findings from the evaluation of the Canada Research Chairs Program (CRCP) that covered the period of April 2015 to March 2022. The purpose of this evaluation was to provide an assessment of the program’s relevance and performance, as well as aspects of design and delivery, including the implementation of equity, diversity and inclusion (EDI) requirements.

Introduced in 2000, the CRCP supports 2,285 research professorships—Canada Research Chairs (CRCs)—in eligible degree-granting institutions across Canada. CRCs aim to achieve research excellence in engineering, natural sciences, health sciences, humanities and social sciences. The administration of the CRCP is the responsibility of the Tri-agency Institutional Programs Secretariat (TIPS), which is managed on behalf of the three granting agencies by the Social Sciences and Humanities Research Council (SSHRC). The CRC award is an institutional grant that can be sought in combination with funding for research infrastructure from the Canada Foundation for Innovation (CFI).

Key findings

After more than 20 years since its implementation, the CRCP continues to play an important role in supporting research in Canada and the program’s objectives remain relevant. The program remained cost-efficient and performed well in terms of achieving its objectives. The findings from this evaluation show that the CRCP (including the accompanying CFI funding) continues to foster research excellence and capacity, as well as attract and retain a diverse cadre of excellent researchers to/in Canadian postsecondary institutions. However, during the period under evaluation, the COVID-19 pandemic created challenges that impacted the degree to which CRCs and their teams could complete planned research and the ability of institutions to attract international researchers.

Since the last evaluation, TIPS has implemented several changes to the design and delivery of the CRCP to further facilitate the program’s success in achieving its objectives. For instance, the Tier 2 stipend played an important role in attracting and retaining and in supporting CRCs in building research capacity. Limiting the number of renewals for the Tier 1 CRC award has provided opportunities for institutions to attract and retain new excellent researchers. Additionally, the implementation of new EDI requirements-particularly equity targets-has supported the attraction and retention of a diverse cadre of excellent researchers by helping the program and institutions identify, mitigate and reduce systemic barriers that prevent participation of members of designated groups.^{Footnote 1}

However, certain CRCP design features are perceived to challenge the program’s success and present opportunities for improvement, some of which build on findings and recommendations from previous evaluations. Findings related to these features, as discussed in this report, led to the identification of three recommendations to help ensure that the CRCP continues to achieve its objectives through its support to CRCs and Canadian postsecondary institutions.

Recommendations

Recommendation 1: Investigate opportunities to increase the value of the Tier 1 and Tier 2 CRC awards with a specific emphasis on dedicating a minimum amount of funding for research.

One of the most notable areas for improvement recognized throughout the evaluation was the value of the CRC awards and the ways the funding is used by institutions: this was a persistent finding across most lines of evidence. CRC awards have approximately 53% less purchasing power now than in 2000 when adjusted for inflation and static funding over the last 20 years. This was perceived by stakeholders, institutional representatives and CRCs to have reduced the prestige and appeal of the award, making it less competitive for attracting international researchers and retaining excellent researchers in Canada. While the full extent to which the CRC award is used for attraction versus retention is unknown: the proportion of CRC awards used to attract researchers has consistently decreased over the 20 years of the program, and may continue to decline, in part, due to the value of the award.

Additionally, the majority of CRCP funds are used for CRC salaries, with approximately 21% of overall funds allocated for research-related expenses (e.g., student salaries, equipment, research stipends). As a result, financial resources to support the CRCs’ research programs are often provided through the institutional support package (not including the CRCP funds) or must be sought by CRCs through other funding sources. Such funding is critical to help CRCs build research capacity (e.g., hiring students). Without it, CRCs may be unable achieve their research goals and support the achievement of CRCP’s objectives.

The value of the CRC awards is a longstanding issue. The two previous CRCP evaluations recommended increasing the funding amount of the awards, possibly indexing to inflation. Increasing the value of the CRC awards and specifically dedicating funds for research provide an opportunity for the program to ensure the continued success of funded CRCs and the achievement of the CRCP’s objectives. While Budget 2018 announced additional funding for the CRC program, it was focused almost exclusively on the Tier 2 level: the number of chair positions was increased by 250 Tier 2 and 35 Tier 1, and a $20,000 research stipend was created for all Tier 2 chairholders during their first terms. While the Tier 2 CRCs’ research stipend is useful to help emerging researchers establish their research program, it may not be sufficient to offset the costs of their research. If the value of the CRC award is not increased, the CRCP may need to re-examine its objectives and the extent to which they can be achieved (e.g., attraction).

Recommendation 2: Examine opportunities to strengthen the support packages offered by institutions, including considering setting a minimum expectation for the financial and non-financial resources offered to CRCs across Canada.

The evaluation also found that the ways in which the CRC award funds are used vary within and across institutions. There are concerns that this will contribute to inequities across Canada. Similar concerns were observed throughout the evaluation in relation to the support packages institutions offer to CRCs. In addition to the CRC award, institutional support packages play a pivotal role in attracting and retaining excellent researchers and the success of a CRC’s research program. The financial and non-financial resources included in a support package are often left to the discretion of the faculty nominating the CRC, which results in wide variations in the supports offered and received by CRCs within and across institutions.

In certain cases, these supports are not competitive with what is offered by other Canadian or international institutions, nor are they robust enough to ensure the success of a CRC’s research program. While some CRCs were able to negotiate their institutional support packages at the time of nomination, others did not know this was a possibility. Additionally, some CRCs did not receive what was promised in their institutional support package and there is a perception among some CRCs that women experienced greater challenges when it came to their support packages, including receiving what was promised.

The quality and amount of supports offered by institutions to ensure the success of CRC research programs and the continued achievement of the CRCP’s objectives is an ongoing concern, noted in the previous evaluation. Setting a minimum standard of financial and non-financial resources institutions must offer as part of their support packages would provide an opportunity to level the playing field for CRCs across Canada, as well as increase the competitiveness and transparency of these packages. Changes to the requirements for institutional support packages should consider any changes to the value of the CRC award, as per recommendation one, and the impact such requirements may have on institutions.

Recommendation 3: Further clarify the definition and application of the concept of research excellence throughout the nomination and review processes, in alignment with the CRCP’s EDI requirements.

Since 2000, the CRCP has been successful at identifying excellent researchers through its nomination and review processes. While the CRCP has made progress towards integrating EDI into its nomination and review processes (e.g., unconscious bias training, equity targets), it was suggested that there is an ongoing preference for traditional measures of research excellence (e.g., the number and impact of peer-reviewed publications) that may not be conducive across contexts and do not necessarily reflect the quality and relevance of research. Additionally, some key informants, case study participants and institutions (through their EDI progress reports) expressed concerns that the focus on publication may disproportionally impact researchers who are members of the designated groups.

There is an opportunity for the CRCP to take a key role in clarifying the definition and application of the concept of research excellence and how it aligns with the program’s EDI requirements, particularly in the wake of the San Francisco Declaration on Research Assessment (DORA) and the growing recognition of the value of using alternative measures of research excellence. For example, the program should develop additional guidance and training around less traditional research approaches and measures of productivity to support the individuals involved in the CRCP’s nomination and review processes. At the same time, it is important to recognize that defining and measuring research excellence is a broader issue within the research ecosystem, with implications beyond CRCP. It therefore requires the involvement of the three granting agencies. As evidenced in recent strategic plans and guidelines, the three granting agencies are examining and broadening the concept of research excellence. There may be opportunities for the CRCP to build on these efforts as the program continues to integrate EDI into its nomination and review processes.

---

1. Evaluation scope and methodology

This report presents findings from the evaluation of the Canada Research Chairs Program (CRCP). The evaluation reviewed the CRCP’s activities from April 2015 to March 2022 and was designed in adherence to the requirements of the Treasury Board’s Policy on Results (2016)^{Footnote 2} and Section 42.1(1) of the Financial Administration Act,^{Footnote 3} which requires that grants and contribution programs be evaluated every five years. Areas examined for this evaluation include the achievement of the CRCP’s expected outcomes and objectives, the extent to which the program’s objectives continue to be relevant, and the efficiency of the program’s design and delivery, including the implementation of equity, diversity and inclusion (EDI) requirements. In the wake of the global pandemic, the evaluation also examined how COVID-19 affected the CRCP, as well as CRCs and their research programs. The CRCP was previously evaluated in 2015-16.^{Footnote 4}

The evaluation was conducted jointly by the Evaluation Division of the Social Sciences and Humanities Research Council (SSHRC) and the Natural Sciences and Engineering Research Council of Canada (NSERC), and Prairie Research Associates (PRA) Inc., an independent consulting firm specializing in program evaluation. Data was collected across multiple lines of evidence, including a review of program documents, key literature, program data, CRC annual reports, institutional annual reports and financial information, as well as from a comparative analysis. Data was also collected through:

• 45 key informant interviews with program stakeholders, unsuccessful nominees to the CRCP, and previous and reallocated CRCs;^{Footnote 5}
• 10 case studies that involved a review of program data and key informant interviews with institutional representatives, CRCs and graduate students supervised by a CRC;
• a survey of institutional representatives (n=40); and
• a survey of Canada Research Chairs (CRCs) (n=1096).^{Footnote 6}

The evaluation also leveraged the opportunity to include data from a student survey conducted as part of the evaluation of the talent-related programs of the Canadian Institutes of Health Research (CIHR), NSERC and SSHRC. Additionally, the Evaluation Division engaged in a contract with Elsevier Canada for a bibliometric and altmetric study of CRCs funded between 2009 and 2018 to assess their productivity, scientific achievements and impacts, compared to a control group of matched researchers and unsuccessful nominees to the CRCP. Data were analyzed by triangulating information gathered from the different lines of evidence listed above with the intention of increasing the reliability and credibility of evaluation findings and conclusions. See Appendix A for further details on the evaluation scope and methodology.

1.1 Evaluation questions

The evaluation examined the following themes and questions:^{Footnote 7}

Relevance

• Is there a continued need for a program to attract and retain a diverse cadre of world-class researchers? (sections 3 and 5)

Effectiveness

• To what extent has the program fostered research excellence and developed research capacity? (section 4)

Efficiency

• What progress have institutions made in the implementation of the program’s EDI requirements? (section 6)
• To what extent does the CRCP design support the effective and efficient management of the program by TIPS and the implementation of the program by institutions? (section 7)

COVID-19

• What impact has the COVID-19 pandemic had on the delivery and performance of the CRCP? (section 4.5 and other sections as relevant)

---

2. Canada Research Chairs Program

Introduced in 2000, the CRCP invests approximately $311 million^{Footnote 8} per year to attract and retain a diverse cadre of some of the world’s most accomplished and promising minds to foster and reinforce academic research excellence and capacity in engineering and the natural sciences, health sciences, humanities and social sciences. The program is also committed to supporting a more equitable, diverse and inclusive Canadian research ecosystem and not perpetuating systemic barriers to the participation of individuals who are members of one or more designated groups, including racialized individuals^{Footnote 9}, Indigenous Peoples, persons with disabilities, women and gender minorities. The CRCP supports up to 2,285 CRC positions in eligible degree-granting postsecondary institutions (and their affiliate hospitals and research institutes), which represents approximately 4.8% of full-time academic positions across the country. CRCs improve our depth of knowledge and quality of life, strengthen Canada's international competitiveness, and help train the next generation of highly skilled people through student supervision, teaching and the coordination of other researchers' work. The logic model outlining the CRCP’s expected outcomes is found in Appendix B.

The administration of the CRCP is the responsibility of TIPS and is managed on behalf of the three granting agencies by SSHRC. The CRC award is an institutional grant and the number of CRC positions allocated to individual postsecondary institutions is proportional^{Footnote 10} to the funding from the three granting agencies received by researchers at the institution over a defined period. The total number of CRC positions is divided across the three federal granting agencies with 837 awards (39%) dedicated to NSERC, 837 awards (39%) dedicated to CIHR, and 474 awards (22%) dedicated to SSHRC. There are also 137 special allocation CRC awards set aside for institutions that have received 1% or less of their total funding by the three granting agencies over the three years prior to the year of the allocation. These CRC awards are not allocated by granting agency and institutions can choose the areas in which they would like to use the CRC. Postsecondary institutions that have established their eligibility and have been allocated one or more CRCs are able to nominate researchers for available positions. Nominations are assessed by a minimum of three members of the College of Reviewers (CoR) through a rigorous multistage peer-review process to ensure that the nominated researcher meets CRCP’s evaluation requirements and high standards of research excellence. Additional details about the nomination and review processes are found in Section 7.8.

There are two types of CRCs funded through the CRCP. Nominees for a Tier 1 CRC are expected to be outstanding researchers who are acknowledged by their peers as world leaders in their fields. For each Tier 1 CRC, the institution receives $200,000 annually for a term of seven years that is renewable once. Nominees for a Tier 2 CRC are expected to be exceptional emerging researchers, acknowledged by their peers as having the potential to lead in their field.^{Footnote 11} For each Tier 2 CRC, the institution receives $100,000 annually for a term of five years that is renewable once. Tier 2 CRCs also receive an annual $20,000 research stipend during their first term. When nominating a researcher for a CRC award, institutions are also able to apply for funding from the Canada Foundation for Innovation (CFI) for infrastructure support to help acquire state-of-the-art equipment that is essential to the success of the nominee’s research program.

As of May 2022, there were 78 postsecondary institutions across Canada with active CRC positions and thus the ability to nominate excellent researchers to the CRCP. At the time of this report, 1,985 of the 2,285 CRC positions across Canada were filled, with almost two-thirds by Tier 2 CRCs (61%). Consequently, the vacancy rate for CRC positions was about nine percent (9%). This is close to the average proportion of vacant CRC positions between 2010-11 and 2020-21, which was approximately 14%. Institutions may have vacant CRC positions for several reasons, including delays between the ending of one CRC’s award and the beginning of another’s and keeping a position vacant in anticipation of the program’s reallocation process (discussed further in Section 7.5).

---

3. Continued relevance of the CRCP

The CRCP’s key objectives are to enable eligible Canadian postsecondary institutions (and their affiliated hospitals and research institutes, as applicable) to:

• Foster research excellence and enhance their role in the global, knowledge-based economy as world-class centres of research excellence;
• Provide specific resources to help attract and retain a diverse cadre of world-class researchers and reinforce academic research excellence in Canadian postsecondary research institutions; and
• Support internationally competitive research and research capacity that generates social, economic and cultural benefits for Canada and Canadians.

The objectives of the CRCP are directly aligned with the federal government’s aim to hire more leading researchers and for Canada to be a leader in current and future economies to help generate economic growth for Canadians.^{Footnote 12} The program’s objective of attracting and retaining a diverse cadre of excellent researchers and commitment to reducing systemic barriers to access and participation in the CRCP for designated groups also align with government expectations to continue address systemic inequities and disparities and ensuring that Canadians see themselves reflected in the government’s work and priorities.^{Footnote 13} Finally, the CRCP’s objectives align with the departmental results for CIHR,^{Footnote 14} NSERC^{Footnote 15} and SSHRC,^{Footnote 16} which expect that Canadian research in health, the natural sciences and engineering, and the social sciences and humanities is internationally competitive; that knowledge in these disciplines is used; that Canada has a pool of highly skilled people in the natural sciences and engineering, social sciences and humanities; and that Canada’s health research capacity is strengthened.

Findings from the evaluation demonstrate that the CRCP continues to be necessary to support research in Canada and that its objectives remain relevant. Most program stakeholders and institutional representatives interviewed believe the need for the program still exists in the same capacity as when it was first implemented 20 years ago, and that its objectives continue to be relevant. Similarly, all institutional survey respondents indicated that the CRCP’s objectives remain relevant, the most relevant being enhancing the role of Canadian postsecondary institutions as world-class centres of research excellence (93%) and attracting a diverse cadre of excellent researchers (73%).

However, some key informants and case study participants noted that the emphasis on and expectations of the CRCP’s objectives may have shifted over time, particularly in relation to notions of attraction and retention, as well as research productivity and excellence. For instance, CRC awards appear to be increasingly used by institutions to retain excellent researchers. The CRCP and institutions also increasingly recognize new measures of productivity and research excellence. Some key informants highlighted the program’s increasing emphasis on EDI requirements and practices, particularly the equity targets that are focused on ensuring that the representation of CRCs in the program better reflects the Canadian population. Additionally, a small number of key informants, institutional representative survey respondents (8%) and case study participants felt that the CRCP was not meeting all its objectives. Reasons include the stagnant amount of the CRC award and the program’s inability to compete internationally in attracting excellent researchers. These changes in emphasis and expectations, as well as challenges in achieving the program’s objectives, will be discussed throughout this report.

Niche of the CRCP and synergies with other funding

Given the 20-years since the implementation of the CRCP, the program has become ingrained in Canada’s research ecosystem and is providing necessary foundational funding for institutions to allow them to foster research excellence and capacity. For example, because award alignment with an institution’s strategic research plan is expected and assessed as part of the review process, institutions often attach CRC awards to faculty positions of strategic importance to support priority research areas. Institutional representatives confirmed that the funding received through the CRCP allows their institutions to dedicate funds to developing priority areas and building research capacity in those areas, including clusters and/or centres. Since 2015, more than half of the institutions with CRCs indicated in their institutional annual reports that they place great importance on the CRCP as a catalyst to support existing research teams, research clusters and/or research centres at their institutions, with larger institutions placing a greater importance on the role of the CRCP. When asked if their institution has/had a research centre or cluster related to their area of research, almost two-thirds of surveyed CRCs (62%) said yes, with almost half of those CRCs (42%, n=492) indicating that their CRC award contributed to the creation of a research centre or cluster.

Evaluation participants noted that having CRCs, as well as research capacity and clusters in strategic areas of research, provides institutions with the ability to seek and receive other prestigious tri-agency funding, such as a CFREF grant. It was further noted that CRCs will often work as team members of a CFREF-funded project and with CERCs. The synergies between CRCs and other tri-agency funding are intentionally designed by institutions to further advance and accelerate research in areas of strategic priority to develop high-quality centres of research excellence and to increase Canada’s ability to be competitive internationally.

Program stakeholders, institutional representatives and CRCs spoke of the distinct nature of the CRCP compared to other funding programs offered by the granting agencies: this continues to make the program relevant in Canada. Among case study participants, the CRCP was viewed to be unique among federal programs due to its prestige, flexible funding, and the associated protected research time and reduced teaching requirements. Additionally, the CRC was considered to be distinctive by some key informants and case study participants because it provides funds for institutions to invest in an individual researcher and their research program, for a longer period of time. By providing researchers with a consistent funding base, they have more time to develop and achieve long-term research goal and support individual projects within that research program. Funding from the CRCP is perceived to be one leg of a three-legged stool, along with institutional support and other funding for research.

Individually, CRCs leverage the prestige associated with their award to seek additional funding to support their research programs. Many interviewed CRCs and institutional representatives explained that CRCs use funding from multiple sources (e.g., not-for-profit and private sectors, other federal grants) to maximize their research capacity. Data extracted from CRC annual reports from 2015 to 2020 show that CRCs are quite successful at securing funding from other sources. Tier 1 CRCs appear to be especially successful as they received, on average, $1,295,900 in funding from sources other than the CRCP and the CFI: this is about three time the average amount received by Tier 2 CRCs ($417,549) (see Figure 1).

According to CRCs, the need for funding from multiple sources is also related to institutions’ primary use of CRCP funding for CRC salaries. Consequently, many CRCs receive about a fifth or less of their award for research-related expenses. While some CRCs noted their frustration with the time needed to write funding applications to be able to support their research, they often believe they are more successful at receiving additional funding because of their CRC title. This was indicated as another way in which the CRC award helps accelerate a CRC’s research program and ultimately their career. Such assertions were supported by findings from the CRC survey, which demonstrate that most CRCs (77%) believe the award had a positive impact on their ability to obtain additional funding from the federal government. The ability to combine CRC funding with other research grants was also highlighted by interviewed CRCs as an important attribute of the CRCP that helps build a research program.

To verify the perception that CRCs are more successful at receiving federal funding, the success rate of CRCs and non-CRCs (i.e., researchers that have never held a CRC position) in receiving funding from certain SSHRC grants (Insight and Partnership grants) and NSERC grants (Alliance, Collaborative Research and Development, Discovery and Engage) was analyzed. As shown in Figure 2, CRCs were more successful in receiving these grants than non-CRCs, confirming this perception.^{Footnote 17}

Improving research programs

In addition to accessing funding, interviewed CRCs noted many other ways in which the CRC award improved their research programs, including increasing opportunities for national and international collaborations, and attracting high-calibre HQP. Measures of the positive impact of the CRC award on the quality and direction of research and the creation and application of new knowledge were also captured in the CRC survey, as shown in Figure 3. Several of the measures outlined in Figure 3 below are directly comparable to findings from the previous evaluation of the CRCP (quality of research, direction of research, attraction of HQP, quality of collaborations) and were either within 1 percentage point from 5 years ago or saw slight increases ranging from 2 to 6 percentage points.

---

4. Fostering research excellence and capacity

4.1 Research productivity

Research publications

A bibliometric study was conducted as part of this evaluation: it examined the average number of research publications produced by individual CRCs during the period prior to and after receiving their award, and that of a matched control group of non-CRC researchers and unsuccessful nominees to the CRCP. However, note that as the population of researchers who were unsuccessfully nominated to a Tier 1 chair was rather small, the bibliometric indicators for unsuccessful nominees were only calculated and discussed for Tier 2 CRC award nominees. To ensure a successful match between the CRCs and researchers in the control group, their level of productivity in terms of publication and citation impact was similar by design. Researchers in the control group were also selected for the size of their affiliated institution and geographical location in order to align their research environment as much as possible. A final important note to consider when reviewing results from the bibliometric analysis is that, since close to half of the CRCP chairholders in the sample have NSERC chairs (45%)^{Footnote 18}, findings for the CRCP as a whole are largely due to/associated with those chairholders.

When moving from the pre-award period to the award period, the CRCs significantly increased their research output (across all three agencies), at a level that surpassed the increases observed for the control group and unsuccessful nominees. In particular, CRCs increased their research output from an annual average of 4.9 publications before the award to an annual average of 7.7 publications after their award. In comparison, the matched control group increased their annual research output from 4.6 to 5.7 publications on average, and the unsuccessful Tier 2 CRC nominees increased their annual research output from 2.2 to 4.0 publications on average (versus a change of 3.5 to 6.3 publications on average each year for Tier 2 CRCs).

Looking at comparisons among CRCs and unsuccessful nominees, findings from the study demonstrate that the average number of research publications produced by CRCs in the period before their award was significantly higher than the average number of research publications produced by unsuccessful nominees, indicating that the CRCP is selecting more productive researchers.

More simply, Figure 4 below presents the yearly average per group for Tier 1 CRCs in the periods prior to and during the award, while Figure 5 presents the same information for Tier 2 CRCs. As illustrated in Figure 4, before their award, the Tier 1 CRCs had very similar publication trends to the matched control group. In the period when CRCs were supported by their award, both groups experienced similar trends in their productivity for the first nine years; however, the productivity of CRCs was steadily higher. Additionally, by year 10 (after the start of their awards), the productivity of CRCs continued to increase while the publication rate of the matched control group began to decrease. During the five-year period before the award in Figure 5, the unsuccessful nominees increased their research output from 1.7 to 2.9 publications per researcher per year. At the same time, the control group and CRCs started at 2.7 publications to respectively reach 3.5 and 4.3 publications per researcher per year. During the award period, unsuccessful nominees reached a level comparable to that of the matched control group, although slightly lower. The CRCs had a substantially higher research output. Note, at the Tier 2 level, a stabilization-or a decrease-in the production of CRCs and the control group is observed for the later years. This is likely, at least in part, due to the effect of the COVID-19 pandemic as the yearly production of scientific papers decreased between 2019 and 2021 in Canada and across the world.^{Footnote 19} The exact effect of COVID-19 is unclear as the research output is not aligned by calendar year but by number of years before or after receiving the award.

Of note: the scientific impact of the CRCs decreased between the pre-award and supported periods, but it remained far above the world level. The scientific impact of the control group also decreased, but to a lower extent than that of the CRCs. Therefore, no positive effect of the program could be observed in terms of the scientific impact of CRCs.

Overall, the findings from the bibliometric study show that the CRCP nomination, review and selection processes help to ensure that higher quality (or excellent) researchers receive the award. The study also shows that researchers continue to be excellent (as measured through publications) following the receipt of their CRC award and suggests that the CRC award contributes to increased rates of publication. Although a decrease in scientific impact was found in comparison to the control group, the scientific impact of CRCs following the award remained above the world level. These findings are similar to those from the bibliometric studies completed for the last two evaluations of the CRCP, which continue to validate these processes.

Many CRCs who participated in the case studies confirmed the bibliometric findings by sharing that their CRC award helped increase their publication rates by increasing their profile and opportunities to collaborate. Some CRCs also perceived the award, particularly its prestige, as having an impact on the calibre of journals in which their research is published, and they found they had further reach to publish in more journals.

Research publications by institution size and CRC Tier

As part of the bibliometric study, the publication rates of CRCs were examined across institution size and by CRC Tier. Some of the key findings from this analysis include an observed higher average number of publications for CRCs working in medium and large institutions, compared to CRCs working in small institutions. These findings align with results of the CRC survey, which show that CRCs working in large institutions reported publishing more refereed journal articles (mean = 18.6) than CRCs working in medium (mean = 12.6) and small institutions (mean = 9.9). The differences in publication rates may be because CRCs located at medium and large institutions tended to supervise more students (mean = 11.5 students and 11.3 students, respectively) than CRCs in small institutions (mean = 8.4 students). Additionally, large and medium institutions are more likely to have a higher proportion of Tier 1 CRCs (39% and 33%) than small institutions (16%). According to the CRC survey, Tier 1 CRCs (on average) tend to publish more refereed journal articles (mean = 20.8 articles) than Tier 2 CRCs (mean = 13.4 articles). The bibliometric study also found that Tier 1 CRCs had higher publication rates than Tier 2 CRCs in all research disciplines.

Alternative metrics^{Footnote 20}

The three granting agencies have acknowledged that traditional notions of research excellence such as bibliometrics can result in biased research assessment, which subsequently impacts funding decisions.^{Footnote 21} It is well documented in the literature that typical research indicators (e.g., citation and impact factors) can be discriminatory^{Footnote 22} (Davies et al., 2021). As a formal commitment to evolving the definition of research excellence, the agencies (alongside CFI) have all signed DORA. In seeking an evolved definition of research excellence, the agencies have identified diversity as a criterion of research excellence. This requires achieving a broader disciplinary mix of researchers across fields and fostering a culture of excellence in knowledge creation and translation. This new approach to interpreting research excellence will recognize fundamental knowledge creation, knowledge mobilization, multiple ways of knowing, and non-traditional research methodologies and outputs as cornerstones of Canadian research.^{Footnote 23},^{Footnote 24},^{Footnote 25}

Looking at alternative metrics, such as social media, book chapters, patents and policies, the altmetric study found that CRCs increased their presence in wider audience media (e.g., Facebook, Twitter and news media) before and after receiving their CRC award. The increase in Twitter mentions was the most significant, particularly for CIHR-funded CRCs, where there was a 47% increase. However, the share of publications by CRCs that were cited in Wikipedia decreased during the award period. The CRC survey also asked CRCs to indicate the number of alternative metric outputs they generated over the past 24 months (see Figure 6). The most common outputs were conference/symposium presentations (average of 11.1 per CRC), followed by conference/symposium posters (average of 5.5 per CRC), multimedia content (average of 2.7 per CRC) and published book chapters (average of 1.5 per CRC). These findings align with the information provided by CRCs through the case studies, specifically that the CRC award helped them to generate research outputs such as presentations at national and international conferences.

With very small differences observed between CRCs and the control group, the altmetric study could not confirm that the CRCP had a clear positive effect on the inclusion of research generated by CRCs in social media. The only exception were SSHRC-funded CRCs who experienced a higher increase in the number of Twitter mentions related to their research, as compared to the matched control group (a difference of 3%).^{Footnote 26} While there were little differences between CRCs and the matched control groups in terms of the presence of their research in social media, CRCs did outpace unsuccessful nominees across all disciplines, with the exception of citations in Wikipedia.^{Footnote 27}

Looking at survey findings about research outputs across granting agencies (see Figure 7), SSHRC-funded CRCs produced, on average, more multimedia content, book chapters and books (mean = 5.3, 3.4, and 0.8, respectively) than CIHR-funded CRCs (mean = 2.2, 1.0, and 0.1) and NSERC-funded CRCs (mean = 1.6, 0.7, and 0.2).

Figure 8 shows that surveyed CRCs who are Indigenous produced, on average, more conference and symposium presentations (mean = 17.1) than CRCs who are not Indigenous (mean = 10.9) and more multimedia content (mean = 5.6) than CRCs who are not Indigenous (mean = 2.7).

During the award period, the citation rates of the research produced by CRCs decreased in policy documents and patents. This may be attributed to the fact that these citation types require more time to mature. With very small differences observed between the citation rates of CRCs and the control group, the altmetric study could not confirm that the CRCP had a clear positive effect on the inclusion of research produced by CRCs in policy documents and patents. The study did uncover a difference in the rate of citations of CRCs and unsuccessful nominees within patents, with unsuccessful nominees being cited more often. The reasons for this difference are unknown.

Knowledge sharing and translation

CRCs often engage in a variety of knowledge translation activities to communicate the insights and findings from their research with audiences outside academia. Data from CRC annual reports show that, between 2015 and 2020, almost all CRCs (89%) indicated that they engaged in knowledge translation “frequently” (47%) or “occasionally” (42%).^{Footnote 28} When CRC survey respondents were asked how often they were invited to engage in various knowledge-sharing activities outside of academia, the most common activities reported were presenting their research to external organizations (66%); participating in working groups that focus on the application of new knowledge, including the CRCs’ research (33%); and sharing with the media (32%). Fewer respondents reported that their research results were referred to in reports, studies and strategic plans (25%), or that they participated in a public or private sector advisory body, commission or consultative group to examine or report on an issue of importance (16%). These data are represented in Figure 9.

The CRC survey also provided evidence that the research produced by CRCs contributed to developing or improving government (47%), not-for-profit (43%), private company (42%) and community-based (38%) processes, policies and products. When examined across the granting agencies, the data demonstrate that SSHRC-funded CRCs were more likely to indicate that their research contributed to developing or improving government, not-for-profit and community-based processes, policies and products every few months; NSERC-funded CRCs were more likely to indicate that their research frequently contributed to developing or improving private company processes, policies and products.

Several of the CRCs who participated in the case studies discussed the ways in which their research was shared outside of academia. These were similar to the findings from the previous CRCP evaluation; for instance, being called on by media to provide interviews, press releases, commentary or opinion articles, and to provide expert advice. In some cases, CRCs consulted and produced reports for federal departments such as Statistics Canada, the Department of Fisheries and Oceans, or the Department of Canadian Heritage; provincial government departments (e.g., ministries of health) and agencies; and government science centres. As for the private sector, CRCs provided examples of consulting with and advising industry stakeholders, as well as sitting on advisory boards.

4.2 Collaborations

Almost all institutional representatives agreed that the CRC award had a positive impact on the number of research collaborations (90%) and the quality of the research collaborations at their institution (88%). At the individual level, most CRC survey respondents (78%) reported that the CRC award had a positive effect on the quality of their research collaborations. This positive impact was also noted by almost all interviewed CRCs, who described an increase in the quality or quantity of their research collaborations since receiving their award. They often attributed this to the prestige associated with the CRCP. The award was perceived to have helped strengthen pre-existing collaborations more so than facilitating the development of new ones. CRCs also provided examples of international and national researchers reaching out to them to collaborate on research projects, as well as receiving positive responses to their own requests to collaborate with international and national researchers. Being recognized as a CRC was perceived as a mechanism for increasing opportunities to attend and present at events, ultimately resulting in more chances to network and develop collaborations. Additionally, researchers often received protected time that released them from teaching and administrative responsibilities, as well as some supplementary funding when they became a CRC. They noted this provides them with more time to focus on developing their collaborations, including travelling to work with collaborators, subsequently allowing them to expand their research capacity.

Data extracted from the CRC annual reports show that a large majority of CRCs (88%, similar to the previous evaluation’s 87%) had significant collaborations with researchers within their own institution, which included building research clusters with their colleagues to help each other apply for grants, share equipment and fund HQPs. It is possible that these collaborations existed before the CRCs received their award and may be why the award was perceived to help strengthen pre-existing collaborations About 70% (similar to the previous evaluation’s 68%) of CRCs also reported having significant collaborations with international institutions, and more than half (58%, similar to the previous evaluation’s 56%) of CRCs collaborated significantly with researchers at other Canadian institutions. CRCs provided examples of leading national research networks; participating in, advising or leading international research organizations; or collaborating with specific researchers and teams at other institutions. When asked about the types of collaborations they were involved with as part of their research, internationally recruited CRCs reported more frequent and significant collaborations with researchers working at international institutions (77%), compared to CRCs who were recruited from the host institution (70%) and those recruited from another Canadian institution (51%). Additionally, Tier 1 CRCs (78%) were more likely to report collaborations with researchers at international institutions than Tier 2 CRCs (63%).

The CRC annual reports further demonstrate the extent to which CRCs engaged in national and international private sector, government and not-for-profit collaborations. Overall, about a fifth (20%) of CRCs collaborated significantly with the private sector in Canada, while a small proportion collaborated significantly with government (14%) and not-for-profit organizations (12%). Figure 10 shows that the trend of these collaborations remained fairly consistent between 2015-16 and 2019-20. At an international level, these percentages decreased to 3% of CRCs reporting significant collaborations with the private sector, while just under one-tenth of CRCs (9%) reported engaging in significant collaborations with international governments or not-for-profit organizations (8%). Some CRCs who participated in the case studies reported having significant national and/or international collaborations with different sectors. Examples of such collaborations included advising private and not-for-profit organizations, as well as collaborations with community groups, initiatives and organizers.

Co-publication rates

The bibliometric study examined co-publication rates for CRCs before and during their award, as well as for a matched control group and unsuccessful nominees to the CRCP. International,^{Footnote 29} national^{Footnote 30} and institutional^{Footnote 31} co-publications were reviewed as part of this analysis as a proxy measure for collaborations. The bibliometric study also examined single-author publication (i.e., the CRC or matched researcher was the sole author). As these indicators are mutually exclusive, when the rate of one type of publication increased it was expected that the rates of other types of publications would concomitantly decrease.

Before their award, CRCs had higher international co-publication rates than the control group. However, their national-only and institutional-only (local) co-publication rates were similar or lower. Overall, CRCs increased their rate of international co-publication following their award by 12 percentage points. However, researchers in the control group demonstrated higher rates of international co-publication than CRCs during the latter group’s award period. This was counterbalanced by CRCs faring slightly better than the control group during their award period in terms of national and institutional co-publication. Although not confirmed by the evidence, it could be presumed that CRCs have increased their national and institutional co-publication versus international co-publication more than the matched control group because many of the CRCs were part of research clusters at their institutions and across Canadian institutions (as noted in Section 3 above).

The rates of single-author publication were very similar for CRCs funded by CIHR or NSERC and their counterparts in the matched control group during the CRC’s award period. However, CRCs funded by SSHRC experienced a decrease in their share of single-author publication compared to their control group. These findings suggest that the CRCP had a greater positive effect on the chances of SSHRC-funded CRCs to participate in collaborations resulting in co-publication. The beneficial effect appears mostly at the national and institutional levels. The larger proportion of SSHRC-funded CRCs working at small institutions may also explain why CRCs at small institutions had higher rates of co-publication and fewer single-author publications.

Overall, unsuccessful nominees to the CRCP did not progress in the same way as the CRCs with respect to collaborations when measured by co-publication rates. In particular, CIHR-funded Tier 2 CRCs had higher rates of international co-publication and SSHRC-funded Tier 2 CRCs had higher rates of institutional co-publication compared to their unsuccessful nominee counterparts. Moreover, the rate of single-author publication from unsuccessful nominees did not decrease to the same extent as they did for the SSHRC-funded Tier 2 CRCs with whom they were matched, suggesting that unsuccessful nominees experienced fewer opportunities to engage in collaborations that resulted in co-publication.

Connecting with other CRCs

A suggestion offered throughout the evaluation in support of collaborations was to have an annual conference / meeting for CRCs. This would create networking opportunities, would allow CRCs to learn about what others were working on at different institutions and help identify potential opportunities for collaboration. It would also provide opportunities for CRCs to share some of their challenges and lessons learned, and to compare their situations, including the support received by their institutions (discussed further in Section 7.7). It was also felt that such gatherings, even though virtual, would have been helpful during the pandemic. There were also perceived opportunities to bring together smaller groups of CRCs working in similar research areas for more focused networking opportunities; for instance, CRCs working with Indigenous communities.

4.3 Cross-disciplinary research

Research that crosses disciplinary and sectoral boundaries is expected to result in more durable socio-economic change.^{Footnote 32},^{Footnote 33} A review of the Strategic Research Plans of the 10 case study institutions reveals the prioritization of interdisciplinary and/or multidisciplinary research, often in areas with one or more CRCs. Also, some of the case study institutions are focusing on developing interdisciplinary research teams and are using their CRC awards to create interdisciplinary faculty positions and attract interdisciplinary researchers. In certain cases, CRCs noted that the interdisciplinarity of the CRC position is what drew them to apply, as much if not more than the fact that the position was tied to the CRCP.

In contrast, some key informants and case study participants believe that the CRCP is not currently designed to support interdisciplinary research positions. The main limiting factor highlighted was that CRC awards are associated with a specific agency, creating a siloed nomination process that does not support interdisciplinary researchers. Some case study participants highlighted the difficulty experienced by interdisciplinary nominees: they felt that the researcher needed to adjust their research program as described in their nomination package to favour one discipline over another to fit within agency-specific funding requirements. A few key informants also highlighted that the peer-review process is sometimes challenged when nominations reflect interdisciplinary research: this has resulted in nominations being returned to the institutions with questions or requests for more information. Consequently, there is a question as to whether the design of the CRCP’s nomination and review processes disadvantaged interdisciplinary researchers.

After receiving the award, however, interviewed CRCs reported that it increased their opportunities for cross-disciplinary research (i.e., interdisciplinary and multidisciplinary). Most surveyed CRCs noted that the award increased their participation in multidisciplinary research collaborations (74%)^{Footnote 34} and interdisciplinary research collaborations (73%).^{Footnote 35} The bibliometric study examined cross-disciplinary indicators to assess the extent to which the program awarded CRC positions to researchers involved in cross-disciplinary research and changes in the rates of cross-disciplinary research following the award. Before receiving their award, CRCs were at par overall with the world average in terms of interdisciplinarity^{Footnote 36} and close to the world average for their share of publications among the 10% most highly interdisciplinary publications. While Tier 2 CRCs were slightly higher than the world average when it came to interdisciplinarity, Tier 1 CRCs were close to or slightly lower than the world average. Moreover, CRCs were higher than the world average in terms of multidisciplinarity;^{Footnote 37} CIHR-funded CRCs had particularly high values compared to the world average. SSHRC-funded CRCs specifically stood out for their high level of interdisciplinarity and multidisciplinarity based on their publications before receiving their CRC award. Of note, the interdisciplinarity and multidisciplinarity publications of unsuccessful SSHRC Tier 2 nominees before their nominations were higher than the world level and above Tier 2 CRCs from all agencies.

During the award period, the level of interdisciplinarity and multidisciplinarity among all CRC publications decreased slightly. When examining specific groups of CRCs, these reductions were mainly attributed to observed decreases in the interdisciplinarity of publications from NSERC-funded CRCs. In comparison, SSHRC-funded CRCs slightly increased their share of publications among the most interdisciplinary publications in the world. In the case of multidisciplinary publications, SSHRC-funded Tier 2 CRCs increased their share of such publications, but SSHRC-funded Tier 1 CRCs had the largest decreases of all CRCs in terms of multidisciplinarity exhibited in their publications, as well as their share of the 10% most multidisciplinary publications.

When compared to the matched control group, the bibliometric analysis shows that CRCs fared slightly better than the control group on cross-disciplinary dimensions. These differences were very small and hard to validate for the overall sample of CRCs. However, when disaggregated by group, the analysis was able to show with more confidence that SSHRC-funded Tier 2 CRCs performed better than their matched counterparts in terms of the level of interdisciplinarity exhibited in their publications, as well as their share of the 10% most multidisciplinary publications. CIHR-funded CRCs performed better than the control group for their share of papers among the most interdisciplinary.

The bibliometric study further shows that unsuccessful CRCP nominees performed slightly better than CRCs on most cross-disciplinary dimensions. For instance, the average level of interdisciplinarity or multidisciplinarity exhibited in unsuccessful nominees’ publications continued to increase compared to the world average: this was not the case for CRCs except for SSHRC-funded Tier 2 CRCs. Additionally, the unsuccessful nominees’ share of publications among the 10% most multidisciplinary publications increased more than those of CRCs. However, CRCs performed better than unsuccessful nominees in increasing their share of publications among the world’s most interdisciplinary publications.

4.4 Students

According to the CRC annual reports, CRCs directly supervised an average of eight students at a time, most of whom were graduate students and one to two postdoctoral researchers. Additionally, they co-supervised approximately two students and sometimes a postdoctoral researcher. When asked about the extent to which their position (including associated CFI funding) enhanced the training they could provide to the students and research staff, CRC annual reports showed that most CRCs (76%) indicated that being a CRC significantly enhanced their ability to provide training.

CRCs reported that the award helped them attract more students to their research program, including more high-calibre national and international students. Results from the survey show that most CRCs indicated that their position increased their ability to attract more students and research staff (82%), as well as attract higher-quality students and research staff (73%). While many interviewed CRCs noted that they were able to leverage their position to attract students, they experienced some challenges engaging students because of a lack of available funding or the inadequate amount offered to students through research stipends. It was noted that the stipends for students were low and often not enough to keep up with the increasing cost of living. In one case, it was further noted that CRCs sometimes missed opportunities to hire the “best” students because the stipend amounts were too low. Studies have shown that financing is a crucial factor in attracting and retaining international students^{Footnote 38} and that there has been a global loss in international student recruitment in academia. These studies partially attributed this to a lack of investment in this area.^{Footnote 39} On average, approximately 4% of CRCP funding was used to fund students. In most cases, CRCs were required to seek additional funding to supplement their research budget for students.

Interviewed CRCs noted that they recruit students through a variety of methods, including online listings (e.g., LinkedIn, institution’s website, job boards), informal networking and peer recommendations, as well as being directly solicited by students interested in working with them. Nearly all interviewed CRCs mentioned taking deliberate action to integrate EDI practices into their hiring processes to support working with a more diverse group of students. Such actions include taking courses to better understand diversity, looking specifically for members of EDI groups when hiring, and taking steps to make all students feel welcomed despite language and cultural barriers. According to some CRCs, it is important to hire students from diverse backgrounds—academic and identity—as diversity supports successful research with opportunities to bring in different perspectives and ideas. More than half (55%) of the surveyed CRCs agreed that being a CRC increased their ability to attract students and research staff who belong to one of the CRCP’s designated groups.

Students who participated in the case studies described the activities and opportunities available to them when working with a CRC, such as:

• participating in data collection and analysis, literature and document reviews, and coding analysis;
• access to specialized labs and equipment, including specialized equipment training;
• participating in theoretical research;
• participating in national and international research collaborations and partnerships;
• attending conferences;
• presenting at events; and
• supporting the development of research publications and co-authoring.

CRC-supervised students surveyed reported having more opportunities than non-CRC-supervised students to practice technical research skills, non-technical skills and professional skills. CRC-supervised students also reported greater opportunities to participate in a variety of knowledge dissemination activities, such as preparing papers, presentations, communications, grant proposals, conferences and workshops. Additionally, as a trainee being supervised by a CRC, students received more training on new equipment, formal or informal specialized training, as well as opportunities to participate in events or activities that have the capability of expanding their network of potential employers.

Evidence from the student survey also showed that more CRC-supervised students reported having an opportunity to participate in collaborations (71%) compared to non-CRC-supervised students (61%). Furthermore, CRC-supervised students reported a higher frequency of collaborations, whether multidisciplinary, multisectoral, or with researchers from academia within and outside of Canada, compared to other students (see Figure 11). CRC-supervised students interviewed as part of the case studies confirmed that their work with CRC resulted in participation in national and international research collaborations and partnerships.

Interviewed students were nearly all positive when discussing their experience working with a CRC. They often praised their supervisor’s personal qualities, such as their passion for work, empathy, knowledge and expertise, or friendliness. Some students also mentioned benefitting from their supervisor’s leadership skills, such as lab management, grant writing, communication and independence. In certain cases, students experienced longer-term positive impacts, noting that their subsequent career and research opportunities benefitted from the training and experience gained working with the CRC. These included awards, tenure-track positions and ivy-league postdoc positions. Such assertions were supported by interviewed CRCs who noted that opportunities to work on their research program has led to students receiving prestigious positions of their own, including high-level positions within large research projects, Tier 2 CRC positions and top prizes for graduates.

4.5 COVID-19

Impact of COVID-19 on CRCs

The COVID-19 pandemic created a challenging situation for many CRCs, particularly during its first two years. The most common challenges identified included the closing of campuses and the inability to continue research in hospitals or for fieldwork, thereby not allowing them (and their students) to have access to laboratories and research spaces. Other challenges included the inability to travel (nationally or internationally) for conferences and collaborative research, community access restrictions that limited data collection, as well as unexpected family and home-life obligations. Such challenges are acknowledged in the literature: COVID-19 was found to have significant negative impacts on the research community^{Footnote 40},^{Footnote 41} and on federal research and development funding initiatives. These included the closure of suppliers / service providers, cancelled conferences, the diversion of resources to research on COVID-19, and unplanned costs of suspending R&D projects.^{Footnote 42} Most CRCs noted that these challenges resulted in some form of research delay or productivity loss. In some cases, CRCs noted that their research projects failed. Others indicated that COVID-19 severely limited access to their community contacts and eroded or destroyed relationships with the communities involved in their research. For instance, they could not continue working with Indigenous communities because they could not travel to the communities that were often closed to non-community members.

The challenges and negative impacts of COVID-19 were perceived by several CRCs and institutional representatives as being particularly onerous for women and Tier 2 CRCs who were more likely to experience disproportionate demands on their time because of family responsibilities. Findings from the literature review show that women academics often have gendered responsibilities (e.g., childcare, housework) that increased during COVID-19 lockdowns.^{Footnote 43},^{Footnote 44} As a result, they often produced less research.^{Footnote 45},^{Footnote 46},^{Footnote 47} Additionally, studies have shown that during COVID-19, early-career researchers (ECRs), such as Tier 2 CRCs, lost months or years of scientific progress because of a lack of access to research spaces, fieldwork opportunities, funding and in-person teaching opportunities^{Footnote 48} (Walker et al., 2022). While there was a general consensus that COVID-19 disproportionately impacted women and younger researchers, the case studies illustrate that women and men noted challenges related to balancing work and family obligations in the face of school and daycare closures, and their own work-from-home situations.

Although most CRCs experienced challenges that impacted their research capacity, a small number reported an increase in productivity in terms of writing and publishing. In general, they did not face the issues mentioned above and some were able to shift their research to focus on COVID-19. For instance, one CRC noted that the two years of the pandemic were the most productive of their career. However, this was an established researcher who shared that they did not have many family responsibilities or other demands on their time, and that their research team had been one of the first to return to the laboratory.

The transition to virtual platforms often did not completely mitigate the challenges experienced by CRCs and even introduced new ones. Many CRCs emphasized that virtual interactions are not a suitable replacement for in-person connections, whether for research conferences, team research meetings, collaborations or impromptu brainstorming sessions. In addition to hindering research progress, there were perceived consequences of missing out on these in-person collaborations and conferences in terms of limiting networking opportunities and future collaborations. The inability to meet in person and the resulting negative impacts on their research and students were more likely to be acknowledged by CRCs who are Indigenous who participated in the case studies and the CRC survey, as well as SSHRC-funded CRCs. Findings from the survey show that almost a third of Indigenous CRCs (30%) noted that working remotely negatively impacted their work and their students, compared to about a quarter (23%) of CRCs who are not Indigenous. Additionally, a little over a third of CRCs who are Indigenous (35%) noted that the pandemic had a negative impact on opportunities to conduct research in person and travel for research, compared to a quarter (25%) of CRCs who are not Indigenous. In certain cases, CRCs indicated that, as a mitigating strategy, they hired Indigenous community liaisons to continue and/or start community research and data collection in Indigenous communities during the COVID-19 pandemic. Having community liaison officers was also perceived to be a positive result for Indigenous communities as they provided them with additional research opportunities and learnings. When looking at the findings by granting agencies, SSHRC-funded CRCs were more likely to indicate that the pandemic impacted their ability to conduct research in person or travel for research (32%, compared to 25% of CIHR-funded and 20% of NSERC-funded CRCs), as well as participate in collaborations and in-person events necessary for networking (21%, compared to 10% of CIHR-funded and 14% of NSERC-funded CRCs).

There is a general perception among CRCs that these challenges and the negative impacts of COVID-19 contributed to a decline in mental health among many researchers and their students, which often resulted in decreased productivity. CRCs from across the case study institutions noted the difficulties they experienced keeping their teams motivated and “energized” during the pandemic. The sudden onset of many of the COVID-19-related issues noted above led to some CRCs feeling overwhelmed or overworked. Some CRCs also noted feelings of anxiety or depression.

Impact of COVID-19 on students

Students also felt that productivity had been lost over the pandemic, with many feeling that they had fallen behind in their studies or research. Students reported―and several CRCs observed―that they also faced mental health challenges during this time and felt overwhelmed, anxious and depressed, which contributed to a loss in productivity. Additionally, students shared their disappointment over their inability to attend in-person conferences, workshops and other events as they felt they had missed important experiences for networking and collaboration. It was noted that virtual communications with their supervisor were not as effective as in-person conversations and were not a suitable replacement.

International students already in Canada were especially prone to feelings of isolation as the lack of in-person events made it challenging to integrate into the community and bond with their research team. These students were often also unable to travel to visit their family due to health restrictions and/or financial reasons, which further increased feelings of isolation.

Moving forward

It was noted that there was value to having a CRC position during the pandemic, as the benefits of the CRC award (e.g., funding, protected time for research) helped mitigate the extent to which CRCs may have experienced COVID-related challenges. In some cases, CRCs credited the extension of their award term offered by the CRCP (which some institutions took advantage of) as a mechanism that provided valuable flexibility and time to adjust and continue their research activities. However, not all CRCs were aware that the CRCP offered institutions an award extension to mitigate the impacts of COVID. A few CRCs noted some useful mitigation strategies implemented by their institutions, such as work-from-home resources, additional teaching release, or providing a safe and accessible research space. Others noted feeling distinctly unsupported by their institution during the pandemic.

When asked what the CRCP and institutions could do to continue to support CRCs as they navigate the pandemic and its lasting effects, there were requests for understanding, recognition and leniency regarding expectations related to the CRCs’ research programs, specifically productivity. Several CRCs emphasized the challenges noted above and the residual impacts on their research capacity, as well as their students’ capacity. It was also indicated that it would take time for many CRCs to rebuild or restart their research programs. As such, it was recommended that the CRCP provide opportunities for nominees to explain any declines in productivity between 2020 and 2022 that may impact their success in receiving a CRC award, adjust their expectations for ECRs, and use the program to fund ongoing COVID-related research. It was also suggested that the CRCP continue to extend the terms of CRC awards for the next few years and provide additional funding to allow CRCs to catch-up or complete their planned research activities. Some CRCs noted that despite any delays in their research, they continued to spend their funding on student salaries and other expenses. As such, they perceived that an extension without funding failed to recognize the full extent of their challenges.

---

5. Attracting and retaining a diverse cadre of excellent researchers

According to stakeholders, the CRCP continues to play an important role at Canadian institutions in attracting and retaining a diverse cadre of excellent researchers. However, trends over time, as described throughout this section, imply that the CRCP is increasingly being used by institutions for retention than attraction. All respondents of the institutional representative survey indicated that the CRCP remains relevant in providing resources to attract and retain excellent researchers to Canadian postsecondary institutions: a majority (85%) indicated that the program had a positive impact on their institution’s ability to attract and retain a diverse cadre of excellent researchers. The case studies, specifically the interviews with institutional representatives further highlighted the importance of the program to attract and retain high-calibre researchers. Additionally, a little less than half of the CRC survey respondents (43%) indicated that they would have likely pursued their research program in a country other than Canada had they not received a CRC award.

Currently, the full extent to which the CRC award is used to attract or retain excellent researchers is unknown, as the notions of attraction and retention may be interpreted in different ways. For instance, a postsecondary institution may recruit (i.e., attract) an excellent researcher to fill an existing faculty position with the promise of a nomination for a CRC award. Consequently, the researcher may already be working at the institution when their nomination is submitted. As a result, their nomination may be perceived and recorded as an instance of retention. Several such instances were identified when comparing the CRCs’ nomination packages, their nomination status according to program data (i.e., whether they were nominated by their existing institution, by another Canadian institution or by an international institution), and findings from the case studies. While a CRC award was used to attract a researcher to an institution, the program data identified the nomination as coming from their existing institution (i.e., that the award was used for retention). There is an opportunity for the program to review and adjust how it collects data on attraction and retention to better understand the extent to which the CRC award is used to achieve these objectives.

Attraction

Program data suggests that between 2010-11 and 2019-20, the annual average proportion of nominations to the CRCP for researchers from other Canadian institutions and international institutions was 6% and 8% respectively. These averages demonstrate a notable change when compared to program data for the first 10 years of the program (i.e., 2000-01 until 2009-10), which indicates that approximately 15% of CRC awards were used to attract researchers from another Canadian institution, while 31% were used to attract researchers from international institutions. When asked where they were working at the time of their nomination, about a quarter (25%) of surveyed CRCs were working at an institution outside of Canada, and about a fifth (18%) were working at a different Canadian institution.

The institutional annual reports and case studies provide several examples of how institutions would attract excellent researchers from other Canadian institutions or internationally by offering a faculty position tied to a CRC. Often the prestige of the CRC award (described in Section 7.1), as well as other items promised to CRC nominees in their institutional support packages (described in Section 7.7), is what attracted them to their current institutions. The CRC survey confirmed that most CRCs that came from another Canadian institution (81%) or an international institution (86%) indicated that the CRC award was important in their decision to move or relocate to their current institution.

According to program data, Tier 2 CRCs were somewhat more likely to be recruited from another Canadian institution (8% vs. 5% Tier 1) or an international institution (12% vs. 6% Tier 1). This data also illustrates that:^{Footnote 49}

• SSHRC-funded nominees were more likely to be recruited from another Canadian institution (14% vs. 4% of NSERC-funded and 5% of CIHR-funded nominees);
• NSERC-funded nominees were more likely to be recruited from an international institution (2% vs. 1%);
• large institution nominees were more likely to be recruited from their nominating institution (84%) compared to small institution nominees (68%);
• a higher proportion of CRCs who are Indigenous (34.8%) were recruited from another Canadian institution compared to CRCs who are not Indigenous (17%); and
• a smaller proportion of francophone CRCs (11%) were recruited from international institutions compared to respondents who primarily speak English (27%) or another language (23%).

The CRC survey findings regarding francophone respondents align with information from the case studies, specifically that francophone institutions perceived greater difficulties in recruiting excellent researchers—nationally and internationally—who can speak French. This evidence appears to explain why francophone institutions primarily use the CRC award for retention purposes.

The most notable factors that respondents of the institutional representative survey reported hindered the attraction of excellent researchers include the amount of funding available to attract and retain students and research staff (20%), as well as the value of the Tier 2 (18%) and Tier 1 (13%) awards. The importance of having access to research funding to attract and hire students and research staff was also noted throughout the case studies. In particular, CRCs spoke about the challenges they experienced advancing their research program when they could not recruit or hire students. COVID-19 also played a role in the challenges experienced by CRCs in recruiting students, particularly those from international institutions, as they often did not receive the necessary documentation in time. Furthermore, one of the more commonly mentioned impacts of the COVID-19 pandemic by stakeholders was the increased challenge of recruiting international researchers, which was often associated with the difficulty of traveling and crossing borders during the pandemic. Stakeholders wondered what impact this has had on retention versus attraction to the program.

Retention

As noted above, the proportion of nominations to the CRCP for researchers from other Canadian institutions and international institutions has been declining. This decline is accompanied by a corresponding increase in the proportion of awards used for retention. Multiple sources confirm the extent to which nominations to the CRCP are used for retention purposes, although the reliability of the data is uncertain. Program data suggest that, between 2010-11 and 2019-20, the majority of nominations (annual average of approximately 86%) are for researchers already working at the nominating institution. This is an increase of a little more than 30% from the annual average of 54% of CRC awards used for retention between 2000-01 and 2009-10. According to the bibliometrics data, more than 75% of CRCs were nominated within the institution they were affiliated with at the time of their nomination. The CRC survey findings also confirm that the majority of CRCs (57%) were working at their nominating institution when they were first nominated to be a CRC. Moreover, the survey findings suggest that the proportion of CRC awards used for retention is continuing to increase, with a higher proportion of current CRCs (60%) indicating they were working at the nominating institution when nominated compared to researchers who are no longer a CRC (50%).

The importance of retaining researchers at their current institutions was evident from the case studies as institutional representatives spoke about the difficulty institutions face trying to retain excellent researchers who are often in high demand and can potentially be “poached” by other institutions. Notably, they highlighted the CRCP’s important role in retaining “prominent and excellent” researchers in whom the institutions have already invested a lot of funding. Offering them a CRC nomination/award appears to be successful: several CRCs noted that the award was important in their decision to remain at the institution in which they were working. Additionally, some institutions will focus their Tier 2 CRC awards on attracting new and prominent researchers and Tier 1 CRC awards on retaining excellent researchers. The CRCP program data support these assertions, showing that Tier 1 CRCs were more likely to be recruited from within the nominating institution. Program data also highlight that CIHR-funded nominees were more likely to be recruited from within the nominating institution. Large institutions were more likely to recruit from within the nominating institution and report that the CRC award is “very important” in retaining top researchers.

Despite the success of using CRC awards to retain excellent researchers, half of the surveyed institutional representatives (50%) noted that the award amount for the Tier 1 and Tier 2 CRCs may also hinder the retention of excellent researchers. A small proportion of surveyed institutional representatives (10%) further noted that researchers left their institution due to inadequate financial support. Many CRCs and institutional representatives participating in the case studies suggested that retaining excellent researchers could be hindered by the uncompetitive funding amount made available to CRCs compared to other international research award programs.

---

6. CRCP’s EDI requirements

EDI-related considerations have been required of the CRCP since a human rights complaint in 2003, as illustrated in Figure 12. This evaluation examined the program’s recognition and integration of EDI since 2015. In 2017, the CRCP implemented an EDI Action Plan to respond to a recommendation from the previous evaluation, which proposed that institutions adopt greater transparency in their allocation, selection and renewal processes for CRCs and make further efforts to meet their equity targets. The CRCP’s EDI Action Plan contained measures to improve the governance, transparency and monitoring of EDI within the program, including the requirement that institutions with five or more CRC allocations develop and implement their own EDI Action Plan. The following year, the CRCP updated its terms and conditions to formally add equity as part of the CRCP’s mandate and objectives, specifically the importance of attracting and retaining a diverse cadre of excellent researchers. In 2020 the program established the population-based equity targets (see Section 6.2). In 2021, the CRCP was the subject of a Canadian Human Rights Settlement Agreement that requires the program to monitor and enforce equity targets at various staggered deadlines: institutions that do not meet their targets will be limited to submitting nominations for researchers who are members of one or more of the designated groups until their targets are met. These groups are women, gender minorities, racialized individuals, Indigenous Peoples and persons with disabilities. Institutions that do not meet the final deadline for the CRCP’s equity targets (set for December 2029) will have their allocation of CRCs reduced. The purpose behind these measures is to identify, address and mitigate the systemic barriers that historically have prevented individuals who are members of one or more of the designated groups from participating in the CRCP.^{Footnote 50}

It is well documented in the literature that researchers from the designated groups experience systemic barriers and challenges that impact their participation within academia.^{Footnote 51},^{Footnote 52} These barriers may be particularly acute for individuals who are members of two or more of the designated groups.^{Footnote 53} For this reason, the CRCP applies and requires institutions with CRCs to apply an intersectional^{Footnote 54} lens to their work to better understand and address the multiple barriers and disadvantages experienced by individuals with intersecting social identities, such as race, gender, sexuality and socio-economic status.^{Footnote 55}

Findings from the CRC survey demonstrate that there are CRCs who faced challenges while becoming and during their time as a CRC that they attributed to their identity. When asked if they experienced challenges in becoming or being a CRC that they would attribute to their gender, approximately one-third of CRCs who are women said “yes” compared to 6% of CRCs who are men (for both categories).^{Footnote 56} This suggests that challenges to participating in the CRCP related to gender identity are more prevalent for women than men. CRCs who are members of other designated groups also reported experiencing challenges while becoming a CRC, ranging from one-quarter to one-half of respondents. Similar proportions reported experiencing barriers during their term as a CRC, except for those with disabilities; almost two-thirds as many more reported experiencing challenges during their term as a CRC. Of the surveyed CRCs who indicated they experienced identity-related challenges while becoming or during their time as a CRC (n = 364), the most common challenges described were gender discrimination between chairs in salary, teaching and recognition (29%), and administrative hurdles at the institution (18%).

When asked about the extent to which the CRCP implemented adequate measures to mitigate challenges for CRCs from one or more of the four designated group, about a third of surveyed CRCs indicated that the CRCP had done so to a “great” or “very great extent” (33%) or to a “slight” or “moderate extent” (32%), while 4% said “not at all.”^{Footnote 57} Looking at the findings across the four designated groups, women (29%, n=457) were less likely than men (37%, n=547) to indicate a “great” or “very great extent”, as were persons with disabilities (18%, n=84) compared to persons without disabilities (34%, n=922). The CRCs who responded to the question about the measures implemented by the CRCP (n=753) were also asked to explain why they believe the CRCP has or has not implemented adequate measures to mitigate challenges for CRCs from one or more of the four designated groups. Responses were quite varied: the most common adequate measure identified was the implementation of equity targets, as they were perceived to work in favour of the designated groups (21%). In terms of not having implemented adequate measures, a common theme was a lack of support from the CRCP before, during and after the award (8%).

6.1 Institutional EDI Action Plans

As noted above, institutions with five or more CRC allocations are required to develop an institutional EDI Action Plan to identify systemic barriers and demonstrate the activities undertaken to address and mitigate these barriers in relation to the CRCP, develop their EDI capacity, and demonstrate the impacts of these efforts.^{Footnote 58} The first institutional action plans were submitted to the CRCP in December 2018. They were formally reviewed by an external panel of experts through a two-phase process during which they were assessed against a five-point rating scale that included the following categories: exceeds, fully satisfies, satisfies, partially satisfies, and does not satisfy. Institutions whose actions plans were rated as “partially satisfy” or “does not satisfy” were given until November 30, 2020 to submit a revised plan and were only allowed to submit new nominations for researchers who were members of one or more designated groups. This requirement was only removed if the institution received a rating of “satisfies” or higher during the next review of their action plan. Institutions whose action plans did not receive a minimum of “satisfies” after resubmission on November 30, 2020 had their peer review results put on hold and were limited to submitting nominations that helped them meet their equity targets. An analysis of the expert panel review of institutional EDI Action Plans showed that following its initial review of institutional EDI Action Plans, large institutions were more successful at receiving a satisfactory rating (87%) than medium (78%) and small institutions (41%). Institutional EDI actions plans were reviewed by an expert panel again in 2022 and almost all the institutions (94%) received a rating of “satisfies” or higher.

Implementation

Many stakeholders noted that the implementation of institutional EDI Action Plans is generating positive results for CRCs and their institutions more broadly. Almost all (95%, n=36) the institutional representatives participating in the survey and whose institution developed an EDI Action Plan indicated that the exercise of developing an action plan helped to guide their efforts for identifying and addressing systemic barriers to sustain the participation and/or address the under-representation of individuals from designated groups in the CRCP (56% said “to a slight or moderate extent”; 39% said “to a great or very great extent”). Institutions were more likely to report that their EDI Action Plan helped alleviate systemic barriers in accessing the CRCP “to a great extent” for women (40%), racialized individuals (38%) and Indigenous Peoples (30%). The proportion was lower for persons with disabilities (20%) and members of LGBTQ2S+ communities (15%). Generally, surveyed institutional representatives working in large institutions were more likely than those in small or medium institutions to indicate that their institution’s EDI Action Plan helped alleviate systemic barriers “to a great extent” for Indigenous Peoples (63% at large institutions vs. 43% at medium and 12% at small institutions) and for persons with disabilities (50% at large institutions vs. 29% at medium and 8% at small institutions).

Most CRCs and institutional representatives who participated in the case studies identified successful compliance with the CRCP’s EDI requirements and noted that a broader notion of these requirements was being “taken seriously” by institution faculty and administration. This, in turn, led to a broad application of EDI guidelines and practices within the institution. Part of this was attributed to the completion of the institutional EDI Action Plan and, for some institutions, the design and implementation of department-level EDI requirements. As a result, almost all interviewed CRCs and institutional representatives identified greater diversity among CRCs and members of institutional committees, more equitable practices in the CRC nomination process, and the successful implementation of training programs such as unconscious bias training. Surveyed institutional representatives confirmed that they implemented strategies to address systematic barriers for the designated groups in relation to the CRCP, such as training (83%), human resources policies and procedures (80%), data collection and reporting (70%), and institutional support systems and resources (70%).

These findings align with those from an analysis of the annual progress reports that institutions with EDI Action Plans are required to submit to the CRCP regarding their progress in implementing their EDI Action Plans and meeting their equity targets. In 2020, 57 institutions submitted a progress report in which they self-declared as having an EDI Action Plan. An analysis of the first three objectives summarized in each of these reports identified eight broad themes under which institutions have made and continue to make progress to advance their equity objectives in relation to the CRCP:

• Human resources, policies and procedures;
• Meeting or exceeding their equity targets;
• Governance structure and leadership;
• Resources and support systems;
• EDI training and education;
• Communication of EDI priorities;
• Data collection; and
• Identification and removal of EDI barriers.

EDI stipend

In April 2020, the CRCP implemented a one-year pilot to provide institutions with an EDI stipend of up to $50,000 to help them implement initiatives that support the CRCP’s EDI requirements and address systemic barriers in their CRCP-related policies, processes and structures. After the pilot, the stipend was offered in 2022 and will be offered in 2023. Through their EDI progress reports, institutions are requested to indicate whether they used the EDI stipend to achieve any of the institutional objectives outlined in their EDI Action Plan and the impact the stipend had on achieving those objectives. An analysis of the 57 progress reports determined that 93% of institutions used the EDI stipend to achieve at least one objective. The progress report analysis further demonstrated that 68% of the institutions reported using funding from the EDI stipend to help them achieve one or more of their top three objectives. These funds were primarily used for hiring external consultants, staff or students; developing EDI materials or communication products; data collection and analysis. For instance, some institutions used part or all of their EDI stipend to hire an EDI officer, advisor or specialist to help them implement one or more initiatives in support of the CRCP’s EDI requirements.

Almost all the institutional representatives who participated in the survey indicated that the EDI stipend was a design and delivery feature that helped them successfully implement the program at their institution (88%). Survey findings also show that institutional representatives from small institutions (92%) were more likely than representatives from medium (86%) and large (75%) institutions to indicate that the EDI stipend had a positive impact on implementing the program. In their progress reports, institutions were also asked to rate the extent to which the EDI stipend impacted their ability to meet their EDI-related objectives using a six-point rating scale: extensive impact, major impact, moderate impact, minor impact, insignificant impact, or don’t know. Figure 13 outlines the extent to which the EDI stipend was reported to have an impact on the achievement of an institution’s EDI-related objectives. Institutions reported that the EDI stipend had an extensive or major impact on achieving almost three-quarters (74%) of the objectives for which the stipend was used. The stipend was only considered to have a minor or insignificant impact for the achievement of less than 10% of the objectives (8%).

Broader impact

In addition to having an impact on the way institutions manage their CRC allocations, the institutional EDI Action Plans were perceived to have brought about positive and effective EDI change at an institutional level. For instance, the EDI Action Plans were perceived to have a ripple effect on broader institutional hiring practices for faculty positions other than CRCs. Moreover, almost all surveyed institutional representatives (97%) indicated that they attribute their institution’s broader efforts to address systemic barriers for designated groups to the EDI requirements and practices of the CRCP. While the findings from individual case studies showed that the extent to which institutions’ progress on EDI-related initiatives varies, participants described significant improvements in their institution’s efforts to advance and support EDI, including increasing diversity among faculty, research teams and the general student population. EDI training, such as unconscious bias training, was seen to greatly raise awareness of EDI issues and solutions within the institution. Many CRCs observed that EDI considerations in general have become more prominent in decision-making and the overall culture at their institution.

Several institutional EDI progress reports submitted to the CRCP outline initiatives that have been implemented to support and advance EDI at an institutional level. Such measures include: establishing governance and leadership structures for EDI; aligning institutional human resource policies and procedures with the EDI principles and practices espoused by the CRCP to widen the pool of candidates; creating institutional communication plans related to EDI; and developing institution-wide data collection, reporting and monitoring processes. Some institutions are also enhancing and expanding their EDI-related training and education, and/or promoting equitable employment opportunities for members of the designated groups. For instance, one institution described that it had developed employment pathways to increase the diversity of its staff at all levels, while other institutions adopted measures to increase the engagement of members of the designated groups and promote their chances at succeeding in staffing competitions. The measures highlighted in the EDI progress reports are similar to those identified by surveyed institutional representatives asked to report on the strategies implemented by their institution to address systemic barriers for designated groups more broadly than the CRCP: training (88%), human resources policies and procedures (88%), data collection and reporting (75%), and institutional support systems and resources (70%).

Challenges and suggested improvements

While there were many positive perceptions about the value and impact of institutional EDI Action Plans, more than half of the surveyed institutional representatives (53%) indicated that it was difficult to develop and implement an institutional EDI Action Plan in accordance with CRCP requirements. These representatives (n=21) offered suggestions for improving the process of developing institutional EDI Actions Plans. These included developing better guidelines and education around equity targets (38%); increasing the flexibility for meeting targets (33%); considering the difficulties faced by smaller institutions when developing their action plan (24%); and improving and/or setting more realistic equity targets (14%). During the case studies, institutional representatives explained being uncertain of what needed to be fixed in the action plans or how to do so. They also did not understand why certain action plans satisfied requirements while others did not. It was therefore suggested that greater clarity was needed about the expectations for institutional EDI Action Plans as well as greater feedback from the program to ensure they were effectively implemented.

6.2 Equity targets

The CRCP requires that institutions with one or more CRCs establish equity targets to ensure that individuals who often face systemic barriers in the program, and in research and academia more broadly―women, persons with disabilities, Indigenous Peoples and racialized individuals―participate in the program. The new equity target-setting methodology launched in March 2021 is based on Canada’s population according to the 2016 Census: women (50.9%), racialized individuals (22%), persons with disabilities (7.5%), and Indigenous Peoples (4.9%). Institutions must meet increased equity targets that match Canada’s population distribution by staggered deadlines leading up to December 31, 2029. The program monitors the institutions’ progress toward meeting their targets. In cases where institutions do not meet their equity targets, they are limited to submitting nominations for individuals that help meet their equity targets until targets for that deadline are met.

Program data indicate that the proportion of nominees who represent one or more of the four designated groups is trending upward in terms of meeting the December 2029 deadline. As shown in Figure 14,^{Footnote 59} there has been a slight but steady increase in the number of women CRCs since 2013-14. In fiscal year 2018-19, women represented approximately 50% of nominees. Since 2019, more women have been nominated to the CRCP than men. A steady increase in the number of Indigenous researchers nominated to the program began in 2017-18, while an increase in the number of nominees who are racialized individuals or have a disability started in 2018-19. In 2019-20, there was a noticeable jump in the number of researchers from one or more of the four designated groups who were nominated to the CRCP. This is likely a response to the launch of the CRCP’s 2017 EDI Action Plan and the 2019 Addendum requirements.

When disaggregated by Tier, nomination data from 2009-10 to 2020-21 showed that members of the four designated groups are more likely to be nominated for a Tier 2 CRC award. For instance, program data illustrate that a greater proportion of Tier 2 nominees are women (42%) compared to Tier 1 nominees (25%), Indigenous researchers are somewhat more likely to be nominated for a Tier 2 CRC (3%) than a Tier 1 CRC (1%), and racialized individuals are somewhat more likely to be nominated for a Tier 2 CRC (19%) than a Tier 1 CRC (15%). According to program data, the same proportion of persons with disabilities were nominated for a Tier 1 and a Tier 2 CRC award (3% respectively).

While nominees for NSERC-funded CRCs (20%) were more likely to identify as a racialized individual than nominees for SSHRC (14%) or CIHR (16%), SSHRC received more CRC nominations for women, persons with disabilities and Indigenous Peoples:

• About half of the nominees for a SSHRC-funded CRC identified as women (50%), compared to 39% of nominees for a CIHR-funded CRC and 26% of nominees for an NSERC-funded CRC.
• A higher proportion of Indigenous researchers were nominated for a SSHRC-funded CRC (7%) compared to a CIHR-funded CRC (1%) or an NSERC-funded CRC (<1%).
• A very small proportion of researchers with disabilities were nominated for a CRC: 4% of nominations for SSHRC-funded CRCs compared to 2% of nominations for CIHR-funded and NSERC-funded CRCs.

Finally, by institution size, nominees from small institutions (5%) were somewhat more likely to identify as Indigenous compared to nominees from medium or large institutions (2% each). Conversely, nominees from small institutions (13%) were less like likely to identify as a racialized individual than from large (18%) or medium-sized institutions (17%).

Currently, as illustrated in Figure 15,^{Footnote 60} the program appears to be exceeding its target for CRCs who identify as racialized individuals and is moving closer to meeting its other equity targets.^{Footnote 61}

Many stakeholders, institutional representatives and CRCs concurred that the CRCP’s equity targets are resulting in a more diverse cadre of CRCs across Canada. Throughout the case studies, several institutional representatives and CRCs noted that most current nominations to the CRCP are of researchers who are members of one or more of the designated groups and expect that this trend will continue. In addition to meeting the program's equity targets, it was also recognized as being an important means of addressing and mitigating persistent systemic barriers and ensuring representation within the CRCP and the research ecosystem.

Unintended consequences and suggested improvements

Through this evaluation, some perceived negative unintended consequences of the CRCP’s equity targets were identified. Two of the more commonly mentioned consequences were related to the notion of tokenism^{Footnote 62} and a perceived loss of excellence. It was mentioned that having targets could create an environment of perceived tokenism, i.e. that some CRCs received their award as a way for institutions to meet targets. The belief that researchers were awarded a CRC because of their identity also led to concerns that meeting diversity targets superseded research excellence. However, findings from the bibliometric and altmetric studies indicate that the introduction of equity targets did not reduce the research excellence associated with the CRCP. In particular, there was no observed change in the quality of researchers nominated to the program following the introduction of EDI requirements, as measured through peer-reviewed publications, social media, book chapters, patents and policies. This provides evidence that the CRCP's recognition and support of research excellence was not compromised by the requirement that institutions meet equity targets.

Some case study participants also suggested that greater focus has, at times, been placed on certain equity groups than others. Specifically, some CRCs perceived a greater focus on gender, with Indigenous researchers and researchers with disabilities receiving much less attention. For example, it was noted that people with disabilities received little to no support as required by the EDI requirements and that cultural differences regarding language and output assessments were not considered in the nomination and reporting processes. In addition, it was noted that reviewers largely ignore the work of Indigenous CRCs and of those working with Indigenous communities when evaluating research outputs (described further in Section 7.8). Consequently, a small number of CRCs did not agree that CRCP’s equity targets have led to positive change at their institution. They believe their institutions and departments remain inequitable. Through the survey, institutional representatives suggested more and better education for staff about the designated groups.

Other unintended consequences perceived by some respondents and mentioned during interviews included:

• CRC awards are not being renewed so institutions can hire new CRCs to meet their equity targets;
• researchers are feeling forced to self-identify so their institution can meet its targets;
• institutions are splitting Tier 1 positions into two Tier 2 positions to meet equity targets;
• CRCs who are members of one or more of the designated groups are being poached by larger institutions;
• based on their size, location and fields of specialization, some institutions have greater difficulty in hiring researchers from the four designated groups.

These unintended consequences highlight the continued need for the CRCP to increase awareness of the role and value of EDI within the program. It was suggested that the CRCP develop better guidelines and education around the equity targets, increase the flexibility to meet targets, consider the difficulties faced by smaller institutions and certain disciplines, and improve and/or set more realistic targets.

---

7. Efficiency of program design and delivery

7.1 Status and prestige of the CRC award

Findings from the evaluation indicated that almost all institutional representatives surveyed noted that the status and prestige of the award facilitated the attraction (90%) and retention (93%) of excellent researchers (see Figure 16). Furthermore, most surveyed CRCs (85%) indicated that the status and the prestige of the CRC award were important factors in their decision to accept a CRC position. These survey results were reinforced by data collected through interviews with CRCs who felt that the prestige of the CRC award was a key factor in their decision to accept a nomination to the CRCP or renew their CRC position. They were also considered main benefits of holding a CRC award, with many CRCs noting them as the main benefit. The CRCs explained that the prestige makes the CRCP stand out from other research funding programs in Canada because it is a strong indicator to their peers and themselves that their career is on an upward trajectory. Being able to showcase their CRC title on their resumé and as part of their email signature was considered important for several CRCs.

In addition to increased recognition, CRCs indicated that the status and prestige of the CRC award has a positive influence on their ability to secure research funding, pursue additional research opportunities, and engage and collaborate with other researchers at the local, national and international levels. The recognition associated with a CRC position was also considered a catalyst for CRCs to receive editorial, advisory and leadership positions in academic journals, and to become affiliated with prominent research groups and international research institutes. Some CRCs also felt that the award helped them attract high-quality students and postdoctoral fellows. However, many of the students who participated in the case studies were often not aware that their supervising researcher was a CRC until after they began working for them or during the process of becoming their supervised student. Generally, students would approach the CRC because they knew that the researcher was excellent and well respected in their field, but not specifically because of their title. The fact that their supervisor was a CRC was considered a bonus as students would often receive several benefits from this connection, as noted in Section 4.4.

Many of the CRCs agree that the CRCP and the award are well known and highly regarded in Canada. However, findings are mixed about the international awareness and recognition of the award. While some CRCs perceived that the award had improved their opportunities to engage and collaborate with international researchers, others were unaware of the existence of the award before they were approached as a candidate for nomination. For instance, several CRCs recruited from international institutions and a few CRCs recruited from a different Canadian postsecondary institution noted that they were not aware of the CRCP before being nominated. Some institutional representatives and CRCs suggested that, despite the status and prestige of the award in Canada, the CRCP is not as competitive internationally because of its stagnant value. which may also reduce awareness of the program at an international level. This led some CRCs to indicate that they would like CRCs and their research contributions to be better recognized in Canada and internationally.

7.2 CFI funding

Funding for research infrastructure is necessary for innovative research and has been shown to be a driver for Canadian institutions to attract and retain sought-after national and international ECRs.^{Footnote 63} To support CRCs’ research, institutions can request funding for research infrastructure from the CFI in combination with a CRCP nomination. The CRCP has a partnership with the CFI in relation to their JELF. The JELF is a key strategic investment that helps institutions recruit and retain outstanding researchers by providing funding to acquire the tools that enable the innovative work of leading researchers. The JELF contributes up to 40% of the total cost of an infrastructure project: the remainder must be provided by the institution or other eligible partners. In many cases, these funding partners have no direct involvement in the research and technology development activities (e.g., provincial governments, suppliers). In other cases, these funding partners have a direct interest or involvement in research and technology development activities (e.g., vendors of specialized instruments who are seeking technological improvements or non-governmental organizations who will use the research results to improve public policies). For small institutions with a special chair allocation through the CRCP, the CFI may fund up to 100% of the project’s eligible infrastructure costs if the total project costs do not exceed $75K. Similar to the CRC award, the JELF is an institutional grant and the decision to apply for this funding in relation to a CRC award is at the discretion of the institution.

Between 2010-11 and 2018-19, approximately 36% of CRCs had JELF funding associated with their CRC award: the average amount of funding received was $168,700. NSERC-funded CRCs were more likely to have JELF funding associated with their CRC award (39%) compared to CIHR-funded and SSHRC-funded CRCs (37% and 27% respectively). During this period, Tier 2 CRCs were more likely to have JELF funding associated with their CRC award than Tier 1 CRCs (40% and 29% respectively). More CRCs working at institutions in the Atlantic region (54%) had had JELF funding than CRCs working in Quebec (36%), Western Canada (35%) and Ontario (33%). Given the concentration of JELF funding in the Atlantic region, it is not surprising that approximately half of CRCs located in small (49%) and medium institutions (42%) had JELF funding associated with their CRC award, while approximately one-third of CRCs located in large institutions (32%) had JELF funding, as illustrated in Figure 17.

CRCs and institutional representatives perceived funding from the CFI to be crucial for attracting and retaining excellent researchers in Canadian institutions. The option to apply for CFI funding as part of the CRC award was highlighted as important by CRCs and institutional representatives as it helps increase the prestige of the CRC award, as well as its competitiveness with international research funding programs. Findings from the surveys suggest that the opportunity to receive funding from the CFI is more commonly used to attract excellent researchers from another institution: the majority of surveyed institutional representatives (78%) and CRCs (59%) noted that CFI funding was a key factor for recruitment. Additionally, the opportunity to receive CFI funding was rated as important by a higher proportion of surveyed CRCs recruited to the nominating institution (66% of CRCs recruited from outside Canada; 64% of CRCs recruited from another Canadian institution) than CRCs who were nominated by their existing institution (53%). When it comes to using CFI funding for retention, data from the institutional annual reports indicate that large institutions were more likely to report that CFI funding was a “very important” factor in retaining excellent researchers (86%), compared to small institutions (57%).

According to the CRC survey, about 59% of CRCs reported that an opportunity to apply for CFI funding was included as part of the support package offered by their institution. When the possibility of applying for CFI funding was discussed during the case studies, several CRCs at various institutions indicated that they were unaware that an application for CFI funding could be associated with their CRC nomination. Additionally, some CRCs were uncertain as to why the institution did not offer them (or provide them) with the opportunity to apply for CFI funding in conjunction with their CRC nomination. It was also noted in the case studies that CFI funding was usually not sought as part of CRC renewals, especially if CFI funding was received as part of the original CRC nomination.

When used in conjunction with CRC funding, CFI funding allowed institutions and CRCs to purchase important research equipment and modernize their research space. Almost all surveyed institutional representatives (90%) agreed that CFI funding had a positive impact on the overall quality of research conducted in their institution, the institution’s capacity to produce new knowledge, and the opportunities to conduct research within their institution. Most institutional representatives further noted that CFI funding had a positive impact on the number (88%) and quality (85%) of research collaborations at their institution, the use of research conducted (83%), and the capacity of CRCs to leverage additional funding (83%). CRCs who had access to CFI funding emphasized its importance to helping them achieve their research outcomes. When asked about the impact of a lack of CFI funding, more than one-third of surveyed institutional representatives (40%) indicated a perceived inability to obtain necessary research infrastructure; about one-third (32%) indicated a loss of or inability to attract HQP. Approximately a quarter (26%) indicated that it would reduce research activity and/or quality.

7.3 Tier 2 stipend

The previous evaluation of the CRCP led to recommendations for TIPS to investigate the feasibility of increasing the award value and/or indexing it to the inflation rate, and to examine options to ensure more robust institutional support packages for CRCs. In Budget 2018, the federal government announced an increase in the funding associated with Tier 2 CRC awards to better support ECRs. In response to the evaluation recommendations and the announcement in Budget 2018, the CRCP implemented an annual $20,000 research stipend for all new Tier 2 CRCs, which the institution would receive during the CRC’s first award term. As this stipend was only for research, it was expected that it would help ensure a more robust support package for Tier 2 CRCs to help them establish their research programs and build capacity.

When asked about the role of the Tier 2 research stipend on attraction and retention, most of the institutional representative survey respondents noted that the stipend facilitated the attraction (65%) and retention (70%) of excellent researchers. Additionally, about 72% of the CRC survey respondents who had received the stipend indicated that the stipend was a factor in their decision to accept the CRC award. Case study findings reveal that the flexibility of these funds—the funding can be used, for example, for hiring students or for travel—is viewed as very important. The $20,000 stipend was generally perceived to have had a positive impact in helping Tier 2s establish their research program. However, a few CRCs reported that the $20,000 stipend was not a significant amount compared to their total research expenses. Some were not aware of the stipend’s existence even though they were eligible to receive these funds. In the latter case, it is possible that some CRCs are unaware of the stipend as the funds are rolled up into their institutional support package.

7.4 Length of the CRC awards

As noted in Section 2, Tier 1 CRC awards are tenable for a term of seven years and renewable once, while Tier 2 CRC awards are tenable for a term of five years and renewable once. Since 2015, 36% (n=843) of the nominations have been renewals. Until 2017, Tier 1 CRC awards were renewable indefinitely; however, TIPS limited the number of renewals in response to a recommendation from the previous CRCP evaluation. Recognizing that the average age of Tier 1 CRCs was increasing, a recommendation was made to mitigate the risk that mid-career researchers would receive fewer opportunities to be nominated for a CRC award. It was also expected that limiting the number of renewals for a Tier 1 CRC award would support the institutions in recruiting new excellent researchers. During the evaluation, a few institutional representatives noted that limiting the number of renewals for Tier 1 CRC awards enabled new and emerging researchers to access the CRCP. It was further noted that it also helped increase the diversity of the program by creating opportunities to nominate researchers who are members of one or more of the designated groups.

According to the surveyed institutional representatives, the length of the Tier 1 CRC award is more likely to help retain excellent researchers (55%) than attract them to an institution (43%). Most surveyed institutional representatives further reported that the length of the Tier 2 CRC award also facilitated the attraction and retention of excellent researchers (55% respectively). The availability of long-term funding can be a key factor in retaining ECRs, such as Tier 2 CRCs, within an institution as it provides sufficient time to develop their projects and research teams.^{Footnote 64} Additional evaluation findings show that most CRCs agreed that the length of the CRC award increased the attractiveness of the position. They view the CRCP as distinct from other research funding because the program provides funds for institutions to invest in an individual researcher for a longer period than most other funding from the three granting agencies. This allows researchers to develop long-term research programs.

Although the length of the awards benefits the institutions and CRCs, some CRCs noted that they left their CRC position before the end of their term because they were unsure what they would do and what supports would be available to them to continue their research program. When opportunities for a new position arose that either provided more clarity for their future or was a promotion, they chose to accept the new position. CRCs interviewed as part of the case studies echoed concerns about the lack of clarity regarding available opportunities following their CRC term. Therefore, while the CRCP supports the development of research programs, the ability to keep them operational after the end of the award concerns some CRCs. It was suggested, primarily by a few Tier 2 CRCs, that a transition period for CRCs nearing the end of their term should be implemented to ease them out of the CRCP and minimize potential disruptions to their research programs.

7.5 Reallocation

At regular published intervals, the number of CRC allocations per institution is recalculated: this results in a redistribution of CRCs across institutions. The timing of when a reallocation exercise will occur is published on the CRCP’s web pages. Reallocation exercises took place in 2008, 2010, 2012, 2014, 2017 and 2020. The reallocation calculation is based on the funding received by researchers from the three granting agencies in the three years before the reallocation exercise. TIPS provides institutions with annual data to help them plan for potential losses or gains of CRC awards during the next reallocation exercise. If an institution loses one or more CRCs during the reallocation exercise, they are given some flexibility in deciding how to adjust. They can choose to return vacant CRCs, thus having no financial impact on their researchers. Alternatively, they can choose to use a deactivation mechanism for active CRCs, resulting in a researcher losing their CRC position and the associated funds over a period of 18 months. If a CRC is deactivated before the end of their term, they retain their title as a CRC until the end of their existing term. Another option is for the institution to use flex moves^{Footnote 65} to try to minimize their loss of active CRCs. An example of a flex move is to shift a vacant CRC in one discipline to another discipline where a CRC was lost (e.g., shifting a vacant SSHRC-funded CRC to become an NSERC-funded CRC).^{Footnote 66}

Evaluation findings suggest that the main consequence of the reallocation exercise is that a CRC loses their position and funding before the end of their term. This was noted as particularly challenging for CRCs if the institution had vacant CRC positions, but chose instead to reallocate a filled position. It was suggested that the CRCs should receive more warning about the reallocation process from their institution and that the reallocation of an active CRC be aligned with the end of their term. In addition to the challenges experienced by CRCs, a few institutional representatives noted difficulties with the reallocation exercise, specifically that it was unclear how to manage this process. One institutional representative explained that it is difficult to plan ahead regarding CRC allocations because of the various factors that need to be considered (e.g., changing financial situation, staffing needs, etc.) and that may change during the reallocation exercise. As such, they find themselves in a situation where they need to make sudden changes to their CRC allocations, which may result in ending a CRC’s term early.

After the previous evaluation of the CRCP, the program was encouraged to investigate ways to minimize the impact of the reallocation exercise on CRCs who lost their positions. The following changes were made:

• Small institutions have a probationary period – Since 2019, if the average funding of an institution drops below the eligibility threshold of $100,000 (annual average) received from the three federal granting agencies, the institution will enter a probationary period until the next allocation exercise and will continue to remain eligible if, during these years, the average funding amount increases beyond the eligibility threshold.^{Footnote 67}
• Unlimited flex moves for a limited period – The CRCP instituted unlimited flex moves as a limited time measure (from November 2017 to December 2019) to help institutions meet their equity and diversity targets and allowed institutions to convert their CRCs across tiers and disciplines beyond the current flexibility limits.^{Footnote 68} Currently, the institutions are once again limited to the number of flex moves they may make.
• Next reallocation exercise in five years – The last reallocation exercise was completed in January 2020. The next one will not occur until 2025 when the frequency of the reallocation exercises will be revisited. In addition, agencies communicate on a yearly basis with the institutions in order to clarify the trend in tri-agency funding and assess any impact it could have in the next reallocation exercise.

Some institutional representatives appreciated the recent changes to the reallocation process and perceived the extension to five years to provide more opportunities to plan ahead. Other benefits of the five-year timeline between allocation exercises include providing institutions with enough time to fill available CRC positions and more stability for CRCs. However, some institutional representatives and CRCs did not appear to be aware of these changes and have not experienced the effects of the above measures (mainly the new five-year published timeline). It was unclear at the time of this evaluation if the new measures will address some of the issues raised by CRCs, such as receiving more warning from their institution about their chair being reallocated and aligning the reallocation with the end of their term.

7.6 Value and use of the CRC award

Value of the CRC award

Institutions receive $200,000 a year for up to 14 years for Tier 1 CRCs and $100,000 a year for up to 10 years for Tier 2 CRCs. Institutions also receive an annual stipend of $20,000 for the first five-year term of Tier 2 CRCs.^{Footnote 69} With the exception of the Tier 2 stipend, these amounts have not changed since the inception of the CRCP in 2000. According to the Statistics Canada Consumer Price Index, there has been a 52.7% increase in the value of assessed goods and services between March 2000 and September 2022.^{Footnote 70} Consequently, if the CRCP awards were indexed to reflect this rate of inflation, the annual valuation of the awards would be $305,400 for Tier 1 CRCs and $152,700 for Tier 2 CRCs.

Although most of the surveyed CRCs (82%) indicated that the availability of research funding from the CRCP was an important factor in their decision to accept the CRC position, many interviewed CRCs, institutional representatives, and key informants noted issues with the CRCP’s static award values. They perceived that the award not being, at a minimum, indexed to inflation has reduced the value of the award in terms of its “purchasing power” year over year. Despite the award value being only one feature a researcher considers when deciding to accept an award, CRCs spoke about the static award values reducing the prestige and appeal of the award, which has made it less competitive in attracting international researchers and retaining excellent researchers in Canada because several other countries have increased their commitment to funding research and innovation in recent years.

Confirming this perspective, a comparative review of countries^{Footnote 71} that have research chair funding programs with similar objectives to the CRCP found that while the grant amounts were variable, some offered annual funding in the range of C$200,000 to C$300,000. For instance, the Australian Laureate Fellowship offers approximately C$264,747 a year plus salary, salary supplement and funding for two post-doc and two postgraduate researchers, The United Kingdom’s Earnest Rutherford Fellowship offers up to about C$261,711 a year in funding, with salaries being agreed to by the host institution. It is important to note, however, that many of these programs offer a small number of grants compared to the CRCP (e.g., the Earnest Rutherford Fellowship offers 10 grants per year and the Australian Laureate Fellowship offers 17 grants per year). Additionally, a few international chairs programs offer a higher award, which may be more comparable to the CERC program. For instance, the Danish National Research Foundation Chair offers grants valued between C$1.02M and C$4.09M per year and does not allow salaries to be covered by the grant.

Use of the CRC award

When the program was first implemented, it was decided that institutions could use the funds in a flexible manner so long as they were used to support the CRC and their research program. Examples of eligible expenditures include: the incumbent’s salary and benefits; salary and benefits of members of the CRC’s team (e.g., students, post-docs); professional/technical services; recruitment costs and relocation expenses; costs associated with outfitting research and office space for the incumbent and their team; administrative costs related to the research program; acquisition, maintenance, operation of research equipment and other research resources; and other costs of research (e.g., travel, workshops, computing, publication costs, material and supplies, etc.). This design feature remains unchanged.

There is a perception among program stakeholders that the reduced value of the CRC awards has resulted in institutions increasingly using the award for CRCs’ salaries and reducing the amount of funds provided to CRCs for research. An analysis of program data revealed that since 2009-10, the largest expense by institutions has consistently been the salary of CRCs, which remained steady at an annual average of around 73% of total CRCP funds. Other ways in which CRCP funds are used include administrative costs (approximately 6% of funds), research related expenses including salaries for non-students such as technicians (10%), salaries for students (4%), equipment and materials (3%), travel (2%), professional services (1%). Less than one percent of the CRCP funding is used for research time stipends.

The case studies provided examples of how the use of CRCP funds varies by and within institutions. Institutional representatives and CRCs confirmed that many institutions use the majority, if not all, of CRC funding to cover the cost of the CRCs’ salaries and for other administrative costs. However, a few institutions provide CRCs with approximately half or most of the funding from their CRC award for their research. One institutional representative described how their institution transitioned from using the majority of CRCP funds for salaries to providing the bulk of the CRCP funding to CRCs in the form of a research grant. This was done to help ensure a commitment from the faculty that would house the CRC as it was required to provide salary funding, as well as ensure the success of the CRC’s research program. While each institution may take a general approach to the amount of the CRC award used for the CRC’s salary, decisions about the use of funds may also be made at the faculty level. As a result, some CRCs within an institution may receive a greater proportion of the CRC award funding for research than other CRCs within the same institution. Concerns about the disparity between and within institutions about their use of the CRCP award funding were raised by institutional representatives and CRCs as this leads to in an inequitable playing field for CRCs across Canada.

During interviews, some program stakeholders suggested that institutions may use CRCP funds to cover most of the costs of a CRC’s salary to free up institutional salary funds to hire another researcher in the same area as the CRC. The intent is for the researcher to collaborate with the CRC, helping develop research capacity in a particular area. When this idea was presented in some of the case studies, a few institutional representatives and CRCs indicated, however, that while this may happen within certain faculties, it was left to the faculty’s discretion. Other CRCs found no evidence that their faculty or institution hired or was planning to hire additional researchers in their department who may then work with them.

As noted above, the static value of the CRC award is an ongoing concern for program stakeholders, institutional representatives and CRCs. In particular, there are concerns that the reduction in the value of the award in terms of its “purchasing power” will negatively impact the extent to which CRCs are able to achieve their research goals and support the achievement of the CRCP’s objectives. There is a considerable desire within the research community for an increase in the amount of funding provided for the CRC awards, with some stakeholders and CRCs suggesting that additional funds could be specifically dedicated for research. However, if additional funding is not possible, some CRCs noted that they would like to see a requirement for a minimum amount of the CRCP funding to be dedicated to research and for the amount of funds institutions can use for salaries and administrative costs to be reduced. Concerns about static funding for the CRCP and the lack of adjustment for inflation were noted in the last two evaluations of this program and resulted in recommendations for the three granting agencies to revise and/or index the award amounts.

7.7 Institutional support

When an institution nominates a researcher for a CRC award, it must submit a nomination package in which it details the researcher’s qualifications, their proposed research program, and how the institution will support the CRC. This support is often referred to as the “institutional support package” and outlines the quality of the institutional environment, as well as the financial and non-financial resources offered by the institution to ensure the success of the CRCs proposed research program. The most common supports offered by institutions included protected time, administrative support, research funding from the CRCP, access to research infrastructure at the institution, funding to support students and/or research personnel, and the opportunity to apply for CFI funding.

The institutional support packages offered as part of the nomination for a CRC award generally play an important role in attracting and retaining excellent researchers in Canadian postsecondary institutions. As evidenced by the findings of the CRC survey, most CRCs indicated that the protected time from teaching and/or administrative duties (80%) and funding to support students and/or research personnel (78%) were important factors in their decision to accept the award. When disaggregated by granting agency and institution size, protected time was of a higher importance for SSHRC-funded CRCs (92%) and CRCs working in small (92%) or medium-sized (87%) institutions, compared to CIHR-funded (77%) or NSERC-funded (76%) CRCs, and by CRCs working in large institutions (76%).

Some CRCs indicated that the support packages offered by their host institutions are not always competitive with what is offered by other Canadian or international institutions. In certain cases, CRCs negotiated to increase the initial support package offered by their nominating institution to ensure they received enough financial and non-financial resources and assistance to facilitate a successful research program. Of the 475 CRC survey respondents who noted that they would have likely pursued their research program in another country if they had not received the CRC award, 103 (22%) indicated it was because the support packages offered by international institutions were more attractive. About a quarter of these CRCs (22%, n=105) further noted that a driving force for considering conducting their research in another country was because the research support and conditions were more attractive at international institutions.

An analysis of the CRC annual reports suggests that almost all CRCs (90%) report receiving protected time for their research. A small number of these researchers noted that receiving protected time was advantageous for their research as it provides more time and opportunities for building research capacity through collaborations and student training. Of CRCs who did not report receiving protected time associated with their CRC award, a small number agreed with this lack of time because of the importance they accorded to teaching and to bringing their experience into the classroom. It was noted by some CRCs (primarily at smaller institutions) that, in practice, having protected time did not lead to less teaching or less work overall. In certain cases, their teaching load did not decrease once they became a CRC. Others felt pressured to continue teaching the same amount to maintain career trajectories or research program success.

Findings from the case studies corroborate those from the CRC survey and annual reports, with CRCs noting that funding for research―including funds to support students―and protected time made available through the institutional support packages were of highest value to them. For instance, some CRCs at two institutions noted that the protected time allowed them to put more energy into starting their laboratory and/or on their research. The availability of institutional facilities was also considered a bonus for their research. Other CRCs noted that the administrative support from the institutions was very helpful.

When asked whether they received what was promised in their support packages, surveyed CRCs generally indicated that they received what was promised. A small number even noted that they received more than what was promised. For example, some CRCs (8%) indicated that they received research funding from the CRCP, funding to support students and/or research personnel, and/or protected time that was not originally included in their support package. However, a few Tier 2 CRCs noted that they did not receive what was promised, including not having access to a research cluster or centre at their institution (6%) and/or not receiving the opportunity to apply for funding from the CFI (4%).

The support packages offered by institutions when they nominate a researcher to the CRCP will often vary within and between institutions, including the amount of protected time and funding provided for research. More than half of the surveyed institutional representatives (58%) indicated that support packages vary somewhat across their institution. Of these institutional representatives (n=24), the majority (83%) further indicated that the support packages for CRCs often vary due to the differing needs according to discipline. Explaining the variation within institutions, some interviewed institutional representatives noted challenges in attracting researchers and indicated that they would improve the offered support packages to better attract these researchers. One representative also noted that there is not a blanket approach to the amount of protected time offered or provided to CRCs within their institution, but CRCs can negotiate their protected time.

There is some evidence that the variation in support packages and what is promised in these packages varies by gender. Of the surveyed CRCs who indicated they experienced an EDI-related challenge in becoming or during their time as a CRC (n=364), most cited gender discrimination between CRCs in terms of salary, teaching requirements and recognition (29%, n=105). Of these respondents, 95% were women CRCs (n=98). Furthermore, during the case studies a few women CRCs indicated that they did not receive what was promised to them in their institutional support package: they were more likely than men to not have the opportunity of applying for CFI (5.0% vs. 1.5%) and/or of developing a research cluster (6.6% vs. 2.6%) although it had been as part of their support package.

Generally, CRCs expressed disappointment that the support packages varied within and across institutions, particularly when the CRC was receiving less than their counterparts in other faculties within their or other institutions. Some CRCs were also unaware they may have been able to negotiate their support package. Several program stakeholders also expressed concerns about the variation across institutional support packages and noted that some of the packages were not robust enough to ensure the success of a nominee’s or a CRC’s research program. In particular, it was suggested that some nominations to the CRCP present a challenge for peer-reviewers and members of the Interdisciplinary Adjudication Committee (IAC) because they are not convinced that the supports offered by the nominating institution are sufficient, although the nominee could be an excellent researcher. In such cases, it was noted that IAC members may exercise the deferred recommendation process, which sends the nomination package back to the institution for clarification. For the 2019, 2020 and 2021 CRCP competitions, it appears that 17% of nomination packages for which the deferred recommendation process was exercised included concerns about the institutional support package. Additionally, several program stakeholders noted that they were limited in the extent to which they could monitor whether institutions were compliant in promising the supports they offered to CRCs.

Suggestions for improving the support packages offered by CRCs and stakeholders include:

• minimum requirements for institutional support packages regarding the amount of funding for research, teaching release, etc. to reduce inequity among CRCs; and,
• increased transparency on institutional support packages and how they differ within and across institution.

As many of these findings related to institutional support and CRCs not receiving what they were promised by institutions were noted in the previous CRCP evaluation, the program has already taken some action. The program is updating its reporting tools so that institutions will be required to indicate what was promised to the CRCs in their nomination package and what they actually received (in terms of financial and non-financial support, excluding funding from the CRCP and the CFI). This reporting update should allow the program to monitor compliance with the promises made in the institutional support packages.

The quality and amount of supports offered by institutions to ensure the success of CRC research programs and the continued achievement of the CRCP’s objectives is an ongoing concern for CRCs and program stakeholders. These concerns were noted in the previous CRCP evaluation, and the institutional supports offered to CRCs was an area of recommendation. In particular, it was recommended that the CRCP examine options to ensure CRCs received robust support packages from their institutions.

7.8 Nomination and review processes

CRC nominations are submitted by institutions that have established their eligibility and have received one or more CRC allocations from the CRCP. Each nomination is peer-reviewed by a minimum of three members of the CoR, which is composed of more than 1,000 volunteer experts (including current CRCs) from a wide range of research fields. The reviewers are chosen for each nomination based on their level of expertise in the nominee's field and their ability to provide a detailed, unbiased, critical review. If the nomination contains interdisciplinary research, TIPS makes efforts to ensure that the reviewers selected either individually or collectively have expertise in all the relevant disciplines.^{Footnote 72}

If the reviewers unanimously agree on funding the nominated CRC, the recommendation goes to the TIPS Steering Committee for the final decision on whether to support the nomination. The Steering Committee is composed of the presidents of NSERC, SSHRC, CIHR, CFI, the Deputy Minister of Innovation, Science and Economic Development Canada and the Deputy Minister of Health Canada. However, if at least one reviewer expresses concerns about a nomination, it is sent to the IAC^{Footnote 73} for review. The IAC either provides a recommendation to the TIPS Steering Committee on whether to approve and fund the nominated CRC or may send the nomination back to the institution through the deferred recommendation process for clarification or updates. The nomination and review processes are presented in Figure 18.

When asked about the CRCP’s nomination and review processes, stakeholders, institutional representatives and CRCs generally favoured these processes and noted that they worked relatively well. However, concerns about the nomination and review processes were noted throughout the evaluation, including:

• the length and complexity of the nomination package;
• a lack of transparency about the program’s expectations for the information included in the nomination package;
• having to connect the nominee’s research to the institution’s strategic direction;
• the length of the review process;
• the inability to respond to reviewers’ feedback;
• some members of the IAC may lack the ability to assess certain research proposals, particularly when they may not be familiar with the discipline or area of research.

As noted in Section 4.3, some stakeholders and case study participants perceived the design of the nomination process not to be conducive to supporting interdisciplinary research as the CRC awards are associated with individual granting agencies. A few CRCs also indicated that their original or renewal nomination occurred during the COVID-19 pandemic, which delayed the review process.

Some program stakeholders were also concerned that more nominees from one or more of the designated groups were referred to the IAC for additional review. Analysis of the program data reveals that between 2009-10 and 2020-21, the IAC received a higher proportion of nominations for Tier 2 CRCs compared to nominations for Tier 1 CRCs (49% and 30% respectively). As noted in Section 6.2, researchers who are members of one or more of the designated groups are more likely to receive Tier 2 nominations than Tier 1 nominations, suggesting that there is an increased likelihood that nomination packages for CRCs who are members designated groups are sent to the IAC. This may help explain why more nomination packages are sent to the IAC for women than men (48% and 39% respectively), for racialized individuals (50% compared to 41% non-racialized individuals), and for nominees with disabilities (52% compared to 43% of nominees without disabilities). The same proportion of nomination packages were sent to the IAC for Indigenous nominees as for non-Indigenous (43%). It is not known why a higher proportion of nominees from most of the designated groups are referred to the IAC (i.e., is it their research programs? their research outputs? institutional support?). However, the program is undertaking a review to see whether systemic barriers may affect access (i.e., success rates) to the program, including whether a nomination is referred to the IAC.

Research excellence

Throughout the evaluation, the definition and application of the concept of “research excellence” was one of the most significant areas of concern related to the nomination and review processes. In academia, the term “research excellence” is widely used but is often ambiguous.^{Footnote 74},^{Footnote 75} Program stakeholders and institutional representatives spoke about the ambiguity of the term: despite having a general sense of the criteria used to assess research excellence, it was not always clear how to apply these criteria.

The criteria most often used to assess research excellence in relation to the CRCP, as identified by institutional representatives and IAC members, included:

• research productivity (often in relation to publications such as number of papers);
• international stature and reputation;
• evidence of leadership;
• ability to obtain funding;
• impact outcomes (often in relation to publications, such as H-index, citation counts)^{Footnote 76};
• collaborative activities;
• significant contributions to research initiatives;
• contributions to research training and ability to attract students;
• fit of research within the institution’s strategic research plan;
• reputation as an expert, which may include accolades, prizes, distinguished fellowships, editorials and editorial activity on prestigious journals.

It was also recognized that traditional measures of research excellence, such as the number and impact of peer-reviewed publications, continue to be favoured during the CRCP’s nomination and review processes. This approach aligns with findings on research excellence in the literature, specifically that many current indicators of research excellence are not robust across contexts, are overly quantitative,^{Footnote 77} and focus on academic impact (e.g., bibliometric indicators) instead of societal impact.^{Footnote 78},^{Footnote 79} Some key informants, case study participants and institutions (through their EDI progress reports) expressed concerns that the focus on publications (e.g., number of publications and citations, H-index) privilege certain groups and types of scholarship over others, thereby reducing the level of diversity within the program. For instance, current measures and guidelines for assessing research excellence may not recognize different systems of knowledge, community-based research, other forms of productivity, or the impacts of a researcher’s personal circumstances on their productivity. Some key informants, case study participants and institutions (through their EDI progress reports) suggested that research excellence cannot always be quantified, and that more holistic evaluative processes should be considered to incorporate people’s life circumstances, challenges or unique accomplishments. It was argued that there needs to be more consideration of non-traditional CVs, career trajectories and research outputs, rather than focus on traditional productivity assessment.

Most key informants, institutional representatives and some CRCs noted that the CRCP nomination and review processes are becoming more flexible in recognizing new measures of productivity and research excellence, and that reviewers and IAC members are taking more and different measures into consideration. Such efforts are likely in response to the fact that the three granting agencies have signed onto DORA (see Section 4.2) and are seeking to evolve their definition of research excellence to recognize fundamental knowledge creation, knowledge mobilization, multiple ways of knowing, and non-traditional research methodologies and outputs as cornerstones of Canadian research. For instance, one of the key priorities outlined in CIHR’s 2021 strategic plan is to develop and promote a renewed concept of research excellence that values EDI.^{Footnote 80} In May 2022, NSERC released guidelines on the assessment of contributions to research that incorporate the principles of DORA in recognizing a broad range of contributions.^{Footnote 81} SSHRC also supports the consideration of a broad range of research contributions^{Footnote 82} and, in 2018, developed the Guidelines for the Merit Review of Indigenous Research to ensure that Indigenous research incorporating Indigenous knowledge systems is recognized as a scholarly contribution and meets SSHRC’s standards of excellence.^{Footnote 83} It was also acknowledged that the CRCP implemented some helpful mechanisms to support EDI-related considerations in the development and review of nomination packages (e.g., unconscious bias training).

There remain, however, some perceived shortcomings with the nomination and review processes related to illustrating and measuring research excellence in the context of the CRCP’s EDI requirements. In particular, it was suggested that more work could be done to clarify the definition and application of “research excellence” to further align this concept with EDI during these processes. For instance, some IAC members continue to struggle when assessing proposed research programs that involve methodologies that are not rooted in traditional, Western science. Additionally, some institutions and CRCs experienced challenges developing nomination packages as they struggled to “fit” the CRC’s research program and/or accomplishments within the program’s and research community’s perceived expectations of what qualifies a researcher as “excellent.” It was requested that the CRCP provide additional clarification and training to support a better understanding of different research approaches, particularly those used for Indigenous research, as well as further guidance about other measures of research excellence that should be considered during the nomination and review processes. Examples of other measures of research excellence suggested include mentoring, working with government committees, producing grey literature, working with and sharing research with Indigenous communities, and knowledge sharing on social media platforms.

7.9 Program expenditures

The cost-efficiency analysis included grants and administrative (both direct and indirect) expenditures from 2013-14 to 2021-22.^{Footnote 84} As presented in Figure 19, between 2013-14 and 2017-18, the CRCP’s annual operating expenditures had been gradually decreasing. In 2018-19, these expenditures began to increase because of the additional funds allocated to the program through Budget 2018 to award 285 more CRCs each year in an effort to increase the number of available Tier 2 CRC positions. In 2021-22, the expenditures once again began to decrease.

For the CRCP, administrative expenditures were examined in relation to grant expenditures to calculate the cost in cents to administer the program for every $1 granted. Since 2013, administering the CRCP costs approximately 2.18₵ per $1 granted (2.05₵/$1 since 2015), compared to the cost-efficiency ratios of a similar program, the CERC Program (5.49₵ per $1 as reported in the CERC 2018-19 evaluation).

Figure 19: Operating ratio for the CRCP between 2013-14 and 2021-22

---

8. Conclusions and recommendations

8.1 Conclusions

Continued relevance for the CRCP

The program’s key objectives of fostering research excellence; attracting and retaining a diverse cadre of researchers; and supporting research that generates social, economic and cultural benefits remain relevant and continue to align with federal government priorities. However, the emphasis placed on some objectives may have shifted over time, particularly in relation to notions of attraction and retention, research productivity and excellence, and EDI. The provision of institutional funding to invest in an individual researcher and their research program and for a longer period of time makes the CRCP a unique program that, when used to build research capacity and clusters in strategic areas of research, provides institutions with the ability to seek out other tri-agency funding (e.g., CFREF). Additionally, CRCs leverage the prestige associated with their award to receive additional grants to fund their research programs. Leveraging other funding is a necessity for CRCs because many institutions primarily use CRCP funding for CRC salaries; consequently, CRCs on average received a fifth or less of their CRC award for research funding.

Fostering research excellence and capacity

The findings from this evaluation show that the CRCP (including accompanying CFI funding) continues to foster research excellence and capacity. Results from the bibliometric and altmetric studies illustrate that the program is successful in identifying excellent researchers and that the CRC award contributes to increased rates of publication. When engaged in knowledge translation, CRCs will often share their research with various sectors through presentations, participation in working groups and the media. Findings from the evaluation also show that CRCs are more likely to collaborate with researchers working at the same institution and that CRCs increased their rate of national and institutional co-publication more than a similar set of non-CRC researchers. Additionally, the program was perceived as increasing opportunities for CRCs to engage in cross-disciplinary research. While the findings from the bibliometrics study indicate that the level of interdisciplinarity and multidisciplinarity among all CRC publications reduced slightly, CRCs fared slightly better than the control group on cross-disciplinary dimensions. However, some program stakeholders and case study participants do not perceive the CRCP as being designed to support interdisciplinary research positions, primarily because CRC awards are associated with a specific granting agency, creating a siloed nomination process. Finally, CRCs credited their award with helping them attract more high-calibre students and enhancing their ability to provide training. Students supervised by CRCs identified having greater opportunities to practice their skills, participate in training and knowledge dissemination activities, as well as attend events and participate in collaborations.

The COVID-19 pandemic created challenges for many CRCs and CRC-supervised students as a result of closed campuses and inability to continue research, travel and collaborations. This often led to reductions in productivity and, in some cases, a decline in mental health. Case study participants highlighted that they perceived the pandemic to be particularly challenging for women and younger researchers (i.e., those who may be more likely to have young children at home) due to potential family responsibilities. These perceptions are supported by the literature that outline challenges experienced by postsecondary researchers during the pandemic. While most of the pandemic’s impacts were negative, a few CRCs saw an increase in productivity: because they did not have many family responsibilities or other demands on their time, they were able to take advantage of unique opportunities when they arose. The extensions offered by the CRCP to help CRCs offset the impact of the pandemic and complete their research was appreciated and is something that CRCs would like the program to continue for the next while. As Canada continues to move out of the pandemic, it is also hoped that the CRCP―and more specifically reviewers―will consider the impacts of the pandemic on research productivity of existing and new CRCs when seeking an award or renewal.

Attracting and retaining a diverse cadre of excellent researchers

Overall, the attraction and retention of excellent researchers remains a key component of the CRCP’s objectives. However, evidence from this evaluation suggests that the use of the CRC award has shifted over time to retaining researchers more than attracting them. Reasons for the shift include the belief that there is a need to keep excellent Canadian researchers in Canada. Additionally, program stakeholders perceived increasing international competitiveness for excellent researchers over the last few years and noted that the award value and/or institutional support packages are not competitive enough to attract excellent researchers to Canadian institutions. During the period of this evaluation, the COVID-19 pandemic also impacted the extent to which institutions were able to recruit international researchers who may have experienced difficulty travelling and crossing borders.

EDI requirements

The CRCP continued to implement EDI-related requirements and mechanisms for the program and institutions in recognition of previous human rights complaints and settlement agreements. Such requirements and mechanisms included EDI Action Plans, the EDI stipend and equity targets, which were perceived to be generating positive results for the program and institutions more broadly. For instance, many institutions have undertaken activities in support of EDI, including training, human resources policies and procedures, data collection and reporting, and institutional support systems and resources. Also, although not all institutions with CRCs are meeting their equity targets, the program overall is exceeding its target of CRCs who are racialized individuals and is moving closer to meeting its other equity targets. While the CRCP and institutions have started to implement measures aimed at addressing systemic barriers, some CRCs who are members of one or more of the designated groups continue to face challenges in becoming a CRC or during their time as a CRC. The equity targets have also had some unintended consequences. Proposed opportunities for improvement by the CRCP in relation to institutional EDI Action Plans and equity targets include: more awareness about the role and value of EDI within the program; better education and guidelines regarding the equity targets and the different designated groups; increased flexibility for meeting targets; consideration for the difficulties faced by smaller institutions and certain disciplines; improving and/or setting more realistic targets; and providing more feedback during the development of action plans.

Efficiency of program design and delivery

The CRCP remains a cost-efficient program and the operating ratio has been decreasing since 2013-14 from 2.59₵ per $1 of grant funding awarded to 1.98₵ per $1 of grant funding awarded. Many of the design and delivery features of the CRCP play a role in attracting and retaining excellent researchers to Canadian postsecondary institutions, including the status and prestige of the award, being able to apply for CFI funding, the Tier 2 stipend, the length of the CRC award, the value of the CRC award, and institutional support packages. These same features are also important contributors to the success of a CRC’s research program and the extent to which it fosters research excellence and capacity. However, the program features listed can either facilitate the achievement of the CRCP’s objectives (e.g., creating opportunities to hire excellent researchers who are members of one or more designated groups, creating opportunities for collaboration, hiring students) or act as a deterrent (e.g., creating inequities among CRC groups, not being able to compete with international institutions, limited funding for research activities and resources).

While attracting and retaining a diverse cadre of excellent researchers and fostering research excellence and capacity were not specifically attributed to the reallocation and nomination and review processes, both processes have a role to play in supporting these CRCP objectives. Through these processes, institutions determine who is nominated for a CRC award and who gets to keep their award (e.g., CRCs are nominated for a second term, CRCs are not reallocated). Members of the CoR and the IAC recommend who receives an award, but, ultimately, the TIPS Steering Committee confirms the award. Moreover, the nomination and review processes may affect the extent to which the program fosters research excellence and capacity based on expectations for institutional support packages.

Opportunities for the CRCP to improve several of its design and delivery features were identified throughout the evaluation, including:

• increase international recognition and awareness of the CRC award;
• implement a transition period for CRCs nearing the end of their term to ease them out of the CRCP and minimize potential disruptions to their research programs;
• provide CRCs with more warning about the reallocation process and have the reallocation of an active CRC align with the end of their term;
• increase the value of the CRC award and specifically dedicate additional funds for research;
• ensure minimum requirements for institutional support packages (e.g., research funding, teaching release) to reduce inequity among CRCs;
• increase transparency of institutional support packages and how they differ within and across institutions;
• clarify the definition and application of research excellence to further align this concept with EDI during the nomination and review processes;
• provide more training to IAC members to support a better understanding of different research approaches, particularly approaches that are used for Indigenous research; and,
• provide further guidance around other measures of research excellence that should be considered during these processes.

8.2 Recommendations

Findings discussed in this report, particularly those related to the design and delivery features of the CRCP, led to the identification of three recommendations to help ensure that the program continues to achieve its objectives through its support to CRCs and Canadian postsecondary institutions.

Recommendation 1: Investigate opportunities to increase the value of the Tier 1 and Tier 2 CRC awards with a specific emphasis on dedicating a minimum amount of funding for research.

One of the most notable areas for improvement recognized throughout the evaluation was the value of the CRC awards and the ways in which the funding is used by institutions: this was a persistent finding across most lines of evidence. CRC awards have approximately 53% less purchasing power when adjusted for inflation and static funding for the last 20 years: this was perceived by stakeholders, institutional representatives and CRCs to have reduced the prestige and appeal of the award, making it less competitive for attracting international researchers and retaining excellent researchers in Canada. While the full extent to which the CRC award is used for attraction versus retention is unknown, the proportion of CRC awards used to attract researchers has consistently decreased over the 20 years of the program, and may continue to decline, in part, due to the value of the award.

Additionally, the majority of CRCP funds are used for CRC salaries with approximately 21% of overall funds allocated for research-related expenses (e.g., student salaries, equipment, research stipends). As a result, financial resources to support CRCs’ research programs are often provided through the institutional support package (not including the CRCP funds) or must be sought by CRCs through other funding sources. Such funding is critical to help CRCs build research capacity (e.g., hiring students). Without it, CRCs may be unable achieve their research goals and support the achievement of the CRCP’s objectives.

The value of the CRC awards is a longstanding issue for the CRCP. The two previous evaluations recommended increasing the funding amount of the awards, possibly indexing to inflation. Increasing the value of the CRC awards and s dedicating funds for research would provide an opportunity for the program to ensure the continued success of funded CRCs and the achievement of CRCP’s objectives. While Budget 2018 announced additional funding for the CRC program, it was focused almost exclusively on the Tier 2 level: the number of chair positions was increased (by 250 Tier 2 and 35 Tier 1) and a $20,000 research stipend was created for all Tier 2 chairholders during their first terms. While the Tier 2 CRCs research stipend is useful to help emerging researchers establish their research program, it may not be sufficient to offset the costs of their research. If the value of the CRC award is not increased, the CRCP may need to re-examine its objectives and the extent to which they can be achieved (e.g., attraction).

Recommendation 2: Examine opportunities to strengthen the support packages offered by institutions, including considering setting a minimum expectation for the financial and non-financial resources offered to CRCs across Canada.

The evaluation found that the ways in which the CRC award funds are used vary within and across institutions. There are concerns that this will contribute to inequities for CRCs across Canada. Similar concerns were observed throughout the evaluation in relation to the support packages offered to CRCs by their institutions. In addition to the CRC award, institutional support packages play a pivotal role in attracting and retaining excellent researchers and the success of a CRC’s research program. The financial and non-financial resources included in a support package are often left to the discretion of the faculty nominating the CRC, which results in wide variations in the supports offered and received by CRCs within and across institutions.

In certain cases, these supports are not competitive with what is offered by other Canadian or international institutions, nor are they robust enough to ensure the success of a (potential) CRC’s research program. While some CRCs were able to negotiate their institutional support packages, others did not know this was a possibility. Additionally, some CRCs did not receive what was promised in their institutional support package and there it was perceived among some CRCs that women experience greater challenges when it comes to their support packages, including receiving what was promised.

The quality and amount of supports offered by institutions to ensure the success of CRC research programs and the continued achievement of the CRCP’s objectives is an ongoing concern noted in the previous evaluation. Setting a minimum standard for the financial and non-financial resources institutions must offer as part of their support packages would provide an opportunity to level the playing field for CRCs across Canada, as well as increase the competitiveness and transparency of these packages. Changes to the requirements for institutional support packages should consider any changes to the value of the CRC award, as per recommendation one, and the impact such requirements may have on institutions.

Recommendation 3: Further clarify the definition and application of the concept of research excellence throughout the nomination and review processes, in alignment with the CRCP’s EDI requirements.

Since 2000, the CRCP has been successful at identifying excellent researchers through its nomination and review processes. While the CRCP has made progress toward integrating EDI into its nomination and review processes (e.g., unconscious bias training, equity targets), it was suggested that there is an ongoing preference for traditional measures of research excellence (e.g., the number and impact of peer-reviewed publications) that may not be conducive across contexts and do not necessarily reflect the quality and relevance of research. Additionally, some key informants, case study participants and institutions (through their EDI progress reports) expressed concerns that the focus on publication may disproportionally impact researchers who are members of one or more of the designated groups.

There is an opportunity for the CRCP to take a key role in clarifying the definition and application of the concept of research excellence and how it aligns with the program’s EDI requirements, particularly in the wake of DORA and a growing recognition of the value of using alternative measures of research excellence. For example, the program should develop additional guidance and training around less traditional research approaches and measures of productivity to support the individuals involved in the CRCP’s nomination and review processes. At the same time, it is important to recognize that defining and measuring research excellence is a broader issue within the research ecosystem, with implications beyond the CRCP. It therefore requires the involvement of the three granting agencies. As evidenced in recent strategic plans and guidelines, the three granting agencies are examining and broadening the concept of research excellence. There may be opportunities for the CRCP to build on these efforts as the program continues to integrate EDI into its nomination and review processes.

---

Appendix A – Methodology

The evaluation was conducted jointly by the NSERC-SSHRC Evaluation Division and Prairie Research Associates (PRA) Inc., an independent consulting firm specializing in program evaluation. The purpose was to examine the relevance and achievement of the CRCP’s expected outcomes and objectives, the efficiency of the program’s design and delivery, the implementation and short-term impacts of the program’s EDI requirements, and the impacts of COVID-19 on the CRCP, institutions and CRCs.

The evaluation examined the following questions and sub-questions.

Relevance
1. Is there a continued need for a program to attract and retain a diverse cadre of world-class researchers?
1.1 What distinctive role does the program play in the university funding environment relative to other federal tri-agency funding programs?
1.2 How have the program and institutions defined and applied the concept of “research excellence” over the years (e.g., in candidate recruitment, nomination and selection)?
Effectiveness
2. To what extent has the program fostered research excellence and developed research capacity?
2.1 Has the program resulted in the attraction and/or retention of a diverse cadre of world-class Canadian and foreign researchers?
2.2 How do the scientific achievements of chairholders compare to a matched group of non-chairholders at Canadian universities?
2.3 What is the impact of the CRCP on chairholders in the longer-term (i.e., on maintaining/expanding their research program following the end of their term as chair) (e.g., productivity, grants secured, prestigious appointments/awards)?
Progress on Implementation of EDI Requirements
3. What progress have institutions made in the implementation of the program’s EDI requirements?
3.1 What unintended impacts (positive or negative), if any, have occurred as a result of the implementation of the program’s EDI requirements?
Efficiency of Program Design and Process
4. To what extent does the CRCP design support the effective and efficient management of the program by TIPS and the implementation of the program by institutions?
4.1 Is the program being delivered in a cost-efficient manner?
COVID-19
5. What impact has the COVID-19 pandemic had on the delivery and performance of the CRCP?

Data for this evaluation were collected using the following methods and sample groups.

Lines of Evidence
Key informant interviews Interviews provided in-depth feedback on the program, drawing on the experiences and perceptions of various stakeholder groups. A total of 42 interviews were conducted with 45 interviewees: TIPS managers and staff / Management committee and Steering committee members (n=16) IAC members (n=6) Unsuccessful nominees (n=8) Early Exit Chairholders (n=5) Reallocated Chairholders (n=6) External stakeholders (n=4)
Surveys Surveys provide efficiency in obtaining information from large groups of stakeholders, allow for the quantification and aggregation of a large volume of data, and afford the ability to obtain outcome information directly from relevant stakeholders. Online surveys were conducted with: Chairholders who were nominated to the CRCP between 2010 and 2020 (1,096 complete, response rate of 36.6%); Representatives with institutions that had at least one CRC award between 2010 and 2020 and that did not participate in a case study (40 complete, response rate of 57.9%); Graduate students: The CRCP evaluation leveraged findings from a survey conducted as part of the NSERC-SSHRC Talent evaluation. The survey was conducted with former and current graduate students who were directly or indirectly funded by at least one of the three granting agencies between 2014 and 2020. A total of 11,611 graduate students completed the survey for the Talent evaluation and approximately 15% of those students (n=1,787) indicated that they were supervised by a CRC. This evaluation analyzed the responses from CRC-supervised students regarding their experiences working with a CRC, including the type and quality of training they received. These findings were compared to the responses from students who were not supervised by a CRC to determine if there were any differences between the two groups.
Case studies Ten case studies were conducted with small, medium and large institutions across Canada, with a combination of anglophone and francophone institutions. The case studies examined the relevance, performance, design and delivery of the CRCP, with a specific focus on identifying the different institutional uses of CRCP funds and the extent to which the allocation of these funds is conducive to achieving program objectives. The case studies involved a review of institutional documents and annual reports, CRC annual reports, and interviews with institutional representatives, CRCs and graduate students supervised by CRCs. A total of 113 interviews were conducted as part of the case studies, 76 with CRCs, 14 with institutional representatives and 23 with graduate students.
Bibliometric and altmetric studies The bibliometric and altmetric studies contributed to this evaluation by demonstrating the effectiveness of the CRCP in attracting a diverse cadre of excellent researchers to Canada. The studies also demonstrated the productivity and collaborations of CRCs in comparison to a matched sample and unsuccessful nominees to the program (i.e., the extent to which the CRC awards contributed to enhanced research capacity at Canadian institutions). The study included a sample of 2,117 CRCs who were nominated between 2009 and 2018, 119 unsuccessful nominees to the CRCP between 2009 and 2018, as well as a sample of 2,117 researchers who were matched to CRCs using the following criteria: volume and impact of publications, country of origin, career stage, discipline, institution size and institution region.
Administrative data review An analysis of program data from nomination packages, institutional annual reports, CRC annual reports, statement of accounts, as well as of success rates, was conducted to provide descriptive statistics pertaining to the demographic composition of CRCP nominees, to address the evaluation questions around cost-efficiency, and to provide indices of productivity and capacity building in the case of the CRCP (e.g., research outputs, knowledge mobilization, collaborations, etc.).
Document and literature review The document review contributed to a strong understanding of the CRCP and the context in which it operates. Documents reviewed include corporate and federal documents, performance measurement frameworks and the CRCP web pages. The literature review focused on the most pertinent publications available to characterize and describe recent changes in the Canadian and international research context that have impacted the program’s ability to attract and retain a diverse cadre of excellent researchers and EDI considerations within academia. The review also examined the impact of the COVID-19 pandemic in these areas.

Some limitations and challenges, along with corresponding mitigation strategies for the evaluation of the CRCP, are presented in the table below.

Challenges	Mitigation
Uncertain environment due to the COVID-19 pandemic. Current requirements to address the pandemic were impacting research and evaluation activities. Many officials were pre-occupied with addressing the impact of the pandemic within their organization and traditional methods such as in-person interviews or case studies involving site visits were not feasible.	Flexibility in timelines and data collection methods was required for the evaluation. Long lead times and pivoting to virtual platforms were used to support project momentum and maintain quality in data collection.
Complexity and diversity of opinions. During the consultations for the CRCP evaluation design, stakeholders were found to have diverse perspectives about the program and the key issues to be examined. It was expected that these disparate views would also be encountered during the evaluation and may make it difficult to reconcile and draw robust conclusions and recommendations that have traction for the organizations.	A high degree of transparency in data collection and analysis was important in these circumstances to ensure balance. It was generally possible to at least partially mitigate this challenge through the triangulation of the lines of evidence (i.e., use of findings from other methods to arbitrate mixed responses obtained in a survey or interviews).
Survey sample size issues. The small population size for the survey of institutional representatives (approximately 70), coupled with challenges in participation levels due to the pandemic, could have increased the magnitude of non-sampling error due to nonresponse, which would considerably reduce the scope of the analysis.	Whenever possible, complementary data from other data collection methods (e.g., case studies) were used to validate the findings from this group.
Use of altmetrics. The use of altmetrics in an evaluative context is far from standardized, and these approaches are even less well established in contexts of rigorous program analysis. There are still many unknowns about the motivations for altmetric citations of peer-reviewed articles. For instance, in certain cases altmetrics may be used by researchers for self-promotion and therefore are not indicative of research uptake. Additionally, some altmetric citations are considered more useful and reliable (e.g., media references, Wikipedia mentions, clinical guidelines, policy documents and patents) than others (e.g., Facebook, Twitter).	The findings from the altmetrics were used sparingly and triangulated with other lines of inquiry. As altmetrics capture different phenomena from peer review and traditional research assessments (i.e., scientific publications) they cannot be considered as a replacement for such measures, but rather as a complement. However, altmetrics may have some potential advantages including inclusion of multiple stakeholders, diversity in the type of outputs measured, and speed in terms of identifying the readership of an article instead of waiting for the uptake of an article’s findings to be identified in subsequent research.
Limited participation in case studies. Institutional representatives, chairholders and students identified as potential interviewees for the case studies could potentially not be willing and/or available to participate in an interview or may lack sufficient familiarity with the program to provide an informed opinion.	To ensure a balance in the number and type of interviewees across cases, back-up interviewees were identified during the case study design phase, and snowball sampling techniques were used to supplement original lists of potential interviewees. Along with triangulation from other sources, qualitative evidence from interviewees was weighted toward those with more knowledge of the program.
Attribution of results. Given that the researchers attracted and retained through the program often obtain funding from other sources to achieve their research programs, chairholders’ achievements cannot be wholly attributed to the program.	The evaluation has contextualized performance data on chairholders within the larger Canadian ecosystem, for example, by controlling for and/or acknowledging the contribution of other sources of funding to the chairholders’ achievements. Moreover, the purpose of the evaluation was not to determine causality or attribution, but rather to determine the extent to which the program is mobilizing a diverse cadre of world-class researchers at Canadian institutions and contributing to their subsequent successes.

---

Appendix B – CRCP Logic Model