Peer Assessment Process Research

The Canada Council and Imprint Consulting Inc. conducted a research project on the Council’s peer assessment process.

Research goal

This project aimed to better understand the effectiveness of the Canada Council’s current peer assessment practices and explore ways to improve the process from the perspectives of peer assessors and staff.

Methods

Imprint Consulting Inc. surveyed 507 peer assessors and held focus groups and interviews with 47 peer assessors and 73 staff members involved in peer assessment.

Outcomes

The findings of this research reflect that peer assessment at the Canada Council is highly regarded. Key strengths include:

• A well-organized, well-supported and well-informed process.
• Interactions among peer assessors that are respectful and constructive.
• Canada Council staff are seen as professional.
• The knowledge, perspective and diversity of committees.
• The integrity, transparency, and inclusiveness of the assessment process.

Some challenges identified include:

• The process is demanding for both assessors and staff, and the increasing volume of applications poses challenges.
• While virtual assessment meetings have improved scheduling flexibility, they have also introduced barriers for some, especially for assessors who are Deaf or have disabilities.
• The shift to multidisciplinary assessment has introduced challenges including ensuring access to disciplinary expertise or representation of priority groups when there are conflicts of interest and supporting peer assessors to feel comfortable assessing applications involving an unfamiliar field of practice.

Here is a summary of recommendations for strengthening the process. The full list of recommendations is in the report below.

• Reducing the demands of the process on peers by improving application forms and the pre-screening process.
• Improving the consistency and organization of how files are grouped for the peer assessment committees.
• Exploring if a return to an in-person process is better for some committees, for example, for Indigenous and Deaf and disability arts committees.
• Enhancing assessment through cross-calibration across committees, use of advisors and communication on multidisciplinary assessment.
• Updating training and resources for assessors on reducing bias and how to apply assessment criteria.
• Improving the application process for applicants by reviewing forms and providing feedback.
• Exploring use of standing committees for peer assessment.
• Reassessing compensation and expectations for peers.
• Providing more accommodations and support to assessors who are Deaf or have disabilities, or who have been impacted by trauma.
• Providing more context for applications from Canada Council’s strategic groups and prioritizing recruitment of peer assessors from different regions.

The Canada Council is integrating the findings and recommendations from this research to guide improvements to its assessment processes.

Questions?

Please contact the Research, Measurement and Data Analytics Section at research@canadacouncil.ca.