Peer Assessment: How the Council Makes its Decisions
The Canada Council for the Arts was created by an act of Parliament in 1957 to support the study, enjoyment and production of works in the arts. As Canada’s national arts funding body, its core business is to provide grants and services and to award prizes to professional Canadian artists and arts organizations throughout Canada, in a wide variety of artistic disciplines and practices.
Two basic operating principles guide the Canada Council’s work:
- its arm’s-length relationship with the federal government, which allows the Canada Council to make artistic decisions free from political interference
- peer assessment as the basis for the majority of its decisions as to which artists, artistic projects and arts organizations receive support
The arm’s-length principle
The Canada Council for the Arts is a Crown corporation operating at arm’s-length from Government. When the Canada Council of the Arts Act was enacted, Parliament recognized the necessity to provide the new cultural organization with an important level of autonomy from Government which would protect the Council’s ability to make judgments and to base its funding decisions on the evaluation of artistic quality. Parliament also recognized that the Council’s governance structure needed to be commensurate with the nature and scope of its mandate and role.
In this relationship, Parliament sets over-arching public policy and determines the amount of money the Canada Council receives, and the federal Cabinet appoints the Members of the Board and the Director for fixed terms. Under the Canada Council for the Arts Act and the Council’s By-laws, Board members are responsible for the conduct and management of the Corporation’s activities. They oversee the Corporation’s ongoing activities and set its strategic directions. In fulfilling its oversight role and governance functions, the Board has to provide advice, challenge management and approve all proposed plans, budgets and future directions.
As a Crown Corporation that administers public funds, the Canada Council has a high degree of responsibility to account fully and openly for its operations. The Canada Council complies with federal legislation and is subject to regular audits by the Auditor General of Canada. Each year, it reports to Parliament through the Minister of Canadian Heritage.
In order to fulfill its legislated mandate, the Council is empowered to set its own priorities, strategic directions, budgets and operational practices. It can independently establish funding programs and make granting decisions free from political interference.
The peer assessment principle
In Canada and elsewhere, peer review is a respected method for assessing quality of achievement in creative and intellectual occupations. It is the preferred method of assessment in agencies that support the arts, sciences and humanities. It is used to evaluate submissions to academic journals and requests for accreditation in professional organizations, as well as for awarding prizes by world-renowned organizations such as the Nobel Foundation.
Over the years, a variety of peer assessment processes have evolved at the Canada Council to become the advisory cornerstone of the Council’s funding decisions. The Council developed policies to govern the role of peers – practising artists and other professionals working in the arts – in their assessment of grant applications and prize nominations.
The Canada Council’s commitment to peer assessment is based on the conviction that:
- while no system is perfect, peer assessment provides the best possible means of identifying outstanding ability and artistic merit in the arts;
- freedom of thought and expression benefits from a decision-making process that encompasses a very wide range of professional expertise; and
- the use of a large number of peer assessors each year provides the best guarantee of accountability and transparency in the Council’s funding decisions.
As a steward of public funds, the Council must make the wisest possible use of its resources. Through the assessment process, peers can attest to the integrity, credibility and fairness of the Council’s procedures, and the Council can involve the arts community directly in its operations.
Canada Council funding decision processes
Currently, the Canada Council receives over 15,500 applications or nominations each year, and awards approximately 6,000 grants and prizes. More than 650 assessors serve each year on 120 committees or juries, which in turn make use of input from several hundred external assessors.
A number of different processes are used to arrive at funding decisions.
Peer assessment committee: This is the method used for most of the Canada Council’s grant programs, where a new committee of peers is usually formed for each competition. Peer assessment committees can be discipline-specific or multidisciplinary.
Standing committee of peers: A standing committee of peers will convene on a regular basis throughout the year in order to provide continuity in a specialized field, such as administration or research. Peers are appointed for terms of up to three years, and serve on a rotating basis. The Council uses this assessment process for theLeadership for Change program, the Killam Prizes and Killam Research Fellowships, for example.
Both peer assessment committees and standing committees of peers allow for a diversity of views to inform a collective decision-making process. Each committee is composed of at least three peers. They work together to rank grant applications, select prize winners or recommend artwork purchases. Decisions are arrived at by consensus or majority agreement. The duration of committee meetings depends on the number and type of applications.
External assessors: The Canada Council routinely calls on individual assessors for specialized expertise, in order to support and complement the work of peer assessment committees or to provide advice directly to Canada Council staff regarding specific grant applications and prize nominations. They can provide written assessments on plays, dance productions, research projects and prize nominations, among other things, but they do not take part in the deliberations of peer assessment committees or standing committees of peers. External assessors are selected on the basis of the expertise needed.
Internal assessment: In a small number of instances, such as Travel Grant programs and Visiting Foreign Artists programs, the Council awards grants based on internal assessment by staff. Neither staff nor Board members are considered as peer assessors within the Council’s practices.
Selection of peer assessors
In selecting people to participate in its peer assessment processes, the Council looks for individuals who have the experience, knowledge and open-mindedness to make a fair and expert evaluation of the comparative merits of applications and nominations to grants and prizes. They are respected within their artistic or scholarly communities, and have professional experience and knowledge directly related to the prize or program assessment criteria and the types of applications or nominations in each competition.
The selection of peers is a major responsibility of the Canada Council staff. To find appropriate peers, they consult the Canada Council database of potential peer assessors, as well as their colleagues and the community at large. The Council is committed to engaging a significant proportion of new assessors each year. Peer assessors can be nominated by other artists, by arts organizations and by the public; they may also nominate themselves.
The Canada Council strives to ensure that its peer assessment system represents a wide variety of experience and perspectives. This is why it has established a buffer period for service on peer assessment committees: normally an individual can be a committee member only once in 24 months. However, the composition of a committee may sometimes require that exceptions be made for assessors who, for example:
- have professional experience in more than one discipline;
- have a specialized expertise necessary for the committee;
- provide continuity between two operating or sequential project committees;
- serve on a standing committee of peers; and/or
- serve on committees for major prizes such as the Governor General's Literary Awards, Molson Prizes, etc.
Exceptions to the 24-month buffer period require approval by the appropriate Divisional Director.
The Canada Council has developed a number of practices over the years to maximize the effectiveness of its peer assessment system. For example, to keep a measure of consistency and efficiency across committees:
- the same committee may be asked to assess more than one program component, or related programs;
- one or two members of a previous jury may be asked to sit on subsequent related committees (with new members);
- different sub-committees may assess distinct groupings of applications in high-volume competitions where the need for specific expertise plays a key role.
Peer assessment at the Canada Council: a fair and effective system
To be fair and effective, any assessment system needs thoughtful and consistent policies and procedures. This is especially true for qualitative evaluations, such as those used in the arts, which cannot be reduced to formulas. The Canada Council has put in place important tools to assist its peer assessment processes to function effectively, including;
- commitment to balanced committees
- clearly articulated Council policies, directions and documentation
- comprehensive prize and program guidelines and assessment criteria
- transparent and consistent procedures for managing conflict of interest
- clear guidelines and practices governing confidentiality of information.
The Council provides ongoing training to its staff to ensure they understand and follow the policies and procedures related to peer assessment at Council.
Commitment to balanced committees
Balanced committees are an important tool to promote a better understanding of artistic merit and outstanding ability within a multiplicity of contexts. The Canada Council for the Arts strives to appoint peer assessment committees that are both knowledgeable and representative by balancing the following factors when selecting committee members:
- Artistic practice – artists and arts professionals with different artistic styles and philosophies, who have an ability to provide aesthetic context.
- Professional specialization – artists and arts professionals who perform a variety of different professional roles in the arts, such as creators, interpreters, administrators, directors, producers, publishers, gallery owners, dealers, curators, critics, educators, etc.
- Language – artists and arts professionals from the two official language communities of Canada, as well as the official-language communities in minority situations such as Anglophones living in Quebec and Francophones living outside Quebec.
- Cultural diversity – artists and arts professionals of African, Asian, Middle Eastern, Latin American or mixed heritage.
- Aboriginal artists and arts professionals – artists and arts professionals of First Nations, Métis and Inuit background.
- Regions – artists and arts professionals from different geographical regions and communities of Canada.
- Gender – men and women.
- Age – artists and arts professionals of different generations.
No single committee can represent all the above attributes, but the Canada Council makes every attempt to maintain a balance over time. It monitors annually the composition of its peer committees and submits a report to the Board.
Clear policies, directions and documentation
Before each peer assessment committee meeting, the Canada Council provides detailed information to the peer assessors, to prepare them for their important task. Each committee member receives:
- A copy of this document, Peer Assessment at the CanadaCouncil for the Arts: How the Canada Council Makes Its Decisions;
- the appropriate prize or program guidelines, assessment criteria and other relevant documentation;
- all eligible applications to be assessed (on digital support or in book form), where applicable;
- support material for these applications, where applicable; and
- where applicable, written assessments from external assessors.
Comprehensive Program and Assessment Criteria
Each program and prize aims to achieve certain goals; the assessment criteria are based on these objectives. The criteria are often weighted – that is, they have different degrees of importance in the overall assessment, as specified in the guidelines. Assessment committees are required to evaluate each application or nomination according to the assessment criteria and the description of the program or prize, as well as Council policies and directions. Respect for artistic merit and artistic excellence remains the most important consideration in awarding grants.
Transparent and consistent procedures for managing conflict of interest
In all its assessment processes, the Canada Council relies on clear procedures to manage potential conflicts of interest in an ethical manner. In selecting committee members actively involved in the fields of the applicants or nominees, the Council makes a conscious effort to identify relationships between peer assessors and applicants or nominees, with a view to avoiding conflicts of interest. Prior to each peer assessment committee meeting, members receive a Conflict of Interest Disclosure Form that defines, in the Council’s view, what is a real or a perceived conflict of interest. They must complete this form and disclose any potential conflict of interest before deliberations begin. They must complete the form even if they have nothing to disclose, to indicate that they have considered the question. The same applies to external assessors and standing committees of peers.
Conflict of interest exists or may exist if peer assessors, external assessors or standing committee members are asked to assess a grant application, prize application/nomination, artwork or artistic production:
- from an employer, a client, or an organization where they are a board member
- from an organization where they acted as a consultant through the Canada Council’s Leadership for Change program
- where they have a direct financial interest in the success or failure of an application or nomination
- where the applicant or nominee is their spouse/partner, or immediate family member
- where their spouse/partner or immediate family member is a senior staff member, contractor or board member with the applicant organization
- where they and the applicant/nominee have been involved in legal proceedings against each other
In addition, peer assessors may judge that they are unable, for any other reason, to assess an application, nomination , or artistic production objectively.
For literary prizes, in addition to all points listed above, conflict of interest exists if the assessor edited one of the books. Conflict of interest may exist for literary prizes if:
- the assessor directly contributed to shaping the contents of one of the books
- the assessor has written a promotional text or review of one of the books
- the assessor’s name is listed in the acknowledgements section of one of the books, in a way that implies a direct contribution to the book
The Killam Program: In addition to all points listed above literary prizes, the academic and scholarly context of this program requires specific guidelines. Conflict of interest exists where the external evaluators, or the members of a standing committee of peers:
- are in the same research unit or university department as the applicant or nominee
- share an administrative link with the applicant or nominee (e.g. head of department, dean of faculty, etc.)
- have collaborated with the applicant or nominee or any member of their research team within the last five years
- have published with the applicant or nominee within the last five years
Assessors who have, or feel they could have, a conflict of interest must contact the program officer immediately to discuss the situation. Conflict of interest situations are reviewed on a case by case basis and determination is made as to whether the committee member is able to serve on the committee in question or contribute to the evaluation of the grant application.
In the case of prizes, the Canada Council only invites assessors who are believed to be free of conflict with the entire pool of applications or nominations, so as to have all committee members present for the entire meeting. The Killam Program’s standing committee is an exception; conflict of interest is handled in a similar manner to the grant program committees.
In a committee awarding grants, peer assessors who declare a conflict of interest must remove themselves from the process while the application in question is being discussed. Other committee members must not talk about this application at any time with the peer assessor who is in conflict.
Art Bank purchase committee members who declare a conflict of interest either leave the meeting room or simply abstain from voting, depending on the significance of the conflict of interest.
Canada Council employees must also disclose any potential conflict of interest with any application(s) submitted to the Canada Council. They must inform their Head of Section and divisional director so that Council can ensure the integrity of its assessment process. They must remove themselves from the assessment of the file or from the entire process, depending on the circumstance.
The Canada Council recognizes that the potential for conflict of interest will always exist when expertise and current knowledge are required in the assessment of competing proposals. To attempt to devise rules that would eliminate all potential for conflict of interest would risk reducing vision and expert judgement to a bureaucratic exercise. The challenge is to recognize that conflict is always possible, to avoid it where feasible and to be ready to manage it when it does occur, so that the ultimate outcome is in the public interest.
At the end of the meeting, all committee members must sign the Peer Assessment Committee Sign-Off Form confirming the committee’s list of final recommendations, the actions taken regarding conflicts of interest and the peer assessors’ commitment to confidentiality and non-disclosure.
Guidelines and practices governing confidentiality of information
Peer assessors must treat both the material that they review and any discussions related to their assessment as confidential. They must not disclose information about grant applications or award nominations. They must not discuss the names of the applicants or nominees, the recommendations, nor any comments made by other peer assessors during a committee meeting. Peer assessment committee members are informed of the Council’s guidelines and practices when they receive information on the nominees, applicants and proposals. They are required to sign a Conflict of Interest Disclosure Form and return it to the Council before the meeting begins. At the end of a committee meeting or individual assessment, peer assessors must return all paper or digital copies of the applications and support material to the chairperson. These documents and other working tools are then shredded or otherwise destroyed.
When providing information, the Canada Council for the Arts aligns itself to federal legislation:
- The Access to Information Act stipulates that Canadian citizens, artists and organizations have a right to request information contained in the Council’s records. The submission of a request does not guarantee complete access to records; exceptions to access are intended to protect information of a particular public or private interest and are the only basis for refusing access to information.
- The Privacy Act protects personal information and prohibits the use or disclosure of this information for any other purpose than originally stated. In general, any information that identifies an individual qualifies as personal information. It may only be disclosed with the consent of the individual or if it is found to be in the public interest. Applicants have a legal right to access the information about themselves in the Council’s files.
The Council does not make public the names of unsuccessful individual applicants. However, as a matter of policy, it publishes the list of grant recipients and prize winners, and the names of the committee members involved. These lists are posted on the Council website annually. In addition, news releases are issued for many prizes, listing the names of the laureates as well as the names of members of the peer assessment committee or standing committee of peers.
Process: During a committee meeting
A program officer usually chairs the peer assessment committee meeting. Occasionally, a head of section or office coordinator may perform this task. At the Art Bank, the director takes this role. The chairperson begins the meeting by formally briefing the assessors on their responsibilities, explaining how the meeting will be conducted, reviewing the Council’s mandate and directions, and highlighting conflict of interest and confidentiality policies. He or she then discusses the purpose and context of the prize or program, and reviews the assessment criteria.
After the introduction, committee members proceed to evaluate each application or nomination according to the assessment criteria, the description of the program or prize, as well as Council policies and directions. Committee members also take into account written assessments from external evaluators, when applicable. In some artistic disciplines, the support material submitted with the applications or nominations cannot be reproduced and forwarded to committee members in advance of the meeting; it is to be played, projected or presented during the meeting. After all the applications have been individually considered, the committee establishes an order of priority for awarding grants. Nominations and applications for prizes and fellowships require the committee to select, as the case may be, finalists, laureates and/or alternate fellowship recipients.
There are a wide variety of practices that help advance the process and build consensus among the members of the committee. In addition to any preparation before the meeting, which may include making notes and preliminary scores on each application, the peer assessors enter into a dynamic participatory process that is moderated by the chairperson. Their comments and the summation of their discussion may be recorded in a variety of formats. The preliminary scores can be used to advance the conversation and reach decisions, as can a show of hands. It is not uncommon that jury members review the applications and nominations more than once during their deliberations. The totality of the committee’s thinking is expected to coalesce at the end of the process, when the committee adopts its final recommendations. Each committee meeting is unique in this respect; the chairperson’s priority is to let all individual members express their views and to foster respect, fairness, and efficacy in the collaborative process of assessment. Facilitation skills are highly prized among Canada Council employees. They are the subject of a strong oral tradition in the workplace, reinforced by a mentorship program and formal training sessions.
The process of assessment usually unfolds with all committee members assembled in a meeting room at the Canada Council for the Arts, at the Art Bank or by teleconference. These meetings may last for a few hours, a few days and occasionally, for a few weeks. For some grant programs, many hundreds of applications must be assessed in a single committee meeting, and variations have been developed to avoid fatigue and increase the efficiency of the evaluation process, while ensuring that it is fair to applicants. One effective method is the pre-scoring of applications by committee members prior to the meeting. In this case, they may receive all proposals and their support material, or view the material on a restricted-access website. They enter a preliminary score for each, on the website or through a common template. A preliminary ranking is created on the basis of these scores and presented to the committee at the beginning of the meeting. This allows the committee to focus the majority of its attention on those applications that were well received at the pre-scoring phase.
In other high-volume programs, the applications are divided into common groupings or pods according to genre, activity type or other commonalities. A separate peer assessment committee is assembled for each grouping, and makes its own recommendations, including a list of highly recommended applications. The process involves balancing the program resources appropriately (both grants and services), and ultimately determining relative merit within and between the pods.
Peer assessors commit to protect the confidential nature of the deliberations. Their notes and scores are destroyed and the chair is the only person who will speak about the committee’s deliberations. The peer assessment committee’s comments are written down only in the case of some operating grant program competitions, i.e., annual or multi-year support to organizations or in some targeted grant programs such as the Capacity Building Initiative. In these instances, the chairperson captures the committee’s response to each application with regards to the program objectives and evaluation criteria. These responses are communicated to applicants along with the final funding recommendations.
Once the final recommendations are determined, each committee member signs the Peer Assessment Committee Meeting Sign-Off Form.
Final recommendation lists
Prior to any grant competition, a budget for each committee meeting is determined. For many committees, their final recommendation list will result in specific grant amounts for successful applicants, as well as lists of highly recommended applications. Highly recommended applications are those that are held in high esteem by the committee but could not be funded within the limits of the available program budget. Highly recommended applications that correspond to an equity priority may receive funding through targeted funding initiatives. Highly recommended applications may receive support at the end of a budget year, if funds are available. Applications that were not identified as highly recommended are not considered eligible for support.
The Killam Research Fellowships use a variation of the highly recommended status, with the ‘alternate’ designation. If one of the successful research projects cannot be pursued by the applicant, a fellowship will then be offered to the highest ranking alternate. Similarly, the Art Bank identifies a few artworks that are to be purchased in the event that one of the selected pieces was not available.
Because of the diversity of practices within the peer assessment system, other models of final recommendation lists may also be used. For example, when the Council needs to ensure that relative artistic merit is considered and responded to among multiple committees within the same program, or among several competitions held in the same period of time, final amounts may be contingent upon a review of the available budgets once all committees and their priority rankings have been finalized and consolidated. This is why some committees’ final recommendation lists may show precise grant amounts while others may show ranges of funding, lists of highly recommended applications, etc.
Assessment of French and English applications
Unilingual French or English-speaking committees are formed to evaluate language-specific prizes, programs and practices in, for example, the Writing and Publishing, Media Arts and Theatre Sections.
Other peer assessment committees assess applications received in both languages. The Canada Council provides for the translation of the core elements of applications and nominations based on the needs of the committees and the nature of the prize or grant program. During the meeting, the committee members are encouraged to speak in the official language of their choice; when necessary, their contribution is interpreted by one of the Council employees present at the meeting. Simultaneous interpretation is provided for some committee meetings.
Assessors’ roles and responsibilities
Members of a Peer Assessment Committee or a Standing Committee of Peers are required to:
- read and analyze the description of the program or prize and the assessment criteria, as well as all applications, nominations and related documentation received prior to and during a peer assessment committee meeting;
- comply with Canada Council policies (conflict of interest, confidentiality, etc.) at all times;
- provide expert advice to the Council;
- assess all the applications or nominations in a competition based on the assessment criteria and description of the program or prize, and in accordance with Council policies and directions;
- establish an order of priority for awarding grants or select recipients of prizes;
- determine and approve the final recommendation list(s) and sign the Peer Assessment Committee Meeting Sign-Off Form.
Committee members are paid an honorarium for each day of the meeting plus reading fees, as well as reasonable expenses for travel, accommodation and meals.
External assessors can be asked to provide:
- written reports about events held by organizations or performances of companies applying for grants and nominated for prizes;
- written reports on manuscripts of literary works written in languages other than French or English;
- written assessments of an application or of the career contribution of a prize nominee;
- input on specific applications or nominations, where their expertise is required.
External assessors base their evaluation on the assessment criteria and the description of the program or prize, in accordance with Council policies and directions. External assessors in dance and theatre are asked to complete the Conflict of Interest Disclosure Form. They cannot serve as external assessors for any application or nomination where they have a conflict to declare. All external assessors must sign a statement indicating they agree to abide by the confidentiality of information clause.
External assessors are paid an honorarium for each report they prepare, except for the Killam Prize, in view of the academic environment it addresses.
Canada Council staff: Roles and responsibilities
Program assistants are required to:
- work closely with program officers to prepare for committee meetings;
- organize the support material presentations and provide assistance during meetings;
- comply with the Canada Council policies (conflict of interest, confidentiality, etc.) at all times;
- assist committee members with travel and accommodation arrangements;
- enter results and prepare letters to applicants or nominators, for approval and signature.
Program officers are required to:
- review applications and nominations for grants, prizes and artwork purchases, and determine their eligibility;
- comply with Canada Council policies (conflict of interest, confidentiality, etc.) at all times;
- recommend committee members, taking into account the Council’s commitment to balanced committees;
- chair committee meetings. As chairpersons, they:
- establish conditions that ensure a serious, fair and open-minded evaluation of every application or nomination;
- ensure that each application or nomination is evaluated according to the assessment criteria and the description of the program or prize, as well as Council policies and directions;
- help the group reach final agreement on an order of priority for awarding grants (priority ranking) or on the selection of finalists (when applicable) and recipients of prizes;
- oversee the approval of the final recommendation(s) and signature of the Peer Assessment Committee Sign-Off Form;
- contribute to the communication of results and ensure follow up, where necessary.
Heads of Section or Office Coordinators are required to:
- comply with the Canada Council policies (conflict of interest, confidentiality, etc.) at all times;
- occasionally chair assessment committees;
- review and approve the program officers’ selection lists of committee members (any exception to the 24-month buffer period must be approved by the appropriate divisional director).
The chairperson may provide factual or background information during a committee meeting. However, it is not the role of Canada Council staff to comment on the artistic merit of an application.
At the Art Bank, the above roles and responsibilities are shared among different employees.
Process: After a committee meeting
After a committee meeting, the chairperson prepares the necessary internal documentation to obtain approval of the grants, prizes and artwork purchases. Authority to approve grants and prizes follows the Delegation of Authority Policy approved by the Board of the Canada Council. As a rule, the decisions that the Board and the Director approve are based on the committees’ final recommendation(s) and are considered final.
Communication of results: Applicants are informed of the result of the competition in writing. Applicants who would like to receive further information on the competition, need clarification about the possibility of reapplying or have follow-up questions about funding decisions may contact the program officer. Peer assessors are not authorized to answer such requests. With the exception of most operating grant programs and some targeted programs, written feedback on an application is not provided.
Peer assessment committee members’ feedback: Assessors are encouraged to provide feedback to the Canada Council staff on the competition and on how to improve the program and the assessment process.