Evaluation of the Class Grant and Contribution Program

Achievement of Expected Outputs

The Program performed well in producing all its intended outputs, such as launching AOs, producing ranked lists of eligible applicants and awarding G&Cs. According to most key informants, effective funding application mechanisms were in place, the AO selection process was fair, and the reporting requirements were similar to those of OGDs. However, information regarding both the nature of the AO proposal selection process and delays in the AO selection process was not provided to applicants consistently across AOs, and constructive feedback on proposals was also provided inconsistently. To address these shortcomings and in keeping with TBS recommendations for transfer payment programs, the CSA should standardize the application, selection, and feedback processes for both solicited and unsolicited proposals, and clearly communicate these processes to the Canadian space community.

Although the Class G&C Program's Performance Measurement (PM) Strategy identifies performance indicators, baseline data, and targets for the program's expected outcomes, there is a need to update the program's PM Strategy to ensure that it contains output-related indicators and baseline data and targets. Also, some of the G&C data stored in the CSA's databases were inaccurate, indicating a need to improve the data entry process in order to ensure archival data validity.

Achievement of Expected Outcomes

The Class G&C Program successfully achieved the expected outcomes of the program and in many cases exceeded the program's targets and key informants' expectations. The extent of knowledge gained by funded projects was considerable, especially with respect to technological or scientific breakthroughs and also with respect to developing new ideas for integration in future space missions, use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. According to key informants, the extent of new knowledge resulting from funded projects either met or surpassed their initial expectations. Also, a considerable number of presentations and publications were produced based on the knowledge gained from the majority of funded projects.

Although there was very little funding awarded to industry and not-for-profit organizations, the program played an important role in supporting a space focus in academia. Half of the funded projects brought new players into space-related research fields during the course of the evaluation period. Furthermore, the number of organizations participating in funded projects' research teams increased considerably and included national and international representation from academia, industry, and other types of organizations. Projects funded by the program also led to notable increases in highly qualified personnel and space-related expertise capacity, as well as to new and maintained national and international collaborations. For example, funding recipients partnered with 225 different organizations in order to carry out their funded projects. The majority of these partnering organizations were from Europe, the United States, and other countries (e.g., Japan, Australia). In addition, multidisciplinary collaborations and achievements surpassed the program's target in which 50% of funded projects resulted in at least one multidisciplinary achievement.

3.1.1 Alignment with Federal Government Priorities

The space and aerospace sectors make critical contributions to Canada's prosperity (Industry Canada, 2014). According to the Aerospace Review Report commissioned by the federal government, "advancing the national interest through space-based activity and fostering a competitive Canadian space industry will require resolve, clear priorities that are set at the highest levels, and effective plans and programs to translate these priorities into practice" (Industry Canada, 2012a, p. 1). The same report concludes that a focus on developing technological capacity can go a long way toward keeping Canada among the global leaders in space (Industry Canada, 2012a). The Class G&C Program helps address these concerns by supporting targeted knowledge development and innovation to sustain and enhance the Canadian capacity to use space to address national needs and priorities in the future. The program objectives are aligned with federal government priorities as set out in the Government of Canada's Whole-of-Government Framework, the Science, Technology and Innovation Strategy, the Space Policy Framework, the 2015 Economic Action Plan, and in the Speech from the Throne and official statements made by the Government that was elected in October 2015.

Government of Canada's Whole-of-Government Framework: The Class G&C Program objectives are aligned with the Economic Affairs sector of the spending areas represented in the Whole-of-Government Framework (TBS, 2011). More specifically, the Class G&C Program is designed to contribute to the Government's expected outcomes to create an innovative and knowledge-based economy (TBS, 2011).

Canada's Science, Technology and Innovation Strategy: Science, technology, and innovation were the focus of a federal government strategy published in 2014. In this strategy, the Government affirms its intention to partner "with industry and the Canadian space research community to leverage existing resources and encourage further technology development opportunities" (Industry Canada, 2014, p. 51). To this end, the Government will "ensure that our policies and programs inspire and assist Canadians to perform at world-leading levels of scientific and technological excellence" (Industry Canada, 2014, p. 13) and will "continue to support and deepen research across a broad spectrum of disciplines that include both discovery- and application-driven research" (Industry Canada, 2014, p. 36). The Science, Technology and Innovation Strategy has identified five research priorities of strategic importance to Canada:

1. environment and agriculture,
2. health and life sciences,
3. natural resources and energy,
4. information and communication technologies, and
5. advanced manufacturing—a priority in which aerospace has been identified as a focus area (Industry Canada, 2014).

For the Government, the five research priorities and focus areas "address the needs of Canada's key industrial sectors, such as space, robotics [and] aerospace" (Industry Canada, 2014, p. 21).

In addition, the Class G&C Program objectives are aligned with the three pillars of the Government of Canada's Science, Technology and Innovation Strategy: People, Knowledge and Innovation. First and foremost, the program contributes to the strategy's People pillar, through which the Government seeks to develop, attract and retain highly qualified and skilled individuals, as well as top experts and leaders needed for Canada to thrive in the global knowledge economy (Industry Canada, 2014).

In addition, the program contributes to the Knowledge pillar by fostering the acquisition of specialized skills in areas of Canadian strength, such as advanced space optics, robotics, and the development of innovative instruments. Through the Knowledge pillar, the Government seeks to support "research and scientific capacity in universities, colleges and polytechnics" (Industry Canada, 2014, p. 2). The Government aims to strengthen its "support for excellence across the spectrum of discovery-driven and applied activities by investing in research and infrastructure" (Industry Canada, 2014, p. 14).

Lastly, the strategy's third pillar, Innovation, aims to "encourage greater partnerships among Canadian businesses, universities and colleges to drive innovation and encourage the adoption of new processes and technologies that help Canadian businesses prepare to compete and win in the global marketplace" (Industry Canada, 2014, p. iii). The Class G&C Program is aligned with the Innovation pillar in that it supports the development or maintenance of partnerships between universities, government, and industry, and it increases international collaboration opportunities for Canadian scientists.

Canada's Space Policy Framework: Published in 2014, Canada's Space Policy Framework is central to the federal government's approach to space. One of the guiding principles outlined in the framework is "Excellence in Key Capabilities," for which the Government highlights that it will "continue to support and advance proven Canadian competencies while keeping a close watch on new niches of technological accomplishment" (CSA, 2014b, p. 10). Within Canada's Space Policy Framework, the Government has also identified four avenues of strategic action, including R&D. To this end, the federal government is committed to working with industry and the Canadian space research community, and to encouraging further opportunities in R&D and innovation by "increasing support for technology development" (CSA, 2014b, p. 11). The Class G&C Program helps the federal government respond to its priority of encouraging further opportunities in R&D and innovation by offering organizations financial aid, which they can use to carry out space R&D activities in priority areas for the CSA.

Economic Action Plan: Further to the 2012 Aerospace Review, chaired by the Honourable David Emerson, the Government announced in Economic Action Plan 2013 and over subsequent months several measures to support the aerospace sector. The measures included launching the Consortium for Aerospace Research and Innovation in Canada to facilitate collaboration among aerospace stakeholders (industry, academia, and research centres) to foster the development of advanced technologies, support the training of highly qualified personnel, and support innovation outreach (Department of Finance Canada, 2015). In Economic Action Plan 2015, the Government of Canada announced its intention to continue to support the pillars of a knowledge-based economy (Department of Finance Canada, 2015). According to the Government, "as the economic opportunities stemming from the application of new knowledge increase, it is crucial for Canada to possess a strong science and technology base, robust research infrastructure and a highly qualified workforce" (Department of Finance Canada, 2015, p. 90). The Class G&C Program objectives of supporting the development of S&T relevant to CSA priorities and fostering the continuing development of a critical mass of researchers and highly qualified personnel (HQP) in areas relevant to CSA priorities contribute to the fulfillment of this Government priority.

Speech from the Throne and official statements of the Government elected in October 2015: After the data gathering period for the Class G&C Program evaluation, the Speech from the Throne was given by the newly elected government. The Throne Speech outlines the policies that the Government is seeking to implement in the months to follow. In the speech, the Government recognized that public investment is needed to create and support economic growth, job creation, and economic prosperity (Government of Canada, 2015a). In this vein, the Minister of Innovation, Science and Economic Development, the Minister of Science, and the Minister of Small Business and Tourism declared, in a joint message to CSA employees, that their overarching objectives will be to promote and support scientific research and the integration of scientific considerations in their investment and policy choices (Message from ministers to CSA employees, November 17, 2015). Finally, in the Minister of Science Mandate Letter, the Prime Minister affirms, "We believe that investments in scientific research, including an appropriate balance between fundamental research to support new discoveries and the commercialization of ideas, will lead to good jobs and sustainable economic growth" (Government of Canada, 2015b). The Class G&C Program contributes to these priorities by funding research organizations, for-profit organizations, and, to a lesser extent, non-profit organizations, with a view to developing knowledge in areas of S&T that are a priority for the CSA.

3.1.2 Alignment with Departmental Priorities

According to the CSA's 2014-2015 Report on Plans and Priorities, the Canadian space industry remains reliant on continued R&D investments to overcome its growth challenges (CSA, 2014c). In addition, "Canada's ability to address national interests through an increasing dependence on space requires a critical mass of HQP in academia, industry and government; therefore, it is vital to develop, mobilize, and retain experts in space and related fields" (CSA, 2014c, p. 10). The Class G&C Program addresses this concern of the CSA's by aiming to increase the number of HQP participating in space S&T initiatives and projects.

In addition, the Class G&C Program supports the CSA's strategic priorities of consolidating CSA's technology development and capability demonstration activities and continuing the implementation of Canada's Space Policy Framework (CSA, 2015c).

The program also contributes to the CSA strategic outcome, which states, "Canada's exploration of space, provision of space services and development of its space capacity meet the nation's needs for scientific knowledge, innovation and information" (CSA, 2015c, p. 5).

As described in the Evaluation Approach and Methods section above, the main focus of the current evaluation was on the program's Research component, which represents 91% of program spending over the course of the evaluation period and 98% of program spending since 2012. As shown in the Resource Allocation section above, only $320K were spent on the Class G&C Program's Awareness and Learning component since 2012, when a decision was made to no longer fund initiatives aimed at elementary and secondary school students. Therefore, the alignment between the Awareness and Learning component and the CSA's priorities is tenuous.

3.1.3 Alignment with Federal Roles and Responsibilities

According to the Canadian Space Agency Act (the Act), the CSA mandate is to:

"...promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Canadian Space Agency Act, 1990, s. 4).

The CSA is delivering on its mandate in collaboration with Canadian industry, academia, Government of Canada organizations, and other international space agencies and organizations (CSA, 2015c). The activities and objectives of the Class G&C Program are aligned with one of the CSA's main functions, which, as set out by the Act, is to "plan, direct, manage and implement programs and projects relating to scientific or industrial space research and development and the application of space technology" (Canadian Space Agency Act, 1990, par. 5(2)(b)). The program is also aligned with paragraph 5(3)(c) of the Act, which states:

"In carrying out its objects, the Agency may...make grants and contributions in support of programs or projects relating to scientific or industrial space research and development and the application of space technology, including projects designed to develop, test, evaluate or apply new or improved processes, products, systems or information relating to space science and technology with a view to determining the commercial potential of that science and technology, but not including any programs or projects relating solely to the commercial exploitation of space science or technology."

According to this provision, the CSA has a major role to play in space R&D and is authorized to implement or support initiatives to collect data, conduct space research and studies, and maintain a critical mass of HQP in the field of space.

Furthermore, the Aerospace Review report highlighted that "space-related public investments are essential for the achievement of fundamental principles of...advancing the development of new technologies, and pushing the boundaries of knowledge" (Industry Canada, 2012a, p. 5). Numerous studies have also demonstrated how important it is for governments to be involved in the initial funding of space R&D (Industry Canada, 2012a; Industry Canada, 2012b; Library of Parliament, 2013; Organization for Economic Cooperation and Development [OECD], 2007; OECD, 2014). In a paper on the federal government's role in the Canadian aerospace industry, the Library of Parliament concluded that "the very long research and product development cycles, combined with the incremental nature of innovation, make early stage funding for R&D—provided in part through government assistance—particularly important" (2013, p. 3). From this perspective, the financial assistance provided to space organizations through the Class G&C Program is aligned with federal roles and responsibilities.

3.1.4 Continued Need for the Program

Several data sources were drawn upon, including information gathered through a document review and an online questionnaire for key informants (CSA employees, as well as recipients and non-recipients of Class G&C Program funding), to determine whether the Class G&C Program continues to address a demonstrable need and whether it is responsive to the needs of Canadians. As shown in Figure 2, the majority of key informants (98% of funding recipients, 71% of non-recipients, and 89% of CSA employees) felt that the Class G&C Program continues to address a demonstrable need.

When asked to explain their answers, more than half of funding recipients as well as some non-recipients indicated that the Class G&C Program is the only federal program entirely dedicated to the development of the space sector. Although there are other programs that offer G&Cs to the space and aerospace sectors, and there may be some overlap, none of those other programs designates space research as a priority. Similarly, the 2008 evaluation of the Class G&C Program also noted that "[t]he CSA is the only organization that is poised to offer focused support to stakeholders in achieving [their] objectives. The Class G&C Program provides the financial support in areas in which other sources of funding do not exist" (CSA, 2008, p. 9). This point will be further detailed in the performance (efficiency and economy) section of this report.

More specifically, when key informants were asked which needs the Class G&C Program addresses, they identified three main needs, which are also supported by the information that was gathered in the document review:

1. the need for capacity development in the Canadian space sector;
2. the need to support collaboration between industry, government, academia, and international partners; and
3. the need to support funding recipients' activities that contribute to the fulfillment of the CSA's mandate and priorities.

The need for capacity development in the Canadian space sector: The first Summative Evaluation of the Class Grant and Contribution Program (CSA, 2008) identified the need for capacity development in the Canadian space sector. The evaluation concluded that "because of the small size of the Canadian space sector, the critical mass to achieve broadly based innovative capacity is difficult to attain within academia and industry without the explicit support and assistance of the government" (p. 9). According to the report, the program played a role in addressing this need by coordinating between different actors and through funding. In addition, according to a 2014 OECD report, "[t]he acquisition and development of space capabilities remains a highly attractive strategic goal, and the number of countries and companies investing in space systems and their downstream applications continues to grow" (OECD, 2014, p. 9).

Of the three needs identified by key informants, the need for capacity development in the Canadian space sector was the most frequently cited. This need can be divided into three sub-categories:

1. The number of HQP, defined as individuals with university degrees at the bachelor's level and above;
2. The extent of space-related expertise; and
3. The availability of useful materials and equipment (e.g., laboratory equipment) for conducting space-related research.

When asked what gaps would exist in the absence of the Class G&C Program, most funding recipients, half of CSA employees, and a few non-recipients stated that there would be a decrease in Canadian capacity and skills in the space sector. They said that, without the Class G&C Program, there would be a lack of resources to support the development of new technologies or to maintain a highly qualified and skilled workforce.

Several non-recipients, some funding recipients, and some CSA employees also mentioned the need for the Class G&C Program to be more attentive to the needs and realities of the Canadian and international space community. These key informants explained that because of the changing global context, when determining project funding priorities, the CSA should give greater consideration to opportunities available to academia and industry, and the challenges these sectors face.

More specifically, funding recipients offered two suggestions to be more responsive to the needs of Canadians. First, the CSA should ensure continuity in the domains targeted by Class G&C Program's AOs, rather than shifting priorities in the projects it decides to fund. In their view, some AOs should be renewed to afford recipients greater opportunities to continue their research activities. Second, funding recipients suggested that the CSA continue funding research projects to develop higher levels of technological maturity. They explained that funding is usually set aside for early-stage R&D projects rather than for the development of more mature technologies.

Lastly, some CSA employees stated that, over the years, the Future Canadian Space Capacity program (PAA 1.3) has benefited greatly from the Class G&C Program, while other CSA programs (Space Data, Information and Services [PAA 1.1] and Space Exploration [PAA 1.2]) have not made sufficient use of it. They explained that the Class G&C Program was designed to support the achievement of priorities across the CSA's programs and that, to be more responsive to the Canadian space community's needs, it should be utilized more extensively by all CSA programs. According to key informants, this approach would support R&D and knowledge gain in a broader variety of space-related domains aligned with CSA priorities.

The need to support collaboration between industry, government, academia, and international partners: In the Space Policy Framework, the Government of Canada recognizes that space is a competitive domain and an expensive undertaking. The Government therefore affirms that it will look to continue partnerships to share the expenses and rewards of major space initiatives (CSA, 2014b, p. 10). Access to multidisciplinary and international collaboration opportunities for Canadian organizations is one of the Class G&C Program's expected outcomes. Several funding recipients as well as some CSA employees and non-recipients indicated that the Class G&C Program addresses the need to support collaboration between industry, government, academia, and international partners.

Some funding recipients and some non-recipients also said that the absence of the Class G&C Program would have repercussions on the capacity of Canadian industry and academia to create national and international collaboration opportunities. They explained that the program allows Canadian researchers to participate in space-related activities and disciplines as well as international research programs.

The need to support funding recipients' activities that contribute to the fulfillment of the CSA's mandate and priorities: This need was primarily cited by nearly a third of CSA employees, who said that G&Cs broaden the range of available funding tools. These CSA employees, as well as some funding recipients stated that the Class G&C Program addresses the need to support activities that contribute to the fulfillment of the CSA's mandate and priorities. Some employees explained that the CSA's financial support for R&D work is critical to the achievement of CSA objectives.

Similarly, some CSA employees specified that a number of pertinent projects funded for the fulfillment of the CSA's mandate and priorities could not be sustained without the Class G&C Program. They said that contracts are not a suitable alternative for funding initiatives geared toward knowledge and skills development.

3.2.1 Achievement of Expected Outputs and Outcomes

The extent to which the Class G&C Program has achieved each of the outputs and outcomes identified in the logic model is evaluated in this section of the report, which is divided into output, immediate outcome, intermediate outcome, and ultimate outcome sub-sections.

3.2.1.1 Outputs

Evaluation Question #5: Have ranked lists of funding priorities been produced? (Op1)

Evaluation Finding #5: Ranked lists of funding priorities for each branch have not been produced and some key informants described, more generally, a lack of clearly articulated, CSA-wide priorities. Evidence suggests that alignment with CSA funding priorities is more pronounced for AO than for unsolicited funding mechanisms.

Information provided by this evaluation's consultative group indicated that ranked lists of funding priorities have not been produced by the branches. Therefore, it was not possible to definitively assess whether the funding mechanisms in place are aligned with each branch's priorities. However, evidence suggests that the AO funding mechanism is better aligned with CSA priorities than is the unsolicited funding mechanism.

Figure 3: CSA employees’ opinions regarding the extent to which the selection mechanisms in place ensure that funded projects are aligned with the CSA’s priorities (%).

	Solicited Projects	Unsolicited Projects
don't know	8%	27%
extremely	31%	19%
considerably	44%	19%
moderately	17%	23%
slightly	0%	13%
not at all	0%	0%

By its very nature, the AO funding mechanism is designed to solicit proposals in DG-approved priority areas. Furthermore, alignment between CSA priorities and solicited proposals is one of the AO proposal selection criteria. As shown in Figure 3, the majority of CSA employees (75%) reported considerable alignment between the AO selection mechanism and the CSA's priorities.

However, others expressed less confidence in this alignment, primarily due to unclear and/or shifting CSA priorities. These employees highlighted the need for CSA-wide strategic planning that takes into account the Canadian space community's needs and is manifested in clear and stable R&D funding priorities.

CSA employees were significantly more likely to report alignment with CSA priorities for AO than for unsolicited funding mechanisms^{Footnote 16}. Most CSA employees either reported slight-to-moderate alignment between the unsolicited funding mechanism and CSA priorities (35%) or indicated that they don't know whether there is alignment (27%; Figure 3), suggesting that alignment is more pronounced for AO than for unsolicited funding mechanisms. Of note, there were no standardized selection criteria in place for unsolicited proposal during the evaluation period.

Evaluation Question #6: Have announcements been made available? (Op2)

Evaluation Finding #6: Ten AOs were made available over the course of the evaluation period, the majority of which were launched by the CSA's Future Canadian Space Capacity program (PAA 1.3). Though most CSA employees reported that effective funding application mechanisms are in place, CSA employees, funding recipients, and non-recipients alike reported challenges stemming from the infrequent and unpredictable timing of AOs. In addition, several funding recipients and non-recipients indicated that the funding application process is arduous and complex, and that there were lengthy delays in announcing the selection results with little communication from the CSA. Despite these concerns, a sizable minority of funding applicants rated the application process favourably. Discrepant perspectives are likely due to variances in the application processes across the ten AOs.

A total of ten AOs were launched over the course of the evaluation period. Among these AOs, five pertained to the CSA's Future Canadian Space Capacity program (PAA 1.3), three pertained to Space Exploration (PAA 1.2), and two pertained to Space Data, Information and Services (PAA 1.1).

In order to gauge the overall quality of the funding mechanisms in place, key informants were asked to compare the Class G&C Program's application process with OGDs' funding application processes. As shown in Figure 4, perspectives varied considerably across the three main key informant groups. Though most CSA employees didn't know enough about OGDs' funding mechanisms to comment, those who did most frequently reported that the application process is comparable to others for both solicited (65%) and unsolicited proposals (58%). In contrast, non-recipients most frequently rated the Class G&C Program's application process as inferior to OGDs' programs (71%). Funding recipients' ratings were more favourable than those of non-recipients^{Footnote 17}, but fairly evenly split with 25% rating the Class G&C Program's application process as superior, 38% rating it as about the same, and 32% rating it as inferior to OGDs' application process.

Figure 4: Key informants’ ratings of the Class G and C Program’s funding application process relative to those other federal departments (%).

	Recipients	Non-Recipients	CSA employees
			CSA employees - Solicited proposals	CSA employees - Unsolicited proposals
don't know	5%	7%	58%	75%
much better	5%	0%	2%	0%
better	20%	0%	4%	2%
about the same	38%	21%	27%	15%
worse	27%	36%	6%	2%
much worse	5%	36%	2%	6%

To gain greater insight into the quality of the Class G&C Program's funding application process, key informants were asked to provide feedback on the AO solicitation process, as well as on the proposal submission processes for both solicited and unsolicited proposals. Though variances across AOs and unsolicited proposals likely contributed to discrepancies in key informants' perspectives, the group data reported below nonetheless provide an overview of the program's application process strengths and weaknesses.

AO Solicitation Process

According to most CSA employees (73%), effective mechanisms are in place for obtaining AO funding applications. However, several CSA employees pointed out that the infrequency and unpredictability of AOs pose challenges for the Canadian space community in terms of planning for and maintaining HQP and infrastructure capabilities. This sentiment was echoed by both funding recipients and non-recipients, several of whom recommended replicating the approach used by other departments of establishing fixed, annual solicitation dates. According to some funding recipients, introducing regular and predictable timing for AOs would not only facilitate and improve the effectiveness of their internal planning, but would also reduce the "time crunch" that some experienced in the past due to short timelines between the launch of an AO and the application deadline.

Though it is not mandatory to standardize the timing of solicitation processes, interviews conducted with OGDs suggest that such standardization can be beneficial both in terms of providing applicants with predictability and in terms of managing forecasted spending within the granting department. However, such standardization may not be aligned with the needs of the CSA if, for example, funding priorities and/or budgets vary considerably from one year to the next. If it is not possible to establish fixed AO solicitation dates then key informants from TBS recommended informing applicants of why a non-standardized approach has been selected.

Because the Class G&C Program does not have standardized solicitation dates, a need for broader dissemination was identified. Given that departments are required to engage their potential applicants, key informants from TBS suggested strategies like email lists, twitter and social media, and notices in journals for engendering greater awareness of funding opportunities. In addition, some OGD interviewees use more personal approaches, like promoting funding opportunities at sector-specific events, society meetings, and trade shows.

In conjunction with the absence of standardized solicitation dates, the service delivery timelines of AO administration varied considerably among the AOs launched during the evaluation period. For example, the number of days between the AO application due date and the funding agreement start date ranged from 33 to 468 days, as shown in Figure 5.

AO funding recipients perceived that the CSA responded to their funding application in a timelier manner than did non-recipients^{Footnote 18} (see frequencies in Figure 6) and some funding recipients expressed appreciation for the speed with which funding decisions were made by the CSA. However, some respondents from both groups explained that the length of time between the submission deadline and the announcement of the results was too lengthy (more than one year in some cases).

In addition to lengthy delays, some recipients and non-recipients indicated that they did not receive communications from the CSA during the waiting period, which created considerable planning challenges for applicants. Furthermore, two of the 14 non-recipient respondents (who had applied to different AOs) reported that they never received notification from the CSA that their proposal had been rejected.

Figure 6: Opinions of funding recipients who submitted solicited proposals and of non-recipients regarding the extent to which the CSA responded to their application in a timely manner (%).

	Recipients	Non-Recipients
don't know	5%	8%
extremely	12%	0%
considerably	35%	8%
moderately	30%	8%
slightly	14%	31%
not at all	5%	46%

Proposal Submission Process

Only a minority of both funding recipients and non-recipients indicated that it was easy to find information about the Class G&C Program on the CSA website (39% of recipients and 21% of non-recipients) and that this information was clear (43% of recipients and 14% of non-recipients), though recipients indicated significantly greater satisfaction in both cases^{Footnote 19}. In addition, some funding recipients (30%) and non-recipients (21%) reported that the level of effort required to complete the funding application was reasonable (Figure 7). However, these data also indicate that most recipients and non-recipients thought the application process could be improved upon.

Figure 7: Funding recipients and non-recipients’ opinions regarding the extent to which the level of effort required to complete the funding application was reasonable (%).

	Recipients	Non-Recipients
don't know	0%	7%
extremely	4%	14%
very	26%	7%
moderately	54%	29%
slightly	9%	14%
not at all	7%	29%

In explaining their responses, almost all of the funding recipients and non-recipients who rated the application process less favourably wrote that the process is arduous and complex – a concern also raised by a few CSA employees. Some respondents explained that the level of effort required to complete the proposal is too high, especially for small G&Cs. However, respondents most frequently explained that there is considerable redundancy within the application process, whereby the same information is requested in different formats across the various sections of the proposal. Other frequently mentioned concerns regarding the cumbersomeness of the application process included that the requested information and the format of the application are inconsistent from one AO to the next, and that the proposal and verification signatures cannot be submitted electronically. In order to lighten the application process, both CSA employees and funding recipients recommended that the CSA introduce a streamlined online application form that does not vary extensively across funding opportunities. CSA employees added that electronic submissions would also improve the efficiency of the proposal selection process.

Though guidelines have been developed regarding the Class G&C Program's application process for solicited and unsolicited proposals, they have not been applied systematically. According to the OGD interviews carried out for this evaluation, TBS recommends an "enterprise-wide" approach whereby both the application process and the application guidelines are standardized (as well as harmonization across government departments, if possible). In line with this recommendation, most of the interviewed OGDs who provide grants or contributions indicated that they have standardized application guidelines, and all have standardized proposal templates for solicited and/or unsolicited proposals.

With respect to the proposal submission format, TBS recommends automation not only for electronic submissions but also for tracking funding recipients across applications. NRC's "Client-Relationship Management" system is a laudable example of such automation, in that their secure log-in, electronic system entails a centralized database that generates and tracks all aspects of the funding process, such as online proposal submissions, approvals, agreements, claims, support documents, reporting and performance reporting, and direct deposit of funding payments.

Evaluation Question #7: Have ranked lists of eligible applicants been produced? (Op3)

Evaluation Finding #7: Ranked lists of eligible applicants were produced for each AO. The selection mechanisms used to create these lists were deemed fair by most CSA employees and funding recipients, but not by non-recipients. All three groups of key informants rated the AO selection process as significantly less transparent than fair. In particular, several funding recipients and non-recipients reported that they did not receive constructive feedback about their proposals. With respect to unsolicited proposals, numerous CSA employees reported concerns with both the fairness and the transparency of the selection process.

A document review identified ranked lists of eligible applicants for each of the ten AOs launched during the evaluation period.

To partially assess quality, CSA employees were asked to compare the Class G&C Program's selection process with that of OGDs^{Footnote 20}. However, many CSA employees reported not knowing enough about OGD selection processes to comment, especially with respect to the selection of unsolicited proposals. For solicited proposals, most (76%) CSA employees who provided ratings reported that the Class G&C Program's selection process is either equivalent or superior to that of OGDs.

Quality was also examined with respect to key informants' perceptions of the fairness and transparency of the selection process. According to CSA employees, the process for selecting both solicited and unsolicited proposals is significantly more fair than it is transparent^{Footnote 21}.

Despite this discrepancy, the majority of CSA employees rated the solicited proposals selection process as both fair (88%) and transparent (77%; Figure 8). In their open-ended responses, several employees commented on the high quality of the AO selection process and attributed its success to the guidance of the CoE. The only suggestion for improvement provided was to use peer or expert-reviews in the selection processes more extensively.

Figure 8: CSA employees’ opinions regarding the extent of fairness and transparency of the selection process (%).

	Solicited proposals		Unsolicited proposals
	Solicited proposals - Fair	Solicited proposals - Transparent	Unsolicited proposals - Fair	Unsolicited proposals - Transparent
don't know	4%	4%	27%	27%
extremely	46%	40%	10%	10%
very	42%	38%	13%	6%
moderately	6%	17%	29%	23%
slightly	2%	2%	10%	17%
not at all	0%	0%	10%	17%

In sharp contrast to the solicited proposal selection process, only a small portion of CSA employees rated the unsolicited proposal selection process as fair (23%) and transparent (17%; Figure 8). Many CSA employees explained that the selection criteria are not well developed and documented. In order to improve the fairness and transparency of unsolicited proposals, CSA employees suggested:

• Decreasing the proportion of G&C funding spent on unsolicited proposals by increasing the frequency of AOs and introducing AOs with broad domains and fixed application dates;
• Introducing standardized application guidelines and selection criteria for unsolicited proposals; and
• Disseminating information on the CSA website about the unsolicited proposal selection process.

Funding recipients rated the CSA's process for selecting funding recipients as fairer and more transparent than did non-recipients^{Footnote 22}. Also, like CSA employees, both funding recipients and non-recipients indicated that the selection process is significantly more fair than it is transparent^{Footnote 23} (see frequencies in Figure 9).

Figure 9: Funding recipients and non-recipients’ opinions regarding the extent of fairness and transparency of the selection process (%).

	Recipients		Non-Recipients
	Recipients - Fair	Recipients - Transparent	Non-Recipients - Fair	Non-Recipients - Transparent
don't know	14%	7%	29%	8%
extremely	13%	9%	0%	0%
very	45%	31%	14%	0%
moderately	27%	31%	7%	23%
slightly	2%	9%	21%	15%
not at all	0%	13%	29%	54%

In terms of fairness, a few funding recipients and non-recipients reported that some AOs appeared to be geared towards the selection of large organizations with ample prior experience. In terms of transparency, numerous recipients and non-recipients expressed concerns about the lack of information provided. Specifically, these respondents explained that

1. the selection process was not clearly articulated,
2. they received little-to-no feedback on their proposals, and
3. the feedback they did receive was not constructive.

Though the type and extent of feedback about unsuccessful proposals varied across AOs, key informants from TBS recommended establishing service standards for feedback whereby unsuccessful applicants are informed of the reasons why their proposals were not accepted. For example, Industry Canada and NRC provide both written feedback and feedback communicated via telephone conversations that include recommendations for strengthening unsuccessful proposals.

Evaluation Question #8: Have G&C agreements been produced and has financial assistance been provided? (Op4)

Evaluation Finding #8: $36M were awarded across 195 G&C agreements pertaining to the Research component of the Class G&C Program during the evaluation period. Half of this funding was awarded to solicited proposals and the other half was awarded to unsolicited proposals, primarily by the CSA's Future Canadian Space Capacity program (PAA 1.3). The vast majority of Research component funding was awarded to universities.

As shown in the Resource Allocation section of this report, $40M in G&Cs were awarded by the Class G&C Program during the course of the evaluation period. The vast majority (91%, $36M) of this funding pertained to the program's Research component, which constitutes the main focus of this evaluation, and 76% of the Research component funds were awarded by the CSA's Future Canadian Space Capacity program (PAA 1.3; Figure 10).

Figure 10: Research component spending by CSA Program, not including smaller than 1% awarded by Internal Services.

	Space Data, Information and Services	Space Exploration	Future Canadian Space Capacity
Percentage	15%	8%	76%

Figure 11: Percent of Research component funding spent on solicited and unsolicited projects

	Solicited projects	Unsolicited projects
Percentage	55%	45%

As per Figure 11, approximately half (55%; $20M) of the Research component's funding was awarded to solicited projects via ten AOs^{Footnote 24} (eight AOs launched during the evaluation period and two AOs launched prior to its commencement). Like the Research component as a whole, most funding for both solicited (95%) and unsolicited projects (54%) stemmed from the CSA's Future Canadian Space Capacity program (PAA 1.3).

The $36M spent on the Research component was used to fund 195 projects (19% of which had start or end dates extending beyond the evaluation period). Of these 195 funded projects:

Figure 12: Research component G and C agreements by type of recipient organization.

	Universities	Private companies	Not-for-profit organizations
Percentage	95%	4%	1%

• 185 (95%) were awarded to universities (Figure 12), representing 98% of the Research component funding spent during the evaluation period;
• 117 (60%) were solicited, awarded an average of $182K per project; and
• 78 (40%) were unsolicited, awarded an average of $228K per project (or an average of 185K per project, if the $3.6M contribution mentioned in Footnote #14 is treated as a data outlier and omitted from the calculation).

Though the range of funding awarded per project was wide, it exceeded $400K for only 14% of solicited projects (Figure 13) and 13% of unsolicited projects (Figure 14).

As described in the Methods section above, most of the funding recipients who responded to this evaluation's online questionnaire (60%) received funding for one project during the evaluation period, though the range of funded projects per recipient was from one to five (Figure 15). In total, 51% of funding recipients received funding for solicited proposals, 25% received funding for unsolicited proposals, and the remaining 25% received funding for a combination of solicited and unsolicited proposals.

Evaluation Question #9: Have assessments of progress and final reports been conducted? (Op5)

Evaluation Finding #9: The vast majority of requested progress and final reports were completed. According to key informants, the reporting requirements are similar to those of OGDs, though over half of funding recipients indicated that the level of effort required to complete progress and final reports was too high.

As described in the Methods section of this report, 297 progress and final reports were completed for projects funded via the Research component, representing 94% of requested reports and 94% of funded projects for which reports were requested. On average, funding recipients completed 1.94 reports per funded project.

When asked how the Class G&C Program's reporting requirements compare with those of OGDs, funding recipients and CSA employees who provided ratings most commonly indicated that the reporting requirements are fairly similar to others (Figure 16). However, most CSA employees did not know how the reporting requirements compare with those of OGDs and therefore did not provide a rating.

Figure 16: Funding recipients and CSA employees’ ratings of the Class G and C Program’s reporting requirements, relative to those of other federal departments (%).

	Recipients	CSA employees
		CSA employees - Solicited projects	CSA employees - Unsolicited projects
don't know	5%	64%	75%
much better	5%	4%	4%
better	12%	4%	2%
about the same	47%	23%	15%
worse	25%	4%	2%
much worse	5%	0%	2%

Funding recipients' perspectives regarding the level of effort required to complete the reporting requirements were divided, with 42% reporting that the level of effort required is very reasonable and the remaining 58% reporting that it is only slightly or moderately reasonable. In explaining their answers, funding recipients voiced discordant perspectives regarding whether or not the degree of reporting is extensive, the information requested is relevant, and the online reporting format is user-friendly.

Though CSA employees were not explicitly asked to comment on the content of the reporting requirements, a few spontaneously mentioned that some of the indicators assessed in progress and final reports (e.g., space S&T capacity building and knowledge dissemination) are not relevant to funding recipients from industry. Similar concerns were not mentioned by funding recipients, though very few recipients were from industry during the evaluation period.

To improve the reporting process, a few funding recipients suggested replacing hard-copy verification signatures with electronic ones. In addition, it was suggested that a follow-up reporting requirement be introduced to capture outcomes that only occur after funded projects are completed. "The metrics that are asked for (papers, talks, etc.) are all things that tend to come to fruition at the very end (or even after) the projects are complete," explained one funding recipients. Though the utility of follow-up reporting would likely be influenced by size and scope of funded projects, it is noteworthy that both NRC and Industry Canada request follow-up reports for up to five years after funded projects are completed.

3.2.1.2 Immediate Outcomes

Evaluation Question #10: Has knowledge been increased from research projects in priority space S&T areas? (Oc1)

Evaluation Finding #10: During the evaluation period, ten AOs were launched and 195 projects were funded by the Research component, with the aim of increasing space S&T knowledge. The extent of knowledge gained by funded projects was considerable, especially with respect to technological or scientific breakthroughs but also with respect to developing new ideas for integration in future space missions, use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. According to most CSA employees and funding recipients, the extent of new knowledge resulting from funded projects either met or surpassed their initial expectations. Conversely, a detrimental impact on knowledge gain was evidenced for proposed projects that didn't receive Class G&C funding. However, some key informants indicated that the potential for knowledge gain was thwarted by both the discontinuation of AOs in targeted space S&T areas and the tendency for only one of the CSA's main programs to extensively use the Class G&C Program. In addition, CSA employees reported closer alignment between the knowledge produced and the CSA's priorities for solicited projects than for unsolicited projects.

Archival data indicates that, over the course of the evaluation period, competitive processes (AOs) were launched and G&C agreements were awarded via the Research component of the Class G&C Program, with the aim of increasing knowledge in priority space S&T areas. As described above, a total of 195 projects were funded and 117 (60%) of these projects were funded as a result of a competitive solicitation process. These solicited projects stemmed from ten AOs, eight of which were launched after the start of the evaluation period. By the end of the evaluation period, two more AOs had been launched (in November and December of 2013), though G&C agreements had not yet been awarded for them.

In the Class G&C Program's PM Strategy (CSA, 2013a), baseline data and targets for the number of AOs launched and the number of new and ongoing projects funded were established based on the 2011-2012 fiscal year. As shown in Table 8, the number of AOs with proposal due dates set for the two years following 2011-2012 were inconsistent (first lower then higher than in 2011-2012), as were the number of new and ongoing projects funded.

The program's PM Strategy (CSA, 2013a) established the baseline data and target for the number of funded projects completed per year based on the 2013-2014 fiscal year, which was the last year of the evaluation period. At this time, 138 (71%) of the 195 funded Research projects had been completed (Table 8). The remaining 57 funded projects (29%) were ongoing, with the latest funded project end dates planned for December 2018.

Table 8: Number of AOs launched, new and ongoing projects funded, and funded projects completed per fiscal year

	2009-2010	2010-2011	2011-2012	2012-2013	2013-2014
Number of AOs launched (based on proposal due dates)	0	2	3	1	4
Number of new and ongoing projects funded
Solicited	52	51	58	30	49
Unsolicited	29	19	27	30	37
Total	81	70	85	60	86
Number of funded projects completedFootnote 25
Solicited	13	14	34	0	15
Unsolicited	17	9	9	6	21
Total	30	23	43	6	36

Extent of Knowledge Increase

Information provided by funding recipients in their progress and final reports, as well as opinions shared by key informants via their responses to the online questionnaires, suggest that, overall, the extent of knowledge increase generated by the funded projects was considerable. However, in some cases, continued project funding and funding awarded by a wider representation of CSA programs would be required in order to achieve a sufficient amount of knowledge gain.

Table 9 shows the number and percent of funded projects for which at least one success was achieved in each of six key indicators of increased knowledge. As explained in the Limitations section of this report, usable data for these indicators was available only for 2011-2012 through 2013-2014 because of variations in the progress and final report templates over time. However, it is important to note that the source of these data was funding recipients' annual progress and final reports regarding their funded projects, many (85) of which were ongoing at the end of the evaluation period. Therefore, the findings reported in Table 9 likely underestimate the extent of knowledge increase that will be achieved by the time all of the funded projects have come to fruition.

As per the findings in Table 9, technological or scientific breakthroughs were the most common types of knowledge increases reported by funding recipients, with two-thirds (66%) of funded projects producing at least one breakthrough by the end of the evaluation period. In addition, about half of funded projects led to the development of new ideas that might be integrated into future space mission (56%), the use of satellite data (51%), and the development of applications and algorithms (43%). Though increased levels of technology and commercial successes were seldom reported (23% and 5%, respectively), the latter finding is not surprising given that, during the evaluation period, the bulk of G&C funding was awarded to academic organizations. Similarly, the limited achievements with respect to increased levels of technology likely reflect the CSA's decisions regarding which projects to fund; as reported in the Relevance section above, funding recipients suggested that, to better address the needs of Canadians, the Class G&C Program should resume discontinued AOs and award more G&Cs for developing higher technology readiness levels of existing technologies.

Table 9: Number and percent of funded projects between 2011-2012 and 2013-2014 (n=121) for which increased knowledge was reported in progress and final reports

Indicator of Increased Knowledge	Number of Funded Projects (%)
Technology or scientific breakthrough	81 (66%)
Development of new ideas that might be integrated in future space missions	69 (56%)
Satellite data used (Canadian or foreign)	63 (51%)
Applications and algorithms development (software readiness level)	53 (43%)
Technology led to a higher level of development (technology readiness level)	28 (23%)
Commercial success	6 (5%)

Of note, though only a little over half of funded projects (56%) had developed new ideas that might be integrated in future space missions by the end of the evaluation period, the potential for Class G&C funding to increase Canadian participation in space missions was spontaneously mentioned by many key informants in their online questionnaire responses, as illustrated by following quotes:

I believe that G&C is the single most important mechanism I have had in the past 5 years for conducting international-level planetary research. It has been crucial in enabling us to participate in the wide range of international planetary missions in which we are now engaged. (written by a funding recipient)

The collaborations formalized from CSA's G&C Program have led to consistent flight opportunities for Canadian scientists who tie in to new or existing national and international mission teams. (written by a CSA employee)

The impact [of not having received funding] is significant. We cannot carry forward our research and mission ideas. They are carried forward by colleagues in other countries. (written by a non-recipient)

Key informants' online questionnaire responses provided several other indications that the extent of knowledge engendered by Class G&C Program funding was considerable. For example, over half (54%) of funding recipients reported that the results of the Class G&C funding they received generated new projects funded through means other than the program (often via international partners); another 39% reported that it was too soon to tell if new projects would be generated.

When specifically asked to identify the extent of knowledge generated by funded projects, most funding recipients (85%) and CSA employees (58%) reported that Class G&C funding contributed either considerably or extremely to increasing knowledge (Figure 17). Those key informants who reported less knowledge gain explained that, though the funded projects were informative, continued funding would have been required to obtain a sufficient amount of knowledge. Incidentally, most of those respondents who did not provide ratings explained that the funded projects with which they were involved were still in the early stages of development; however, a few CSA employees explained that they would require more specific and targeted performance data in order to meaningfully rate the extent of knowledge generated by funded projects.

Figure 17: Funding recipients and CSA employees’ opinions regarding the extent to which G and C funding increased space S and T knowledge (%).

	Recipients	CSA employees
don't know	5%	17%
extremely	32%	17%
very	53%	41%
moderately	11%	21%
slightly	0%	5%
not at all	0%	0%

Key informants were also asked via online questionnaires to indicate how closely the extent of increased knowledge matched their initial expectations. The vast majority of funding recipients (93%) and over half of CSA employees (56%) reported that the extent of knowledge generated by Class G&C funding either matched or surpassed their initial expectations. However, according to 14% of CSA employees (or 20% of those who provided a rating), the extent of knowledge gain had been less than they expected. These employees explained that, to match their initial expectations, the Class G&C Program would need to be used more extensively across the CSA's programs, rather than primarily by the CSA's Future Canadian Space Capacity program (PAA 1.3). As described earlier in this report, this theme also emerged from CSA employees' responses pertaining to the relevance of the program, in which they explained that more extensive use of the Class G&C Program across the CSA would be required in order to better meet the needs of Canadians.

Another means of assessing the extent of knowledge increase is to examine the impact on proposed projects that were not funded by the program. According to non-recipients' online questionnaire responses, 21% were able to carry out their proposed projects without Class G&C funding. However, these non-recipients explained that the funding they received for their proposals from other sources was a stop-gap measure (i.e., insufficient in the long-run). Furthermore, they reported that receiving Class G&C funding would have had an important positive impact on their projects in terms of timelines, scope, and quality. These responses suggest that the absence of Class G&C funding can have a detrimental impact on knowledge gain.

Regardless of whether or not non-recipients were able to carry out their proposed projects, most of them described a negative impact on the potential for knowledge gain resulting from lack of G&C funding. For example, one non-recipient explained that "It is difficult to estimate the damage of not having these results [from the proposed project], but I am convinced that it is influencing the competitiveness of our industry."

Alignment between Knowledge Increase and the CSA's Priorities

To fully achieve the expected outcomes of the Class G&C Program, the knowledge gained from funded projects should be aligned with the CSA's space S&T priorities – especially given that supporting activities that contribute to the CSA's mandate and priorities was identified by CSA employees as a key need that should be addressed by the program (see Relevance section). As reported in the Outputs section of this report, alignment with the CSA's priorities was found to be more pronounced for AO than for unsolicited funding mechanisms. Therefore, it is not surprising that a similar trend was evidenced regarding the alignment between knowledge increase in priority areas and the type of mechanism used to fund projects. As illustrated by the frequencies presented in Figure 18, those CSA employees who provided ratings were significantly more likely to report that the knowledge produced by solicited projects was aligned with the CSA's priorities than the knowledge produced by unsolicited projects^{Footnote 26}.

Figure 18: CSA employees’ opinions regarding the extent to which the knowledge produced was aligned with the CSA’s space S and T priorities (%).

	Solicited projects	Unsolicited projects
don't know	6%	26%
extremely	30%	11%
very	49%	34%
moderately	11%	28%
slightly	4%	2%
not at all	0%	0%

Evaluation Question #11: Has space focus in universities, post-secondary institutions, and not-for-profits and for-profit organizations been maintained and/or increased? (Oc2)

Evaluation Finding #11: Half of the funded projects brought new players into space-related research fields during the course of the evaluation period. In addition, the number of organizations participating in funded projects' research teams increased considerably and included national and international representation from academia, industry, and other types of organizations. Further evidence of increased focus on space as a result of the Class G&C Program was provided by key informants, most of whom reported that the program has played an important role in supporting a space focus in academia. However, for the period of time under review, very little Class G&C funding was awarded to industry and not-for-profit organizations; therefore, the program's contribution to maintaining a space focus in these types of organizations was limited.

In their annual progress and final reports, funding recipients identified which other organizations were related to the research teams for their funded projects. Some of the annual data pertaining to the identification of these organizations were missing and reliable data were only available for the last three years of the evaluation period (2011-2012 to 2013-2014). However, based on available data for these three years, a total of 242 different organizations (other than the funding recipients' organizations) participated in at least one funded project's research team for at least one year. Given that several of these organizations were related to more than one funded project's research team, the average number of other organizations related to the research team per funded project was 4.48 (SD = 4.69), with a wide range of zero to 34.

Among the 242 other organizations that participated in at least one funded project's research team, 42% were Canadian (Figure 19). The remainder were organizations based in Europe, the United States, and other regions (e.g., Costa Rica, Australia, China).

Figure 19: Organizations that participated in research teams by geographic location.

	Canadian	American	European	Other
Percentage	42%	25%	27%	6%

Keeping in mind that 95% of funding recipients during the evaluation period were from academic institutions, it follows that the most common type of organization participating in research teams was universities (representing 49% of organizations that participated in at least one funded project's research team). However, as shown in Figure 20, the remaining 51% of the research team organizations were private companies, research centres, national government entities, and other types of organizations (e.g., provincial entities, not-for-profit organizations, colleges).

Figure 20: Organizations that participated in research teams by type of organization

	University	Research center	National/federal governement department or entity	Private company	Other
Percentage	49%	14%	9%	18%	10%

As shown in Figure 21, the number of other organizations participating in research teams each year increased considerably in the last two years for which reliable data were available. In 2013-2014, the annual number of research team organizations nearly doubled the baseline data and target established in 2011-2012 for this performance indicator, as per the Class G&C Program's PM Strategy (CSA, 2013a). Given that the number of funded projects per year did not increase as steeply, these data indicate that more organizations were participating in funded projects' research teams by the latter part of the evaluation period. Indeed, according to annual progress and final reports, half (51%) of the funded projects brought new players into space-related research fields.

Figure 21: Number of organizations participating in research teams by fiscal year

	2011-2012	2012-2013	2013-2014
Frequency	92	162	170

When funding recipients were asked in their online questionnaires about the extent of space focus within their organizations, two thirds (68%) indicated that the Class G&C Program has contributed either considerably or extremely to maintaining a focus on space within their organizations. As one funding recipient explained, "This program is the most important source of funding for our space-related research. Without it, it is unlikely we would have near the space focus that we have." However, other funding recipients reported less impact on the space focus within their organizations because, as a couple of recipients explained, Class G&C funding cannot be relied upon as a source of ongoing support.

CSA employees expressed similar opinions to those of funding recipients regarding the Class G&C Program's contribution to maintaining a focus on space in academic institutions. Because academic institutions represented the vast majority of funding recipients during the course of the evaluation period, most CSA employees (58%) indicated that the program successfully maintains an academic space-focus. However, few CSA employees (21%) reported benefits of maintaining a space focus within industry and most (65%) were unsure about any such benefits for not-for-profit organizations (which have not been a main target population of the program).

Following the end of the evaluation period, the Class G&C Program's target population was broadened to focus more on industry (see Limitations section above), which will likely produce different findings regarding the program's contribution to maintaining a space focus in for-profit organizations at the time of the next program evaluation.

Evaluation Question #12: Have partnerships been established and/or sustained, has there been access to international collaborations for Canadian organizations, and have partner contributions been leveraged? (Oc3)

Evaluation Finding #12: The majority of funding recipients established and/or maintained partnerships in order to carry out their funded projects and over two-thirds of these collaborations were with organizations outside of Canada. Key informants indicated that Class G&C funding effectively increased beneficial collaborations with new and pre-existing partners, which contributed to the success of funded projects. In addition, about half of the funded projects leveraged partners' contributions. The estimated leverage ratio was $1.86 (cash or in-kind) for every $1 of Class G&C funding invested, but this estimated ratio should be interpreted with caution due to methodological limitations.

Partnerships Established and/or Sustained

According to findings stemming from the first evaluation of the Class G&C Program (CSA, 2008), the program played a critical role as an enabler of collaboration between stakeholders, which led to the success of numerous funded projects. As reported above in the Relevance section of this report, the key informants surveyed for the current evaluation identified an important continued need for the Class G&C Program to foster collaboration between national and international partners, in order for Canadians to participate in space-related activities.

According to annual progress and final reports (for those years during which reliable data were available, 2011-2012 to 2013-2014), the majority (84%) of funding recipients maintained or established partnerships in order to carry out their funded projects. In total, partnerships were established with 225 different organizations during this period of time. However, partnering organizations were often implicated in more than one funded project, resulting in an average of 3.98 (SD = 4.16) partnerships per funded project and a range of zero to 17.

Figure 22: Partnering organizations by type of organization

	University	Research center	National/federal governement department or entity	Private company	Other
Percentage	49%	20%	10%	11%	10%

The composition of the 225 different partnering organizations resembled that of organizations participating in research teams (reported above) with respect to the types of organization. As shown in Figure 22, about half of partnering organizations were universities (49%) and the remainder were research centres, private companies, national government entities, and other types of organizations (e.g., space agencies, not-for-profit organizations).

As shown in Figure 23, the number of organizations per year that partnered with funding recipients on at least one funded project fluctuated over the three years of the evaluation period for which reliable data were available (2011-2012 to 2013-2014). The baseline data and target for this indicator were established in 2011-2012 (CSA, 2013a).

Figure 23: Number of partnering organizations by fiscal year.

	2011-2012	2012-2013	2013-2014
Frequency	129	140	95

To gain a richer understanding of the quantitative findings reported above, key informants were asked about their perceptions of the partnerships formed as a result of Class G&C Program. When asked to compare the collaborations that resulted from Class G&C funding with their initial expectations, CSA employees and funding recipients alike indicated that their expectations were either met or surpassed with respect to the extent of collaboration with both new partners (74% of CSA employees and 91% of funding recipients) and partners that funding recipients had collaborated with in the past (70% of CSA employees and 94% of funding recipients; Figures 24 and 25). To further promote collaboration, it was suggested to add development of partnerships and collaborations to the selection criteria in future AOs. Of note, a similar practice is incorporated into Industry Canada's Technology Demonstration Program and NSERC's Industrial Research Chairs and Collaborative Research and Development programs, which focus on the development of partnerships and collaborations.

Figure 24: Extent to which collaboration with new and pre-existing partners matched CSA employees’ initial expectations (%).

	new partners	pre-existing partners
too soon to know/ don't know	21%	26%
much more than expected	2%	0%
more than expected	30%	14%
same as expected	42%	56%
less than expected	5%	5%
much less than expected	0%	0%

Figure 25: Extent to which collaboration with new and pre-existing partners matched recipients’ initial expectations (%).

	new partners	pre-existing partners
too soon to know/ don't know	5%	6%
much more than expected	9%	6%
more than expected	40%	26%
same as expected	42%	62%
less than expected	4%	2%
much less than expected	0%	0%

According to most funding recipients, the Class G&C funding they received considerably increased their collaboration with new partners (70%) and with partners they had collaborated with in the past (61%). Furthermore, most funding recipients reported that they have greatly benefited from the collaborations they developed with new partners (72%) and pre-existing partners (61%). Several funding recipients explained that the partnerships forged or strengthened as a result of Class G&C funding fostered continued collaboration following project completion. Others remarked that the success of their funded projects would have been diminished in the absence of collaboration from partners.

In a similar vein, most non-recipients indicated that, had they received the Class G&C funding, this funding would have generated considerable collaboration both with new partners (58%) and pre-existing partners (73%). In the absence of Class G&C funding, a couple of non-recipients explained that their opportunities to collaborate with partners have diminished.

Access to International Collaborations

As reported in the section directly above, funding recipients partnered with a total of 225 different organizations in order to carry out their funded projects. Among these partnering organizations, less than one-third (29%) were Canadian (Figure 26). The remainder were from Europe, the United States, and other geographic locations (e.g., Japan, Australia).

Figure 26: Partnering organizations by geographic location

	Canada	United States	Europe	Other Country
Percentage	29%	25%	38%	8%

As shown in Figure 27, most CSA employees (65%) and funding recipients (86%) reported that the extent of international collaborations developed to carry out funded projects either met or exceeded their initial expectations. As a result of having received Class G&C funding, most funding recipients reported increased collaboration with international partners (64%) and over half of funding recipients indicated that they benefited greatly from these international collaborations (57%). As one funding recipient explained, "G&C funding has been key to enabling many of these collaborations."

Further evidence that Class G&C funding fosters international collaboration was provided by non-recipients, about half (55%) of whom indicated that their collaborations with international partners would have increased notably had they received Class G&C funding. Some described their lack of Class G&C funding as a missed opportunity for better positioning Canada on the international space scene.

Figure 27: Funding recipients and CSA employees’ opinions regarding the extent to which collaboration with international partners matched their initial expectations (%).

	Recipients	CSA employees
too soon to know/ don't know	7%	26%
much more than expected	14%	0%
more than expected	35%	26%
same as expected	37%	40%
less than expected	4%	9%
much less than expected	4%	0%

Leveraged Partner Contributions

Unlike some federal transfer payment programs (e.g., NSERC's Collaborative Research and Development program, Industry Canada's Technology Demonstration Program), the CSA's Class G&C Program does not include the level of leveraged funds among its selection criteria. However, according to funding recipients' annual progress and final reports, partners' contributions were leveraged for 52% of the projects that received Class G&C funding between 2011-2012 and 2013-2014 (the years for which reliable data were available). As shown in Figure 28, the numbers of funded projects that leveraged partners' contributions in both 2012-2013 and 2013-2014 were higher than the baseline data and target established in 2011-2012, as per the program's PM Strategy (CSA, 2013a).

Figure 28: Number of funded projects per year that leveraged partners’ contributions

	2011-2012	2012-2013	2013-2014
Frequency	18	41	38

Among those funding recipients who completed the online questionnaire, 79% reported having leveraged funds (cash and in-kind) for at least one project funded by the Class G&C Program^{Footnote 27} (which approximates the proportion of non-recipients who anticipated that they would have been able to leverage funds had they received Class G&C funding; 82%). Among those funding recipients who leveraged funds, 66% indicated that they had already benefited greatly from this leveraging. Furthermore, the majority of funding recipients (91%) and CSA employees (69%) indicated that the extent of contributions leveraged through the Class G&C Program either matched or exceeded their initial expectations.

According to estimates provided by funding recipients in their online questionnaires, an average of $1.86 was leveraged from partners or other sources for every $1 of Class G&C funding (Figure 29). This leverage ratio is comparable to that of NSERC's Collaborative Research and Development program ($1.56 per $1 invested) and lower than NRCan's Class G&C program, as reported in program evaluation reports ($3.26 per $1 invested; NSERC, 2010; NRCan, 2012). However, leverage ratio comparisons should be interpreted with caution because, though these programs share similarities with the CSA's Class G&C Program, their nature and design differ from the CSA's program. Furthermore, the validity of the leverage data is questionable because funding recipients provided a retrospective estimate that, in some cases, combined leveraged funds across multiple funded projects. Of note, leverage data were also collected via funding recipients' annual progress and final reports, but the validity of these data was also questionable and the data were not accessible in an aggregated form that is conducive to data analysis.

3.2.1.3 Intermediate Outcomes

Evaluation Question # 13: Has the availability of space-related knowledge and information in priority areas been increased? (Oc4)

Evaluation Finding #13: A considerable number of presentations and publications were produced based on the knowledge gained from the majority of funded projects. Knowledge was disseminated nationally and internationally, as well as within academic circles and for the general public. Furthermore, following project completion, most funding recipients continued to disseminate knowledge to communities outside of their organizations.

The first evaluation of the Class G&C Program (CSA, 2008) found that a broad dissemination of funded projects' research results had not yet been achieved due to the short history of the program at that time. In contrast, findings from this evaluation indicate space-related knowledge generated from funded projects is now widely disseminated. According to funding recipients' progress and final reports, most funded projects (69%) led to important achievements with respect to outreach and general scientific awareness (for the years for which reliable data were accessible, 2011-2012 through 2013-2014).

Throughout the evaluation period, the majority of the projects funded produced presentations (83%) and publications (69%).

• Most of the presentations were conferences, seminars, or workshops (75%), and the remainder were presentations to media or the general public (18%) and other types of presentations (8%).
• Most of the publications were peer reviewed (73%), and the remainder were non-peer reviewed (25%) and books (2%).
• Most of these publications (74%) contained an acknowledgement of the CSA's funding.
• For fiscal years 2012-2013 and 2013-2014 (the years for which reliable data were available), 37% of the new and ongoing funded projects also produced other types of publications (e.g., materials published on the internet, technical reports, training manuals).

As shown in Figures 30 and 31, an average of 17.52 presentations and 13.71 publications were produced per funded project. However, the ranges were very wide and only a few funded projects resulted in 50 or more presentations (6%) or publications (5%). In particular, one funded project produced 539 publications, far above the number produced by other funded projects. When this outlier was removed from the calculation, the average number of publications per funded project was 10.52 (SD = 16.49).

As per Figure 32, the numbers of presentations and publications per year increased steadily over the course of the evaluation period. The baseline data and target for these indicators were established in 2011-2012 (CSA, 2013a) and, in the two following years, the numbers of both presentations and publications exceeded the target.

Figure 32 : Number of presentations and publications per year (Frequency)

	2009-2010	2010-2011	2011-2012	2012-2013	2013-2014
Presentations	184	254	537	868	838
Publications	63	546	367	636	485

Information provided by funding recipients in response to the online questionnaire for this evaluation suggests that the knowledge gained from funded projects continued to be widely disseminated following project completion. Specifically, 70% of funding recipients reported a considerable amount of continued knowledge dissemination to communities outside their organizations.

When asked how closely the extent of knowledge dissemination matched their initial expectations, the majority of both funding recipients (89%) and CSA employees (83% of those who provided ratings) indicated that their initial expectations were either matched or exceeded. They explained that the knowledge produced by Class G&C-funded projects was widely disseminated both nationally and internationally, as well as both within academic circles and for the general public. However, some respondents suggested that the CSA could facilitate an even wider dissemination of knowledge by making publications more accessible both within the CSA and to the general public. In addition, some respondents noted that publications and presentations are less pertinent measures of performance for funding recipients from industry than from academia.

Evaluation Question # 14: Has space-related S&T capacity in target areas been increased? (Oc5)

Evaluation Finding #14: Space-related S&T capacity has been increased as a result of the Class G&C Program, particularly with respect to HQP and space-related expertise capacity, but also with respect to the availability of materials and equipment for conducting space-related research. During the evaluation period, capacity-building was most evident within academic settings for carrying out space science research, as evidence by findings that most members of funded projects' research teams (HQP) were students or faculty. However, the discontinuation of AOs in previously targeted space S&T areas threatens the maintenance of some of the capacities developed to date.

Overall Impact on Capacity

As reported in the Relevance and Immediate Outcomes section of this report, when asked what needs are addressed by the Class G&C Program, key informants most frequently identified the need to build capacity in the Canadian space sector. During the course of the evaluation period, the Class G&C Program's Research component funded 195 space S&T projects, with the goal of sustaining and enhancing the Canadian capacity to use space in addressing national needs and priorities. These funded projects resulted in an increase in space S&T knowledge and, according to most funding recipients and CSA employees, Class G&C funding also contributed to maintaining a space focus in Canada.

Figure 33: Funding recipients and CSA employees’ opinions regarding the extent to which the program’s impact on capacity matched their initial expectations (%).

	Recipients	CSA employees
too soon to know/ don't know	9%	33%
much more than expected	7%	2%
more than expected	37%	16%
same as expected	46%	37%
less than expected	2%	12%
much less than expected	0%	0%

Against this backdrop, it is not surprising that most funding recipients (77%) reported a strong beneficial impact of the Class G&C Program on their organizations' overall capacity for conducting space-related S&T research. As illustrated by the following quote, some funding recipients explained that their organizations' capacity would have diminished dramatically without Class G&C funding: "Without it, we'd have next to nothing; it has been absolutely critical." Similarly, most CSA employees (69% of those who provided ratings) reported a beneficial impact on Canada's overall capacity for conducting space-related S&T research.

In addition, the majority of both funding recipients (89%) and CSA employees (68%) indicated that the extent of the program's impact on capacity either matched or surpassed their initial expectations (Figure 33).

As mentioned earlier in this report (see Immediate Outcomes section), only 20% of non-recipients who completed the online questionnaire carried out their proposed projects and, in the absence of Class G&C funding, the timelines, scope, and quality of these projects were hindered. When asked what effects receiving Class G&C funding would have had, 62% indicated that this funding would have considerably increased their organizations' overall capacity for conducting space-related S&T research. In their open-ended responses, several non-recipients reported challenges maintaining their organizations' capacity without Class G&C funding.

Though the data reported above demonstrate that Class G&C funding effectively increases space-related S&T capacity, non-recipients and funding recipients did not differ significantly with respect to their ratings of their organizations' current capacity for conducting space-related S&T research. A partial explanation for this lack of statistical difference was offered in the open-ended questionnaire responses provided by funding recipients and CSA employees. Several respondents from both groups explained that they have recently witnessed a detrimental effect on capacity, brought about by the discontinuation of AOs in previously targeted space S&T areas. In particular, they pointed to challenges with maintaining HQP and expertise in space science domains. A recipient explained that "the real question is what happens from here, now that these programs are done. How do we maintain our expertise in space-related research? We risk losing it."

Types of Capacity-Building

In their online questionnaires, funding recipients, non-recipients, and CSA employees were asked about the extent to which the Class G&C Program beneficially impacts three specific types of capacity building:

• HQP^{Footnote 28},
• space-related expertise, and
• availability of useful materials and equipment (e.g., laboratory equipment) for conducting space-related research.

The responses to these online questions aligned with the findings reported in the Relevance section concerning the capacity needs addressed by the Class G&C Program.

The Class G&C Program's Research component was not explicitly designed to increase the availability of materials and equipment for conducting space-related research. Nonetheless, as shown in Figures 34 through 36 below, several key informants pointed to the effectiveness of the program in increasing materials and equipment capacity. Specifically, half of funding recipients (51%) reported a considerable beneficial impact on the materials and equipment available within their own organizations and 26% reported beneficial impacts that extended to organizations outside of their own. Similarly, a notable minority of non-recipients anticipated that Class G&C funding would have greatly benefited their organizations' (42%) and other organizations' (42%) materials and equipment capacity. Though several CSA employees were not able to provide ratings, 48% of those who did so reported considerable materials and equipment benefits for Canadians.

Though the Class G&C Program appears to have an unintended positive impact on materials and equipment capacity, all three groups of key informants indicated that the main capacity-building benefits of the program pertain to HQP and space-related expertise (Figures 34, 35, and 36). According to most recipients, the Class G&C funding they received beneficially impacted both HQP (74%) and space-related expertise (77%) capacity within their organizations. In addition, a minority of funding recipients reported beneficial impacts on HQP (36%) and expertise (43%) in other organizations. In somewhat similar proportions, non-recipients indicated that, had they received Class G&C funding, their organizations and others would have experienced beneficial impacts on both HQP (67% and 17%, respectively) and expertise capacity (67% and 58%, respectively). Likewise, most of the CSA employees who provided ratings reported a beneficial impact in Canada on HQP (64%) and space-related expertise (60%) resulting from the Class G&C Program. Key informants from all three groups of respondents explained that the program is particularly beneficial for creating new jobs, attracting new talent (both from within Canada and abroad), and training students and personnel.

Figure 34: Funding recipients’ opinions regarding the extent of beneficial impact that the program had (within their own organization and in other organizations) on HQP, space-related expertise, and materials and equipment (%).

	HQP		Expertise		Materials and equipment
	HQP - Within own organizations	HQP - Outside their organizations	Expertise - Within own organizations	Expertise - Outside their organizations	Materials and equipment - Within own organizations	Materials and equipment - Outside their organizations
too soon to know/ don't know	2%	17%	2%	11%	4%	17%
extremely	32%	11%	32%	15%	21%	13%
considerably	42%	25%	46%	28%	30%	13%
moderately	23%	23%	14%	25%	16%	17%
slightly	2%	9%	7%	13%	14%	13%
not at all	0%	15%	0%	8%	16%	26%

Figure 35: Non-recipients’ opinions regarding the extent of beneficial impact that program would have had (within their organization and in other organizations) on HQP, space-related expertise, and materials and equipment (%).

	HQP		Expertise		Materials and equipment
	HQP - Within own organizations	HQP - Outside their organizations	Expertise - Within own organizations	Expertise - Outside their organizations	Materials and equipment - Within own organizations	Materials and equipment - Outside their organizations
too soon to know/ don't know	25%	25%	17%	17%	25%	25%
extremely	25%	0%	17%	8%	25%	17%
considerably	42%	17%	50%	50%	17%	25%
moderately	8%	42%	17%	8%	8%	8%
slightly	0%	0%	0%	8%	17%	8%
not at all	0%	17%	0%	8%	8%	17%

Figure 36: CSA employees’ opinions regarding the extent of beneficial impact in Canada that the program has on HQP, space-related expertise, and materials and equipment (%).

	HQP	Expertise	Materials and equipment
too soon to know/ don't know	23%	19%	41%
extremely	14%	9%	5%
considerably	35%	44%	7%
moderately	19%	14%	29%
slightly	9%	14%	10%
not at all	0%	0%	10%

Funding recipients' annual progress and final reports corroborate the finding that the Class G&C Program effectively supports HQP capacity. As shown in Table 10, the number of HQP working on funded projects' research teams increased over the course of the evaluation period and, in the last two years, it approximately doubled the baseline data and target established in 2011-2012 for this indicator (CSA, 2013a). However, when interpreting this data, it should be noted that some HQP likely worked on more than one funded project's research team in the same fiscal year and, therefore, were counted more than once. Furthermore, the data reported in Table 10 do not take into account the amount of time that each HQP dedicated to working on a funded project. Data pertaining to full-time equivalents (FTEs) were collected toward the end of the evaluation period, but these data were not accessible in a format conducive to analysis for this evaluation.

Table 10: Type, number, and percent of HQP involved in funded projects' research teams by fiscal year

Type of HQP	2009-2010	2010-2011	2011-2012	2012-2013	2013-2014
Faculty	41 (21%)	84 (32%)	191 (38%)	374 (34%)	324 (34%)
Students & Post-Doctoral Fellows	128 (66%)	159 (61%)	256 (51%)	584 (53%)	435 (46%)
Other	26 (13%)	20 (8%)	53 (11%)	140 (13%)	198 (21%)
Total	195	263	500	1098	957

On average, 10.21 (SD = 11.66) HQP per fiscal year worked on each funded project's research team, with a range of 1 to 103 HQP per research team per year. In terms of composition, about half (52%) of funded projects' research members were students and post-doctoral fellows, 34% were faculty, and 15% were other types of HQP (e.g., scientists, engineers, technicians from industry).

The large proportion of students and faculty on research teams is a result of most Class G&C funding having been awarded to academic institutions. In their open-ended responses, some CSA employees questioned the relevance of HQP as a performance indicator for G&Cs awarded to industry, noting that maintaining HQP capacity is less of a concern for industry than it is for academic funding recipients. As the Class G&C Program evolves to include more funding recipients from industry, the pertinence of retaining HQP capacity as an indicator for these recipients should be examined.

Evaluation Question #15: Has multi-disciplinary and/or institutional collaboration been increased? (Oc6)

Evaluation Finding #15: Multi-institutional and multidisciplinary collaborations were forged by funding recipients, with half of funding recipients reporting a considerable increase in their multidisciplinary collaborations as a result of Class G&C funding. The extent to which multidisciplinary collaborations took place either matched or exceeded most funding recipients and CSA employees' initial expectations. Furthermore, half of the funded projects resulted in at least one achievement related to multidisciplinary research.

Data from progress and final reports pertaining to the number of multidisciplinary collaborations established by funding recipients were not accessible in a format conducive to analysis for this evaluation. However, as reported in the Immediate Outcomes section, the vast majority of funding recipients were from academic institutions and about half of the organizations that they worked with in their research teams and that they collaborated with through partnerships were from private companies, research centres, national government entities, and other types of non-academic organizations. These findings indicate that multi-institutional collaborations were established by funding recipients in order to carry out their funded projects.

According to most funding recipients (90%) and CSA employees (67%), the extent to which funded projects led to multidisciplinary collaborations either matched or exceeded their initial expectations. About half of funding recipients reported that their multidisciplinary collaborations increased considerably as a result of Class G&C funding (51%) and that they benefited greatly from these collaborations (46%). As one funding recipient explained, "For the first time… we have created a multidisciplinary team. This has been a fantastic move and has greatly improved both the training opportunities and the effectiveness of the project – much more than I expected." In a similar vein, most non-recipients (64%) indicated that Class G&C funding would have increased their collaborations with multidisciplinary teams, had they received funding for the proposed projects.

According to funding recipients' progress and final reports, several of the multidisciplinary collaborations were particularly fruitful. Specifically, 50% of the funded projects resulted in one or more achievement related directly to multidisciplinary research (during the years for which reliable data were available, 2011-2012 through 2013-2014).

Figure 37: Number of multidisciplinary achievements produced by funded projects per year.

	2011-2012	2012-2013	2013-2014
Frequency	25	32	30

As shown in Figure 37, in both 2012-2013 and 2013-2014, the number of multidisciplinary achievements produced by funded projects each fiscal year was slightly higher than the target for this indicator, which was established based on data from 2011-2012 (CSA, 2013a).

3.2.2 Demonstration of Efficiency and Economy

In order to evaluate the Class G&C Program's efficiency, CSA employees' experiences and opinions regarding the program's overall efficiency and the efficiency of its governance and delivery structure were solicited. Additional indicators of program efficiency were provided by key informants' opinions of the program's value with respect to the use of public funds, including perspectives regarding the overall quality of the program.

Efficiency of Resource Utilization and Governance Structure

Overall Efficiency of Resource Utilization: Divergent perspectives were voiced by CSA employees regarding the extent to which the Class G&C Program has been delivered efficiently with respect to resource utilization.

According to several key informants (32%), the program has been delivered in a very efficient manner (Figure 38). These CSA employees explained that laudable efforts have been made by the CoE to optimize resource utilization (e.g., through the development of templates and a checklist). Furthermore, they commended CoE personnel for the efficient and helpful manner in which they support funding initiatives.

In sharp contrast, other CSA employees (43%) described a less efficient program delivery, burdened with heavy bureaucratic processes and too much oversight from the CoE (Figure 38). Several of these CSA employees expressed concern that the burdensome bureaucratic process leads to excessive delays in the launch of AOs, resulting in a rush to award funding agreements at the end of the fiscal year and, occasionally, in lapsed program funds.

As the volume of funding agreements increased and the number of CoE personnel decreased over time, CoE personnel indicated that a large portion of their time was dedicated to oversight of funding initiatives, leaving little time for carrying out the remainder of their functions. Though a direct comparison with OGD transfer payment programs was not possible due to fundamental differences between programs, it is interesting to note that the oversight function of G&C CoEs differs across federal programs. For example, the CoE of NRC's Industrial Research Assistance Program conducts little oversight of specific funding initiatives because many of their processes are automated (e.g., online funding application tools). To determine if guidelines and templates are followed uniformly, NRC relies upon a quality assurance assessment conducted by their Finance directorate, as well as internal audits and evaluations. In contrast, the CoE of NRCan's Class G&C Program includes an Agreement Review Group that conducts oversight of all high dollar value and medium-to-high risk funding agreements, as well as any changes to approved templates and a 10-15% sampling of other agreements. NRCan's CoE then delivers a quarterly report to senior management.

Governance Structure: As with the divergent opinions expressed regarding the overall efficiency of the Class G&C Program, contradictory views were also expressed with respect to the program's governance structure. As shown in Figure 39, when asked to rate the overall appropriateness of the program's governance structure, 27% of CSA employees indicated that it is very or extremely appropriate, whereas 48% indicated that it is either not at all, slightly, or moderately appropriate.

In response to more specific questions about the program's governance structure, a sizable minority of CSA employees (25 to 40%) reported that the CoE's accountabilities and roles and responsibilities are both clearly defined and appropriate. Conversely, between 50 and 60% of CSA employees indicated that this is not the case and that better communication and/or revision of the CoE's accountabilities and roles and responsibilities are required. In similar proportions, divergent views were expressed regarding the extent to which accountabilities and roles and responsibilities are clearly defined and appropriate for employees who work on G&C initiatives (e.g., directors, managers, staff).

Across the sub-groups of CSA employees surveyed, those who expressed that accountabilities and roles and responsibilities are appropriate and clearly defined tended to be program directors, managers, and staff who make use of the Class G&C program. However, other directors, managers and staff disagreed, as did most CoE personnel, Financial Management personnel, and senior executives.

Relationship between Perceived Purpose of G&Cs and Perceived Program Efficiency: In their open-ended responses to the online questionnaire, a few of the CSA employees who indicated dissatisfaction with the efficiency of the program's governance structure explained that it is outdated and should be revised to better align with both Canada's Space Policy Framework (CSA, 2014b) and the CSA's new project management approach.

However, much of the discord among CSA employees with respect to program efficiency and governance structure stemmed from the lack of a shared vision regarding the purpose of G&C funding agreements. Essentially, CSA employees disagree about whether:

1. like contracts, G&Cs should be treated like a means of funding projects that are solely designed to support the objectives of other programs in the CSA's PAA, or whether
2. G&Cs should be treated as components of a broader, CSA-wide program with its own specific objectives, processes, and expected outcomes.

Those CSA employees who viewed G&Cs merely as a means of funding projects in support of other CSA programs perceived the CoE's oversight efforts geared at ensuring consistencies in program delivery and results as inefficient. They explained that expertise regarding the Canadian space sector's and the CSA programs' needs lies with program directors, managers, and staff, rather than with the CoE. In order to improve efficiency, they recommended revising roles and responsibilities accordingly.

Conversely, those who view G&Cs as contributing to a CSA-wide program cautioned that G&Cs should not be confused with contracts. They expressed concern that there is a lack of harmonization across the CSA with respect to the AO development process and the selection, management, and reporting of funded projects, often because the guidelines and tools are not applied uniformly. Consequently, these key informants perceived large discrepancies in the development and administration of different AOs and in their resulting quality.

Several proponents of this view recommended further streamlining the Class G&C Program and better communicating the established accountabilities and roles and responsibilities, especially to newer users of the program who are accustomed to awarding contracts. However, a few CSA employees proposed allocating greater responsibilities and accountabilities to the CoE in order to better harmonize program delivery and results across the CSA.

Evidence collected for the purpose of the current evaluation supports the rationale for a CSA-wide G&C program with standardized objectives, processes, and expected outcomes that are uniformly applied across the CSA:

• The Terms and Conditions of the CSA's Class G&C Program, as approved by TB in 2009, demarcate a program designed to support the overriding objectives of the CSA, including a program implementation strategy and expected outcomes that are unique to this program and applicable to all initiatives funded through it.
• As further delineated in the CoE's Terms of Reference (2010a), the CoE's primary role entails harmonization and standardization of program delivery and reporting, including providing G&C expertise, portfolio services in G&C management, program oversight and performance, and support of G&C Advisory Committee activities.
• Though there are opportunities for improvement, as a whole, the Class G&C Program in its current form effectively responded to the space-related needs identified by key program stakeholders over the course of the evaluation period (as described in the Ultimate Outcomes section of this report).
• As described in the Outputs section of this report, TBS recommends standardization of processes, systems, and procedures for the management and delivery of transfer payments within departments and harmonization across federal departments.
  • As per TBS's Policy on Transfer Payments (2008), TBS, ministers, and deputy heads each have responsibilities associated with the achievement of the policy's expected result that "collaboration exists within and among departments to harmonize transfer payment programs and standardize their administration, when appropriate."

However, the perception held by some CSA employees that the program's current oversight and standardization methods are inefficient cannot be ignored. A review of the Class G&C Program's Terms and Conditions, as well as of the Terms of Reference related to the program, could prove beneficial in order to ensure that the program is structured in the most efficient way to meet the needs of the Canadian space sector and CSA program priorities while, simultaneously, adhering to the TBS Policy on Transfer Payments. The results of this review and any subsequent amendments to the program's governance structure should be clearly communicated across the CSA to ensure buy-in and alignment.

Value with Respect to the Use of Public Funds

As shown in Figure 40, the majority of CSA employees (66%) and funding recipients (88%) reported that, overall, the Class G&C Program has produced good value with respect to the use of Canadian public funds. As one funding recipients wrote, "Canadians would be proud of the way we have spent each cent from this program!" Though most non-recipients indicated that they couldn't provide a rating, those who did so also reported good use of public funds (83%).

In explaining their answers, respondents from all three groups of key informants pointed to the program's success in meeting its expected outcomes, including knowledge generation and dissemination, capacity building and maintaining a focus on space, and facilitation of important partnerships and collaborations both nationally and internationally. According to one non-recipient, "Without [the program], Canada would quickly disappear from being essential participants [in international space programs] to followers and observers at most." In addition, several key informants described the Class G&C Program as providing a very good return on modest investments. For example, one funding recipient described "The 'bang for buck' in terms of science, publications, HQP, collaborations, etc..." In a similar vein, several CSA employees and a few funding recipients expressed particular satisfaction with the flexibility of G&Cs relative to contracts. They explained that G&Cs afford freedom to define project objectives, timelines, and budget without deliverables and in an efficient manner that is managed entirely by the CSA. "For certain activities, this program is an excellent way of financing university or industry teams, in comparison with contracts that can be much more demanding in terms of documentation," wrote one CSA employee. Similarly, a funding recipients wrote that "All of the funds contributed go directly to supporting research; the efficiency of other programs… has decreased dramatically to the point that many researchers struggle to find value in contract research."

Another means by which the Class G&C Program produced good value with respect to use of public funds was by raising the caliber of Canada's reputation for space-related R&D. Though only funding recipients explicitly identified this benefit to Canadians in their explanations of why the program produces value for money, key informants from all three groups of respondents spontaneously mentioned the program's positive impact on Canada's reputation at various points in their open-ended responses to the online questionnaire. The following quotes illustrate the credibility built by the Class G&C Program:

[The funded project] promises to put Canada firmly on the map as a solid and respected location for this type of research. (written by a funding recipient)

We are seen as credible players with good expertise and facilities, enabling us to play with the "big boys." (written by a funding recipient)

[Funding recipients] publish and present their results internationally, which makes Canada shine and attracts international collaborations. (written by a CSA employee)

Although difficult to specify the exact impact the funding would have had, it is clear that it would have represented a significant capacity-building to the benefit of Canada on the international scene. (written by a non-recipient)

Despite general satisfaction with the value produced by the Class G&C program, several key informants offered suggestions for improvement. According to some CSA employees, value for money could be improved upon by better aligning the funding spent on unsolicited proposals with the CSA's priorities (as described in the Outputs section of this report) and by reducing the level of oversight performed by the CoE to improve efficiency (as described in the sub-section directly above).

According to some funding recipients, ongoing CSA support to bring projects funded by past AOs to higher levels of maturity is the primary means by which better value for money could be achieved. As one funding recipient explained, without continued CSA funding, "… we cannot translate or bring the CSA's G&C to higher value for Canadians at this point." Writing from a broader perspective, another funding recipients explained that the better alignment of the CSA's future road map with the direction of funded projects is needed, stating that, "I believe transparency and clear definition of CSA direction are needed to improve the use of public funds." The needs for continuity of funding in priority domains, funding to bring projects to higher levels of maturity, and, more broadly, clearly articulated, CSA-wide priorities have emerged as findings of this evaluation at several points in both the Relevance section and the Achievement of Expected Outputs and Outcomes section of this report.

Overall Program Quality: In their online questionnaires, key informants were asked to rate their level of overall satisfaction with the Class G&C Program, as well as to indicate how the program compares with other federal G&C programs. With respect to overall satisfaction, most funding recipients (74%) and CSA employees (68%) indicated that they are generally satisfied (Figure 41). However, non-recipients expressed significantly less overall satisfaction, with 57% indicating that they are dissatisfied with the program^{Footnote 29}.

Similarly, most funding recipients (84%) and CSA employees (27% or 64% of those who provided ratings) indicated that the program is about the same or better than others, whereas most non-recipients (72%) indicated that the program is worse than others (Figure 42). In terms of between-group differences, funding recipients and CSA employees were significantly more likely than non-recipients to rate Class G&C Program favourably compared to others. Also, funding recipients were significantly more likely than CSA employees to rate the program favourably compared to others.

To better contextualize these ratings of the program's overall quality, key informants were asked to identify what they value most and what they value least about the Class G&C Program. Not surprisingly, the themes that emerged from the information provided paralleled many of the key findings reported upon in the Relevance and Performance sections above. Specifically, the most valued aspects of the program cited frequently by key informants were that:

1. The program funds Canadian space-related activities (cited by several funding recipients and non-recipients, as well as some CSA employees);
2. G&Cs offer more flexibility than do contracts in terms of project objectives, budget, and timelines (cited by several funding recipients and CSA employees, as well as some non-recipients);
3. The program is instrumental in building Canadian space-related capacity (cited by several funding recipients and non-recipients); and
4. The CoE provides useful support and expertise (cited by several CSA employees).

The least valued aspects of the program cited frequently by key informants were that:

1. The timing of funding opportunities (AOs) is both infrequent and unpredictable, and the space-related priorities targeted by AOs also shift unpredictably (cited by several funding recipients and non-recipients, as well as some CSA employees);
2. There is insufficient transparency and feedback in the selection process for solicited proposals, accompanied by lengthy delays between the submission due date and the announcement of selection decisions (cited by several funding recipients and non-recipients, as well as some CSA employees);
3. The processes in place for launching AOs and awarding funding are overly bureaucratic and onerous (cited by several CSA employees).

Three facets of economy were examined for the purposes of this evaluation:

1. economy in program spending,
2. the appropriateness of program inputs for supporting expected results, and
3. redundancy or overlap with other programs.

Economy in Program Spending

Forecasted versus Actual Dollar Amount Awarded to G&C Agreements: Based on the financial data presented in Table 1 (in the Resource Allocation section), the actual amount awarded to funded projects via the Class G&C Program exceeded the forecasted budget by 8% (or $3,242K) over the combined five years of the evaluation period. The increased spending on G&C agreements relative to the program's forecasted budget took place in the earlier years of the evaluation period, when the program's current Terms and Conditions were initially being implemented. Conversely, during the last two years of the evaluation period (2012-2013 and 2013-2014), the forecasted budget exceeded actual spending by 5 to 8%.

An examination of forecasted and actual spending on G&C agreements by type of funding showed that, over the course of the evaluation period, the surplus in actual spending was primarily awarded via grants (Table 11). In terms of the CSA's four main programs, the Future Canadian Space Capacity program and, to a lesser extent, the Space Data, Information and Services program spent more than their forecasted budgets on G&C agreements. In contrast, the Space Exploration program spent less than its forecasted budget, as did the Internal Services program (though the budget for Internal Services was relatively small). Of note, it was not possible to examine forecasted and actual spending with respect to solicited and unsolicited projects, as well as with respect to the program's Research and Awareness and Learning components, because budget forecasts were not apportioned in this manner.

Allocation of Resources: Financial data regarding spending allocated specifically to each of the five activities or to each of the five outputs identified in the Class G&C Program's logic model were not available. In addition, the ratio of administrative costs to program expenditures could not be evaluated because, with the exception of costs associated with the CoE, financial data regarding the cost of managing and administering the Class G&C Program across the CSA had not been tracked.

The costs associated with the CoE are presented in Table 5 of the Resource Allocation section. In the years following the creation of the CoE, the forecasted FTEs and salary dollars remained fairly stable, but the forecasted budget for O&M decreased, resulting in a 27% reduction in the CoE's overall forecasted budget from 2010-11 to 2013-2014. In terms of actual spending, the CoE cost 32% less than its forecasted budget. A comparison of CoE costs in 2010-2011 with CoE costs in 2013-2014 shows that the actual cost of the CoE decreased by 48%, primarily due to considerably less O&M in the latter years when the program's implementation was better established. In addition to decreased spending on O&M, the number of FTEs allocated to the CoE in 2013-2014 decreased from about 5 to 3.5, representing a reduction of approximately 22% in salary dollars. Of note, a total of 3 CoE FTEs were budgeted for 2015-2016 (when the current report was written) and the actual FTEs for this year also approximated 3 FTEs.

In contrast to the decrease in CoE costs over the evaluation period, the amount spent on funded projects via the Class G&C Program was 24% higher (and 36% higher for the Research Component) in 2013-2014 than it had been five years earlier in 2009-2010 (see Tables 1 and 3 in the Resource Allocation section). Furthermore, the program's budget for all but the first of the following four fiscal years (2014-2015 to 2017-2018) was forecasted to more than double the actual spending on funded projects in 2013-2014 (Table 2). This projected budget increase is primarily accounted for by larger forecasted spending for projects funded via contributions, much of which are expected to be awarded to industry (as described in the Limitations section).

Across the evaluation period, an average of $0.07 was spent on the CoE for every dollar awarded to G&C agreements (Table 12). When tracked year-by-year, the cost of the CoE for every dollar awarded to funded projects decreased over time, from $0.10 in 2010-2011 (when the CoE was first created) to $0.05 in 2013-2014.

Appropriateness of Program Inputs

When CSA employees were asked about the appropriateness of the Class G&C Program's inputs, about one quarter of respondents (23%) reported that the inputs are appropriate and another 45% reported that they are insufficient for achieving program delivery (Figure 43).

The most commonly cited reason for insufficient program inputs was that there were too few human resources allocated to the CoE, causing delays in the administration of funding initiatives and limiting the CoE's ability to carry out its functions. In explaining their responses, almost half of CSA employees pointed to the recent increase in the use of the Class G&C Program (i.e., expanded program budget and volume of funding agreements) and the simultaneous decrease in FTEs allocated to the CoE, as described in the sub-section above on allocation of resources. In addition, according to a few CSA employees, the human resources allocated to the CoE should be expanded enough to not only cover their current roles and responsibilities, but also to take on additional functions, such as participating in the writing of AOs and answering funding applicants' questions.

In contrast, some of those CSA employees who indicated that the human resources allocated to the CoE are either adequate or excessive explained that the current CoE FTEs could be utilized more efficiently by reducing the amount of oversight performed by CoE personnel – a perspective described in detail in the Efficiency section of this report. Whether due to insufficient human resources allocated to the CoE or to inefficiency of the CoE in carrying out its oversight function, several CSA employees voiced concerns about the negative ramifications of lengthy delays between AO application due dates and the funding agreement start dates, as per the finding reported above in the Output section of this report.

The need to increase the level of G&C funding awarded to the Canadian space sector and especially to academia was also commonly cited by CSA employees who reported insufficient program inputs. With the discontinuation of funding for priorities targeted by prior AOs and increased funding currently being awarded to industry, concerns were expressed that the capacity and space focus in academia was diminishing.

Though funding recipients and non-recipients were not explicitly asked to comment on the appropriateness of the Class G&C Program's inputs, several respondents from both groups nonetheless identified concerns regarding insufficient G&C funding for academia in recent years and in the years to come. For example, one funding recipient wrote, "I hope that CSA will continue to value the partnerships with the academic community that have come through the G&C Program... and continue to support an active research community in all the areas of space science; this has been a very valuable tool for developing my research group and without the support from CSA I would not be at the point I am now." In addition, a few non-recipients commented that the extent of funding awarded by the CSA pales in comparison to that of other space-oriented countries.

Redundancy with Other Programs

An internet search did not reveal any comparable transfer payment programs aimed specifically at funding Canadian space-related S&T development. In addition, most CSA employees (57% or 77% of those who provided a rating) reported that the Class G&C Program is not redundant with other Canadian funding programs. As one CSA employee explained, "Although there are other funding sources available, few (if any) really cater to the space domain. No doubt that leveraging opportunities exists, but the reality remains that the CSA has a unique mandate and requirements that are best served by an internal program."

The absence of redundancy was further supported by the finding reported above that only 21% of non-recipients were able to carry out their proposed projects via other funding sources and, even then, the funding they received was insufficient. As one non-recipient (whose proposed project was never actualized) explained, "I very much appreciate having the opportunity to apply for funding from CSA, as it is essential for my research and not available elsewhere." Similarly, several funding recipients commented that there are only a few other funding opportunities for space-related S&T development and 88% of recipients reported that they most likely would not have been able to carry out their projects if Class G&C funding were unavailable.

Historically, the Class G&C program has collaborated on occasion with NSERC for specific funding initiatives. It was suggested that closer collaboration between the Class G&C Program and granting councils could prove beneficial for leveraging funds. To facilitate this collaboration, a few CSA employees suggested that the delivery mechanisms of the Class G&C Program (e.g., application and performance reporting processes) should be aligned with those of granting councils. Incidentally, efforts to harmonize the Class G&C Program with OGDs would be simplified by the fact that the federal granting councils have harmonized their Terms and Conditions and standardized their processes (according to interview with OGDs conducted for this evaluation).

Achievement of Expected Outputs

Evaluation findings showed that the following G&Cs products were directly produced by the Class G&C Program and its Research component during the evaluation period (October 1, 2009 to March 31, 2014):

• Ten AOs were launched through the program's Research Component (two of which had yet to award funding agreements by the end of the evaluation period);
• Ranked lists of applicants were produced for each of the eight AOs for which funding agreements had been awarded;
• $40M in G&Cs were awarded by the Class G&C Program;
• 91% ($36M) of all Class G&C funding pertained to the program's Research component;
  • 76% of Research component funding was awarded by the CSA's Future Canadian Space Capacity program (PAA 1.3);
  • 98% of Research component funding was awarded to universities;
• 195 Research component funding agreements were awarded;
  • 40% stemmed from unsolicited proposals;
  • 60% were solicited via AOs (two of which were launched prior to the approval of the Class G&C Program's Terms and Conditions in 2009); and
• 94% of the requested progress and final reports were completed by funding recipients, representing 94% of Research component-funded projects.

According to most key informants, effective funding application mechanisms were in place, the AO selection process was fair, and the reporting requirements were similar to those of OGDs. Though ranked lists of funding priorities applicable to all G&C initiatives (a planned program output) had not been produced by each of the CSA's branches, an assessment of whether the ten AOs launched during the evaluation period were aligned with CSA priorities was conducted on a case-by-case basis. The CSA should standardize the process for identifying ranked lists of funding priorities applicable to all G&C initiatives.

With respect to solicited proposals, evaluation findings showed that information regarding both the nature of the proposal selection process and delays in the selection process was not provided to applicants consistently across AOs, and constructive feedback on proposals was also provided inconsistently. In addition, there were redundancies within AO application forms and inconsistencies across AOs application processes. To address these shortcomings and in keeping with TBS recommendations for transfer payment programs, the CSA should standardize the application, selection, and feedback processes across AOs and clearly communicate these processes to the Canadian space community. To facilitate funding applicants' internal planning, key informants also suggested increasing the predictability of the timing of AOs.

Unlike solicited projects funded via AOs, evaluation findings indicate that there are no clearly articulated selection criteria for unsolicited proposals. Furthermore, key informants raised concerns about the fairness and transparency of the unsolicited proposal selection process, as well as about the extent of alignment between unsolicited projects and CSA funding priorities. As with AOs for solicited proposals, introducing standardized and transparent application, selection, and feedback processes for unsolicited proposals would address the challenges identified with the unsolicited proposal funding mechanism. Furthermore, the recommendation made above regarding producing a ranked list of funding priorities for each branch of the department would ensure alignment between unsolicited projects and CSA priorities.

The Class G&C Program's PM Strategy identifies performance indicators, baseline data, and targets for the program's expected outcomes, but not for expected outputs, indicating a need to update the program's PM Strategy to ensure that it contains output-related indicators and baseline data and targets. Also, some of the G&C data stored in the CSA's databases were inaccurate (e.g., more than one identification number per agreement in some cases, erroneous list of completed progress and final reports), indicating a need to improve the data entry process in order to ensure archival data validity.

Achievement of Expected Outcomes

Evaluation findings showed that the projects funded by the Class G&C Program's Research component successfully contributed to increasing space S&T knowledge – though clear alignment between the types of knowledge gained and the CSA's priorities was more evident for solicited projects than for unsolicited projects, thereby reinforcing the need for a standardized unsolicited proposal selection process. Evidence also showed that the knowledge produced by funded projects resulted in important technological or scientific breakthroughs, as well as in the development of new ideas for integration in future space missions, the use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. Furthermore, the extent of knowledge dissemination increased; presentations and publications were widely disseminated both nationally and internationally, as well as within academic circles and for the general public. However, according to CSA and funding recipient key informants, the lack of continuity in funding for S&T domains targeted by prior AOs curtailed some funding recipients' abilities to bring their R&D projects to a higher level of maturity and reap their full potential for knowledge gain.

In addition to increasing knowledge generation and dissemination, the Class G&C Program effectively increased Canada's focus on space. For example, funding recipients brought new players into space-related research fields and the number of organizations participating in funded projects' research teams increased considerably over the course of the evaluation period. In addition, projects funded by the Research component led to notable increases in HQP and space-related expertise capacity within those organizations that received Class G&C funding, as well as outside of these organizations – though several funding recipients expressed concern that the capacities developed could not be maintained following project completion.

Class G&C funding also fostered new and maintained collaborations between funding recipients and both national and international partners, which produced leveraged funds for about half of the funded projects. Furthermore, as a result of the diversity among organizations participating in funded projects' research teams and among partnering organizations, funding recipients established multidisciplinary and multi-institutional collaborations that produced achievements specifically related to multidisciplinary research in half of the funded projects.

Due to changes in the progress and final report templates over time, approximately one-third of the report data provided by recipients were only accessible one year at a time and could not be linked to related data collected during other years in the evaluation period, thereby precluding some aggregated, multi-year analyses of the program's performance. This methodological limitation indicates a need to ensure that performance data from recipients' progress and final reports are systematically stored in an accessible, multi-year format conducive to supporting evaluation.