An expert panel has designed a sophisticated methodology for assessing the impacts of Ontario's innovation policies and programs although the province – like nearly all jurisdictions – lacks the indicators and data and measurement capabilities to implement it. The Ministry of Research and Innovation requested that the Council of Canadian Academies (CCA) assemble the Expert Panel on the Socio-Economic Impacts of Innovation Investments to get a better sense of how effective the $600-700 million it spends annually on innovation is and how its investments can have a greater impact.

The panel — chaired by Esko Aho — senior fellow at Harvard Univ, a consultative partner for Nokia Corporation and former prime minister of Finland — was charged with recommending steps that could be taken to determine whether the $3.2 billion in innovation-related investments made since 2005 are having the desired impact.

"The data are a key problem. Nobody is there in terms of what the panel envisioned so the report is visionary in that sense," says Dr Margaret Dalziel, associate professor at the Univ of Waterloo's Conrad Business, Entrepreneurship and Technology Centre, co-founder and VP research with The Evidence Network (and an expert panel member). "Creating an intervention is difficult so we wanted to isolate impact and bring it all back in to ensure it works within the whole system."

For Ontario, billions in innovation investment are at stake as the province seeks to determine whether its policies and programs are having the desired effect, namely an increase in competitiveness, productivity and economic activity.

The report notes that while Ontario is by far the largest economy in Canada, generating $655 billion in GDP in 2011, it still has a trade deficit of $100 billion. Its R&D spending is the second highest in the country at 2.23%, second only to Quebec (2.49%). In April 2008, MRI launched its Ontario Innovation Agenda which facilitated investment in key sectors including the bio-economy and clean technologies; advanced health technologies (oncology, regenerative medicine, and neuroscience); pharmaceutical research and manufacturing; and digital media and information and communications technologies. Yet without clear objectives and measurement tools, it has not been able to measure the impact of its investments.

To effectively gauge impact, the panel put forward a suite of measurement methodologies: case studies; scorecards and benchmarking; indicator-based frameworks; general econometric approaches (CDM and growth accounting); difference-in- difference estimation; matching estimation; regression discontinuity design; and, random field experiments.

"If the goal of measurement is to produce estimates of short-term impact, the best source of data is a properly designed client-based survey that minimizes the subjectivity of responses. If the goal of measurement is to firmly establish a rigorous, reliable, and long-term causal estimate of program impact, state-of- the-art approaches, like random field experiments and regression discontinuity design, require specific program design, a substantial quantity of data, and a significant amount of time." —
CCA Expert Panel report