Innovation by numbers: A plea to stop counting and start investigating

By Dr Richard Hawkins

As one who has always advocated the expansion of capabilities to monitor and measure innovation, I venture what follows with trepidation. Because I find myself on the verge of recommending a moratorium. The tail is now wagging the dog, and this must stop.

My concern is prompted by the comprehensive review, issued recently by the University of Manchester, of what has and has not been demonstrated empirically about innovation in the scientific literature spanning some 60 years. But it is amplified by the simultaneous appearance of the Thompson Reuters Top 100 Global Innovators for 2014, which is based on essentially the same kinds of metrics that underpin ever increasing amounts of this literature.

Perhaps because of its unprecedented scope, the Manchester exercise impressed upon me as never before just how few of the assumptions that drive innovation strategy and policy these days have strong scientific support. I was even more struck by how ambiguous most of the evidence really is as to which specific institutional, legal, policy and management factors affect innovation in any replicable way. And then, looking at the Top Global Innovators list, I was struck by how it manages to ignore just about everything that has been established reliably through half a century of scientific work.

So if the scientific measurement of innovation has so far produced such ambiguous outcomes and perverse impacts, why are we doing it? There is no doubt that we have become much better at it. The technical apparatus is now astoundingly sophisticated. The benchmarks for publishing in the field are now much more rigorous, and mostly oriented towards making methodological strides in quantitative analysis. And yet, from careful review by some of the leading scholars in the field, few genuinely new insights appear to have emerged that have any obvious utility outside of the monastic academic orders.

On the other hand, as a product of one of the world's highest profile data factories, the Thompson-Reuters product is likely to achieve massive circulation and to be taken seriously. Policy makers will no doubt inhale it. But all it does is to list alphabetically 100 companies that, according to the (undisclosed) method employed, demonstrate high efficiency in acquiring and exploiting lots of patents. That's it. The whole movie.

Extreme reductionism

Moreover, criticizing it on scientific grounds would be pointless. Its only egregious diversion is that it slyly relates the performance of listed companies to such policy-linked factors as higher education, fiscal measures, R&D spend and the like. None of these factors have ever been shown conclusively or consistently to have any necessary or predictable relationship to one another, let alone to innovation. Except of course if you engage in reductionism so extreme that you exclude all reality.

But are these really the 100 most innovative companies in the world? Of course not. That is a ridiculous claim. To accept it requires that we exclude all knowledge that can't be patented from significance as a factor in innovation. We must also exclude the vast multitude of factors that contribute to the business acumen of firms, which is really where innovation happens or not.

World-beaters or basket cases

We must also confine the ability to innovate solely to producers of technology, thus marginalizing better than 95% of the economy. And then we would have to ignore that membership in this club is no indication at all of the health of these companies, or of their sectors, or of the jurisdictions in which they operate. On exactly the same metric, some can be world-beaters, others basket cases.

So what we have in the end is a metric that there is no point in disputing on scientific grounds, but that tells us absolutely nothing useful, and, were we to act on it, would lead in mindlessly futile directions. But as readers may have guessed by now, my beef is not with Thompson-Reuters or their like. They do what they do. What concerns me is how exercises like this also illuminate the apparent impotence or irrelevance for policy and strategy of what is coming out of academic research. This certainly is not because of any failure to come up with more advanced metrics. Indeed, the field has become obsessed with them. So why can something as crude as the Top 100 Innovators still find air to breathe?

Some years ago, governance expert Gilles Paquet wrote eloquently about the epistemological chasm into which investigation of social and economic phenomena can fall when an obsession with appearing to be "scientific" degenerates into "scientism". This happens pretty quickly once metrics become goals in themselves and the phenomena being investigated become defined by the technical apparatus available to asses them rather than the other way around. As Prof Paquet noted sagely, the implications for policy making are profound as numbers cease to correspond to anything, as policy stakeholders settle for confirmation of the status quo, and as space for experimentation evaporates.

Innovation hit parade

My fear is that we have fallen into this void. Thus, innovation "hit parades" continue with impunity to stake bogus claims to the leading edge of techno-metrics. But as we should know by now, fighting numbers with numbers is a mug's game. What the "scientific" community needs to work on are some fundamentally new narratives, based on what we know to be sure, but more importantly on what we now know we don't know. Which, thanks to our Manchester colleagues among many others, turns out to be quite a lot.

To bridge the gap, and to start moving forward, we need to get back to the real spirit of scientific inquiry which is figuring out what to measure in the first place, or whether measurement is even helpful or possible. I for one am happy to suspend the numbers game until we sort this out. Then maybe we can recalibrate and put our rapidly accumulating technical expertise to some useful work. But we also need some fresh thinking from our client communities and some demands from them to start thinking outside of the box again.

Dr Richard Hawkins is a professor in the Science Technology and Society Program at the University of Calgary.