In our latest paper published in Social Science and Medicine we critique the way ‘impact’ is measured within the UK’s Research Excellence Framework* (REF) and in doing so we propose an alternative measurement, one based on enlightenment and process rather than outcomes.
In recent years the academic landscape has been shifting, and has been significantly affected by the introduction of an ‘impact agenda’. Academics are increasingly expected to demonstrate their broader engagement with the world and the evidence-related outcomes of their work. This change has profound implications for our universities and for the staff within them. In this paper we chart the rise of the impact agenda within the UK and using a case study of a recent research project on tobacco and alcohol we outline how we tried to capture the ‘impact’ that arose from this work. In doing so we use Oliver et al’s (2014) systematic review to organise our critique into thematic clusters (contact and relationships, organisations and resources, research and researcher characteristics, policymaker characteristics and policy characteristics). In addition we add a sixth theme of time. Our paper shows that, for us, the research was not immediately ‘impactful’, nor was the route to eventual impact linear.
On a personal level, my experience of working with external organisations was overwhelmingly positive. I thoroughly enjoyed it, and thrived in this academic/non-academic interface. Our commitment to this outreach has been rewarded, both by citations in many non-academic reports (see for example here and here) and in the relationships that we have cemented outside of academia. Our work was recognised by organisations and individuals outside of the University, indeed a motion in the Scottish Parliament signed by 27 MSPs congratulated us on our efforts. In addition we were exposed to lay understandings of our research that we may not otherwise have been, which helped us to clarify our message. For example, acting as a discussant on the Kaye Adams BBC Radio Scotland phone-in can concentrate the mind! Not all of the responses were positive and we also had inevitable feedback from other sectors (see here).
Despite all of this we are wary of the current framing of the impact agenda. We are not arguing that research should not be taken out of the academy, quite the opposite. Instead we argue that a rather more nuanced notion of impact should be attentive to connections, conversations and impressions, none of which can be measured in a metric sense. In this paper we propose a move towards an ‘enlightenment’ model based on Weiss (1977). Rather than measuring only outcomes and assuming a direct path from research to policy (or other kinds of) change, we might give greater recognition to process. This would involve recognising that our research is just one small part of wider understandings and whilst we can contribute to change, this often takes time and our work may not represent the defining determinant of change. In such a model we would alter the focus of REF impact from demonstrable impact to demonstrable knowledge exchange and public engagement.
We highlight three reasons why we should consider such a shift. First, it would encourage researchers to engage with audiences beyond academia as a matter of course. We would not have to specifically plan for the ‘impact’: we would not have to try to guess what impact our research might have before we even carry out the research (the tail that wags the dog). It might also stop us from constructing ‘fairy tales of influence’ (Dunleavy, 2012). Second, it would reduce the extent to which the outcomes of impact are dependent upon serendipitous circumstances – such as the circumstances that helped increase the impact of our work. Empirically-informed theories of policy change consistently emphasise the role of external events over which researchers have little control. The current system of REF is, at least to some extent, likely to be rewarding luck. Third, critical academic work cannot – and should not – be used to meet the short-term and immediate needs of policy makers. This is important in health geography, concerned as it is with questions over the broader determinants of health, such as housing, wealth, education and power. It is precisely because these are of importance that we need to reconsider the impact agenda.
* The REF process seeks to measure the quality of research in UK universities. Higher-scoring universities subsequently attract more government funding.
REFERENCES
Dunleavy, P., 2012. REF Advice Note 1: Understanding HEFCE’s Definition of Impact. LSE Blogs accessible here: http://blogs.lse.ac.uk/impactofsocialsciences/2012/10/22/dunleavy-ref-advice-1/.
Oliver, K., et al., 2014. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research 14 (2).
Weiss, C.H., 1977. Research for Policy’s sake: the enlightenment function of social research. Policy Analysis 3 (4), 531e545.
CRESH 