Broadly speaking, my doctoral project is examining the increasing use of foresight methods for tackling sustainability problems, and their effects. A key focus that’s emerged is inquiring into related practices for enhancing the handling of more “wicked” sustainability problems – for example, as part of participatory governance approaches – and for preventing worse challenges emerging in the future. These practices often incorporate or adapt methods such as scenario-building.
Research conducted to-date reveals a need to more fully consider whether such practices actually produce ‘better’ results. And it poses questions about how such evaluations might be done, and about what could or should be the focus of such research. I’m currently developing a collaborative project with a consultancy, Reos Partners, which I hope will help to address these questions.
This post summarises some of the related key learnings and questions identified to-date.
Evaluating ‘foresight’ and better understanding its ‘value proposition’
Practitioners often report that evaluation tends to be anecdotal. Similarly, academics critique some of the futures literature as often being, in effect, post-rationalised accounts of success. Also, practitioners tend to publish papers about their own processes, limiting independent evaluation. To my surprise, technology foresight is similar: scholars lament the ‘immaturity of evaluation’ and point to a scarce empirical-base for impact assessment. Leading scholars in The Handbook of Technology Foresight state that the “evidence of the worth of foresight is therefore largely anecdotal” (p.349).
That said, some types of evaluation are occasionally done to evaluate technology foresight. Some are akin to quality assurance, whereby external experts review a process (i.e. like peer review). The key concept of “additionality” framed the recent evaluation of the UK Government Foresight program. However, a key gap is understanding and measuring impacts, particularly over the longer-term. The approaches used in peripheral fields like technology assessment may offer learnings.
More practitioners and scholars are discussing these issues, including ambiguity regarding “success” criteria (e.g. here). As practitioners moved away from predictive accuracy as a success or quality measure I suggested no coherent alternatives have been put in its place. A consensus seems to be emerging that “success” criteria are context-dependent and require much more thought.
Another emerging focus is examining ‘how’ the value/desired outcome is created in foresight, and what shapes this (e.g. particular methodological choices, contextual factors, etc). Scholars such as Rene Rohrbeck have called for more research to address these questions.
Evaluating interventions in complex sustainability problems
As I’ve begun to focus more on evaluation I’ve begun to think more about the areas of practice I’m looking at. This reveals new complexities and suggests new questions. For example, in Reflexive Governance for Sustainable Development European social scientists argue it’s impossible to predict the effects of interventions in sustainability problems and that outcomes/results cannot, and should not, be predefined. How does this shape the potential for evaluation and what methods and foci should be adopted in this research? They also argue the “high probability of unintended consequences needs to be assumed as an essential condition of problem-solving strategies” (p.12). The literature on “wicked” problems I’ve reviewed also raised questions about the scope of evaluation.
Reading this literature on ‘reflexive governance’ brought to mind some themes in early discussions with practitioners. They reported key challenges posed by ‘emergent complexity’; that is, they were similarly of the view that intervention results cannot and should be predefined.
Nonetheless, these European social scientists suggest the ‘working’ of processes and governance arrangements can be better evaluated, i.e. procedural quality. They also suggest greater focus on change indicators (e.g. of structural change) and monitoring. Moreover, they argue ‘reflexive governance’ strategies need to be adaptive and “should feature experimentation, monitoring and evaluation”. Broader aims of ‘reflexive’ governance could be evaluated, such as:
- Improving the quality of problem definitions
- Achieving societal goals without generating second-order problems (i.e., breaking that cycle)
- Evaluating and reconsidering societal ends
Different ‘schools’ of evaluative research
I’ve learnt that these issues and questions can be related to contrasting approaches to evaluation such as quasi-experimental evaluation (QEE), constructivist, or realist evaluation. QEE aims to prove what ‘works’ and does not ‘work’, via careful study designs that seek to isolate particular causal factors. This is akin to the randomised controlled laboratory tests of a new drug. Constructivists critiqued these methods and proposed methods that incorporate subjective perceptions, e.g. of program stakeholders. Realist evaluation is a form of theory-driven evaluation; ‘realists’ questions whether QEE is achievable in “open” social systems and advocate a different model of causality grounded in a realist philosophy of science. For them programs don’t ‘work’; instead programs are ‘active’, meaning “the triggers of change in most interventions are ultimately located in the reasoning and resources of those touched by the programme” (Pawson & Tilley, 2004). Human volition is central: “active programmes… only work through the stakeholders’ reasoning”. Programs are also seen as open systems: they cannot be fully isolated or kept constant (as required in QEE), and they can change the conditions that made them work.
The following aspects of ‘realist’ evaluation appear to be well-suited to evaluating foresight processes:
- The focus inspecting the reasoning of ‘program’ subjects and stakeholders. This is based on the following distinction: “It is not programmes that work but the resources they offer their subjects to make them work” (Pawson & Tilley, 2004); Realist evaluators argue experimental evaluation (and QEE) tends to problematically excludes these causal agents;
- The focus on the processes activated in a program (termed ‘mechanisms’) which make sense of, or may be responsible for, changes: given the increasing focus on ‘how’ foresight exercises create value/outcomes, causal mechanisms could be examined;
- The focus on interventions contexts, which links with the context-dependency of foresight: ‘realists’ recognise that causal ‘mechanisms’ are only active under particular circumstances – which is the ‘for whom’ and ‘in what circumstances’ part of realist evaluation;
- Evaluation findings that provide practitioner guidance: ‘realists’ argue that evaluative research hasn’t adequately addressed the question “would it work here?” New approaches could help practitioners to know where and when to conduct particular interventions; and
- The focus on coming to grips with the complexity of interventions and the provision nature of findings: a ‘program’ is seen as “a complex social system introduced amidst complex social systems. Both of these systems are open” (Pawson & Tilley, 2004).
Wider related trends and theoretical frameworks
The first wider trend that is suggested by my reading is a recent growth in collaborative foresight, such as multi-organisational scenario building and other practices in inter-organisational contexts. These papers suggest these new context present important new challenges that need to be better understood, such as power dynamics and the strategic games that occur in them.
The second wider trend is the use and adapting of futures methods for different purposes, which primarily have an influencing function (not anticipatory). Reos Partners work is a good example of this and collaboration with them would provide opportunities to examine this further. Practitioners are taking futures thinking into new areas, particularly regarding sustainability issues.
My research has also identified additional relevant theoretical frameworks and concepts which may be useful in my analysis, or even shape the specific research questions, including:
- The Sociology of Expectations (SoE): this theoretical framework explores that roles and social dynamics of future expectations – both general expectations about the future, and normative expectations (such as visions). Van Lente (see this paper), from the Copernicus Institute of Sustainable Development in The Netherlands, has argued that the social dynamics and roles of future expectations is important for the efficacy of foresight exercise – for example, a foresight exercise could consequently further embed a “lock-in” situation rather than disrupt it! Very interesting. SoE has focused on the “production” and circulation of expectations in science and technology (in STS research) but perhaps it can be extended;
- The concept of reflective practice as a form of learning that seeks inquiry about the most fundamental assumptions and premises behind practices. According to Raelin it involves probing to a much deeper level than “trial-and-error” experience (see paper);
- The concept of reflexive governance which describes new problem-handling approaches and more distributed ‘steering’. It “take[s] up the repercussions of reflexivity: uncertainty, ambiguity and dispersion of social control” (source: Reflexive Governance for Sustainable Development, p.xvi) and is argued to capture the major governance implications of sustainability; and
- Post-normal science: A recent paper in Futures on future scenarios for Venice argues that “futuring activities” are collective exercises that need to “happen in a post-normal context”, “i.e. involving all relevant social actors in safe and legitimate debate spaces”. Moreover, such ‘post-normal futuring’ (that’s their terminology) is suggested to be able to open “spaces where sustainability conversations could take place”. This paper is co-authored by one of the creators of post-normal science theory (Silvio Funtowicz), making it a very important reference.
A ‘realist evaluation’ approach, if adopted, has its own epistemology and ontology. This includes a realist philosophy of science, a focus on “open” systems, and particular aspects of programs (e.g. mechanisms) which are viewed as an important part of reality that needs to be probed.
Finally, a range of related concepts and theories are relevant to institutional and structural change. These include the concepts of ‘lock-in’, ‘system innovation’, organisational learning and so on. The theory and practice of ‘large systems change’ is also a rapidly emerging field (e.g. see here).
Thinking about potential contributions to knowledge
My research to-date suggests there is an important contribution to be made in investigating how ‘foresight’ activities can be better evaluated, and examining related challenges. Some scholars I’ve liaised with are quite skeptical about if/how this can be done. The capacity of practitioners to evaluate their work is also an obvious and important issue. Nonetheless, it is clearly something that many practitioners are struggling with and are seeking greater clarity on, and there is increasing interest in evaluation. This is also relevant to additional fields conducting interventions into sustainability problems.
A related critique of foresight argues it is a “practice-led” field, with some academics questioning if it has an adequate theoretical and methodological core. This brings to mind Senge and Scharmer’s model of a knowledge creation system (see paper) which included ‘theory and methods’, ‘tools’ and ‘practical know-how’ and their interconnections. They discussed breakdowns that can occur in the major linkages between these elements. Thus, there is a potential contribution in helping to connect them – for example, through new practical know-how informing new theory development, or by better connecting ‘theory and method’ with the development of new tools. It’s time to re-read this paper!
Reviewing the identified theoretical frameworks and concepts a few questions stand out: what is the contribution of new prospective practices to achieving ‘reflexive’ governance? What is the potential and limitation of these practices for disrupting “lock-in” situations? (Regarding this question I’ve identified the related concepts of “innovation impasses” and “waiting games”, which can occur in situations of high uncertainty [see special issue], and “path dependency”). How does the Sociology of Expectations help to explain the outcomes and dynamics of participatory/discursive foresight processes?
Going back to the ideas in my original PhD proposal, there is an intensifying need to better probe the roles of ‘foresight’ activities in different contexts, and for a better understanding of how it does/doesn’t play these roles and why. Some ideas are emerging, which can be drawn and built on in my doctoral research, but there is precious little evidence. More clarity on such roles, and related underpinning theory, would also help with identifying specific context-relevant “success” measures.
Pawson, R. & Tillet, N. (2004), “Realist Evaluation”, accessed at http://www.communitymatters.com.au/RE_chapter.pdf
Also see this excellent review of Reflexive Governance for Sustainable Development which concisely outlines many of the key points in what is complex, “academic” book – on the last two pages of this pdf.