Over the last week or so I’ve come to realise a lot of my reading and thinking has an implicit focus, which I ought to make much more explicit. That is: the capacity of futures work (or “applied foresight” or “futures research” as many folk call it) to help us to more effectively address complex twenty-first century problems. Sometimes they’re called “super wicked” problems, or just “wicked” problems. Others like Otto Scharmer refer to “hyper complex” problems.
Interestingly, the famous 1973 paper that introduced the concept of “wicked problems” made the (then) novel argument that decision tools should be developed based on the type of problem that is faced (e.g. by policy makers) and their key features.
This line of inquiry, then, continues this search and broad approach. First, some definitions:
-According to Scharmer a ‘hyper complex problem’ is characterised by: dynamic complexity (cause and effect are distant in time and space; it’s hard to properly grasp the problem from first-hand experience); social complexity (conflicting worldviews, cultures and interests among the key stakeholders; those involved see things very differently); and emergent complexity (or ‘generatively complex’ in Adam Kahane’s terms), that is they unfold in unpredictable and unfamiliar ways.
-A ‘super wicked problem’ is said to have four key features that makes it more challenging that an ordinary ‘wicked problem’ (see this paper): time is running out (i.e. urgency); those who cause the problem also seek to provide a solution; the central authority needed to address it is weak or non-existent; and, partly as a result, policy responses discount the future irrationally. In addressing such problems the authors contend that we need to find new ways of intervening in the tendency to give more weight to immediate interests over long-term interests (echoing many others).
The latter issue is core to Dennis Meadows’ distinction between easy and difficult problems
A pessimistic perspective (or, perhaps, realistic?)
Adapt by Tim Harford (Abacus books, 2011) can be read as a 300 page long argument against foresight. The book focusses on the challenge of dealing with complex problems in a complex world, arguing “the more complex and elusive our problems are, the more effective trial and error becomes, relative to the alternatives” (p.35). Its messages are similar to Reframe by Eric Knight. Harford questions assumptions about planning and the “planner’s dream” (of the capacity to calculate the best decisions or even the future through ever-more data rich modelling and analysis), which he describes as a seductive vision we should move on from but can’t seem to. That point made me think of “big data” evangelists and the emergence of new forms of predictive analytics.
Several headings in the opening chapter tell some of the story: e.g. ‘The experts are humbled’, ‘We are blinder than we think’, and so on. He challenges “the story we tell ourselves about how change happens: that the solution to any problem is a new leader with a new strategy” (p.59) – a football club sacking its coach, the buzz when President Obama was elected – arguing in reality, in most cases, such as those highlighted from US military strategy, “it is impossible to know in advance what the correct strategy will be” (p.65). Besides, the constantly changing nature of many problems and their wider landscape (e.g. the market you’re competing in, or the battlefield being fought on, and so on) strongly militate against foresight – that’s the central argument, anyway.
He adapts the related metaphor of a “fitness landscape” from evolutionary theory. It goes like this: imagine a landscape of possible solutions for a particular problem (each solution taking a ‘square’ on this landscape), and the better the solution the higher the altitude of the square it contains — creating a jumble of cliffs, summits (good solutions), and valleys (bad solutions). In an economy, or a biological ecosystem, the peaks keep moving and changing in ever-shifting landscapes, and “as one peak subsides, others may not be clearly visible” (p.15). He asserts:
Evolution is effective because, rather than engaging in an exhaustive, time-consuming search for the highest peak – a peak that may not even be there tomorrow – it produces ongoing ‘works for now’ solutions to a complex and ever-changing set of problems (p.16)
This argument immediately brought to my mind the continual development of energy technologies and the characteristics of modern energy systems and energy usage — which problematises any “blueprint” of a low or zero carbon energy future. When someone creates a detailed vision and proposal it’s soon out-of-date as energy trends change, technologies evolve, and new ideas emerge. It is a strong argument against rigid visions; however, advocacy requires closure.
In also reminds me of Nate Silver’s response to the question ‘are we getting better at forecasting the future?’ He said: “as-a-whole not by a lot, maybe a little bit at a time. Technology gradually improves, and scientific knowledge becomes more complete, but it’s still very incomplete relative to the scope of questions that we have. And in some things, and in human endeavour, society becomes more complex, so you might have better methods, arguably, and more data, but you’re also are running against a moving target when the system itself becomes more complicated.”
Harford often adopts and adapts evolutionary perspectives. E.g. “In biological evolution, the evolutionary process has no foresight. It is the result of pure trial and error over hundreds of millions of years. Could that also be true in an economy, despite the best efforts of managers, corporate strategists and management consultants?” (p.17). He draws on research suggesting effective planning and leadership is far rarer than we think in the modern world.
(NOTE: the argument made is a stark contrast with some key perspectives in the ‘wicked problems’ literature. The latter argues that, when addressing these problems, there is little or no opportunity to learn from trial-and-error, moreover, that there is little opportunity to be ‘wrong’).
The group of scholars that first argued climate change should be characterised as a ‘super wicked problem’ also argue that an ‘applied forward reasoning’ orientation is needed to address this new class of problem (see paper). Such forward reasoning involves “looking forwards in time to elucidate how generating path dependencies might foster desired policy outcomes in the future” (p.124) – the operative word being ‘how’. They further argue that this can help to “nurture “a policy process in which our long-term interests gain sway over our short-term interests (p.129).
Like Tim Harford, the author of Adapt, they are critical of conventional policy analysis tools that “assume a relatively linear and predictable world” and attempts at probabilistic prediction. They assert that ‘super wicked problems’ “occur in open, non-linear systems, where human beings may also interact in reflective and unpredictable ways to change their environment” (p.139). However, rather than abandon forward-looking analysis, they call for more effort to “reason forward to how the problem and interventions might unfold over time”, e.g. via scenario-building. They then proceed to identify different causal processes through which interventions to create “low emissions” pathways could potentially ‘work’ (e.g. lock-in, self-reinforcing, increasing returns, and positive feedback) and, crucially, specify ‘logics’ which can be incorporated into scenario building.
The stated purpose of scenario analysis is as follows: to “assess whether there is a plausible logic that an intervention, or set of interventions, is likely to unleash a path-dependent process that can change behaviour before the time runs out to address the super wicked problem in question” (p.132). They stress the need “to consider non-linear and unfolding causal, yet unpredictable, policy trajectories” (p.138) and to pay attention to potential indirect or unintended effects.
In making this argument they also contend that single-shot “big bang” policies are the wrong way to go, due to the likelihood of producing “shocks” that hamper implementation and compliance and the difficulty gaining adequate support for paradigmatic change. Accordingly, they call for “progressive incrementalism” in which policy development occurs in smalls steps which accumulate to produce significant results. In this respect they are in agreement with Harford.
Transformative Scenario Planning by Adam Kahane (Berrett-Koehler, 2012) also sits in the optimistic category. Kahane’s book addresses the ‘hyper complex’ problems characterised by Otto Scharmer, and frames the transformative scenario method as an application of Scharmer’s “theory U process” (pp.22-23). Whilst Kahane clearly states that there are no “sure bets” when dealing with such problems in complex social systems, he also states the following:
The transformative scenario planning is one of a family of stories about how to transform social systems collaboratively… Transformative scenario planning is a particularly effective way for a team of actors to generate collaborative forward movement on a complex, stuck, problematic situation (pp.91-92)
In essence, Kahane is calling for a shift from using scenario planning as aid for ‘adaptive’ planning and strategies (e.g. for risk management) to using scenario planning as a process to influence the future (an ‘activist’ orientation). How this occurs is not addressed in-detail; the main ideas are that the process of scenario-building helps to relax and evolve our mental models, and that, when done collaboratively, this can help actors to work effectively and creatively with diverse others. A further claim is the scenarios can change deeper myths and metaphors (a possible link to Causal Layered Analysis [CLA]) and even become myths (p.90). See Chapter 8, titled ‘New Stories Can Generate New Realities’. The main success stories to support these views are Kahane’s work in conflict-filled countries in the process of complex transitions such as South Africa and Columbia.
Kahane’s argument is consistent with broader arguments about the ‘internal’ dimensions of change. For example, Alex Steffen has argued that the process of creating zero carbon cities is “is first and foremost a struggle of vision”. Similarly, Sohail Inayatullah has pioneered forms of “narrative foresight” (that’s his term), which focusses on the ‘exploration and creation of alternative and preferred futures and the worldviews and myths/metaphors that underlie them’. It has a particular focus on the deepest layer of CLA – that is, on deep myths and metaphors.
The book also led me to consider other forms of collaborative planning and what could be called ‘anticipatory coordination’ (a term coined by Arie Rip). For example, the semiconductor industry is famous for its use of technological roadmapping exercises, a process that is sponsored by the five leading chip manufacturing regions in the world (Europe, Japan, Korea, Taiwan, and the United States). STS scholars have argued that ‘Moore’s Law’ should be viewed as a “self-fulfilling prophecy” and not as a “prediction” in-part because of influential planning exercises like these. Additionally, because industry players expect Moore’s Law to continue the planning and R&D become framed by these expectations, also contributing to the emergence of such future pathways.
Finally, some futures scholars have suggested that complexity theory might be used to strengthen scenario analysis (rather than to critique it) – albeit with a shift away from probability analysis and, instead, a focus on ‘plausibility-based scenario practices’. This new type of scenario might address some of the concerns of skeptics like Tim Harford, but I doubt it. Nonetheless, this work indicates how a richer theoretical grounding could be developed to improve futures practices.
Additional perspectives and thoughts
- Assumptions about change and ‘change management’ in complex systems: On the one hand, those harnessing complexity thinking and engaging with the indeterminacy of the future are questioning these. (Also see the great paper ‘Coping with chaos in change processes’ co-authored by Arie Rip). Indeed Kahane states “the mystery at the heart of transformative scenario planning is that we cannot know the future. We can investigate it and influence it, but we cannot calculate or control it” (p.96). On the other hand, futures scholars and practitioner suggest we can do better than the trial-and-error approaches championed by Harford.
- The limits of the dominant “prophetic tradition” of sustainable development (which has a predictive orientation), as epitomised by the likes of Paul Ehrlich and, more recently, Jorgen Randers (in his book 2052: A Global Forecast for the Next Forty Years) and Paul Gilding (in The Great Disruption). The literature on ‘super wicked’, ‘wicked’ problems, and complexity theory and thinking – emphasising non-linearity, open systems, uncertainty and chaos – problematises this approach, along with new tools utilising increasing computational power and data gathering to ‘calculate’ the future. Recently, we’ve seen the re-emergence of strong prophetic aspects to environmental discourses and science, which some argue represents the re-emergence of ‘environmental determinism’. As futures studies emerged as, in part, a critique of determinism and deterministic future projections foresight practitioners are critical of such trends. (Full disclosure: I am critical of these trends; however, I recognise the need to more proactively consider the future to address sustainability risks. It’s a tough balancing act).
- The emergent ideas of the “optimists camp” perhaps provide an initial outline of the “new domains of praxis” called for by scholars like Mike Hulme
Change processes are a central concern in formulating and enacting responses to ‘super wicked problems’ and ‘wicked problems’. Many current global environmental problems, e.g. climate change, require consideration of the medium-term and longer-term future, raising the second issue.
It is also important to consider the potential utility of different futures techniques for better addressing the challenges raised by the ‘pessimists camp’ and other scholars. For example, the ‘futures wheel’ technique could assist with anticipating non-linear and unintended effects from planning interventions. Changes to the ever-evolving “fitness landscape” (the evolutionary theory metaphor) might be better spotted and considered via horizon scanning or trend analysis.
Some of these ideas are discussed in the following paper I published last year: McGrail, S. 2012, ”Cracks in the System’: Problematisation of the Future and the Growth of Anticipatory and Interventionist Practices’, Journal of Futures Studies, vol. 16, no. 3 (March), pp. 21-46.