One of the most important sustainability-related articles you have never read

This may be a big call, but I think one of the most important sustainability-related articles which few people have read is ‘Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences’ by Naomi Oreskes et al (published in Science back in 1994).** Sure it sounds dry, but it usefully considers the increasing use of numerical simulation models in the earth sciences, which is one the most important scientific trends of recent decades. E.g. models developed to evaluate or forecast physical processes such as: predicting the behavior of the climate system in response to increased rising CO2 concentrations; or resource estimation models used to predict petroleum reserves. The paper addresses key issues and principles related to their predictive value, model validity and model verity.

Given the increase in the use of these models, and related predictions, we need to understand their validity and value. We also need to be intelligent consumers of the products of modellers.

The article examines the philosophical basis of the terms “verification” and “validation” as applied to such models in the earth sciences (although the issues and principles are generic). As they write “to say that a model is verified is to say that its truth has been demonstrated, which implies its reliability as a basis for decision-making”; validation means a model is said to reflect real world behavior.

Verification issues: the core issue is that “verification is only possible in closed systems in which all the components of the system are established independently and are known to be correct”. In contrast, natural systems are never closed and models require input parameters that are incompletely known. Oreskes et al write: “the degree to which our assumptions hold in any new study can never be established a priori. The embedded assumptions thus render the system open”. They go on to discuss issues related to non-unique model results and underdetermination.

A related key issue is that a faulty model may (misleadingly) appear to be correct. For instance, they write that “a subset of the problem of non-uniqueness is that two or more errors in auxiliary hypotheses may cancel each other out”. Here an example might be helpful.

Climate scientist James Annan recently blogged about a paper published in Nature in 1972 that apparently made a remarkably accurate prediction of 0.6C warming over the next 30 years. (Also see the related article published by The Guardian). Annan points out that “it looks like he made a number of significant errors which end up cancelling out”. The “eventual outcome was due to the fortuitous cancellation of several factors which he did not account for” and, he argues, it shouldn’t be interpreted that we “understood the climate system really really well” (back in the early 1970s), as is claimed.

Similar comparisons of projections made in the ‘Limits to Growth’ study with later trends are conducted to claim that the “truth” of the underlying model has been demonstrated – even though many of its assumptions have been shown to be incorrect. This is highly misleading, at best.

Validation issues:

Oreskes et al cites some scientists who note that “[t]he most common method of validation involves a comparison of the measured response from in situ testing, lab testing, or natural analogs with the results of computational models that embody the model assumptions that are being tested”. A famous example is the climate model testing conducted by James Hansen. Amongst other model verification strategies, he used a natural event – one of the largest volcanic eruptions of the 20th century (the Pinatubo eruption) – to test his climate models. However, as Oreskes et al argue:

“the agreement between any of these measures and numerical output in no way demonstrates that the model that produced the output is an accurate representation of the real system. Validation in this context signifies consistency within a system or between systems. Such consistency entails nothing about the reliability of the system in representing natural phenomena.”

Drawing on the verification issues and principles they further argue that establishing that a model accurately represents actual processes occurring in a real system is not even a theoretical possibility. Oreskes et al go on to explicate further issues such as regarding model calibration.

However, the real kicker is that:

“Even if a model result is consistent with present and past observational data, there is no guarantee that the model will perform at an equal level when used to predict the future. First, there may be small errors in input data that do not impact the fit of the model under the time frame for which historical data are available, but which, when extrapolated over much larger time frames, do generate significant deviations. Second, a match between model results and present observations is no guarantee that future conditions will be similar, because natural systems are dynamic and may change in unanticipated ways.”

Oreskes et al conclude that both model verification and validation are impossible; and the terms are used by scientists “in ways that are contradictory and misleading”. It is hard to overstate the importance given the prominence of models and model-based projections in sustainability discourses.

How, then, could or should such models be interpreted and used?

Based on these principles, Oreskes et al argue that numerical models should be understood to “always represent complex open systems in which the operative processes are incompletely understood and the required empirical input data are incompletely known. Such models can never be verified”. Furthermore, we shouldn’t think about and evaluate numerical models in an either/or way: “In practice, few (if any) models are entirely confirmed by observational data, and few are entirely refuted”.

The value of models is as a heuristic. A model may resonate with nature, but it is not the real thing, and should be interpreted and used accordingly. This is useful reminder – it’s easy to get impressed by the fancy computers and massive server-farms that power today’s sophisticated computer-based models. However, they are still models which don’t accurately represent the actual real-world processes.

Key implications

A key implication is noted by Oreskes et al: “we must admit that a model may confirm our biases and support incorrect intuitions”. They further suggest that “models are most useful when they are used to challenge existing formulations, rather than to validate or verify them”.

Additionally, if the predictive value of such models is always open to question, this suggests the need to consider when are these predictions, or model-based future projections, seen as credible (despite being inherently open to question), when aren’t they seen as credible, and why. For example, as part of my doctoral research I’m examining CSIRO’s ‘Future Forum’ process which typically includes some modelling exercises which, amongst other outputs, are conducted to support and influence decision-making. The CSIRO brand may be enough for the predictions to be taken seriously. Or perhaps other factors are important. If we consider a social construction of reality perspective we might consider the role of power in shaping the future (e.g. those viewing and using model outputs might consider whether the assumptions are consistent with the intentions of powerful actors whose choices will influence the futures that are being modelled). Many additional perspectives could also be considered.

Finally, the realities of modeling have many implications for environmental advocacy and environmental politics. Oreskes et al don’t argue that we shouldn’t use models, but we do need to be mindful of their limits and the associated issues. Personally, I worry that environmentalists (and many other folk, too) currently rely too heavily on the numerical simulation models developed in the earth sciences – as well as similar models developed in other fields like economics and other social sciences.

For the full argument see the article which can be downloadable here.

**Although according the Google Scholar the article has been cited 2,129 times. Naomi Oreskes is better known for co-authoring Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming. She is currently a Professor at Harvard University.