Making rational decisions in the face of uncertainty poses a challenge to all decision-makers from those involved in emergency hazard management to global climate change policy. Decision-makers can be influenced by how the information is framed (e.g., Taylor et al. 1997; Wernstedt et al. 2019), presented (e.g., Cox et al. 2013; Mulder et al. 2017; Miran et al. 2020), and anchored (e.g., Whyte and Sebenius 1997; Englich et al. 2006). The decision-maker’s risk sensitivity can be informed by political challenges, such as the impact of false alarms and blame in the event of poor forecasts (Demeritt et al. 2010). There can also be issues with end-users understanding, or ignoring, complex forecast output (e.g., Demeritt et al. 2010, 2016). To address these issues this paper will demonstrate how bespoke frameworks, designed in advance by decision-makers, can help to optimize the actions taken, thus minimizing potential losses associated with the subjective risk sensitivities of individual users.
Uncertainty is an unavoidable part of making decisions in a complex and often fast-moving environment. Research has shown that providing uncertainty information can encourage more economically rational decisions (e.g., Nadav-Greenberg et al. 2008; Riveiro et al. 2014), promote user confidence, and reflects the current state-of-the-art science for many environmental situations (World Meteorological Organization 2008). However, the end-users of the uncertainty information do not always interpret it in the way the producers of the information think they should (e.g., Demeritt et al. 2007; Morss et al. 2010). Plus, in many areas, there is no universally agreed method on the design and presentation of uncertain information (Hogan Carr et al. 2018), and this can influence how it is used (Hogan Carr et al. 2016a,b).
One way to quantify uncertainty is to perform several plausible realizations of a situation (known as an ensemble). These realizations are constructed by sampling parameters that are used as input to a simulation or within the simulator itself. Depending on the complexity of the simulator, the number of realizations (or ensemble members) could range from approximately 20 for a global numerical weather prediction model [e.g., the Met Office global weather forecast ensemble (MOGREPS-G) has 18 members; Bowler et al. 2008] to billions for the hydrochemical model of Iorgulescu et al. (2005). Once completed, the output from the ensemble of simulations needs to be condensed in some way so that a decision-maker can use the information. It is possible for many graphics to be produced for different times, locations, and environmental hazard specific thresholds. This would be overwhelming, plus the interpretation of the ensemble relies on the decision-maker’s experience and risk sensitivity (e.g., Mulder et al. 2017; Hogan Carr et al. 2021). This risk sensitivity is generally not transparent, i.e., it depends on the undisclosed judgment of the individual decision-maker, which can lead to issues surrounding defensibility and culpability. In many situations the ensemble mean is the metric of choice used by decision-makers. However, there are many other metrics, such as ensemble spread, ensemble agreement (i.e., how many ensemble members agree that there is going to be an impact in a particular location), and a reasonable best/worst case scenario, which may be more informative.
Bayesian decision theory is a branch of statistics that provides a transparent framework that explicitly discloses the losses associated with each possible action in the decision-making process and leads to optimum decision-making under uncertainty. The losses are adaptable depending on the hazard and the operational priorities of the decision-maker, meaning that the framework can be applied to many different circumstances. This approach has previously been applied to warnings of daily severe precipitation over the United Kingdom by Economou et al. (2016) and by Western et al. (2018), who applied it to the detection of volcanic ash during the 2010 eruption of Icelandic volcano Eyjafjallajökull and the 2011 eruption of Puyehue-Cordón Caulle in Chile.
To address the applicability of the technique presented in Western et al. (2018) to other natural hazards, a decision theoretical approach is applied to an ensemble of Volcanic Ash Transport and Dispersion Model (VATDM) simulations used for the purpose of airline flight planning, focusing on the 2019 eruption of the Russian volcano Raikoke. The atmosphere is categorized into regions of High, Medium, and Low hazard based on volcanic ash concentrations and three idealized sets of losses to show the impact of different operational losses on the optimal action. Finally, to better visualize the potential disruption to aviation operations, the optimal hazard action is projected on the representative flight tracks across the Pacific Basin. By providing this information in this form, it is possible to provide decision-makers with information relevant for their applications and consistent with their risk sensitivity.
What is decision theory?
Bayesian decision theory provides a transparent and coherent framework for making optimal decisions under uncertainty. A decision-maker’s goal is to choose the action, within some finite set of actions,
The construction of a loss table, such as the one shown in Table 1, displays the loss associated with taking each action given the true state of the hazard. Note that losses can be positive or negative (i.e., a gain), but this study only considers non-negative losses, which assumes it is satisfactory that there is no loss in making a correct decision.
An example loss table, where am are the potential actions to be taken and xn are the potential states of the hazard. L(am, xn) are the loss functions for each potential action taken given the actual state of the hazard.
Application of a decision theoretical approach to ensemble volcanic ash forecasts of Raikoke 2019
One area where decisions need to be made using uncertain information is in airline operations following a volcanic eruption, where decisions need to be made about which flight routes are safe and economical to fly before there is complete information about the exact nature of the eruption or dispersion of the volcanic ash particles. If a plane encounters high levels of ash, it can cause temporary engine failure and lead to permanent engine damage, but if it encounters moderate or low levels of ash, this can lead to a need for increased maintenance (Casadevall 1994; Guffanti et al. 2010). Currently these decisions are aided by advisories issued by Volcanic Ash Advisory Centers (VAACs). These advisories are a combination of observations of the ash cloud (ground based and from satellites), output from a VATDM, and forecaster judgment. They indicate the expected geographical position of the ash cloud but contain no quantitative information about ash concentration or any indication of uncertainty. However, the guidelines for the production of these advisories are periodically under review and the Roadmap for International Airways Volcano Watch in Support of International Air Navigation states that from 2025 not only will quantitative ash forecasts need to be provided but also uncertainty information, potentially through the use of an ensemble (Meteorology Panel 2019).
Prata et al. (2019) present a methodology that uses a risk matrix approach to condense the multiple streams of data from a VATDM ensemble into a graphic that can be used to make fast and robust decisions in an emergency response situation. The approach identifies the geographical regions that are considered potentially hazardous to aircraft based on the probability of exceeding low, medium, and high concentrations as defined in Civil Aviation Authority (2017). This approach considers potential impact of encountering high ash concentrations but does not incorporate any losses that would be incurred if wrong action was taken. Examples of losses include the costs associated for scheduling a flight and then encountering a region of high ash concentration and reputational damage caused by cancelling flights along route that a competitor operates.
Case study description.
Raikoke is a small volcanic island located at 48.2°N, 153.3°E in the northwest Pacific Ocean. The eruption that is the focus of this study started at approximately 1800 UTC 21 June 2019, when a series of nine explosive events occurred until approximately 0600 UTC 22 June 2019. The initial height of the eruption plume was estimated to be 10–14 km above mean sea level (Global Volcanism Program 2019a). Both sulfur dioxide and ash were dispersed throughout the troposphere and lower stratosphere, including being caught up in a nearby cyclone. Over 40 airplanes were diverted following the eruption (Global Volcanism Program 2019b).
Ensemble VATDM simulations.
The VATDM simulations analyzed in this study were performed using the Numerical Atmospheric-Dispersion Modelling Environment (NAME; Jones et al. 2007) driven by weather forecast data from the Met Office global weather forecast ensemble (MOGREPS-G; Bowler et al. 2008). This VATDM has been developed at the Met Office and is used by the London VAAC for producing the ICAO defined ash advisories and graphics following an eruption in the North Atlantic.
A 1,000-member ensemble of NAME simulations was produced by perturbing nine parameters, similar to those perturbed in Prata et al. (2019), within plausible ranges for this eruption informed by Harvey et al. (2018). The parameters include the height of the eruption plume, ash density, distal fine ash fraction (the fraction of ash is available for long-range transport), duration of the eruption, parameters within the NAME turbulence parameterization, and driving meteorology. The parameter values are selected using Latin Hypercube sampling. In this study the start time of the eruption is not perturbed. The simulations output ash concentration (g m−3) every 6 h on a global grid with a resolution of 0.45° × 0.3° (approximately 40 km in the midlatitudes). Full details of the ensemble creation can be found in Capponi et al. (2022).
Before applying the decision theoretical approach to the ensemble of VATDM simulations, it is useful to understand the evolution of the ash plume following the eruption. Initially, the plume travels zonally to the east (Fig. 1a) before being transported around a cyclone (Fig. 1b). There is a small branch that travels toward the west. This is ash that is dispersed near the surface. The main part of the plume wraps around the cyclone transporting ash back toward the west and into Russia. By 0000 UTC 24 July, the plume extends across large parts of the North Pacific, almost reaching Alaska (Fig. 1c). As the plume is transported away from the site of the eruption the mean column loading values reduce from peak values of approximately 200 g m−2 along the plume axis at 0000 UTC 22 June to 2.2 g m−2 by 0000 UTC 24 June as the plume dispersed. Although this quantity shows the mean evolution of the plume and is broadly consistent with satellite retrievals of ash column loading (not shown), it does not show any information about the range of column loading values predicted by the ensemble or any information about the height of the ash and so is not suitable for making an informed decision about airline operations.
Ensemble-mean ash column loading at 0000 UTC (a) 22, (b) 23, and (c) 24 Jun 2019. The red triangle indicates the location of Raikoke.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
Another way to view the ensemble output is to calculate the fraction of ensemble members that agree on a specific threshold being exceeded in each NAME grid box. Figure 2 shows the evolution of the fraction of ensemble members that have ash column loading values greater than 0.2 g m−2. This is qualitatively similar to the evolution of ash column loading. As with the column loading, the agreement values are the highest along the axis of the plume, with peak agreement values dropping from 0.993 to 0.77 by 0000 UTC 24 June. This graphic illustrates the variability within the ensemble, but judgments would need to be made regarding the level agreement and column loading values that are needed to impact flight planning decisions. Thus, decisions made using such graphics can be subject to the specific risk sensitivity of the individual decision-maker.
Fraction of ensemble members that agree on the presence of ash greater than 0.2 g m−2 at 0000 UTC (a) 22, (b) 23, and (c) 24 Jun 2019. The red triangle indicates the location of Raikoke.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
A further way to view the ensemble information is to determine the fraction of ensemble members that have peak ash concentrations within the thresholds in the Civil Aviation Authority (2017) guidelines at each of the flight levels (FLs; measured in hundreds of feet) that are required for the VAAC advisories and graphics. Currently the concentration thresholds are: 200–2000 μg m–3 (Low), 2000–4000 μg m−3 (Medium), and >4000 μg m−3 (High) (Civil Aviation Authority 2017). An example of this is shown in Fig. 3, which shows these fractions for FL350–550 at 0000 UTC 24 June. For the majority of the plume, the highest fractions are less than the 200 μg m−3 threshold, but at 52°N, 175°W the highest fraction (approximately 0.35) is greater than 4000 μg m−3. This suggests that this region should potentially be avoided by aircraft. Analyzing these figures takes time and experience. Furthermore, in an emergency response situation there would also be figures for FL0–200 and FL200–350 for several different times. This could lead to information overload in an emergency response situation and hinder decision-making.
The fraction of ensemble members that have FL350–550 peak ash concentrations (a) <200, (b) 200–2000, (c) 2000–4000, and (d) >4000 μg m−3 at 0000 UTC 24 Jun 2019. The red triangle indicates the location of Raikoke. The gray line indicates 180° longitude. The black cross indicates the location of the example grid point used in Fig. 4.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
Application of the decision theoretical approach.
In this section, decision theory is applied to the ensemble of volcanic ash simulations of Raikoke. Here we assume the prior probability of all ash states, P(x), are equally likely and this term is proportional to a value of one in our determination of
To illustrate the impact of decision-makers with different operational loss models, three different hypothetical loss tables are used. L0(a, x) (Table 2) defines the loss table when there is no information about losses. In this situation, all of the off-diagonal losses are uniform, i.e., L0(am, xn) = 1, where m ≠ n, and the optimal action
L0(a, x): uniform loss table with no information about losses. L1(a, x): symmetrical loss table used to represent decision-maker with equal costs associated with flight cancellation/rerouting and engine maintenance. L2(a, x): skewed loss table used to represent a decision-maker with high costs associated with engine maintenance compared to flight cancellation or rerouting.
Figure 4 illustrates the calculation of the optimal action for a single grid point at FL350–550 for three different loss tables. The grid point is in the North Pacific at 50.30°N, 185.4°E and is shown in Fig. 3 as a black cross. The most likely ash concentration at this location is less than 200 μg m−3 (Fig. 4a), with 48% of the members having ash concentrations below this threshold. Figure 4b shows the likelihoods from Fig. 4a multiplied by loss values constructed to have uniform losses for all situations apart from where the optimal action matches the state [i.e., L0(am, xn) = 1, where m ≠ n]. These cells (on the diagonal of the table) are set to zero. Also shown in the last column is the sum of the losses for each action. The optimal action is the action that results in the minimum loss. As expected, the optimal action given a uniform loss table is the same as using the most likely concentration range. In this case the optimal action is no flight rerouting (no action, highlighted in gray). Figure 4c shows the same variables as Fig. 4b but for loss table L1(a, x). In this case, the optimal action is to load more fuel for potential flight rerouting (Low action, highlighted in yellow), despite there being only 14.2% of the ensemble members that have ash concentrations in this range. Figure 4d shows the same variables again as Fig. 4b but for loss table L2(a, x). In this case, the optimal action is cancelling the flight (High action, highlighted in red), despite there being only 32% of the ensemble members having ash concentrations in this range.
(a) The fraction of ensemble members that have concentration values within each of the concentration states (None, Low, Medium, High) for a grid point at 50.30°N, 185.4°E. Tables show the determination of the optimal hazard action
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
The impact of using the decision theory framework on the whole ash plume can be seen in Fig. 5. At 0000 UTC 24 June 2019, using the uniform loss table to inform optimal hazard action (Figs. 5a–d), the majority of the hazard comes from FL0–200 with a band of the highest hazard action extending south east from Raikoke before wrapping round the cyclone. There is no hazard action identified at FL350–550. The total combined hazard action area (maximum hazard action over all flight levels) is 1.71 × 106 km2 with 38% of it at the highest hazard action level.
The spatial distribution of optimal hazard action based on the uniform loss approach L0(a, x) for (a) FL0–200, (b) FL200–350, (c) FL350–500, and (d) maximum risk level at each point at 0000 UTC 24 Jun 2019. (e)–(h) The optimal hazard action determined using the symmetrical loss approach L1(a, x). (i)–(l) Optimal hazard action determined using the skewed loss approach with high maintenance costs compared to cancellation and rerouting L2(a, x). Yellow shading indicates regions of Low hazard action, orange shading indicates Medium hazard action, and red regions indicate the High level of hazard action. The black triangle indicates the location of Raikoke. The gray line indicates 180° longitude. The black cross indicates the location of the example grid point used in Fig. 4.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
Using the symmetrical loss table L1(a, x) with large costs associated with both rerouting and engine maintenance, the pattern of hazard action identified (Figs. 5e–h) is similar to that using the uniform loss approach, with the addition of a region of Low hazard action at FL350–550 and with the region of High hazard action greatly reduced compared to the uniform loss approach. The total combined hazard action area (where the action has been taken to classify the hazard action above the level of “None”) is 2.85 × 106 km2 with only 1% at the highest hazard action level. This is a reduction in the area of the highest hazard action by a factor of approximately 25. Using this loss table also instructs the action to classify areas of Medium hazard action, which are not identified using the uniform loss approach. Thus, using the symmetrical loss table would likely result in fewer cancelled flights (High hazard action decision routes) but more rerouted flights (Medium hazard action decision routes) and more flights requiring additional engine checks on arrival (Low hazard action decision routes) compared to the uniform loss table.
When using the skewed loss table L2(a, x) that has high engine maintenance costs compared to the cost of cancellation/rerouting, the overall pattern of optimal decisions (Figs. 5i–l) is similar to the symmetrical loss table. However, there are areas where the optimal action is high, i.e., cancellation of flights at all three FLs considered here. The area identified requiring an action greater than none is much more extensive than the uniform loss approach, with the total combined action area—of Low, Medium, or High hazard—of 5.60 × 106 km2 with 43% at the highest action level. This is an increase of the High hazard action area of a factor of 3.65 compared to the uniform loss approach, and a factor of 90 compared to using the symmetric hazard approach. Thus, using L2(a, x) has the potential to have a much greater impact on aviation operations than when using L1(a, x) or the uniform loss approach.
Figure 6 shows the impact of using the decision theory approach on flight tracks across the Pacific Basin at 0000 UTC 24 June 2019. Representative eastbound and westbound time–optimal routes from Sapporo (CTS) to Honolulu (HNL) and San Francisco (SFO) to Shanghai (PVG) international airports were calculated by solving a time–optimal control problem as described in Wells et al. (2021). Using the uniform loss approach L0(a, x) (Fig. 6a) shows that although there is a large area of the highest hazard action, only a small fraction of the flight track between PVG and SFO is impacted by this. There are regions of the tracks between SFO and PVG and CTS and HNL that encounter regions of the lowest hazard action. The route between CTS and HNL is not impacted by the ash at this time.
The optimal hazard action projected onto representative flight tracks between San Francisco (SFO) and Shanghai (PVG) and between Honolulu (HNL) and Sapporo (CTS) International airports at 0000 UTC 24 Jun 2019 for (a) the uniform loss approach using L0(a, x), (b) the symmetrical decision theory approach using L1(a, x), and (c) the skewed decision theory approach using L2(a, x). Light gray shading indicates the flight track with action necessary. Yellow shading indicates regions of Low hazard action, orange shading indicates Medium hazard action, and red regions indicate the High level of hazard action. The black triangle indicates the location of Raikoke. The gray line indicates 180° longitude. The black airplane icons indicate the direction of travel between the international airports.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0258.1
Using the symmetrical risk table L1(a, x) shows a very similar impact on the flight tracks as when the uniform loss approach is used but with the High hazard action region replaced by Medium hazard action. However, when the skewed loss table L2(a, x) is applied there are numerous regions along the flight tracks which are impacted by the High optimal hazard action. This could potentially lead to severe disruption to aviation operations.
Clarkson et al. (2016) advocate that the assessment of the risk to an aircraft from volcanic ash should be performed using dosage (how much ash is encountered in total) over the whole flight trajectory rather than only avoiding regions of high ash concentration. In this case, the potential actions taken by the decision-maker could be in the form of a set of flight paths, where the state x is the dosage encountered along each flight path.
Conclusions
Decision theory provides a robust framework to identify regions of potential risk to aviation from volcanic ash using a large ensemble of VATDM simulations. This framework has been applied to the case study of the 2019 Raikoke eruption. It demonstrates the impact of moving from a most probable (uniform loss) approach to one where uncertainty is treated explicitly by associating a loss with each possible action. The construction of a representative loss table is nontrivial and may require intimate knowledge of the business model and costs of the individual decision-makers. However, as shown in the example, application of different loss tables can greatly impact the optimal action to be taken.
Using a decision theoretical approach operationally would allow the determination of risk to be shared between scientists, forecasters, and decision-makers, where the loss table can be constructed for, or even by, the end-user. The losses used in the table could be subjective or parameterized by a loss function. There is also the possibility to define a reasonable best and worst case scenario loss table to give a range of plausible scenarios. The collaboration between scientists and end-users would also help to build trust and understanding in the products that are produced, and therefore also contribute to robust and transparent decision-making. It is important to note that the use of a decision theoretical framework does not remove the need to perform an appropriately designed set of ensemble simulations. The construction of ensemble simulations for operational use is not the focus of this study, but it is crucial to ensure that each member sampled is representative of the real world.
The current approach is limited as the “best” decision is determined on a pixel-by-pixel basis, whereas real-world natural hazards tend to have a level of spatial coherence. This could be extended by using multivariate spatial analysis similar to those used in satellite detection of ash clouds (e.g., Pavolonis et al. 2015). Further extensions could focus on the formulation of more complex, parameterized loss functions (e.g., Economou et al. 2016). We are unaware of any operational applications of decision theory within meteorological decision-making but the approach used in this study could easily be applied to any decision-making process using probabilistic information.
Acknowledgments.
We thank Sarah-Ellen Calise at the University of Reading for useful discussions about this work during the production of her MSc dissertation. We also thank Cathie Wells for the providing the time optimal flight tracks used in Fig. 6. Natalie Harvey, Helen Dacre, and Antonio Capponi gratefully acknowledge funding from NERC Grant NE/S005218/1.
Data availability statement.
The NAME ensemble simulation outputs are available in the Lancaster University Research Directory at
References
Bowler, N. E., A. Arribas, K. R. Mylne, K. B. Robertson, and S. E. Beare, 2008: The mogreps short-range ensemble prediction system. Quart. J. Roy. Meteor. Soc., 134, 703–722, https://doi.org/10.1002/qj.234.
Capponi, A., N. J. Harvey, H. F. Dacre, K. Beven, C. Saint, C. Wells, and M. R. James, 2022: Refining an ensemble of volcanic ash forecasts using satellite retrievals: Raikoke 2019. Atmos. Chem. Phys., 22, 6115–6134, https://doi.org/10.5194/acp-22-6115-2022.
Casadevall, T. J., 1994: The 1989–1990 eruption of redoubt volcano, Alaska: Impacts on aircraft operations. J. Volcanol. Geotherm. Res., 62, 301–316, https://doi.org/10.1016/0377-0273(94)90038-8.
Civil Aviation Authority, 2017: Guidance regarding flight operations in the vicinity of volcanic ash. CAP Doc. 1236, Civil Aviation Authority, 31 pp., https://publicapps.caa.co.uk/docs/33/CAP%201236%20FEB17.pdf.
Clarkson, R. J., E. J. Majewicz, and P. Mack, 2016: A re-evaluation of the 2010 quantitative understanding of the effects volcanic ash has on gas turbine engines. Proc. Inst. Mech. Eng., 230G, 2274–2291, https://doi.org/10.1177/0954410015623372.
Cox, J., D. House, and M. Lindell, 2013: Visualizing uncertainty in predicted hurricane tracks. Int. J. Uncertainty Quantif., 3, 143–156, https://doi.org/10.1615/Int.J.UncertaintyQuantification.2012003966.
Demeritt, D., H. Cloke, F. Pappenberger, J. Thielen, J. Bartholmes, and M.-H. Ramos, 2007: Ensemble predictions and perceptions of risk, uncertainty, and error in flood forecasting. Environ. Hazards, 7, 115–127, https://doi.org/10.1016/j.envhaz.2007.05.001.
Demeritt, D., S. Nobert, H. Cloke, and F. Pappenberger, 2010: Challenges in communicating and using ensembles in operational flood forecasting. Meteor. Appl., 17, 209–222, https://doi.org/10.1002/met.194.
Demeritt, D., E. M. Stephens, L. Créton-Cazanave, C. Lutoff, I. Ruin, and S. Nobert, 2016: Communicating and using ensemble flood forecasts in flood incident management: lessons from social science. Handbook of Hydrometeorological Ensemble Forecasting, Q. Duan et al., Eds., Springer, 1–30, https://doi.org/10.1007/978-3-642-40457-3_44-1.
Economou, T., D. B. Stephenson, J. C. Rougier, R. A. Neal, and K. R. Mylne, 2016: On the use of Bayesian decision theory for issuing natural hazard warnings. Proc. Roy. Soc., 472A, 20160295, https://doi.org/10.1098/rspa.2016.0295.
Englich, B., T. Mussweiler, and F. Strack, 2006: Playing dice with criminal sentences: The influence of irrelevant anchors on experts’ judicial decision making. Pers. Soc. Psychol. Bull., 32, 188–200, https://doi.org/10.1177/0146167205282152.
Global Volcanism Program, 2019a: Report on Raikoke (Russia)—August 2019. Bulletin of the Global Volcanism Network, A. E. Crafford and E. Venzke, Eds., Smithsonian Institution, 44 pp., https://doi.org/10.5479/si.GVP.BGVN201908-290250.
——, 2019b: Report on Raikoke (Russia), 19 June-25 June 2019. Weekly Volcanic Activity Report, S. K. Sennert, Ed., Smithsonian Institution and U,S, Geological Survey, https://volcano.si.edu/showreport.cfm?doi=GVP.WVAR20190619-290250.
Guffanti, M., T. J. Casadevall, and K. E. Budding, 2010: Encounters of aircraft with volcanic ash clouds: A compilation of known incidents, 1953–2009. Data Series 545, U.S. Geological Survey, 12 pp., https://pubs.usgs.gov/ds/545/DS545.pdf.
Harvey, N. J., N. Huntley, H. F. Dacre, M. Goldstein, D. Thomson, and H. Webster, 2018: Multi-level emulation of a volcanic ash transport and dispersion model to quantify sensitivity to uncertain parameters. Nat. Hazards Earth Syst. Sci., 18, 41–63, https://doi.org/10.5194/nhess-18-41-2018.
Hogan Carr, R., B. Montz, K. Maxfield, S. Hoekstra, K. Semmens, and E. Goldman, 2016a: Effectively communicating risk and uncertainty to the public: Assessing the national weather service’s flood forecast and warning tools. Bull. Amer. Meteor. Soc., 97, 1649–1665, https://doi.org/10.1175/BAMS-D-14-00248.1.
Hogan Carr, R., B. Montz, K. Semmens, K. Maxfield, S. Hoekstra, and E. Goldman, 2016b: Motivating action under uncertain conditions: Enhancing emergency briefings during coastal storms. Wea. Climate Soc., 8, 421–434, https://doi.org/10.1175/WCAS-D-16-0028.1.
Hogan Carr, R., B. Montz, K. Semmens, K. Maxfield, S. Connolly, P. Ahnert, R. Shedd, and J. Elliott, 2018: Major risks, uncertain outcomes: Making ensemble forecasts work for multiple audiences. Wea. Forecasting, 33, 1359–1373, https://doi.org/10.1175/WAF-D-18-0018.1.
Hogan Carr, R., K. Semmens, B. Montz, and K. Maxfield K. Maxfield, 2021: Improving the use of hydrologic probabilistic and deterministic information in decision-making. Bull. Amer. Meteor. Soc., 102, E1878–E1896, https://doi.org/10.1175/BAMS-D-21-0019.1.
Iorgulescu, I., K. J. Beven, and A. Musy, 2005: Data-based modelling of runoff and chemical tracer concentrations in the Haute-Mentue research catchment (Switzerland). Hydrol. Processes, 19, 2557–2573, https://doi.org/10.1002/hyp.5731.
Jones, A., D. Thomson, M. Hort, and B. Devenish, 2007: The UK Met office’s next-generation atmospheric dispersion model, NAME III. Air Pollution Modeling and its Application XVII, C. Borrego and A.-L. Norman, Eds., Springer, 580–589, https://doi.org/10.1007/978-0-387-68854-1_62.
Lindley, D. V., 1971: Making Decisions. Wiley-Interscience, 195 pp.
Meteorology Panel, 2019: Roadmap for International Airways Volcano Watch (IAVW) in support of international air navigation. International Civil Aviation Organization, 17 pp., www.icao.int/airnavigation/METP/MOGVA%20Reference%20Documents/IAVW%20Roadmap.pdf.
Miran, S. M., C. Ling, and J. J. James, 2020: People’s thresholds of decision-making against a tornado threat using dynamic probabilistic hazard information. Int. J. Disaster Risk Reduct., 42, 101345, https://doi.org/10.1016/j.ijdrr.2019.101345.
Morss, R. E., J. K. Lazo, and J. L. Demuth, 2010: Examining the use of weather forecasts in decision scenarios: Results from a US survey with implications for uncertainty communication. Meteor. Appl., 17, 149–162, https://doi.org/10.1002/met.196.
Mulder, K. J., M. Lickiss, N. Harvey, A. Black, A. Charlton-Perez, H. Dacre, and R. McCloy, 2017: Visualizing volcanic ash forecasts: Scientist and stakeholder decisions using different graphical representations and conflicting forecasts. Wea. Climate Soc., 9, 333–348, https://doi.org/10.1175/WCAS-D-16-0062.1.
Nadav-Greenberg, L., S. L. Joslyn, and M. U. Taing, 2008: The effect of uncertainty visualizations on decision making in weather forecasting. J. Cogn. Eng. Decis. Making, 2, 24–47, https://doi.org/10.1518/155534308X284354.
Pavolonis, M. J., J. Sieglaff, and J. Cintineo, 2015: Spectrally enhanced cloud objects—A generalized framework for automated detection of volcanic ash and dust clouds using passive satellite measurements: 2. Cloud object analysis and global application. J. Geophys. Res. Atmos., 120, 7842–7870, https://doi.org/10.1002/2014JD022969.
Prata, A. T., H. F. Dacre, E. A. Irvine, E. Mathieu, K. P. Shine, and R. J. Clarkson, 2019: Calculating and communicating ensemble-based volcanic ash dosage and concentration risk for aviation. Meteor. Appl., 26, 253–266, https://doi.org/10.1002/met.1759.
Riveiro, M., T. Helldin, G. Falkman, and M. Lebram, 2014: Effects of visualizing uncertainty on decision-making in a target identification scenario. Comput. Graph., 41, 84–98, https://doi.org/10.1016/j.cag.2014.02.006.
Taylor, H. A., C. E. Renshaw, and M. D. Jensen, 1997: Effects of computer-based role-playing on decision making skills. J. Educ. Comput. Res., 17, 147–164, https://doi.org/10.2190/Y1CX-XU4G-KDFK-JBFJ.
Wells, C. A., P. D. Williams, N. K. Nichols, D. Kalise, and I. Poll, 2021: Reducing transatlantic flight emissions by fuel-optimised routing. Environ. Res. Lett., 16, 025002, https://doi.org/10.1088/1748-9326/abce82.
Wernstedt, K., P. S. Roberts, J. Arvai, and K. Redmond, 2019: How emergency managers (mis?)interpret forecasts. Disasters, 43, 88–109, https://doi.org/10.1111/disa.12293.
Western, L. M., J. Rougier, and I. M. Watson, 2018: Decision theory-based detection of atmospheric natural hazards from satellite imagery using the example of volcanic ash. Quart. J. Roy. Meteor. Soc., 144, 581–587, https://doi.org/10.1002/qj.3230.
Whyte, G., and J. K. Sebenius, 1997: The effect of multiple anchors on anchoring in individual and group judgment. Organ. Behav. Hum. Decis. Process., 69, 75–85, https://doi.org/10.1006/obhd.1996.2674.
World Meteorological Organization, 2008: Guidelines on communicating forecast uncertainty. WMO/TD-1422, 25 pp., https://library.wmo.int/doc_num.php?explnum_id=4687.
