1. Introduction
The energy sector has long been a key user of weather and climate information across time scales: short-range weather forecasts (e.g., Taylor and Buizza 2003; Costa et al. 2008), longer-range forecasts out to several weeks ahead (e.g., Dubus 2014), and projections of possible future climates decades ahead (e.g., McColl et al. 2012; Wang et al. 2014). These forecasts are all used to inform the planning, development, management, and running of energy systems on those time scales. The energy sector has also been a leader in demonstrating demand for seasonal-to-decadal climate prediction (Buontempo et al. 2010; Bruno Soares and Dessai 2015). In particular, seasonal forecasts of the climate in the coming 3-month period have the potential for providing real added value, in both the practical sense and the financial sense, across a range of areas within the energy sector (Troccoli 2010; Doblas-Reyes et al. 2013; Dessai and Bruno Soares 2015; Bruno Soares and Dessai 2015). Lack of skill in many areas of the globe has limited the uptake of this kind of information within the sector, however.
Energy demand is strongly related to air temperature (e.g., Valor et al. 2001; Hor et al. 2005; Apadula et al. 2012; Zhang et al. 2014; Thornton et al. 2016), and the potential use of seasonal climate forecasting in demand management has been recognized for many decades (e.g., Brown Weiss 1982 and references therein). The need to reduce both greenhouse gas emissions and air pollution has driven an increase in the amount of electricity supplied by renewable sources. This has, in turn, resulted in an increase in the weather dependence of energy supply systems and therefore an increase in the possible utility of weather and climate forecasting to the sector.
The energy sector in China faces more severe issues than do the energy sectors of other countries. China is the coldest country during winter, and the hottest during summer, among countries located at the same latitudes (excluding desert regions), and therefore the demands for heating in winter and cooling in summer are high. Under the background of global climate change, extreme climate events have changed significantly in China over the past 60 years. The increase in the number of high-temperature and drought days (Qin et al. 2015) has exacerbated the energy gap for cooling and irrigation when such extremes occur. Other particular features are 1) the increases in demand that are due to rapid urbanization (Wang 2014; Lin and Ouyang 2014) and 2) the recent large growth in both installed and planned renewable energy capacity (e.g., Hong et al. 2013; CNREC 2014; Lo 2014; Qiang et al. 2016). Any shift in the future from coal-driven combined heat and power facilities to more electricity-based building heating would also have significant impacts on the use of renewable energy sources (e.g., Zhang et al. 2016).
If skillful, seasonal forecasts would be very useful to the Chinese energy sector. For example, seasonal forecasts could give an early warning of a season of high demand such as a particularly cold winter or hot summer, or of reduced supply as a result of low wind speeds, or more cloudy/hazy periods that reduce solar-power generation. Predicting conditions that could damage energy infrastructure, such as storms, could also be valuable. In all of these cases, seasonal forecasts could allow improved mitigation plans to be put in place: rescheduling maintenance work, making early decisions around staff availability, and financial planning for the coming 3-month period. The information could be used by a range of people, including industry regulators, network operators, energy-production companies, maintenance contractors, and traders in the financial markets. In many cases, using a seasonal forecast might require changes in an organization’s decision-making process, however.
Seasonal forecasts are most useful if they have sufficient skill to allow decision-making. Furthermore, what “sufficient” means (beyond being statistically significant) will depend on the particular use case. If forecasts are not sufficiently skillful, then, although they might be happy to receive forecast information, organizations might not be able to make a decision that is based on that information.
Although skillful forecasts for some variables in some parts of the world have been possible for some time (e.g., Arribas et al. 2011), recent advances in seasonal forecasting systems have led to major improvements in the skill of extratropical features such as the North Atlantic Oscillation (NAO; e.g., Athanasiadis et al. 2014, 2016; Butler et al. 2016; Smith et al. 2016, and references therein). Scaife et al. (2014) demonstrated skill in NAO forecasts from version 5 of the Met Office’s Global Seasonal Forecasting System (GloSea5; MacLachlan et al. 2015). This has led to the development of seasonal climate services for various sectors in the United Kingdom, including hydrology (Svensson et al. 2015), transport (Palin et al. 2016), and energy (Clark et al. 2017). Seasonal forecasting has a long history in China, but the traditional low skill from dynamical models has led to a wide quantity of literature in statistical downscaling and statistical forecasting (e.g., Xiao et al. 2012; Wang et al. 2015; Xing et al. 2016; Golding et al. 2017, and references therein). Much of this literature has focused on forecasting the monsoon and associated precipitation (e.g., Ke et al. 2011; Ying and Ke 2012; Tung et al. 2013; Peng et al. 2014; Wu and Yu 2016). Development of wind speed predictions has focused on short-term (weather) forecasts, although there has been some work done on the longer-term monthly time scale (Guo et al. 2011). It is timely therefore to examine the skill of GloSea5 in China for direct seasonal forecasts of energy-relevant climate variables. The results could allow the development of future climate services (Golding et al. 2017), that is, the provision and use of climate information to enable better-informed decisions.
The development and use of such climate services has become a major undertaking worldwide, with international coordination being facilitated by the Global Framework for Climate Services (GFCS; Hewitt et al. 2012). The GFCS focuses on five priority sectors, one of which is the energy sector. China is developing its own framework aligned to the GFCS, called the China Framework for Climate Services (http://www.cma.gov.cn/en2014/20150311/2015323e/2015323e07/2015323e09/201503/t20150317_276866.html; Golding et al. 2017). It brings together actors involved in scientific research and climate-service development with service providers and users to ensure that available capability and services meet users’ needs.
In this paper we assess the skill of seasonal forecasts of wind speed, irradiance, and temperature across China from GloSea5 and consider the implications for the wind-power, solar-power, and energy-demand sectors. We first describe the datasets and methods used in section 2. We then present our results in section 3, considering a China-wide overview of each variable before focusing on some specific areas of interest. We discuss our conclusions in section 4.
2. Data and analysis methods
a. Datasets
In this paper we use the hindcast dataset produced to assess the version of GloSea5 that was deployed operationally at the Met Office in February of 2015. It is based on the second Global Coupled configuration (GC2) of the third Hadley Centre Global Environmental Model (HadGEM3), described in Williams et al. (2015). HadGEM3-GC2 uses the “GA6.0” Global Atmosphere configuration of the Met Office Unified Model (version 8.4) as its atmospheric component, on an “N216” grid1 (a horizontal resolution of 0.83° in longitude and 0.55° in latitude) and 85 vertical levels reaching a height of 85 km near the mesopause (Walters et al. 2017). This is coupled to the “GL6.0” Global Land configuration of the Joint U.K. Land Environment Simulator (JULES) land surface model (Best et al. 2011), the “GO5.0” Global Ocean configuration of the Nucleus for European Modelling of the Ocean (NEMO) model with a 0.25° nominal resolution and 75 vertical levels (version 3.4; Megann et al. 2014; Madec 2008), and the “GSI6.0” Global Sea Ice configuration of the Los Alamos Sea Ice Model (CICE, version 4.1; Rae et al. 2015; Hunke and Lipscomb 2010). GloSea5 is described in full in MacLachlan et al. (2015) and references therein.
The assessment hindcast was produced to examine the skill of the system in forecasting for winter (December–February: DJF) and summer (June–August: JJA) only. Lagged ensemble “forecasts” were produced by collating eight-member ensemble forecasts initialized on 25 October, 1 November, and 9 November for DJF and on 25 April, 1 May, and 9 May for JJA. This method yields a total of 24 members for each hindcast season. The DJF hindcasts cover from (boreal) winter 1992/93 to winter 2011/12, and the JJA hindcasts cover (boreal) summers 1992–2011. The details of the initialization are described in MacLachlan et al. (2015).
Our ability to robustly assess forecast skill is limited by the size of the hindcast, both in terms of the number of years and the number of members. The operational configuration (the forecast version) of GloSea5 uses 42 members, rather than the 24 available in the hindcast used here, and therefore the probability distributions inferred by using the hindcast will be less well resolved than they would be operationally. Furthermore, it has been shown that the skill itself depends directly on the size of the ensemble, because it allows better identification of predictable signals (Scaife et al. 2014; Eade et al. 2014; Dunstone et al. 2016; Li et al. 2016; Lu et al. 2017), although the size of this effect in GloSea5 is less strong in China than in the extratropical North Atlantic Ocean. For our purposes, the ensemble size means that we can regard the levels of skill that are shown here, where significant, to be lower limits on the actual skill that could be realized in the operational system. The robustness of the skill estimates is also limited by the number of years in the hindcast (Kumar 2009), because this places a restriction on the number of different types of event that are sampled in the period of study. The impact of the limited hindcast period is quantified by assessing the statistical significance of the correlations between the hindcast and observations, as described in the next section.
We use the ERA-Interim reanalysis data (ERAI; Dee et al. 2011) as a proxy for observations in this paper. ERAI and other reanalyses are frequently used to validate temperature and wind data from climate models, but their use as a proxy for irradiance observations is more contested (e.g., Boilley and Wald 2015). We compared some of our irradiance results with the satellite-derived observational Surface Solar Radiation Data Set—Heliosat based on Meteosat-East (SARAH-E; Huld et al. 2016; Amillo et al. 2014) and found that, for the seasonal means averaged over the large areas that we consider here and in the standardized units that we use, ERAI compares very well. The use of climatological aerosols in both ERAI and SARAH-E, and indeed in GloSea5, means that the impact of aerosols on interannual variability remains an important uncertainty.
b. Skill-assessment method
In this paper we focus on three meteorological variables of interest to the energy sector in China: near-surface air temperature (related to energy demand), 10-m wind speed (linked2 to wind-power generation), and downwelling shortwave irradiance at the surface (related to solar-power generation). Precipitation is also of great importance for the energy sector, because China has a very large, and growing, hydroelectric industry. Li et al. (2016) have already shown that GloSea5 has significant skill in forecasting summer precipitation in the Yangtze River basin, where the Three Gorges Dam is located, and this has led to the development of a trial forecast service (Golding et al. 2017). Further work by Lu et al. (2017) found high levels of skill in GloSea5 forecasts of winter precipitation over southeastern China.
We take the following approach to skill assessment. For each variable, we first map the Pearson correlation between the hindcast ensemble mean and the observations. Although the limited time span and ensemble size of the hindcast means that forecasts from single grid cells (or even small regions) are likely to be very noisy and not robust, these maps give a good indicative overview and provide context when selecting larger geographical areas of interest for subsequent analysis. These regions are selected on the basis of their interest to the energy sector: we consider the current and likely future development of substantial energy supply or demand and whether they appear to have some promising skill (regions are not selected on the basis of skill maps alone).
We then assess the skill in each region in more detail. We consider three types of plots, each with an associated skill score:
standardized3 time series (using the hindcast ensemble mean), with the Pearson correlation r [we also estimate the uncertainty in r (95% confidence intervals) using the Fisher z transformation],
reliability and sharpness diagrams, with the Brier skill score (BSS), which show the joint distribution of hindcast probabilities and observed frequencies for a particular class of event (the BSS measures how much better the forecast system is relative to use of climatological means in that case) (see appendix A for further details), and
“relative operating characteristic” (ROC) diagrams, with the ROC skill score (ROCSS), which describe the ability of the forecast system to distinguish between events occurring or not occurring (see appendix B for further details).
It is useful to have an indication on the correlation maps of where the skill might be significantly nonzero. We use the Fisher z transformation4 to draw contours around areas that would be significantly nonzero at the 5% level.5 Note that we are not correcting for multiple testing here, and therefore it should be expected that some of the regions marked as notionally significant on maps will be false positives; the significance contour should not be taken as definitive.
There are some cases in which there is a clear trend running through the data. The reproduction of such a trend by the hindcast is a genuinely useful aspect of skill, because it shows that the forecast system is capable of maintaining the impact of whatever forcing caused the trend, after being initialized. It can, however, hide information about the ability of the model to evolve correctly away from its initialized state more generally, which is also of interest when assessing the model. We have therefore also looked at the correlation skill after detrending, which we perform by simply removing the linear least squares regression fits to the hindcast ensemble mean and observational time series, separately. Note that we are not making any assumptions as to the cause or significance of any trends. The time series is sufficiently short that natural interannual and decadal-scale variability will be very important, even before considering anthropogenic drivers of climate change such as carbon dioxide emissions, land-use change (affecting effective surface roughness and hence wind speed), and aerosol emissions6 (affecting surface irradiance and temperature). We simply remove the empirical linear trend.
Our reliability and ROC diagrams are made in terms of probabilistic forecasts of particular types of event: we consider the probability of the variable in question being above the median; in the upper, middle, or lower tercile; or in the top or bottom quintile of its historical distribution. These quantiles are calculated for the hindcast and observational datasets independently, from their own climatological characteristics (defined by the period covered by the hindcast). This means that our reliability diagrams and Brier skill scores are insensitive to a simple bias in the mean state between the two datasets.
For each type of event (e.g., upper tercile), the distribution of ensemble members each year7 provides the forecast probability of that event occurring, in each grid cell. Using bins of probability with width 0.1, we then consider all of the years in which the event was forecast to occur with probability in a given bin and count the frequency of times that it was observed to actually occur. The counting is done in each grid cell, and we then pool the counts from all of the grid cells in the region using cosine-of-latitude-weighted sums to calculate the skill scores and reliability/ROC plots.
Although the quantities we consider here are insensitive to a mean bias, there are possible cases in which absolute values can be important. Transfer functions (power curves) for relating meteorological quantities to electrical power output are usually threshold based (e.g., Bett and Thornton 2016 and references therein). It remains to be seen if such threshold-based transformations are necessary in a seasonal forecasting context, but in these cases the hindcast climatological values can be used for bias correction (Arribas et al. 2011; MacLachlan et al. 2015).
3. Results
Here, we show maps of the correlation between GloSea5 and ERAI for each variable. In each case, we then go on to examine particular regions in more detail through their regional time series, reliability, and ROC diagrams.
a. Wind speed
Maps of the correlation between ERAI and the GloSea5 hindcast for 10-m wind speeds are shown in Fig. 1 for winter and summer. Because there are no significant trends in the wind speeds over the hindcast period, maps of the correlation of detrended time series are practically indistinguishable (not shown).
Maps of the correlation of seasonal mean 10-m wind speed between the GloSea5-GC2 hindcast and ERAI for (top) winter and (bottom) summer. Green boxes highlight regions discussed in the text, and yellow contours mark correlations (|r| > 0.44) that are suggestive of significant skill.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
Although there are some areas of significant positive skill for wind speed in DJF, they are patchy. A major highlight, however, is the very high skill in the South China Sea, off the southern and southeastern coasts of China, with some skill being retained inland. This good performance is likely to be related to the skill in forecasting the El Niño–Southern Oscillation (r ≈ 0.9 for the DJF Niño-3.4 index;8 see MacLachlan et al. 2015) and its teleconnections over China; these are shown in Fig. 2 in terms of correlations between the Niño-3.4 index and wind speed. The overall response in the region differs in detail between GloSea5 and ERAI, but the significant anticorrelation in the South China Sea is present in both, with weaker wind speeds correlated with El Niño events. In this region, as part of the East Asian winter monsoon, there is strong northeasterly flow around the southeastern edge of the Siberian–Mongolian high pressure feature (Chang et al. 2006). During El Niño events, there is increased subsidence over the Maritime Continent region of Southeast Asia (Ramage 1968), increasing the surface pressure over that region and thus reducing the land–sea pressure gradient and resulting monsoonal winds (Zhang et al. 1996).
Maps of the correlation of the DJF Niño-3.4 index and 10-m wind speeds in (top) ERAI and (bottom) the ensemble members of the GloSea5-GC2 hindcast. Note that we do not use the ensemble mean here, because we are showing the internal correlation within the model: correlating the model’s ensemble means with each other would result in a stronger signal because it would remove natural noise that is still present in the single “realization” of ERAI.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
Figure 3 shows skill and reliability for winter wind speeds in this South China Sea region (the southeastern green box in Fig. 1). The deterministic ensemble-mean forecast has a correlation of r ≈ 0.8, and the reliability diagrams show that this region also exhibits skillful, reliable, and sharp probabilistic forecasts of above-median wind speed events. Results are also very good for upper- and lower-tercile events and for the upper and lower quintiles, although these latter are more noisy. This result is reflected in the sharpness diagrams: high-probability forecasts of outer-quintile events in particular are very poorly sampled by the hindcast, whereas forecasts of above-median events are well sampled across the full range of probabilities. It is also important to note that this skill in wind speed is not retained in summer (as seen in the bottom panel of Fig. 1): in that case, the skill scores of r = 0.13, BSS = −0.07, and ROCSS = 0.004 are not significantly different from zero (not shown).
Plots to assess skill and reliability for winter wind speeds in the South China Sea region marked in Fig. 1. (top) Standardized time series, using the ensemble mean of the seasonal hindcast. Also shown are (middle) the reliability and sharpness diagrams (see appendix A for details) and (bottom) ROC plots (see appendix B) for the three event definitions: (left) above-median winds, (center) terciles, and (right) outer quintiles, as labeled. In each panel, the corresponding skill score is given in the legend. With the time series, an estimate of the 95% confidence interval on the correlation is shown in parentheses (the presence of an asterisk in this and subsequent figures indicates significance).
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
Some other regions also appear to have reasonably high levels of skill. Figure 4 shows the time series for winter wind speed in north-central China and southern Mongolia (the northern green box in Fig. 1). This is a region of particularly high wind resource (e.g., CNREC 2014; Davidson et al. 2016), and so being able to forecast it could be of great practical use. The skill in this particular region is marginal, however (r = 0.42; although the 95% confidence interval just crosses zero, we expect this to be an underestimate of the true forecast skill, as discussed in section 2). Research is ongoing to understand the origin of this skill, but current results suggest that it is related to the model’s skill in predicting the upper-level Middle Eastern jet stream and, to some extent, the Arctic Oscillation. It has previously been demonstrated that these features are connected to winter climate over China (e.g., Zuo et al. 2015; Yang et al. 2004; He et al. 2017), and research is under way to determine the physical mechanism.
Standardized time series for winter wind speeds in the northern-central China/southern Mongolia region (the northern box marked in Fig. 1). The correlation and its 95% confidence interval are marked in the legend.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
Yunnan Province, in southern China, shows modest but significant skill for wind speeds in both winter and summer (Fig. 5; the region is also marked in Fig. 1). Yunnan is very mountainous, and energy production has traditionally been dominated by hydroelectricity. In recent years there has been a substantial drive to utilize the available wind resource (Liang et al. 2015), however, and a seasonal forecast could prove useful.
Standardized time series for (top) winter and (bottom) summer wind speeds in the Yunnan region (the southwest box marked in Fig. 1). The correlation and its 95% confidence interval are marked in the legend.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
The wind forecast skill across most of China in summer is indistinguishable from zero. Wind industry developers should be aware of these limitations in skill and focus on using seasonal forecasts for winter decision-making, in skillful regions.
b. Irradiance
Figure 6 shows correlation maps for irradiance in winter and summer. In winter, there is a broad area of promising skill in eastern China and the East China Sea. This result bears a strong resemblance to the patterns of skill in winter precipitation shown by Lu et al. (2017), perhaps unsurprisingly, because both rainfall and irradiance are strongly related to cloudiness. Lu et al. (2017) determined that the key drivers of precipitation predictability here are 1) ENSO and 2) rainfall in the eastern Indian Ocean/Bay of Bengal. It is reasonable to assume that the same processes that affect winter precipitation in this region also affect cloudiness and therefore downwelling shortwave irradiance at the surface.
Maps of the correlation of seasonal mean irradiance between the GloSea5-GC2 hindcast and ERAI for (top) winter and (bottom) summer. As in Fig. 1, contours mark regions of notionally significant skill. The green box highlights the eastern/coastal region discussed in the main text.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
Figure 7 shows the winter irradiance skill in eastern China in more detail, using the region in the eastern green box marked in Fig. 6: probabilistic forecasts remain reasonably skillful and reliable for tercile and outer-quintile events. While this is not a region of particularly high solar radiation resource within China (CNREC 2014), it is an area of high population density and many urban centers, including Shanghai. The potential for high levels of demand modulated by large numbers of roof-mounted solar panels means that being able to forecast winters with more or less solar generation than usual could be of value.
Skill and reliability assessment of winter irradiance for the easternmost coastal region of China marked with a green box in Fig. 6. As in Fig. 3, we (top) show the standardized time series using the ensemble mean and also show (middle) reliability and sharpness diagrams and (bottom) ROC plots for (left) above-median, (center) tercile, and (right) outer-quintile events. Each panel shows its corresponding skill score, and further details on the reliability and ROC plots are given in appendixes A and B, respectively.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
In summer, the correlation overall across China is much poorer, with the region that we considered above in eastern China now having r = 0.03, which is consistent with zero (not shown). Regions farther west appear to have higher levels of skill, but the skill is still very patchy.
As already discussed, because both ERAI and GloSea5 use climatological aerosols, we are unable to assess the impact of interannual aerosol variability on seasonal irradiance forecasts. This might have a strong impact in urban areas that are affected by haze, for example, and it remains an important uncertainty when considering the application of these results to solar-power generation.
c. Temperature
Figure 8 shows correlation maps for temperature, including a comparison with detrended data. It is clear that there is significant skill in predicting summer temperatures over large areas of China but that in many regions these areas of skill are due in large part to the model reproducing the observed positive trends over the hindcast period; in such cases, using a climate model for seasonal forecasting may not be necessary. In other areas, the skill persists after the trend is removed. The winter temperatures are less affected by trends and, indeed, show very little skill overall. As with winter wind speed, research is under way to improve the forecasts through the use of statistical models that are based on larger-scale atmospheric drivers.
Maps of the correlation of seasonal mean air temperature between the GloSea5-GC2 hindcast and ERAI, for (left) winter and (right) summer for the (top) standard and (bottom) detrended data. As in Fig. 1, contours mark regions of notionally significant skill. The northern (Beijing area) and southern (Yunnan province) green boxes are discussed in the text.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
One exception is Yunnan Province in south-central China, which we also highlighted for wind speed skill: here the province shows positive skill for both winter and summer temperatures, before and after detrending. Yunnan is less urban than many more-eastern parts of China, and therefore the utility of a temperature forecast in energy-demand planning is more limited. Agriculture and tourism are both very important for Yunnan, which could thus still benefit from a skillful seasonal temperature forecast.
It is clear that the potential for useful seasonal forecasts of energy demand and hence temperature is greatest in urban centers. In particular, energy demand in Beijing is strongly related to temperature in summer (Zhang et al. 2014). It is important therefore that the region around Beijing (the northern green box in Fig. 8) also shows some skill for summer temperatures, before and after detrending, with correlations of ~0.5–0.6 (Fig. 9). There is less skill, however, for probabilistic forecasts that are more detailed than the simple “above average” case (note that the trend seen in the time series will also make a positive contribution to the reliability), although this skill might be improved by looking at a larger region and thus reducing statistical noise.
Skill and reliability assessment for summer temperatures in Beijing (as marked with a box in Fig. 8), as in Figs. 3 and 7. We include the time series of both the (top left) standard and (top right) detrended data; the other results are produced without detrending.
Citation: Journal of Applied Meteorology and Climatology 56, 11; 10.1175/JAMC-D-17-0070.1
4. Discussion and conclusions
Our results have shown that, although overall skill for energy-relevant variables in China remains patchy, some specific areas have significant skill: winter wind speeds in the South China Sea (reflecting a known relationship with ENSO), winter solar irradiance in eastern/southern China, and summer temperatures across much of China (due to the trend), including Beijing (even when detrended). Taken together with similarly promising results for skillful summer precipitation forecasts in the Yangtze River basin (Li et al. 2016) and winter precipitation forecasts in southeastern China (Lu et al. 2017), there are clear opportunities to develop useful seasonal climate services for specific cases within China. Indeed, taking our results and those on precipitation skill together, there are clear potential climate-service applications beyond the energy sector: for example, forecasting risks to agriculture and transport and risks of flooding.
It is important to note that, outside the key regions highlighted here, the skill mostly remains indistinguishable from zero. This can be widespread in particular cases, such as for wind speeds and irradiance in summer or temperatures in winter.
We have only considered the skill of the direct model output from the GloSea5 hindcast here. This represents a minimum level of forecast skill, in two senses. First, the operational forecast ensemble is larger than that available in this hindcast, and it is well established that the forecast skill in GloSea5 increases with the size of the ensemble (e.g., Scaife et al. 2014; Li et al. 2016; Dunstone et al. 2016; Lu et al. 2017).
Second, statistical models linking larger-scale drivers directly to the impact variable of interest may offer further improvement in predictability (e.g., Scaife 2016, chapter 9). This technique has been used for seasonal forecasts in the United Kingdom (e.g., Svensson et al. 2015; Palin et al. 2016; Clark et al. 2017) and is often used already in China (e.g., Xiao et al. 2012; Wang et al. 2013; Peng et al. 2014; Wang et al. 2015; Xing et al. 2016). Research is ongoing to understand the predictability of larger-scale drivers in GloSea5 and how they can be used to improve sector-specific forecasts.
Furthermore, the most user-relevant services are likely to be forecasts of the particular impact of interest to the user, such as energy supply or demand. A next step in developing seasonal climate services on the basis of these results should therefore be to assess the skill of GloSea5 against such direct impacts data, where available from a potential user. The way that forecasts are communicated and handled also affects the usefulness of the forecast (e.g., Taylor et al. 2015; Davis et al. 2016): user engagement is therefore key to optimizing a climate-prediction service. Nevertheless, if codeveloped with users and communicated carefully, our results show some areas of very promising skill, allowing the development of improved, skillful seasonal climate forecasting services for specific parts of the energy sector and other sectors in China.
Acknowledgments
This work and its contributors (PB, JL, AS, CH and NG) were supported by the U.K.–China Research and Innovation Partnership Fund through the Met Office Climate Science for Service Partnership (CSSP) China as part of the Newton Fund. Author CL was supported by the National Natural Science Foundation of China (Grant 41320104007). Author PB thanks Hongli Ren, Jo Camp, Robin Clark, and Margaret Gordon for helpful discussions.
APPENDIX A
Reliability and Sharpness Diagrams and Brier Skill Scores
Model reliability is a description of how closely the forecast probabilities of an event correspond to the frequency of that event being observed in historical data, assessing a conditional bias in the forecast system: for example, we might find that every time the event is forecast to occur with 70% probability it actually occurs only 60% of the time. Once such discrepancies have been characterized, they can then be removed through calibration, resulting in improved forecasts. A full description can be found in Wilks (2011, chapter 8) but we describe the key points for interpreting our plots here.
The reliability diagram for a given class of event is a plot of the observed frequency of the event at times when it was forecast to occur with a given probability. As described in section 2b, we use bins of probability of width 0.1 and pool the event counts from all grid cells in the chosen region of interest. For the set of years in which the event was forecast to occur with a given probability, we plot on the vertical axis the fraction of those years in which the event actually occurred, and then we join the points from each bin with a line.
We mark additional lines in our reliability diagrams (sometimes called an attributes diagram; Hsu and Murphy 1986). The black, solid 1:1 line marks “perfect reliability,” differentiating between underconfident and overconfident forecasts—these will have steeper or shallower reliability lines, respectively, than the “perfect” case. The climatological frequency (e.g., ⅓ for terciles) is marked with a dotted horizontal “no resolution” line: if the reliability line coincides with this line, then it cannot resolve different events into different probabilities, because all forecasts occur at the climatological rate. A dashed line midway between perfect reliability and the no-resolution line is called the “no skill” line, because only points above this line make a positive contribution to the BSS. We shade this region of skill in green.
Below each reliability diagram, we include a sharpness diagram, which is a histogram of the distribution of forecasts made in each probability bin. If the histogram is flat, then the hindcast has sampled the full range of possible forecast probabilities and is described as sharp. If it is strongly peaked at the climatological frequency for the event, then the system has no sharpness and mostly just predicts the climatological frequency (climatology). Taken together, the sharpness and reliability diagrams provide a complete description of the joint distribution of observed frequencies and forecast probabilities.
APPENDIX B
ROC Diagrams and ROC Skill Scores
the number of hits NH(p)—that is, the number of times that the event was forecast with probability > p and was observed to occur,
the number of misses NM(p)—that is, the number of times that the event was observed but was not forecast with probability > p,
the number of false alarms NFA(p)—that is, the number of times that the event was forecast with probability > p but was not observed to occur, and
the number of correct rejections NCR(p)—that is, the number of times that the event was not observed to occur and was not forecast with probability > p.
REFERENCES
Amillo, A., T. Huld, and R. Müller, 2014: A new database of global and direct solar radiation using the eastern Meteosat satellite, models and validation. Remote Sens., 6, 8165–8189, doi:10.3390/rs6098165.
Apadula, F., A. Bassini, A. Elli, and S. Scapin, 2012: Relationships between meteorological variables and monthly electricity demand. Appl. Energy, 98, 346–356, doi:10.1016/j.apenergy.2012.03.053.
Arribas, A., and Coauthors, 2011: The GloSea4 ensemble prediction system for seasonal forecasting. Mon. Wea. Rev., 139, 1891–1910, doi:10.1175/2010MWR3615.1.
Athanasiadis, P. J., and Coauthors, 2014: The representation of atmospheric blocking and the associated low-frequency variability in two seasonal prediction systems. J. Climate, 27, 9082–9100, doi:10.1175/JCLI-D-14-00291.1.
Athanasiadis, P. J., and Coauthors, 2016: A multisystem view of wintertime NAO seasonal predictions. J. Climate, 30, 1461–1475, doi:10.1175/JCLI-D-16-0153.1.
Best, M. J., and Coauthors, 2011: The Joint UK Land Environment Simulator (JULES), model description—Part 1: Energy and water fluxes. Geosci. Model Dev., 4, 677–699, doi:10.5194/gmd-4-677-2011.
Bett, P. E., and H. E. Thornton, 2016: The climatological relationships between wind and solar energy supply in Britain. Renewable Energy, 87, 96–110, doi:10.1016/j.renene.2015.10.006.
Boilley, A., and L. Wald, 2015: Comparison between meteorological re-analyses from ERA-Interim and MERRA and measurements of daily solar irradiation at surface. Renewable Energy, 75, 135–143, doi:10.1016/j.renene.2014.09.042.
Brown Weiss, E., 1982: The value of seasonal climate forecasts in managing energy resources. J. Appl. Meteor., 21, 510–517, doi:10.1175/1520-0450(1982)021<0510:TVOSCF>2.0.CO;2.
Bruno Soares, M., and S. Dessai, 2015: Exploring the use of seasonal climate forecasts in Europe through expert elicitation. Climate Risk Manage., 10, 8–16, doi:10.1016/j.crm.2015.07.001.
Buontempo, C., A. Brookshaw, A. Arribas, and K. Mylne, 2010: Multi-scale projections of weather and climate at the UK Met Office. Management of Weather and Climate Risk in the Energy Industry, A. Troccoli, Ed., NATO Science for Peace and Security Series C: Environmental Security, Springer, 39–50, doi:10.1007/978-90-481-3692-6_3.
Butler, A. H., and Coauthors, 2016: The Climate-system Historical Forecast Project: Do stratosphere-resolving models make better seasonal climate predictions in boreal winter? Quart. J. Roy. Meteor. Soc., 142, 1413–1427, doi:10.1002/qj.2743.
Chang, C. P., Z. Wang, and H. Hendon, 2006: The Asian winter monsoon. The Asian Monsoon, B. Wang, Ed., Springer Praxis, 89–127, doi:10.1007/3-540-37722-0_3.
Clark, R. T., P. Bett, H. Thornton, and A. Scaife, 2017: Skilful seasonal predictions for the European energy industry. Environ. Res. Lett., 12, 024002, doi:10.1088/1748-9326/aa57ab.
CNREC, 2014: China wind, solar and bioenergy roadmap 2050. China National Renewable Energy Centre Rep., 25 pp., https://www.iea.org/publications/freepublications/publication/china_wind.pdf.
Costa, A., A. Crespo, J. Navarro, G. Lizcano, H. Madsen, and E. Feitosa, 2008: A review on the young history of the wind power short-term prediction. Renewable Sustainable Energy Rev., 12, 1725–1744, doi:10.1016/j.rser.2007.01.015.
Davidson, M. R., D. Zhang, W. Xiong, X. Zhang, and V. J. Karplus, 2016: Modelling the potential for wind energy integration on China’s coal-heavy electricity grid. Nat. Energy, 1, 16086, doi:10.1038/nenergy.2016.86.
Davis, M., R. Lowe, S. Steffen, F. Doblas-Reyes, and X. Rodó, 2016: Barriers to using climate information: Challenges in communicating probabilistic forecasts to decision-makers. Communicating Climate-Change and Natural Hazard Risk and Cultivating Resilience, J. L. Drake et al., Eds., Advances in Natural and Technological Hazards Research, Vol. 45, Springer, 95–113, doi:10.1007/978-3-319-20161-0_7.
Dee, D. P., and Coauthors, 2011: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system. Quart. J. Roy. Meteor. Soc., 137, 553–597, doi:10.1002/qj.828.
Dessai, S., and M. Bruno Soares, 2015: Report summarising users’ needs for S2D predictions. European Provision of Regional Impact Assessment on a Seasonal-to-Decadal Timescale (EUPORIAS) Deliverable 12.3, 115 pp., http://www.euporias.eu/system/files/D12.3_Final.pdf.
Doblas-Reyes, F. J., J. García-Serrano, F. Lienert, A. P. Biescas, and L. R. L. Rodrigues, 2013: Seasonal climate predictability and forecasting: Status and prospects. Wiley Interdiscip. Rev.: Climate Change, 4, 245–268, doi:10.1002/wcc.217.
Dubus, L., 2014: Weather and climate and the power sector: Needs, recent developments and challenges. Weather Matters for Energy, A. Troccoli, L. Dubus, and S. E. Haupt, Eds., Springer, 379–398, doi:10.1007/978-1-4614-9221-4_18.
Dunstone, N., D. Smith, A. Scaife, L. Hermanson, R. Eade, N. Robinson, M. Andrews, and J. Knight, 2016: Skilful predictions of the winter North Atlantic Oscillation one year ahead. Nat. Geosci., 9, 809–814, doi:10.1038/ngeo2824.
Eade, R., D. Smith, A. Scaife, E. Wallace, N. Dunstone, L. Hermanson, and N. Robinson, 2014: Do seasonal-to-decadal climate predictions underestimate the predictability of the real world? Geophys. Res. Lett., 41, 5620–5628, doi:10.1002/2014GL061146.
Golding, N., C. Hewitt, P. Zhang, P. Bett, X. Fang, H. Hu, and S. Nobert, 2017: Improving user engagement and uptake of climate services in China. Climate Serv., 5, 39–45, doi:10.1016/j.cliser.2017.03.004.
Guo, Z., J. Zhao, W. Zhang, and J. Wang, 2011: A corrected hybrid approach for wind speed prediction in Hexi Corridor of China. Energy, 36, 1668–1679, doi:10.1016/j.energy.2010.12.063.
He, S., Y. Gao, F. Li, H. Wang, and Y. He, 2017: Impact of Arctic Oscillation on the East Asian climate: A review. Earth-Sci. Rev., 164, 48–62, doi:10.1016/j.earscirev.2016.10.014.
Hewitt, C., S. Mason, and D. Walland, 2012: The Global Framework for Climate Services. Nat. Climate Change, 2, 831–832, doi:10.1038/nclimate1745.
Hong, L., N. Zhou, D. Fridley, and C. Raczkowski, 2013: Assessment of China’s renewable energy contribution during the 12th Five Year Plan. Energy Policy, 62, 1533–1543, doi:10.1016/j.enpol.2013.07.110.
Hor, C.-L., S. J. Watson, and S. Majithia, 2005: Analyzing the impact of weather variables on monthly electricity demand. IEEE Trans. Power Syst., 20, 2078–2085, doi:10.1109/TPWRS.2005.857397.
Hsu, W.-r., and A. H. Murphy, 1986: The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2, 285–293, doi:10.1016/0169-2070(86)90048-8.
Huld, T., R. Müller, A. Gracia Amillo, U. Pfeifroth, and J. Trentmann, 2016: Surface solar radiation data set—Heliosat, Meteosat-East (SARAH-E)—Edition 1. Satellite Application Facility on Climate Monitoring (CM SAF), doi:10.5676/DWD/JECD/SARAH_E/V001.
Hunke, E. C., and W. H. Lipscomb, 2010: CICE: The Los Alamos Sea Ice Model documentation and software users’ manual, version 4.1. Los Alamos National Laboratory Tech. Rep. LA-CC-06-012, 116 pp., http://oceans11.lanl.gov/trac/CICE.
Ke, Z.-J., P.-Q. Zhang, L.-J. Chen, and L.-M. Du, 2011: An experiment of a statistical downscaling forecast model for summer precipitation over China. Atmos. Ocean. Sci. Lett., 4, 270–275, doi:10.1080/16742834.2011.11446941.
Kumar, A., 2009: Finite samples and uncertainty estimates for skill measures for seasonal prediction. Mon. Wea. Rev., 137, 2622–2631, doi:10.1175/2009MWR2814.1.
Li, C., and Coauthors, 2016: Skillful seasonal prediction of Yangtze River valley summer rainfall. Environ. Res. Lett., 11, 094002, doi:10.1088/1748-9326/11/9/094002.
Liang, Y., Y. Zhu, and W. Tan, 2015: Research on the utilization of small stand-alone wind power generation in Yunnan mountainous areas. Proc. 2015 Int. Conf. on Applied Science and Engineering Innovation, Jinan, China, Henan University, doi:10.2991/asei-15.2015.297.
Lin, B., and X. Ouyang, 2014: Energy demand in China: Comparison of characteristics between the US and China in rapid urbanization stage. Energy Convers. Manage., 79, 128–139, doi:10.1016/j.enconman.2013.12.016.
Lo, K., 2014: A critical review of China’s rapidly developing renewable energy and energy efficiency policies. Renewable Sustainable Energy Rev., 29, 508–516, doi:10.1016/j.rser.2013.09.006.
Lu, B., A. A. Scaife, N. Dunstone, D. Smith, H.-L. Ren, Y. Liu, and R. Eade, 2017: Skillful seasonal predictions of winter precipitation over southern China. Environ. Res. Lett., 12, 074021, doi:10.1088/1748-9326/aa739a.
MacLachlan, C., and Coauthors, 2015: Global Seasonal Forecast System version 5 (GloSea5): A high-resolution seasonal forecast system. Quart. J. Roy. Meteor. Soc., 141, 1072–1084, doi:10.1002/qj.2396.
Madec, G., 2008: NEMO ocean engine. Institut Pierre-Simon Laplace Note du Pôle de modélisation 27, 386 pp., http://www.nemo-ocean.eu/About-NEMO/Reference-manuals.
McColl, L., E. J. Palin, H. E. Thornton, D. M. H. Sexton, R. Betts, and K. Mylne, 2012: Assessing the potential impact of climate change on the UK’s electricity network. Climatic Change, 115, 821–835, doi:10.1007/s10584-012-0469-6.
Megann, A., and Coauthors, 2014: GO5.0: The joint NERC–Met Office NEMO global ocean model for use in coupled and forced applications. Geosci. Model Dev., 7, 1069–1092, doi:10.5194/gmd-7-1069-2014.
Palin, E. J., A. A. Scaife, E. Wallace, E. C. D. Pope, A. Arribas, and A. Brookshaw, 2016: Skillful seasonal forecasts of winter disruption to the U.K. transport system. J. Appl. Meteor. Climatol., 55, 325–344, doi:10.1175/JAMC-D-15-0102.1.
Peng, Z., Q. J. Wang, J. C. Bennett, P. Pokhrel, and Z. Wang, 2014: Seasonal precipitation forecasts over China using monthly large-scale oceanic-atmospheric indices. J. Hydrol., 519, 792–802, doi:10.1016/j.jhydrol.2014.08.012.
Qiang, L., C. Yi, T. Chuan, Z. Xiao-Qi, and L. Jun-Feng, 2016: Strategic deliberation on development of low-carbon energy system in China. Adv. Climate Change Res., 7, 26–34, doi:10.1016/j.accre.2016.04.002.
Qin, D., J. Zhang, C. Shan, and L. Song, 2015: China National Assessment Report on Risk Management and Adaptation of Climate Extremes and Disasters (Refined Edition). Science Press, 123 pp, https://www.purpleculture.net/china-national-assessment-report-on-risk-management-and-adaptation-of-climate-extremes-and-disasters-refined-edition-p-22726/.
Rae, J. G. L., H. T. Hewitt, A. B. Keen, J. K. Ridley, A. E. West, C. M. Harris, E. C. Hunke, and D. N. Walters, 2015: Development of the Global Sea Ice 6.0 CICE configuration for the Met Office Global Coupled model. Geosci. Model Dev., 8, 2221–2230, doi:10.5194/gmd-8-2221-2015.
Ramage, C. S., 1968: Role of a tropical “maritime continent” in the atmospheric circulation. Mon. Wea. Rev., 96, 365–370. doi:10.1175/1520-0493(1968)096<0365:ROATMC>2.0.CO;2.
Scaife, A. A., 2016: The North Atlantic and Arctic Oscillations: Climate variability, extremes, and stratosphere–troposphere interaction. Dynamics and Predictability of Large-Scale, High-Impact Weather and Climate Events, J. Li et al., Eds., Special Publications of the International Union of Geodesy and Geophysics, Cambridge University Press, 122–132, http://www.cambridge.org/9781107071421.
Scaife, A. A., and Coauthors, 2014: Skillful long-range prediction of European and North American winters. Geophys. Res. Lett., 41, 2514–2519, doi:10.1002/2014GL059637.
Smith, D. M., A. A. Scaife, R. Eade, and J. R. Knight, 2016: Seasonal to decadal prediction of the winter North Atlantic Oscillation: Emerging capability and future prospects. Quart. J. Roy. Meteor. Soc., 142, 611–617, doi:10.1002/qj.2479.
Svensson, C., and Coauthors, 2015: Long-range forecasts of UK winter hydrology. Environ. Res. Lett., 10, 064006, doi:10.1088/1748-9326/10/6/064006.
Taylor, A., and Coauthors, 2015: Report describing formulation of strategies for communicating confidence levels for S2D forecasts. European Provision of Regional Impact Assessment on a Seasonal-to-Decadal Timescale (EUPORIAS) Deliverable 33.3, 46 pp., http://www.euporias.eu/system/files/D33.3.pdf.
Taylor, J. W., and R. Buizza, 2003: Using weather ensemble predictions in electricity demand forecasting. Int. J. Forecasting, 19, 57–70, doi:10.1016/S0169-2070(01)00123-6.
Thornton, H. E., B. J. Hoskins, and A. A. Scaife, 2016: The role of temperature in the variability and extremes of electricity and gas demand in Great Britain. Environ. Res. Lett., 11, 114015, doi:10.1088/1748-9326/11/11/114015.
Troccoli, A., 2010: Seasonal climate forecasting. Meteor. Appl., 17, 251–268, doi:10.1002/met.184.
Tung, Y. L., C.-Y. Tam, S.-J. Sohn, and J.-L. Chu, 2013: Improving the seasonal forecast for summertime South China rainfall using statistical downscaling. J. Geophys. Res., 118, 5147–5159, doi:10.1002/jgrd.50367.
Valor, E., V. Meneu, and V. Caselles, 2001: Daily air temperature and electricity load in Spain. J. Appl. Meteor., 40, 1413–1421, doi:10.1175/1520-0450(2001)040<1413:DATAEL>2.0.CO;2.
Walters, D., and Coauthors, 2017: The Met Office Unified Model Global Atmosphere 6.0/6.1 and JULES Global Land 6.0/6.1 configurations. Geosci. Model Dev., 10, 1487–1520, doi:10.5194/gmd-10-1487-2017.
Wang, B., B. Xiang, and J.-Y. Lee, 2013: Subtropical high predictability establishes a promising way for monsoon and tropical storm predictions. Proc. Natl. Acad. Sci. USA, 110, 2718–2722, doi:10.1073/pnas.1214626110.
Wang, B., R.-Y. Ke, X.-C. Yuan, and Y.-M. Wei, 2014: China’s regional assessment of renewable energy vulnerability to climate change. Renewable Sustainable Energy Rev., 40, 185–195, doi:10.1016/j.rser.2014.07.154.
Wang, H., and Coauthors, 2015: A review of seasonal climate prediction research in China. Adv. Atmos. Sci., 32, 149–168, doi:10.1007/s00376-014-0016-7.
Wang, Q., 2014: Effects of urbanisation on energy consumption in China. Energy Policy, 65, 332–339, doi:10.1016/j.enpol.2013.10.005.
Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences, 3rd ed., International Geophysical Series, Vol. 100, 704 pp., doi:10.1016/B978-0-12-385022-5.00008-7.
Williams, K. D., and Coauthors, 2015: The Met Office Global Coupled model 2.0 (GC2) configuration. Geosci. Model Dev., 8, 1509–1524, doi:10.5194/gmd-8-1509-2015.
WMO, 2010: Standardized verification system (SVS) for long-range forecasts (LRF) (Attachment II-8), in Manual on the Global Data-Processing and Forecasting System: Volume 1—Global aspects. World Meteorological Organization Rep. WMO-No. 485, 11.8-1–11.8-17, https://www.wmo.int/pages/prog/www/DPFS/documents/485_Vol_I_en.pdf.
Wu, Z., and L. Yu, 2016: Seasonal prediction of the East Asian summer monsoon with a partial-least square model. Climate Dyn., 46, 3067–3078, doi:10.1007/s00382-015-2753-4.
Xiao, Z., B. Liu, H. Liu, and D. Zhang, 2012: Progress in climate prediction and weather forecast operations in China. Adv. Atmos. Sci., 29, 943–957, doi:10.1007/s00376-012-1194-9.
Xing, W., B. Wang, and S.-Y. Yim, 2016: Long-lead seasonal prediction of China summer rainfall using an EOF-PLS regression based methodology. J. Climate, 29, 1783–1796, doi:10.1175/JCLI-D-15-0016.1.
Yang, S., K. M. Lau, S. H. Yoo, J. L. Kinter, K. Miyakoda, and C. H. Ho, 2004: Upstream subtropical signals preceding the Asian summer monsoon circulation. J. Climate, 17, 4213–4229, doi:10.1175/JCLI3192.1.
Ying, L., and F. Ke, 2012: Improve the prediction of summer precipitation in the southeastern China by a hybrid statistical downscaling model. Meteor. Atmos. Phys., 117, 121–134, doi:10.1007/s00703-012-0201-0.
Zhang, N., X. Lu, M. B. McElroy, C. P. Nielsen, X. Chen, Y. Deng, and C. Kang, 2016: Reducing curtailment of wind electricity in China by employing electric boilers for heat and pumped hydro for energy storage. Appl. Energy, 184, 987–994, doi:10.1016/j.apenergy.2015.10.147.
Zhang, R., A. Sumi, and M. Kimoto, 1996: Impact of El Niño on the East Asian monsoon. A diagnostic study of the ’86/87 and ’91/92 events. J. Meteor. Soc. Japan, 74, 49–62, doi:10.2151/jmsj1965.74.1_49.
Zhang, Z. Y., D. Y. Gong, and J. J. Ma, 2014: A study on the electric power load of Beijing and its relationships with meteorological factors during summer and winter. Meteor. Appl., 21, 141–148, doi:10.1002/met.1313.
Zuo, J., H.-L. Ren, and W. Li, 2015: Contrasting impacts of the Arctic Oscillation on surface air temperature anomalies in southern China between early and middle-to-late winter. J. Climate, 28, 4015–4026, doi:10.1175/JCLI-D-14-00687.1.
That is, 432 cells east–west × 324 cells north–south.
Wind-turbine hub heights are around 80–100 m. The skill measures that we use here focus on the relative changes to each variable, however, and on the time scales in which we are interested—seasonal means—there is no significant difference between relative changes in wind speeds at 10 and 100 m.
By “standardized” we mean that we subtract the average and then divide by the standard deviation σ. The result is an anomaly time series in units of σ. We use the 20-yr period of the hindcast to determine the long-term average and σ, for both the observations and the hindcast itself.
For this test to apply exactly, the data must be Gaussian and independent. In our case, the central limit theorem means that this will be a reasonable approximation for our seasonal-average data and that the test will provide a good indication of nonzero correlation.
In our case of having 20 years of data, the threshold in correlation corresponding to the 5% significance level is |r| > 0.44.
Although ERAI uses climatological aerosols, it assimilates other observations that could be forced by anomalous aerosol emissions, making it particularly complicated to assess in this way.
We use cross validation when calculating the quantile values, with a window length of 1 yr (i.e., one DJF or JJA period), following the method of WMO (2010): the quantile in question is calculated separately for each year from the 19 years of data remaining after that year is masked out.
The Niño-3.4 index is the time series of sea surface temperature anomalies in the region extending from 120° to 170°W and from 5°S to 5°N.
In general, the BSS compares the forecast system with any reference forecast, but here we use the climatological rate of the event in question (referred to as climatology) as the reference.