• Binder, P., and Coauthors. 1996: MAP Mesoscale Alpine Programme design proposal. MAP Programme Office, 77 pp. [Available from MAP Programme Office c/0 Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Bougeault, P., and Coauthors. 1998: MAP Mesoscale Alpine Programme science plan. MAP Programme Office, 64 pp. [Available from MAP Programme Office c/0 Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Frustaci, G., and Coauthors. 2000: ITAMAP: The data set of non-GTS surface hourly data gathered during the MAP field phase (Sep–Nov 1999): Version 1.2, September 2000. MAP Programme Office, 64 pp. [Available from MAP Programme Office c/o Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Ghelli, A., and Lalaurette F. , 2000: Verifying precipitation forecasts using upscaled observations. ECMWF Newsletter, Vol. 87, 9–17.

  • Gregory, D., Morcrette J-J. , Jakob C. , Beljaars A. , and Stockdale T. , 2000: Revision of convection, radiation and cloud schemes in the ECMWF Integrated Forecasting System. Quart. J. Roy. Meteor. Soc., 126 , 16851710.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hirter, H., and Richner H. , 1996: MAP Mesoscale Alpine Programme data centre description. MAP Programme Office, 77 pp. [Available from MAP Programme Office c/o Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Jakob, C., and Klein S. A. , 2000: A parameterisation of the effects of cloud and precipitation overlap for the use in general-circulation models. Quart. J. Roy. Meteor. Soc., 126 , 25252544.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jakob, C., and Coauthors. 2000: The IFS cycle CY21r4 made operational in October 1999. ECMWF Newsletter, Vol. 87, 2–9.

  • Karl, T. R., Wang W-C. , Schlesinger M. E. , Knight R. W. , and Portman D. , 1990: A method of relating general Circulation model simulated climate to the observed local climate. Part I: Seasonal statistics. J. Climate, 3 , 10531079.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Katz, W. R., and Murphy A. H. , 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

  • Kim, J-W., Chang J-T. , Baker N. L. , Wilks D. S. , and Gates W. L. , 1984: The statistical problem of climate inversion: Determination of the relationship between local and large-scale climate. Mon. Wea. Rev., 112 , 20692077.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and Buizza R. , 2001: Quantitative precipitation forecasts over the United States by the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 129 , 638663.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., . 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Ritchie, H., Temperton C. , Simmons A. J. , Hortal M. , Davies T. , Dent D. , and Hamrud M. , 1995: Implementation of the semi-Lagrangian method in a high resolution version of the ECMWF forecast model. Mon. Wea. Rev., 123 , 489514.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting, 5 , 570575.

  • Skelly, W. C., and Henderson-Sellers A. , 1996: Grid box or grid point: What type of data do GCMs deliver to climate impacts researchers? Int. J. Climatol., 16 , 10791086.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simmons, A. J., Burridge D. M. , Jarraud M. , Girard C. , and Wenge W. , 1989: The ECMWF medium-range prediction models: Development of the numerical formulations and the impact of increased resolution. Meteor. Atmos. Phys., 40 , 2860.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Teixeira, J., 1999: The impact of increased boundary layer resolution on the ECMWF forecast system. ECMWF Tech. Memo. 268, 55 pp.

  • Teixeira, J., . 2000: Boundary layer clouds in large scale atmospheric models: Cloud scheme and numerical aspects. Ph.D. dissertation, University of Lisbon, 190 pp. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

    • Search Google Scholar
    • Export Citation
  • Tiedtke, M., 1989: A comprehensive mass flux scheme for cumulus parameterization in large-scale models. Mon. Wea. Rev., 117 , 17791800.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tiedtke, M., . 1993: Representation of clouds in large-scale models. Mon. Wea. Rev., 121 , 30403061.

  • Untch, A., Simmons A. J. , Hortal M. , and Jakob C. , 1999: Increased stratospheric resolution in the ECMWF forecasting system. Proc. SODA (Satellite Ozone Data Assimilation) Workshop on Chemical Data Assimilation, No. 188, De Bilt, Netherlands, Royal Netherlands Meteorological Institute, 45–52.

    • Search Google Scholar
    • Export Citation
  • Wigley, T. M. L., and Santer B. D. , 1990: Statistical comparison of spatial fields in model validation, perturbation, and predictability experiments. J. Geophys. Res., 95 ((D1),) 851865.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Wilson, L. L., and Lettenmaier D. P. , 1992: A hierarchical stochastic model of large-scale atmospheric circulation patterns and multiple station daily precipitation. J. Geophys. Res., 97 , 27912809.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • View in gallery

    Station density (grayscale) for 0.5° × 0.5° grid boxes, averaged over the whole period under investigation. The rectangle defines the area selected for the MAP-ALPS experiment

  • View in gallery

    Distribution of the 24-h accumulated precipitation on 20 Sep 1999 (0600 UTC) as observed from (a) the high-resolution network stations and (b) after the upscaling technique has been applied. (c) The same field as observed from the GTS SYNOP stations

  • View in gallery

    USO (solid line) and GTS-SO (dashed line) cumulative precipitation [mm (24 h)−1] distributions. The distributions have been normalized by their respective totals

  • View in gallery

    Mean observed 24-h accumulated precipitation for each forecast category for the period 8 Sep–16 Nov 1999 plotted against the forecast categories. The forecasts are range (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

  • View in gallery

    FBI for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line for USO and dashed line for GTS-SO

  • View in gallery

    ETS for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

  • View in gallery

    TSS for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

  • View in gallery

    Observed distribution of precipitation for seven closed classes for the MAP-ALPS experiment

  • View in gallery

    As in Fig. 5a but for the MAP-ALPS experiment (forecast range t + 42 h)

  • View in gallery

    As in Fig. 6a but for the MAP-ALPS experiment (forecast range t + 42 h)

  • View in gallery

    As in Fig. 7a but for the MAP-ALPS experiment (forecast range t + 42 h)

  • View in gallery

    Time series of FBI (weekly moving average) for the 0.1-mm (top line) and 1-mm (bottom line) threshold. The forecast range is t + 42 h

  • View in gallery

    Forecast (color shaded) and upscaled observation (numbers) for the 24-h cumulative precipitation field at 0600 UTC 14 Oct 1999. The forecast is relative to 12 Oct 1999, range t + 42 h

  • View in gallery

    IR satellite image at 0300 UTC 20 Sep 1999

  • View in gallery

    Observed 24-h accumulated precipitation at 0600 UTC averaged over the period 15–30 Sep 1999 superimposed onto the 24-h accumulated precipitation (range t + 42 h; color shaded). The forecast field is an average of all the t + 42 h forecasts verifying between 15 and 30 Sep. Shading is as in the legend

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 151 146 1
PDF Downloads 56 52 1

Verification of Precipitation Forecasts over the Alpine Region Using a High-Density Observing Network

View More View Less
  • 1 University of L'Aquila, L'Aquila, Italy
  • | 2 ECMWF, Reading, United Kingdom
© Get Permissions
Full access

Abstract

The demand for verification of forecasting systems to ascertain their strengths and weaknesses is increasing dramatically as models evolve more rapidly. Precipitation forecasts have always been of great interest to forecasters because they influence daily life. The recent flooding over Europe has also shown how important it is to know how models can reproduce these events. The issue of precipitation verification is addressed here, starting from the assumption that model spatial scales have to be verified against data representing similar scales. Only in this way may the skill of forecasting system used herein be determined. The performance of the European Centre for Medium-Range Weather Forecasts model in predicting precipitation is discussed. The study concentrates on the period September to November 1999 during which high-density observations were available for the Alps. The high-resolution observing network over the Alpine region has been used to reconstruct a precipitation analysis that contains smoothed small-scale variability and represents with sufficient accuracy the average behavior of the observed field in the model grid box. The precipitation forecast is verified against both the precipitation analysis and the surface synoptic observations (SYNOP) available in real time via the Global Telecommunication System. Both verification approaches show that for the Alpine region, during autumn 1999, the model overestimates the precipitation amount. Overestimation is smaller when the forecast is compared with the precipitation analysis. It is also shown that verification against irregular and scattered observations (SYNOP data) is highly influenced by the variability of the precipitation in a grid box. A precipitation analysis is, therefore, important if model skill has to be defined.

Corresponding author address: Dr. Anna Ghelli, ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom. Email: a.ghelli@ecmwf.int

Abstract

The demand for verification of forecasting systems to ascertain their strengths and weaknesses is increasing dramatically as models evolve more rapidly. Precipitation forecasts have always been of great interest to forecasters because they influence daily life. The recent flooding over Europe has also shown how important it is to know how models can reproduce these events. The issue of precipitation verification is addressed here, starting from the assumption that model spatial scales have to be verified against data representing similar scales. Only in this way may the skill of forecasting system used herein be determined. The performance of the European Centre for Medium-Range Weather Forecasts model in predicting precipitation is discussed. The study concentrates on the period September to November 1999 during which high-density observations were available for the Alps. The high-resolution observing network over the Alpine region has been used to reconstruct a precipitation analysis that contains smoothed small-scale variability and represents with sufficient accuracy the average behavior of the observed field in the model grid box. The precipitation forecast is verified against both the precipitation analysis and the surface synoptic observations (SYNOP) available in real time via the Global Telecommunication System. Both verification approaches show that for the Alpine region, during autumn 1999, the model overestimates the precipitation amount. Overestimation is smaller when the forecast is compared with the precipitation analysis. It is also shown that verification against irregular and scattered observations (SYNOP data) is highly influenced by the variability of the precipitation in a grid box. A precipitation analysis is, therefore, important if model skill has to be defined.

Corresponding author address: Dr. Anna Ghelli, ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom. Email: a.ghelli@ecmwf.int

1. Introduction

Forecast verification is the process of determining the quality of forecasts through the assessement of the degree of similarity between forecast and observed conditions (Murphy and Winkler 1987; Murphy 1993).

Verification processes give an insight into weaknesses and strengths of forecasting systems, thus allowing a more complete use of the information contained in the forecasts. Forecast verification encompasses many different methodologies to define the quality of the forecast, using either a deterministic or a probabilistic approach. For an extensive review see Katz and Murphy (1997).

In this paper we address the question of verification of deterministic forecasts against observations. Models predict precipitation on spatial scales different from the observed ones. Therefore, verification against irregularly distributed data, as surface synoptic observations (SYNOP) on the GTS (a complete list of acronyms is available in Table 1) might be, is liable to misinterpretation. Moreover, interpolation of model fields to station locations is necessary to compare forecasts to observations; this process does not create new information, it only increases the spatial precision of the field, while introducing further uncertainties (Skelly and Handerson-Sellers 1996, hereinafter referred to as SHS). Furthermore, the interpolation methods commonly used assume that the underlying field is continuous. This assumption is not generally true of precipitation fields.

Ghelli and Lalaurette (2000, hereinafter referred to as GL) analyzed model performance using a high-resolution observing network over France. They found that gridded observations better represent the gridbox behavior described by the model.

The question of whether GCM results pertain to gridpoint or gridbox area has been largely discussed in the context of climate change. SHS pointed out that when dealing with variables that are implicitly areal (as they result from subgrid parameterizations like convection, precipitation, radiation, etc.), GCM output should be treated as areal quantities. A “gridpoint approach” is probably most suitable when dealing with finite-difference or spectral methods that produce point, rather than areal, values. Kim et al. (1984) and Karl et al. (1990) have treated simulated precipitation as an areal quantity. Alternatively Wigley and Santer (1990) and Wilson and Lettenmaier (1992) have accepted for the simulated quantities the gridpoint approach. In the present paper, simulated precipitation is treated as an areal quantity.

The high-resolution precipitation data used in this study have been collected as part of MAP (Binder et al. 1996). The upscaling procedure is as used in GL, and it consists of a simple averaging procedure of all the observations contained in a model grid box.

The upscaled observations are used to verify ECMWF precipitation forecasts over the Alpine region. The verification period starts on 8 September and ends on 16 November 1999. Two areas have been taken into consideration: (i) a region extending from 42° to 50°N and from 0° to 20°E (hereinafter referred to as MAP-LARGE), and (ii) an area centered on the Alps from 44° to 48°N and from 7° to 14°E (hereinafter referred to as MAP-ALPS).

Verification against SYNOPs has also been carried out for both the areas during the same period. Three different measures of model skill are examined to assess the two verification approaches.

The paper is organized as follows: section 2 describes briefly the ECMWF model, the observation dataset specifications, and the upscaling techniques used; verification procedures are discussed in section 3. Results are presented in section 4. In section 5 conclusions are drawn.

2. Model overview and data specification

a. The ECMWF model

For the period of this study, the ECMWF GCM had spectral horizontal resolution TL319. A detailed description of the model can be found in Simmons et al. (1989) and Ritchie et al. (1995). During the verification period the model vertical resolution changed (on 12 October 1999) from 50 (Untch et al. 1999) to 60 levels (Teixeira 1999; Jakob et al. 2000). The increased number of levels doubled the vertical resolution below 1500 m and brought the lowest level down to 10 m above the surface as opposed to 33 m above the surface in the previous 50-level version.

Stratiform and convective clouds are represented with a prognostic cloud scheme (Tiedtke 1993). Clouds are generated by large-scale ascent, diabatic cooling, and boundary layer turbulence. They are dissipated through evaporation, turbulent mixing with unsaturated environmental air, and precipitation processes. Convective precipitation, vertical momentum fluxes, and temperature changes in the atmosphere due to release of latent heat or cooling in connection with evaporation are calculated in the convective scheme. The subgrid vertical fluxes of mass heat water vapor and momentum are computed at each model level using a simple mass flux model interacting with its environment. The scheme is applied to penetrative convection, shallow convection, and midlevel convection (Gregory et al. 2000; Tiedtke 1989). In October 1999 a new parameterization scheme was been introduced into the model (Jakob and Klein 2000). It differs from Tiedtke's original method in that the formulation of the precipitation/evaporation explicitly accounts for the vertical distribution of cloud layers.

A more detailed description of the model's parameterization can be found in Teixeira (2000).

b. Verification data

High-resolution precipitation observations (from the MAP dataset) are available at MDC (Hirter and Richner 1996) and consist of precipitation reports collected by national and regional meteorological centers for about 4000 stations (synoptic, automatic, and climatological).

The subset of the MAP dataset used in this study covers the period 8 September–17 November 1999 (Bougeault et al. 1998; Frustaci et al. 2000), which is referred to as SOP. The precipitation observations are accumulated over 24 h from 0600 UTC. A simple quality control procedure has been carried out on the data to remove possible suspect reports. The mean and median of local precipitation distributions, built using all the stations within a 30-km radius around each grid point, are calculated. Whenever the mean and median are more than a fixed threshold apart, the station reports associated with the local distribution are flagged (thresholds used: 25, 50, and 100 mm). Flagged reports are discarded or kept after an analysis of satellite imagery and a consistency check with neighboring stations.

Figure 1 depicts the mean spatial distribution of the station density for the Alpine region considered in this study. The highest station density is on the French and Swiss side of the Alps, while the Italian side is less sampled. The Slovenian network also provides good coverage. Data for the northeast side of the Alps were not available at the time of this study. The box in Fig. 1 defines the area selected for the MAP-ALPS experiment.

Each station of the high-density observing network is assigned to a grid box and an average of all the observations within each box is calculated and assigned to the relative grid point. The upscaled observations (USO) are used to verify the model precipitation forecast. Other averaging methods, that is, function of the station–gridpoint distance and function of precipitation intensity, have been tested and the results showed little or no dependence on the averaging method chosen.

The upscaling is applied if there are more than three stations in a grid box. This threshold has been chosen as a compromise between the increased accuracy of the verification analysis as the station density increases (Mullen and Buizza 2001) and the loss of information on the Alpine area for higher thresholds, as the station density is not as high in the Alps (Fig. 1).

Figure 2 shows the precipitation distribution for a chosen day as reported by the high-resolution observing system (panel a) and the distribution after upscaling (panel b). The upscaled field maintains the characteristics of the initial field but the very small-scale variability is lost in the averaging procedure.

The precipitation values (GTS-SO) obtained from the GTS are accumulated over a 24-h period from 0600 UTC. In order to compare the GTS-SO with the forecast value, four grid points surrounding the station location are chosen and the values are linearly interpolated to the station location itself. The GTS-SO distribution is presented in Fig. 2c.

Figure 3 depicts the GTS-SO (dashed line) and the USO (solid line) cumulative distributions normalized by their totals. There is a qualitative similarity between the cumulative distributions, though caution should be used in drawing conclusions on the superiority of one verification approach versus the other. They represent two different methods and one of them (GTS-SO) suffers from “representativity” problems, as discussed in the introduction.

3. Verification scores

In this section, the accuracy of a deterministic forecasting system in a dichotomous situation (yes–no) is under investigation. Murphy and Winkler (1987) and Murphy (1991) state that, while testing the performance of an individual forecast system, it is important to remember the complexity and dimensionality of the verification problem. Therefore, enough accuracy measures have to be used to fully estimate the value of a forecast. In this study three measures are used to evaluate the skill of the ECMWF model forecasts. Following GL, contingency tables for different thresholds have been built for both the USO and SO. Table 2 is an example: a is the number of correct forecasts of a precipitation category, b is the number of forecasts incorrectly predicting precipitation, c is the number of forecasts failing to predict an observed event, and d is the correct forecast of no precipitation. The precipitation thresholds chosen for this study were 0.1, 1, 2, 4, 8, and 16 mm (24 h)−1. The forecasting system has been verified against the two datasets using FBI, ETS, and TSS. A detailed explanation of such scores can be found in Wilks (1995).

The measures used are written as
i1520-0434-17-2-238-e1
where
i1520-0434-17-2-238-e3
and
i1520-0434-17-2-238-e4

FBI measures the event frequency with no regard for the forecast accuracy. Its value is 1 for a perfect forecast, and it is larger (smaller) than 1 if the system is overforecasting (underforecasting). The ETS (Schaefer 1990) is a modified version of the threat score rendered equitable by taking away the random forecast [R(a)]. Therefore, a chance forecast will score 0, as will a constant forecast. A perfect forecast will have an ETS equal to 1. The TSS can also be written as the probability of detection [a/(a + c)] minus the probability of false detection [b/(b + d)]. As in the ETS, the random and constant forecasts receive a 0 score, while a higher score is obtained if a rare event is forecast correctly.

4. Results

a. The MAP-LARGE experiment

In this section the 1200 UTC forecasts (range t + 42 and t + 66) for the period 8 September–16 November 1999 are verified against GTS-SO (dashed line) and USO (solid line). Figure 4 represents the mean observation value relative to each forecast category. The forecasts are divided into categories [from 5 to 35 mm (24 h)−1 every 5 mm (24 h)−1]. For each forecast category the observations pertaining to the forecast–observation pair are averaged. This gives an indication on the under/overestimation of the rainfall amounts. The diagonal represents the perfect forecast; values above (below) the diagonal indicate under- (over)forecasting. Both ranges, t + 42 (Fig. 4a) and t + 66 (Fig. 4b), show that the forecast overestimates precipitation for values larger than 10 mm (24 h)−1. An average overestimation of 8 mm (24 h)−1 is assigned to a forecast (range t + 66) of 30 mm (24 h)−1 for GTS-SO (dashed line), while the overestimation is smaller for USO (solid line). Small amounts of rainfall are forecast accurately on average.

The two verification approaches show substantial differences in terms of FBI. FBIs for USO (solid line) and GTS-SO (dashed line) is depicted in Fig. 5. Ideally, for a perfect forecast the FBI should be equal to 1 (the event is forecast as often as is observed). For an FBI greater than 1 the event is forecast more often than is observed (overforecast), and vice versa for an FBI of less than 1 (underforecast). FBI for GTS-SO indicates overestimation for both forecast ranges and all the thresholds. A much better picture is obtained when verifying model forecasts against USO. The model overforecasts rainfall events for small thresholds at t + 42 and for all the thresholds at t + 66, though the overestimation is substantially reduced. Apart from a few well-defined events, SOP is characterized by dry periods alternated with localized precipitation events, the latter being associated with weak systems or shallow troughs. In these situations, the representativity issue becomes important, and the model skill will not be high if compared with irregularly distributed GTS-SO, unless its resolution is able to resolve very local effects. A typical example is when small portions of the grid box have rainfall events. The model may predict this fraction accurately, but if it is verified by a SYNOP located where there was no precipitation, the model will have bad scores and in general it will show overforecasting. Many stations inside the same grid box will represent the behavior of each portion of the grid box, and their average is likely to compare better with the model forecast. Therefore, GTS-SO might depict a distorted scenario leading to overestimation of the precipitation forecast in periods similar to the present SOP, while USOs describe the average behavior in the grid box and transfer this information onto the model grid point.

Figure 6 shows the ETS for the ranges t + 42 (panel a) and t + 66 (panel b). High values of the score indicate a more skillful forecasting system. Thresholds 1 and 2 mm (24 h)−1 give the highest forecast accuracy, while at the extreme thresholds the forecasting system is less skillful for both the GTS-SO (dashed line) and USO (solid line) verification. As already observed for FBI, the model shows better scores and it is less penalized at the extreme categories when verified against USO. The slight deterioration in the accuracy measure for the t + 66 range reflects the “natural” deterioration of the forecast with time.

TSS is depicted in Fig. 7 [t + 42 (panel a) and t + 66 (panel b)] and offers a picture similar to that for ETS. Extreme categories score better when USOs (solid line) are used in the verification.

b. The MAP-ALPS experiment

The “representativity” becomes of primary importance in regions with complex orography. Because of the coarse representation of mountain ranges in a model, grid points and station locations have significantly different orographic height, thus rendering verification results very difficult to interpret. The upscaling technique partly overcomes this problem.

In this section, the beneficial effect of verifying model forecasts against USO for the alpine area selected in Fig. 1 is investigated. The area is centered over the Alps with 64 GTS-SOs and 35 USOs. FBI, ETS, and TSS have been calculated for the range t + 42 and t + 66. The results for the two ranges are similar; therefore, only scores for the range t + 42 will be discussed.

Figure 8 describes the observed distribution of precipitation for seven closed classes. About 20% of the cases in SOP show precipitation above the 8 mm (24 h)−1 threshold, while the majority of the cases have very little to no rainfall.

Figure 9 shows FBI for the verification against USO (solid line) and against GTS-SO (dashed line). Both verification techniques indicate overestimation of precipitation in the Alpine area. In the GTS-SO case, small and large thresholds are largely overestimated. The improvement in these two extreme thresholds is substantial when USOs are used. The ETS and TSS scores for the same area are depicted in Figs. 10 and 11. They provide a measure of accuracy of the forecast and confirm that for both verification against GTS-SO (solid line) and USO (dashed line) the model is less skillful for the lowest threshold value. The model skill for the SOP improves as the threshold increases, indicating that for the cases with rainfall above 8 mm (24 h)−1 (20% of cases) the forecast has been quite accurate. The accuracy is higher in the verification against USOs, as the latter represent the gridbox behavior rather than a local phenomenon.

The scatterplot of forecast versus observed values (not shown) indicates a strong correlation between forecast and observed precipitation values (0.74) in the case of USO. Such correlation decreases if the model is verified against the GTS-SO (0.58).

c. Scores time series: Case study

In this section the time series of FBI for the whole SOP is discussed as a possible alternative way to look at scores. The period considered is relatively short. Therefore, to support any conclusions, we have analyzed the synoptic situation in detail.

The time series relative to the verification against USO has been calculated for the MAP-LARGE experiment using a 7-day moving sample mean (FBI is calculated for a 7-day period) over the 70-day period. Figure 12 shows the scores relative to the t + 42 range forecast for 0.1- (top line) and 1-mm (bottom line) thresholds. Time series for higher thresholds are affected by noise due to the small sample size.

FBI indicates, for the period under investigation, overforecasting of rainfall events in the Alpine region. This seems to disagree with the hypothesis of underestimation of the orographic effects on the precipitation field put forward by GL. We believe, though, that there is no contradiction in the two results. Ghelli and Lalaurette (2000) have examined a mixed terrain area. In this paper attention is concentrated on the Alpine region with a large number of meteorological stations located in the Alps. Moreover, both studies concentrate on different time periods and, therefore, drawing general conclusions concerning the behavior of the forecast would be inappropriate.

FBI has higher values (both ETS and TSS indicate loss of accuracy for the same period, not shown) around the middle of October. The period was characterized by a lack of precipitation events, and the forecast overestimated the precipitation on the only rainy day. The model change on 12 October 1999 cannot be responsible for the deterioration of the scores, as subsequent forecasts show relatively good skill. Synoptic analysis shows that, from the second week in October, high pressure dominated the Mediterranean basin for several days. On 13 October, a trough approached Europe affecting the Alps with high precipitation mainly on the windward side of the mountains. The trough moved eastward slowly because of the presence of the high pressure area. The 1200 UTC forecast of 11 October 1999 (range t + 42) shows a trough and the associated rainband to the south of the observed position. The following 1200 UTC forecast corrects the position but overestimates the amount of precipitation!

Figure 13 depicts the t + 42 precipitation forecast 1200 UTC 12 October, verifying at 0600 UTC 14 October 1999. The forecast field is shaded and numbers denote USO. The maximum value of observed precipitation is 30 mm (24 h)−1, while the forecast value goes up to 41 mm (24 h)−1. Moreover, the precipitation field is overestimated near the mountains.

The second half of September was dominated by a heavy precipitation event associated with perturbation characterized by a well-defined large-scale forcing. A trough extended south into North Africa with strong warm and moist advection over the Alpine area and southerly flow at low levels. Figure 14 shows the infrared satellite imagery relative to 20 September 1999. The cloud band covers the western Mediterranean, central Europe, and the British Isles. The precipitation patterns ranged from light stratiform rain to strong orographic precipitation (see reports at MDC for a detailed description). Figure 15 shows the observed 24-h accumulated precipitation (0600 UTC) averaged over the period 15–30 September superimposed on the forecast field (shaded). The 24-h accumulated precipitation forecast field has been obtained by averaging all t + 42 h forecasts that verify in the period mentioned. An indication of the model performance can be gained by the positive bias (forecast minus observed value) in the alpine region. The model overestimates precipitation amounts, as was already observed in the FBI time series.

The beginning of November was characterized by heavy precipitation events in the Alpine region and in the central Mediterranean basin. An upper-level trough with an associated cold front dominated the synoptic scenario. A strong convergence area (southwesterly and easterly flow) was observed over the Italian Po valley and maxima of precipitation were recorded along the Italian side of the Alps. The FBI time series indicates that, in general, the forecast slightly overestimates precipitation amounts.

5. Discussion and conclusions

In this paper two approaches to precipitation verification are discussed. Precipitation forecasts are usually validated using SYNOP data available in real time via the GTS. Their spatial distribution is quite irregular, with areas of intense coverage and areas of very little sampling. Verification against such data implies an interpolation of model fields to station location. Representativity becomes an important issue, and its relevance increases if verification over complex terrain is carried out.

The second approach uses a high-resolution observing network to reconstruct an observed field on the model grid. The technique consists of assigning each high-resolution observation to a grid box. All the observations within the same grid box are averaged and the mean value is assigned to the grid point. Different averaging techniques have been tested, but sensitivity studies have shown little to no difference to the averaging techniques used.

The high-resolution data are a subset of the MAP dataset and cover a period of 70 days between September and November 1999. ETS, TSS, and FBI have been used to assess the skill of the forecasting system.

The model overestimation of precipitation amount is a common feature of the two verification approaches. Overestimation is smaller when the precipitation forecast is compared with USO and, interestingly, FBI does not seem to change substantially between the lowest and the higher categories. On the contrary, validation against GTS-SO shows overestimation is larger for smaller amounts of precipitation. An accurate analysis of the period under examination shows high variability of the observed field with small and localized precipitation events. The upscaling procedure takes away part of the observed small-scale variability, therefore, comparing better to the model forecast.

Both ETS and TSS have shown higher values when the model is verified against USO, as opposed to verification against GTS-SO, for the forecast ranges considered in the study. Another interesting aspect shown by both ETS and TSS is the better performance of the model when forecasting rainfall in the 1–2 mm (24 h)−1 thresholds. Extreme thresholds always score worse.

Similar results have been obtained for the smaller area centered over the Alps. The question of how representative stations are becomes crucial in this case because of the complexity of the terrain. We believe the upscaling technique is actually able to produce a valuable alternative for verifying the model. The averaging method used reduces the small-scale variability that the model is unable to forecast and, at the same time, the averaged value retains some information on the complexity of the terrain. FBI has shown overestimation in all categories for both verification procedures, but they are smaller for USO. One relevant aspect of the verification against USO is that ETS also has large values for large thresholds of precipitation, which leads us to conclude that the model has some skill in forecasting such events in the Alpine region.

The overestimation of the observed precipitation noted in this study does not contradict the hypothesis in GL that orographic precipitation is underestimated. It must be taken into account that the chosen areas in the two studies are different, the periods analyzed in GL are those defined as standard seasons, and the two datasets do not necessarily have similar climatological frequencies.

Time series of FBI using a 7-day moving sample for small precipitation thresholds [0.1 and 1.0 mm (24 h)−1] have been shown for verification against USO. A spike around the middle of October 1999 indicates a general loss of skill of the model. The synoptic analysis has revealed that an inconsistent forecast and an overestimation of the precipitation for the only rainy day of the period has led to the spike in the FBI trend.

We feel that the use of high-resolution observations addresses the problem of representative stations correctly. The upscaling technique is designed to reconstruct an observed precipitation field smoothing small-scale variability that general circulation models are not able to simulate yet. The paper does not discuss in any great length possible errors associated with the upscaling technique. More work is needed in this direction, as well as a complete discussion on the statistical properties of the observation dataset. In particular, a more detailed investigation on climatological frequency would be welcome, but historical data are not available for the two areas considered in the present paper.

The improvement in the model behavior when verified against USO emphasizes the need for an analysis of precipitation. The differences between the two verification approaches shows a large contribution for the variability within the grid boxes; therefore, great care must be taken when assessing a forecasting model to GTS-SO, as forecast errors could be due to “representativity” problems. A better understanding of weaknesses and strengths of a forecasting system would be gained if model spatial scales were compared with similar observed scales.

Acknowledgments

We thank the MAP Data Centre for providing datasets of precipitation used in the study. We thank the reviewers for their careful revisions, which helped to improve the manuscript. Moreover, useful discussions with one of the reviewers are acknowledged.

REFERENCES

  • Binder, P., and Coauthors. 1996: MAP Mesoscale Alpine Programme design proposal. MAP Programme Office, 77 pp. [Available from MAP Programme Office c/0 Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Bougeault, P., and Coauthors. 1998: MAP Mesoscale Alpine Programme science plan. MAP Programme Office, 64 pp. [Available from MAP Programme Office c/0 Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Frustaci, G., and Coauthors. 2000: ITAMAP: The data set of non-GTS surface hourly data gathered during the MAP field phase (Sep–Nov 1999): Version 1.2, September 2000. MAP Programme Office, 64 pp. [Available from MAP Programme Office c/o Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Ghelli, A., and Lalaurette F. , 2000: Verifying precipitation forecasts using upscaled observations. ECMWF Newsletter, Vol. 87, 9–17.

  • Gregory, D., Morcrette J-J. , Jakob C. , Beljaars A. , and Stockdale T. , 2000: Revision of convection, radiation and cloud schemes in the ECMWF Integrated Forecasting System. Quart. J. Roy. Meteor. Soc., 126 , 16851710.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hirter, H., and Richner H. , 1996: MAP Mesoscale Alpine Programme data centre description. MAP Programme Office, 77 pp. [Available from MAP Programme Office c/o Swiss Meteorological Institute, Krähbülstrasse 58, CH-8044 Zürich, Switzerland.].

    • Search Google Scholar
    • Export Citation
  • Jakob, C., and Klein S. A. , 2000: A parameterisation of the effects of cloud and precipitation overlap for the use in general-circulation models. Quart. J. Roy. Meteor. Soc., 126 , 25252544.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jakob, C., and Coauthors. 2000: The IFS cycle CY21r4 made operational in October 1999. ECMWF Newsletter, Vol. 87, 2–9.

  • Karl, T. R., Wang W-C. , Schlesinger M. E. , Knight R. W. , and Portman D. , 1990: A method of relating general Circulation model simulated climate to the observed local climate. Part I: Seasonal statistics. J. Climate, 3 , 10531079.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Katz, W. R., and Murphy A. H. , 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

  • Kim, J-W., Chang J-T. , Baker N. L. , Wilks D. S. , and Gates W. L. , 1984: The statistical problem of climate inversion: Determination of the relationship between local and large-scale climate. Mon. Wea. Rev., 112 , 20692077.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and Buizza R. , 2001: Quantitative precipitation forecasts over the United States by the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 129 , 638663.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., . 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Ritchie, H., Temperton C. , Simmons A. J. , Hortal M. , Davies T. , Dent D. , and Hamrud M. , 1995: Implementation of the semi-Lagrangian method in a high resolution version of the ECMWF forecast model. Mon. Wea. Rev., 123 , 489514.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting, 5 , 570575.

  • Skelly, W. C., and Henderson-Sellers A. , 1996: Grid box or grid point: What type of data do GCMs deliver to climate impacts researchers? Int. J. Climatol., 16 , 10791086.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simmons, A. J., Burridge D. M. , Jarraud M. , Girard C. , and Wenge W. , 1989: The ECMWF medium-range prediction models: Development of the numerical formulations and the impact of increased resolution. Meteor. Atmos. Phys., 40 , 2860.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Teixeira, J., 1999: The impact of increased boundary layer resolution on the ECMWF forecast system. ECMWF Tech. Memo. 268, 55 pp.

  • Teixeira, J., . 2000: Boundary layer clouds in large scale atmospheric models: Cloud scheme and numerical aspects. Ph.D. dissertation, University of Lisbon, 190 pp. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

    • Search Google Scholar
    • Export Citation
  • Tiedtke, M., 1989: A comprehensive mass flux scheme for cumulus parameterization in large-scale models. Mon. Wea. Rev., 117 , 17791800.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tiedtke, M., . 1993: Representation of clouds in large-scale models. Mon. Wea. Rev., 121 , 30403061.

  • Untch, A., Simmons A. J. , Hortal M. , and Jakob C. , 1999: Increased stratospheric resolution in the ECMWF forecasting system. Proc. SODA (Satellite Ozone Data Assimilation) Workshop on Chemical Data Assimilation, No. 188, De Bilt, Netherlands, Royal Netherlands Meteorological Institute, 45–52.

    • Search Google Scholar
    • Export Citation
  • Wigley, T. M. L., and Santer B. D. , 1990: Statistical comparison of spatial fields in model validation, perturbation, and predictability experiments. J. Geophys. Res., 95 ((D1),) 851865.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Wilson, L. L., and Lettenmaier D. P. , 1992: A hierarchical stochastic model of large-scale atmospheric circulation patterns and multiple station daily precipitation. J. Geophys. Res., 97 , 27912809.

    • Crossref
    • Search Google Scholar
    • Export Citation

Fig. 1.
Fig. 1.

Station density (grayscale) for 0.5° × 0.5° grid boxes, averaged over the whole period under investigation. The rectangle defines the area selected for the MAP-ALPS experiment

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 2.
Fig. 2.

Distribution of the 24-h accumulated precipitation on 20 Sep 1999 (0600 UTC) as observed from (a) the high-resolution network stations and (b) after the upscaling technique has been applied. (c) The same field as observed from the GTS SYNOP stations

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 3.
Fig. 3.

USO (solid line) and GTS-SO (dashed line) cumulative precipitation [mm (24 h)−1] distributions. The distributions have been normalized by their respective totals

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 4.
Fig. 4.

Mean observed 24-h accumulated precipitation for each forecast category for the period 8 Sep–16 Nov 1999 plotted against the forecast categories. The forecasts are range (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 5.
Fig. 5.

FBI for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line for USO and dashed line for GTS-SO

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 6.
Fig. 6.

ETS for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 7.
Fig. 7.

TSS for the period 8 Sep–16 Nov 1999. Forecast ranges are (a) t + 42 h and (b) t + 66 h. Solid line is for USO and dashed line is for GTS-SO

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 8.
Fig. 8.

Observed distribution of precipitation for seven closed classes for the MAP-ALPS experiment

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 9.
Fig. 9.

As in Fig. 5a but for the MAP-ALPS experiment (forecast range t + 42 h)

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 10.
Fig. 10.

As in Fig. 6a but for the MAP-ALPS experiment (forecast range t + 42 h)

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 11.
Fig. 11.

As in Fig. 7a but for the MAP-ALPS experiment (forecast range t + 42 h)

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 12.
Fig. 12.

Time series of FBI (weekly moving average) for the 0.1-mm (top line) and 1-mm (bottom line) threshold. The forecast range is t + 42 h

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 13.
Fig. 13.

Forecast (color shaded) and upscaled observation (numbers) for the 24-h cumulative precipitation field at 0600 UTC 14 Oct 1999. The forecast is relative to 12 Oct 1999, range t + 42 h

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 14.
Fig. 14.

IR satellite image at 0300 UTC 20 Sep 1999

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Fig. 15.
Fig. 15.

Observed 24-h accumulated precipitation at 0600 UTC averaged over the period 15–30 Sep 1999 superimposed onto the 24-h accumulated precipitation (range t + 42 h; color shaded). The forecast field is an average of all the t + 42 h forecasts verifying between 15 and 30 Sep. Shading is as in the legend

Citation: Weather and Forecasting 17, 2; 10.1175/1520-0434(2002)017<0238:VOPFOT>2.0.CO;2

Table 1.

List of acronyms

Table 1.
Table 2.

Contingency table

Table 2.
Save