Editorial Type:
Article
Article Type:
Research Article

Analysis of the Relative Operating Characteristic and Economic Value Using the LAPS Ensemble Prediction System in Taiwan

Hui-Ling Chang Central Weather Bureau, Taipei, and Department of Atmospheric Sciences, National Central University, Jhong-Li, Taiwan

Search for other papers by Hui-Ling Chang in
Current site
Google Scholar
PubMed Close
,
Shu-Chih Yang Department of Atmospheric Sciences, National Central University, Jhong-Li, Taiwan

Search for other papers by Shu-Chih Yang in
Current site
Google Scholar
PubMed Close
,
Huiling Yuan School of Atmospheric Sciences, and Key Laboratory of Mesoscale Severe Weather/Ministry of Education, Nanjing University, and Jiangsu Collaborative Innovation Center for Climate Change, Nanjing, China

Search for other papers by Huiling Yuan in
Current site
Google Scholar
PubMed Close
,
Pay-Liam Lin Department of Atmospheric Sciences, National Central University, Jhong-Li, Taiwan

Search for other papers by Pay-Liam Lin in
Current site
Google Scholar
PubMed Close
, and
Yu-Chieng Liou Department of Atmospheric Sciences, National Central University, Jhong-Li, Taiwan

Search for other papers by Yu-Chieng Liou in
Current site
Google Scholar
PubMed Close
Print Publication:
01 May 2015
DOI:
https://doi.org/10.1175/MWR-D-14-00189.1
Page(s):
1833–1848
Restricted access

Abstract

Measurement of the usefulness of numerical weather prediction considers not only the forecast quality but also the possible economic value (EV) in the daily decision-making process of users. Discrimination ability of an ensemble prediction system (EPS) can be assessed by the relative operating characteristic (ROC), which is closely related to the EV provided by the same forecast system.

Focusing on short-range probabilistic quantitative precipitation forecasts (PQPFs) for typhoons, this study demonstrates the consistent and strongly related characteristics of ROC and EV based on the Local Analysis and Prediction System (LAPS) EPS operated at the Central Weather Bureau in Taiwan. Sensitivity experiments including the effect of terrain, calibration, and forecast uncertainties on ROC and EV show that the potential EV provided by a forecast system is mainly determined by the discrimination ability of the same system. The ROC and maximum EV (EVmax) of an EPS are insensitive to calibration, but the optimal probability threshold to achieve the EVmax becomes more reliable after calibration. In addition, the LAPS ensemble probabilistic forecasts outperform deterministic forecasts in respect to both ROC and EV, and such an advantage grows with increasing precipitation intensity. Also, even without explicitly knowing the cost–loss ratio, one can still optimize decision-making and obtain the EVmax by using ensemble probabilistic forecasts.

Corresponding author address: Dr. Shu-Chih Yang, Department of Atmospheric Sciences, National Central University, Jhong-Li, 32001, Taiwan. E-mail: shuchih.yang@gmail.com

Abstract

Measurement of the usefulness of numerical weather prediction considers not only the forecast quality but also the possible economic value (EV) in the daily decision-making process of users. Discrimination ability of an ensemble prediction system (EPS) can be assessed by the relative operating characteristic (ROC), which is closely related to the EV provided by the same forecast system.

Focusing on short-range probabilistic quantitative precipitation forecasts (PQPFs) for typhoons, this study demonstrates the consistent and strongly related characteristics of ROC and EV based on the Local Analysis and Prediction System (LAPS) EPS operated at the Central Weather Bureau in Taiwan. Sensitivity experiments including the effect of terrain, calibration, and forecast uncertainties on ROC and EV show that the potential EV provided by a forecast system is mainly determined by the discrimination ability of the same system. The ROC and maximum EV (EVmax) of an EPS are insensitive to calibration, but the optimal probability threshold to achieve the EVmax becomes more reliable after calibration. In addition, the LAPS ensemble probabilistic forecasts outperform deterministic forecasts in respect to both ROC and EV, and such an advantage grows with increasing precipitation intensity. Also, even without explicitly knowing the cost–loss ratio, one can still optimize decision-making and obtain the EVmax by using ensemble probabilistic forecasts.

Corresponding author address: Dr. Shu-Chih Yang, Department of Atmospheric Sciences, National Central University, Jhong-Li, 32001, Taiwan. E-mail: shuchih.yang@gmail.com
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133, 1076–1097, doi:10.1175/MWR2905.1.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., M. Leutbecher, and L. Isaksen, 2008: Potential use of an ensemble of analyses in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 134, 2051–2066, doi:10.1002/qj.346.

    • Search Google Scholar
    • Export Citation

  • Chang, H. L., H. Yuan, and P. L. Lin, 2012: Short-range (0-12 h) PQPFs from time-lagged multimodel ensembles using LAPS. Mon. Wea. Rev., 140, 1496–1516, doi:10.1175/MWR-D-11-00085.1.

    • Search Google Scholar
    • Export Citation

  • Dalcher, A., E. Kalnay, and R. N. Hoffman, 1988: Medium range lagged average forecasts. Mon. Wea. Rev., 116, 402–416, doi:10.1175/1520-0493(1988)116<0402:MRLAF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Donner, L. J., 1988: An initialization for cumulus convection in numerical weather prediction models. Mon. Wea. Rev., 116, 377–385, doi:10.1175/1520-0493(1988)116<0377:AIFCCI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Ebert, E., 2001: Ability of a poor man's ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129, 2461–2480, doi:10.1175/1520-0493(2001)129<2461:AOAPMS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Fang, X., and Y.-H. Kuo, 2013: Improving ensemble-based quantitative precipitation forecasts for topography-enhanced typhoon heavy rainfall over Taiwan with a modified probability-matching technique. Mon. Wea. Rev., 141, 3908–3932, doi:10.1175/MWR-D-13-00012.1.

    • Search Google Scholar
    • Export Citation

  • Fang, X., Y.-H. Kuo, and A. Wang, 2011: The impacts of Taiwan topography on the predictability of Typhoon Morakot's record-breaking rainfall: A high-resolution ensemble simulation. Wea. Forecasting, 26, 613–633, doi:10.1175/WAF-D-10-05020.1.

    • Search Google Scholar
    • Export Citation

  • Fujita, T., D. Stenstrud, and D. C. Dowell, 2007: Surface data assimilation using an ensemble filter approach with initial condition and model physics uncertainties. Mon. Wea. Rev., 135, 1846–1868, doi:10.1175/MWR3391.1.

    • Search Google Scholar
    • Export Citation

  • Gourley, J. J., J. Zhang, R. A. Maddox, C. M. Calvert, and K. W. Howard, 2001: A real-time precipitation monitoring algorithm—Quantitative Precipitation Estimation Using Multiple Sensors (QPE-SUMS). Preprints, Symp. on Precipitation Extremes: Prediction, Impacts, and Responses, Albuquerque, NM, Amer. Meteor. Soc., 57–60.

  • Grell, G., J. Dudhia, and D. Stauffer, 1995: A description of the fifth-generation Penn State/NCAR Mesoscale Model (MM5).NCAR Tech Note NCAR/TN-398+STR, 138 pp.

  • Hamill, T. M., 1997: Reliability diagrams for multicategory probabilistic forecasts. Wea. Forecasting, 12, 736–741, doi:10.1175/1520-0434(1997)012<0736:RDFMPF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550–560, doi:10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and J. Juras, 2006: Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., 132, 2905–2923, doi:10.1256/qj.06.25.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2006: Reforecasts: An important dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87, 33–46, doi:10.1175/BAMS-87-1-33.

    • Search Google Scholar
    • Export Citation

  • Hartmann, D. L., R. Buizza, and T. N. Palmer, 1995: Singular vectors: The effect of spatial scale on linear growth of disturbances. J. Atmos. Sci., 52, 3885–3894, doi:10.1175/1520-0469(1995)052<3885:SVTEOS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Harvey, L. O., Jr., K. R. Hammond, C. M. Lusk, and E. F. Mross, 1992: The application of signal detection theory to weather forecasting behavior. Mon. Wea. Rev., 120, 863–883, doi:10.1175/1520-0493(1992)120<0863:TAOSDT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Heckley, W. A., 1985: Systematic errors of the ECMWF operational forecasting model in tropical regions. Quart. J. Roy. Meteor. Soc., 111, 709–738, doi:10.1002/qj.49711146904.

    • Search Google Scholar
    • Export Citation

  • Hsiao, L.-F., and Coauthors, 2013: Ensemble forecasting of typhoon rainfall and floods over a mountainous watershed in Taiwan. J. Hydrol., 506, 55–68, doi:10.1016/j.jhydrol.2013.08.046.

    • Search Google Scholar
    • Export Citation

  • Hsu, W.-R., and A. H. Murphy, 1986: The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2, 285–293, doi:10.1016/0169-2070(86)90048-8.

    • Search Google Scholar
    • Export Citation

  • Jian, G.-J., S.-L. Shieh, and J. McGinley, 2003: Precipitation simulation associated with Typhoon Sinlaku (2002) in Taiwan area using the LAPS diabatic initialization for MM5. Terr. Atmos. Oceanic Sci., 14, 261–288.

    • Search Google Scholar
    • Export Citation

  • Jolliffe, I. T., and D. B. Stephenson, 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. Wiley, 254 pp.

  • Kalnay, E., and A. Dalcher, 1987: Forecasting forecast skill. Mon. Wea. Rev., 115, 349–356, doi:10.1175/1520-0493(1987)115<0349:FFS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Katz, R. W., and A. H. Murphy, 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

  • Lang, S. T. K., M. Leutbecher, and S. C. Jones, 2012: Impact of perturbation methods in the ECMWF ensemble prediction system on tropical cyclone forecasts. Quart. J. Roy. Meteor. Soc., 138, 2030–2046, doi:10.1002/qj.1942.

    • Search Google Scholar
    • Export Citation

  • Mason, I. B., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30, 291–303.

  • Mason, S. J., and N. E. Graham, 1999: Conditional probabilities, relative operating characteristics, and relative operating levels. Wea. Forecasting, 14, 713–725, doi:10.1175/1520-0434(1999)014<0713:CPROCA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73–119, doi:10.1002/qj.49712252905.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1977: The value of climatological, categorical and probabilistic forecasts in the cost–loss ratio situation. Mon. Wea. Rev., 105, 803–816, doi:10.1175/1520-0493(1977)105<0803:TVOCCA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281–293, doi:10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., and M. Ehrendorfer, 1987: On the relationship between the accuracy and value of forecasts in the cost–loss ratio situation. Wea. Forecasting, 2, 243–251, doi:10.1175/1520-0434(1987)002<0243:OTRBTA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Mylne, K. R., 1999: The use of forecast value calculations for optimal decision making using probabilistic forecasts. Preprints, 17th Conf. on Weather Analysis and Forecasting, Denver, CO, Amer. Meteor. Soc., 235–239.

  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126, 649–667, doi:10.1002/qj.49712656313.

    • Search Google Scholar
    • Export Citation

  • Richardson, D. S., 2001: Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size. Quart. J. Roy. Meteor. Soc., 127, 2473–2489, doi:10.1002/qj.49712757715.

    • Search Google Scholar
    • Export Citation

  • Roebber, P. J., and L. E. Bosart, 1996: The complex relationship between forecast skill and forecast value: A real-world analysis. Wea. Forecasting, 11, 544–559, doi:10.1175/1520-0434(1996)011<0544:TCRBFS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/arw_v3_bw.pdf.]

  • Su, S. H., H. C. Kuo, L. H. Hsu, and Y. T. Yang, 2012: Temporal and spatial characteristics of typhoon extreme rainfall in Taiwan. J. Meteor. Soc. Japan, 90, 721–736, doi:10.2151/jmsj.2012-510.

    • Search Google Scholar
    • Export Citation

  • Swets, J. A., 1973: The relative operating characteristic in psychology. Science, 182, 990–1000, doi:10.1126/science.182.4116.990.

  • Swets, J. A., 1979: ROC analysis applied to the evaluation of medical imaging techniques. Invest. Radiol., 14, 109–121, doi:10.1097/00004424-197903000-00002.

    • Search Google Scholar
    • Export Citation

  • Swets, J. A., and R. M. Pickett, 1982: Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press, 253 pp.

  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74, 2317–2330, doi:10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 3297–3319, doi:10.1175/1520-0493(1997)125<3297:EFANAT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Toth, Z., Y. Zhu, and T. Marchok, 2001: The use of ensembles to identify forecasts with small and large uncertainty. Wea. Forecasting, 16, 463–477, doi:10.1175/1520-0434(2001)016<0463:TUOETI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Wandishin, M. S., and H. E. Brooks, 2002: On the relationship between Clayton’s skill score and expected value for forecasts of binary events. Meteor. Appl., 9, 455–459, doi:10.1017/S1350482702004085.

    • Search Google Scholar
    • Export Citation

  • Wei, M., Z. Toth, R. Wobus, and Y. Zhu, 2008: Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system. Tellus, 60A, 62–79, doi:10.1111/j.1600-0870.2007.00273.x.

    • Search Google Scholar
    • Export Citation

  • Whitaker, J. S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev., 126, 3292–3302, doi:10.1175/1520-0493(1998)126<3292:TRBESA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

  • Wilks, D. S., and T. M. Hamill, 1995: Potential economic value of ensemble-based surface weather forecasts. Mon. Wea. Rev., 123, 3565–3575, doi:10.1175/1520-0493(1995)123<3565:PEVOEB>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Yeh, T. C., and R. L. Elsberry, 1993: Interaction of typhoons with the Taiwan orography. Part I: Upstream track deflections. Mon. Wea. Rev., 121, 3193–3212, doi:10.1175/1520-0493(1993)121<3193:IOTWTT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Yuan, H., J. A. McGinley, P. J. Schultz, C. J. Anderson, and C. Lu, 2008: Short-range precipitation forecasts from time-lagged multimodel ensembles during the HMT-West-2006 campaign. J. Hydrometeor., 9, 477–491, doi:10.1175/2007JHM879.1.

    • Search Google Scholar
    • Export Citation

  • Zhu, Y., 2005: Ensemble forecast: A new approach to uncertainty and predictability. Adv. Atmos. Sci., 22, 781–788, doi:10.1007/BF02918678.

    • Search Google Scholar
    • Export Citation

  • Zhu, Y., Z. Toth, R. Wobus, D. S. Richardson, and K. Mylne, 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 73–83, doi:10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 4804 4269 51
PDF Downloads 419 63 5