1. Introduction
The National Weather Service (NWS) River Forecast Centers (RFCs) have been using the Meteorological Ensemble Forecast Processor (MEFP; Schaake et al. 2007; Wu et al. 2011; NWS 2016, 2017) to produce bias-corrected ensemble precipitation forecasts for over a decade in support of the Hydrologic Ensemble Forecast Service (HEFS; Demargne et al. 2014) (see the appendix for a list of acronyms and abbreviations). The MEFP precipitation forecasts are generated based on single-valued forecasts such as the ensemble mean precipitation forecast from the Global Ensemble Forecast System (GEFS; Toth and Kalnay 1997; Cui et al. 2012). The operational experience with the MEFP indicates that, whereas the MEFP-generated precipitation ensembles are unbiased in the mean when verified against all amounts of precipitation (Brown et al. 2014), they tend to underforecast heavy-to-extreme precipitation rather significantly (Georgakakos et al. 2015; Whitin and He 2015; Seo et al. 2019). Consequently, the HEFS streamflow forecasts tend to be unconditionally unbiased but conditionally biased which limits their utility and value for flood and inflow forecasting (Jozaghi et al. 2021). To address the above, Kim et al. (2025) proposed conditional bias-penalized regression (CBPR) with magnitude-dependent optimization of the regression coefficient for the MEFP as an alternative to ordinary least squares regression (OLSR). Conditional errors may be differentiated into type I and type II as described below (Kim et al. 2025). Type I error, which is associated with false alarms, is defined as
The purpose of this work is to apply CBPR in the MEFP, validate the resulting CBPR-aided MEFP via hindcasting experiments, and assess the margins of improvement in ensemble precipitation hindcasts for heavy-to-extreme precipitation events, or large events for short, and in ensemble streamflow hindcasts forced by the CBPR-aided MEFP hindcasts. This paper differs from Kim et al. (2025) in that, whereas Kim et al. (2025) deal with the theory and its proof-of-concept evaluation for potential application in the MEFP, this paper applies the theory to the MEFP, describes the resulting prototype CBPR-aided MEFP, and evaluates its impact on precipitation and streamflow forecasts. The main contributions of this work are comparative verification of precipitation hindcasts from the CBPR-aided MEFP for large events, comparative verification of streamflow hindcasts forced by the above precipitation hindcasts, improved understanding of the parametric uncertainty in the CBPR-aided MEFP for large events, and advances in understanding of the type I versus type II error trade-off for the CBPR-aided MEFP in hydrologic ensemble forecasting.
2. Events, locations, data, and tools used
The large events used in this work are those that produced the largest 2-day mean areal precipitation (MAP) in the 20-yr period of 2000–19 at the headwater basins in the HEFS testbed (Lee et al. 2020; Kim et al. 2025). We also considered using the largest 1-day, rather than 2-day, MAP which would include more convective events in the warm season for which the GEFS is less skillful (Sukovich et al. 2014; Kim et al. 2025). The hydrologic models for headwater basins, such as the Sacramento Soil Moisture Accounting Model (SAC; Burnash et al. 1973) and unit hydrograph (UHG; Chow et al. 1988) used in the NWS’s Community Hydrologic Prediction System (CHPS; Roe et al. 2010), are generally less skillful for flashy events. Hence, the use of the largest 1-day MAP would increase not only future input uncertainty but hydrologic uncertainty as well (Krzysztofowicz 1999; Seo et al. 2006) which is not very favorable for comparative evaluation. Using the largest 3-day or longer-duration MAP, on the other hand, would be more favorable to the GEFS and the current operational hydrologic models in the CHPS but is less likely to offer a very robust comparative assessment of the CBPR-aided MEFP. Hence, the choice of the largest 2-day MAP for event selection is a practical compromise. The precipitation events thus selected include Hurricanes Barry (July 2019), Irene (August 2011), Irma (August–September 2017), and Harvey (August 2017) and Tropical Storms Lee (September 2011) and Allison (June 2001), as well as notable mesoscale convective complexes and atmospheric river events (Kim 2024).
For the same depth of precipitation, the return period for MAP is larger than that for point precipitation (Pavlovic et al. 2016). Hence, in terms of precipitation frequency (Perica et al. 2018), the largest 2-day MAP events in a 20-yr period are nominally larger than 20-yr 48-h (point) precipitation. In north Texas, hydraulic structures for flood mitigation, such as open channels and culverts, are designed for 25–100-yr 24-h (point) precipitation (North Central Texas Council of Governments 2009). Similar design criteria are used elsewhere. Hence, the precipitation events used in this work, a number of which exceed 100-yr 24-h (point) precipitation, represent high-impact events that would require flood mitigation infrastructure.
Selecting streamflow locations requires particular care. For some locations, hydrologic uncertainty may be too large to discern significant improvement or deterioration in future input uncertainty. Such situations may occur where the movement and storage of water are heavily controlled but not well modeled, unknown (and hence unmodeled) hydrologic processes are at work, the hydrologic models lack calibration, large uncertainties exist in the initial conditions (ICs), or unknown sources or sinks exist in the hydrologic system. Assessing hydrologic uncertainty requires hydrologic analysis and uncertainty decomposition which is well beyond the scope of this work. Instead, we selected among the candidate streamflow locations in each RFC’s service area those that report the largest peak mean daily flows (QME) from the largest 2-day MAP events. The above approach amounts to selecting basins that are more likely to produce larger surface runoff (i.e., have larger runoff ratio) and that are likely to route the surface runoff more quickly to the basin outlet (i.e., have smaller time to peak). Whereas the locations selected in this way are not likely to fully represent the diversity of the hydrologic response of the candidate basins, they are likely to include catchments that are more responsive to large precipitation and hence better suited for comparative assessment of the CBPR-aided MEFP. Figure 1 shows the 82 MAP basins and 48 stream gauge locations used in this work, the latter of which are identified by the five-character location names used in the NWS. If the MAP basin is divided into multiple subbasins, the NWS identifier has additional characters to differentiate the relative locations of the subbasins, whereas the first five characters identify the basin outlet to which the subbasins drain. For example, the letters “L,” “M,” “U,” and “G” within the additional characters indicate lower, middle, upper, and glacier subbasins, respectively. In some cases, the additional characters indicate the forecast group (typically the name of the river) to which the location belongs. Throughout this paper, we abbreviate the RFC names with their first two characters and use the NWS identifiers for the MAP basins or subbasins for brevity. For example, CN/NFDC1HUF refers to the upper subbasin that drains to NFDC1 (North Fork of the American River at North Fork Dam) located in the service area of the California–Nevada RFC (CNRFC).
Eighty-two MAP basins and 48 streamflow locations used in this work. The latter are identified by the five-character NWS location name.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
The hindcast period of 2000–19 is chosen in accordance with the HEFS baseline validation practices in the NWS (Lee et al. 2018, 2020). The single-valued precipitation hindcasts used as input to the MEFP are the ensemble mean from the GEFS, version 12, reforecast dataset (Guan et al. 2022; Hamill et al. 2022). The MAP data used for verification of precipitation hindcasts are the MAP time series produced by the respective RFCs. The QME data used for verification of streamflow hindcasts are from the U.S. Geological Survey. The CHPS configurations used to generate ensemble streamflow hindcasts from ensemble precipitation hindcasts are from the respective RFCs and reflect their operational setup. The MEFP parameters are estimated using the MEFP Parameter Estimation (MEFPPE) Program (NWS 2016). The CBPR regression coefficients λ are estimated using the CBPR Parameter Estimation Program (Kim 2024). The primary verification tool used is the Ensemble Verification System (Brown et al. 2010).
3. Methods
This section describes the CBPR-aided MEFP, the selection of the CEs, and the design of the validation experiments. To describe the CBPR-aided MEFP, it is necessary to describe the MEFP. Below, we provide a brief conceptual description of the MEFP (Schaake et al. 2007; Wu et al. 2011; NWS 2016, 2017) and how the CBPR-aided MEFP differs.
a. CBPR-aided MEFP
Given the single-valued forecast in real time, the MEFP samples synthetic ensemble traces from the conditional cumulative distribution function (CDF) of observed precipitation for each CE (Wu et al. 2011). Figure 2 shows the complete set of CEs used for the CBPR-aided MEFP in this work. Contrary to the name, CEs are not events in the hydrometeorological or probabilistic sense but forecast subhorizons of varying durations and, hence, are equivalent to support in geostatistics. The MEFP then uses the Schaake shuffle (SS; Clark et al. 2004; NWS 2017) to provide the synthetic traces of precipitation with the spatiotemporal variability in the ordinal sense of historically observed precipitation over the forecast horizon.
CEs used for the CBPR-aided MEFP. The solid red dots indicate the CBPR-active CEs.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
The MEFP starts the SS with the historical traces of observed precipitation at the input time step of the hydrologic models (usually 6 h) valid over the forecast subhorizon of the CE (step 2 in Fig. 3). The traces are then aggregated to the duration of the CE and sorted in the ascending order (step 3 in Fig. 3). The MEFP samples synthetic traces from the conditional CDF associated with the CE (step 4 in Fig. 3) which are sorted in the ascending order. Each of the sorted, or rank-ordered, synthetic traces is then assigned the historical year in which the observed precipitation of the same rank occurred (step 5 in Fig. 3). For example, if the CE covers 0–96 h of lead time and the largest historically observed precipitation in this 4-day period occurred in 2017 for the basin of interest, the largest synthetic trace sampled from the conditional CDF is assigned an ensemble member identifier of “2017.” The SS then updates each trace from the latest shuffling of the ensemble forecast of 6-h precipitation (recall that this is initially the climatological ensemble forecast) by multiplicatively adjusting the 6-h precipitation uniformly within the CE such that the adjusted total precipitation is the same as the synthetic trace sampled from the conditional CDF (step 6 in Fig. 3). The above process is repeated for all CEs beginning with the CE with the smallest Pearson product–moment correlation (correlation for short hereafter) ρ in the bivariate standard normal space between the positive single-valued QPF and the positive observed precipitation and ending with the CE with the largest correlation (step 8 in Fig. 3). The above operation cumulatively reflects the temporal patterns of granularity in 6-h precipitation that may be gleaned from the successive adjustments while exploiting all available scale-dependent skill in the input single-valued QPF.
Summary description of the MEFP. Different colors track the individual ensemble members through the sequence of the steps depicted.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
Once the above steps are completed for all CEs, one has replaced the initial historical traces of observed precipitation over the forecast horizon with the same number of ensemble traces of synthetic precipitation which reflect the multiscale predictive skill in the input single-valued QPF over different parts of the forecast horizon and share the same rank correlation and spatiotemporal variability in the ordinal sense with the climatological ensemble forecast. The scale-dependent predictive skill in the GEFS ensemble mean forecast enters into the MEFP through step 4 in Fig. 3, whereas all other steps are for stitching together the randomly sampled ensemble traces from the predictive distributions for different CEs over the forecast horizon.
b. Selection of canonical events
For the CBPR-aided MEFP, selecting CEs is a two-step process: 1) choose CEs for the MEFP and 2) choose CEs for CBPR among the CEs selected in step 1. The CEs chosen in step 2 are referred to as the CBPR-active CEs. The CEs in the MEFP should be chosen with care such that the collective skill in the GEFS ensemble mean forecast over the different forecast subhorizons is maximally utilized. For example, if the CEs consist only of very short durations, precipitation and precipitation forecasts associated with them will have very limited predictability and predictive skill, respectively, resulting in small ρ across the entire forecast horizon. In such a case, the MEFP ensemble forecasts are not likely to improve materially over the resampled climatological ensemble forecasts generated with the MEFP with ρ = 0 for all CEs, leaving much of the predictive skill that may exist in the GEFS ensemble mean forecast untapped. In this work, a single 67-CE configuration (see Fig. 2) was used for all RFCs at all times which largely encompasses all existing RFC-operational CE configurations (Kim 2024).
Once the CEs are selected for the MEFP, the CBPR-active CEs may be chosen based on the magnitude of CB inferred from the λ-versus-ρ relationship. For example, if the λ values differ little from ρ values for some CEs, little CB exists in the single-valued forecast over the forecast subhorizons associated with the CEs. There is hence little need to select such CEs as CBPR active. If, on the other hand, λ values differ significantly from ρ values, significant CB exists in the single-valued forecast over the CEs. Hence, such CEs should be selected as CBPR active. In practice, location-specific or seasonally varying selection of the CEs and CBPR-active CEs is impractical due to insufficient sample size and the risk of mis-stratification. For this reason, the current operational practice is to use a single set of CEs at all times for all locations within the RFC’s service area regardless of the hydroclimatological variations therein. In accordance with this practice, we used a single set of CBPR-active CEs in this work for all locations at all times. Selecting a large number of CBPR-active CEs not only increases computation for the optimization of λ but may also introduce unnecessary random variations from repeated sampling. If the number of CBPR-active CEs is too small, on the other hand, the CBPR results for the CBPR-active CEs may be overridden during the SS by the OLSR results that are associated with the non-CBPR-active CEs.
The order in which CBPR is applied among different CEs is tied to the order in which the SS operates, i.e., the CE with the smallest and largest ρ is processed first and last, respectively. In the current MEFP software, CBPR has no control over this order. It is hence important that, among the CBPR-active CE candidates, those associated with the largest ρ values are chosen across different stretches of the forecast horizon. In this way, the CBPR correction is less likely to be overridden by non-CBPR-active CEs during the SS. In Fig. 2, the solid red dots identify the CBPR-active CEs used in this work for all RFCs. Once the CEs are selected, the CBPR Parameter Estimation Program (Kim 2024) is run for all CBPR-active CEs to optimize λ by minimizing the conditional mean continuous ranked probability score (CRPS) (Hersbach 2000; Kim et al. 2025). The rest of the MEFP process is the same as when CBPR is not used (NWS 2016, 2017). For brevity, the RFC-operational MEFP and the CBPR-aided MEFP are referred to hereafter as the MEFP and the CBPR, respectively, unless the potential for confusion arises.
c. Design of validation experiments
For comparative evaluation, hindcasting experiments were designed and carried out for dependent and independent validation. Dependent validation is aimed at assessing the margin of improvement or deterioration by the CBPR for large events in the absence of parametric uncertainty arising from the variability of precipitation at various scales. For parameter estimation, all available data were used in the 2000–19 reforecast period within the 2-month window centered at the time of the largest 2-day MAP. Hindcasting and verification were then carried out for the 2 months of the particular year in which the largest 2-day MAP event occurred. The choice of a 2-month window is tied to the 61-day sampling window used operationally for the MEFP parameter estimation. The MEFPPE estimates the MEFP parameters valid for a specific day of the year using all forecast–observation pairs within the 61-day window centered at that day (NWS 2016). The 2-month window thus includes the large event selected in section 2 for each location along with any smaller events that may have occurred within the 2-month period.
For independent validation, 10-fold cross validation was carried out for the 2-month period corresponding to the 61-day sampling window centered at the day of the year of the largest 2-day MAP event. For example, if the largest 2-day MAP event occurred on 31 August–1 September of some year at some location, independent validation was performed for the location for the months of August and September only. For 10-fold cross validation, a 2-yr period was withheld for each fold from the 20-yr reforecast period for parameter estimation. Hindcasts were then generated for the withheld 2-yr period for validation. This process is repeated for all 10 folds, resulting in cross-validated precipitation and streamflow hindcasts valid for the 2-month window in every year of the 20-yr reforecast period. Table 1 summarizes the validation experiments carried out in this work. The resulting hindcasts thus allow the assessment of the margin of improvement or deterioration by the CBPR for 1) the largest 2-day MAP and other events that occurred within the 2-month period of the 61-day sampling window (referred to as 1-yr validation in Table 1) and 2) all events that occurred in the same 2-month window over the 20-yr reforecast period (referred to as 20-yr validation in Table 1). Note that 1-yr independent validation is based on the parameters estimated using only the single fold that excludes the 2-yr period in which the large event occurred, whereas 20-yr validation uses all 10 folds.
Summary of validation experiments. The asterisk indicates validation for the 2-month window centered at the time of the large event.
Large-sample cross validation of the HEFS hindcasts is a rather time-consuming process due to the interactive nature of the MEFPPE and the CHPS. In this work, we reduced the number of cross-validation locations by screening the candidate locations based on the assessment of parametric uncertainty. If the MEFP and CBPR parameters show little parametric uncertainty, dependent validation likely suffices. Hence, by comparing ρ and λ values associated with the fold that does not include the large event with those associated with all other folds, one may assess how the independent validation results may, or may not, differ from the dependent validation results before actually carrying out cross validation. The locations selected for cross validation in this work are those that are most significantly impacted by parametric uncertainty among all candidate locations. Hence, the locations selected likely offer the most stringent test for the CBPR.
Figure 4 shows examples of ρ and λ for the CBPR-active CEs in the 67-CE configuration (see Fig. 2) from the 10 different parameter estimation runs associated with the 10 folds. Table 2 shows the forecast subhorizons associated with the CBPR-active CEs. Fold 1 corresponds to the period of 2002–19 for parameter estimation (2000–01 withheld). Similarly, fold 2 corresponds to the combined period of 2000–01 and 2004–19 for parameter estimation (2002–03 withheld) and so forth. In Fig. 4, the data points are connected as lines to aid differentiation of fold-specific results. By visually examining the parameter values across all folds and recognizing which fold, if any, may stand out, one may identify the 2-yr period in which the large event occurred at that location. In Fig. 4, the blue and red lines represent ρ and λ values associated with the fold that excludes the large event-bearing 2-yr period, respectively, whereas the cyan and pink lines represent ρ and λ values associated with the nine folds that include the large event-bearing 2-yr period, respectively.
Examples of ρ and λ for the CBPR-active CEs estimated from 10 different parameter estimation runs associated with 10-fold cross validation. The title in each panel indicates the RFC name and the NWS identifier for the MAP basin or subbasin.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
CBPR-active CEs.
The top four panels in Fig. 4 show that, for GLOO2 and BOLI2, excluding the large event tends to reduce ρ and λ, an indication that the largest 2-day MAP event is more predictable than the rest of the events that occurred in the same 61-day window in the 20-yr reforecast period. For ORTW1G, on the other hand, the opposite is observed, indicating that the largest 2-day MAP event at this location is less predictable than the rest. For NFDC1HUF, the GEFS ensemble mean forecast shows the same level of predictive skill whether the largest 2-day MAP event was included or not. At many locations, the parametric uncertainties of ρ and λ exhibit little sensitivity to the inclusion or exclusion of the large event, similarly to NFDC1HUF. For such locations, one may expect the independent and dependent validation results to be similar. For this reason, attention is paid primarily to those locations that are most sensitive to the inclusion or exclusion of the large event. Based on the examination of ρ-versus-λ plots as well as the probability distributional parameters for all locations, a total of 16 locations were selected for independent validation for precipitation: AB/GLOO2, AP/CHLA2LWR, CB/OAWU1HMF, CN/NFDC1HUF, CN/SESC1HOF, LM/BSLL1, LM/GLML1, MA/MAWN4RMP, MB/SLDM8LWR, NC/BOLIS, NE/GTBM3SNE, NW/ORTW1G, OH/VRNI3, SE/MDLF1, and WG/HGTT2, WG/PICT2 where the first two characters identify the RFC followed by the NWS location identifier as explained in section 2.
The bottom four panels of Fig. 4 show ρ and λ for SESC1HOF, MAWN4RMD, MDLF1, and HGTT2 among the 16 locations selected for which the largest 2-day MAP events are due to atmospheric river and Hurricanes Irene, Irma, and Harvey, respectively. Note that the differences in ρ or λ between the large event including and excluding folds tend to be larger at these locations than those in the upper panels. One may hence expect the results from independent and dependent validation to differ significantly at the lower-panel locations. It is important to add, however, that if parametric uncertainty is large for λ, it tends to be large for ρ as well. Such strong collinearity between λ and ρ stems from the fact that they both represent the slope in slope-only linear regression in bivariate normal space which has minimal degrees of freedom. Hence, comparative performance between independent and dependent validation is not likely to differ as much as the parametric uncertainty in λ alone might suggest.
4. Results and discussion
This section presents the validation results for precipitation and streamflow and offers discussions.
a. Precipitation
The dependent validation results are presented first, followed by the independent validation results.
1) Dependent validation
Figure 5 shows the difference in CRPS between the MEFP and CBPR hindcasts, i.e., ΔCRPS = CRPSMEFP − CRPSCBPR, versus the common verifying observation of 24-h precipitation for days 2, 4, 6, 8, 10, and 12. Beyond day 12, ΔCRPS scatters around 0 across all ranges of the verifying observation. Note that CRPSMEFP and CRPSCBPR are not mean CRPS but CRPS of individual hindcasts. Hence, each data point in Fig. 5 represents a head-to-head comparison of the CBPR versus MEFP hindcast for the common verifying observation. In each plot, there are, on average, 4880 data points representing the same number of hindcast pairs for 24-h precipitation for 82 locations in 13 RFCs. A large positive ΔCRPS indicates a large improvement by the CBPR over the MEFP. A negative ΔCRPS indicates that the CBPR deteriorates accuracy relative to the MEFP.
Difference in CRPS between the MEFP and CBPR hindcasts for all locations vs the common verifying observation of 24-h precipitation for days 2, 4, 6, 8, 10, and 12.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
Figure 5 indicates that the larger the verifying observation is, the larger the margin of improvement by the CBPR over the MEFP tends to be; that the shorter the lead time is, the larger the improvement tends to be; that the CBPR tends to underperform the MEFP for smaller verifying observations; and that the performance of the CBPR and the MEFP becomes increasingly similar as the lead time increases. The above findings are wholly consistent with the CE-specific quasi-analytical results of Kim et al. (2025), an indication that the 67-CE configuration with 13 CBPR-active CEs (see Fig. 3) successfully translates the predictive distributions of CBPR into ensemble hindcasts via the SS.
Whereas Fig. 5 shows increasingly superior performance of the CBPR for increasingly larger amounts of precipitation, in some plots, there are one or two data points for which the CBPR underperforms the MEFP significantly. These data points are associated with the GEFS ensemble mean forecasts that are large overforecasts of relatively small verifying observations. Such exacerbation of type I error (i.e., false positive) by the CBPR-aided MEFP stems from the fact that the CBPR reduces type II error (i.e., false negative) at some expense of type I error. Occurrences of such rare but significant deterioration of type I error by the CBPR are in effect the price one pays for the greatly reduced type II error without having to improve the accuracy of the conditioning single-valued forecast by improving the GEFS. In certain applications, it may be desirable to reduce such rare but significant exacerbation of overforecasts by the CBPR. The above may be achieved by applying more stringent acceptance criteria as described in Kim et al. (2025) but at some expense of reduced performance for large amounts of precipitation.
(top) CRPSS and (bottom) RMSESS of the CBPR hindcasts for 24-h precipitation for days 1–14 for 82 locations in 13 RFCs.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
Because the climatology of observed precipitation varies in space, the precipitation amount represented by a conditioning percentile varies from location to location. In addition, the quality of the RFC-operational CE configuration and the MEFP parameters varies from RFC to RFC. For these reasons, Fig. 6 mainly serves to ascertain the relative conditional and unconditional performance of the CBPR in periods of large events rather than to assess how the relative performance of the CBPR may vary according to the absolute magnitude of the verifying observation. Figure 6 indicates that the CBPR improves over the MEFP for most locations not only conditionally but also unconditionally and that the margin of improvement is generally larger for heavier precipitation. The second observation above is consistent with the CE-specific quasi-analytical results (Kim et al. 2025). The first observation might appear inconsistent with Kim et al. (2025) who found slight deterioration in unconditional performance by the CBPR. Recall, however, that the event space for the 1-yr validation in this work is limited only to the large event-bearing 2-month periods. Hence, though described as “unconditional” to differentiate from the conditional results, the 0th-percentile results in Fig. 6 are in fact conditioned on the largest 2-day MAP event being included in the verifying observations. Because the CBPR tends to perform better for larger events, it is not very surprising that the CBPR outperforms the MEFP not only conditionally but also unconditionally when validated for the large event-bearing periods only. Figure 6 also indicates that, for the Northwest River Forecast Center (NWRFC), the relative performance of the CBPR perceptibly deteriorates over the last 4 days of the forecast horizon. The above observation suggests that adding a CBPR-active CE of 4-day duration for days 11–14 to the 67-CE configuration (see Fig. 3) may improve the performance of the CBPR. For the assessment of reliability and discrimination using the reliability diagram and relative operational characteristic curves for specific CEs, respectively, the reader is referred to Kim et al. (2025).
Many users of the HEFS products and services rely on ensemble mean forecasts to whom reduction in RMSE is of larger interest. The lower panels in Fig. 6 show that the RMSESS results are similar to the CRPSS results but somewhat more and less favorable to the CBPR at lower and higher thresholds, respectively. That the RMSESS is more favorable (or the CRPSS is less favorable) to the CBPR relative to the MEFP at lower thresholds is a reflection that, for smaller precipitation amounts, the ensemble quality of the MEFP hindcasts is relatively high. That the RMSESS is less favorable (or the CRPSS is more favorable) to the CBPR at higher thresholds is a reflection that the CRPS scores the CBPR more favorably for getting the ensemble spread more accurately particularly for large precipitation amounts (Kim et al. 2025). Because ensemble mean directly reflects the presence of outlying ensemble members, the RMSE of ensemble mean forecast is an extremely useful measure in detecting unrealistically large ensemble members. The RMSESS results in Fig. 6 indicate that the CBPR hindcasts are free of such members and that the CBPR ensemble mean hindcasts capture the predictive skill in the central tendency of the ensemble hindcasts very well.
2) Independent validation
The independent validation results are presented in two parts. The first part, referred to as the 1-yr independent validation, comparatively evaluates the CBPR and MEFP hindcasts valid for the 2-month period in which the large event occurred. Recall that these hindcasts are based on the parameters estimated using the fold that excludes the large event-holding 2-yr period. This result hence allows a head-to-head comparison with the 1-yr dependent validation result presented above. The second part, referred to as the 20-yr independent validation, evaluates the hindcasts valid for the 2-month period in each year of the 20-yr hindcast period. This result hence allows an independent comparative evaluation of the CBPR versus the MEFP for all events of varying magnitudes that occurred within the 61-day window during the 20-yr reforecast period. For example, if the large event occurred in the wet season at some location, the 20-yr validation hence amounts to an independent validation of the CBPR versus the MEFP for the 2-month wet season at the location.
Figure 7 shows the mean CRPS versus lead time of the CBPR (solid lines) and MEFP (dotted lines) hindcasts from 1-yr dependent validation (top panels) versus those from 1-yr independent validation (middle panels) for two of the 16 locations selected in section 3 for independent validation. Note that the y axis is in log scale so that both the conditional and unconditional mean CRPS may be plotted together. Figure 7 shows that the reduction in mean CRPS by the CBPR is increasingly larger for larger conditioning thresholds and that the margins of reduction in mean CRPS, i.e.,
Examples of the mean CRPS of the CBPR (solid lines) and MEFP (dotted lines) hindcasts vs lead time from (top) 1-yr dependent validation vs those (middle) from 1-yr independent validation and (bottom) the ME of the hindcasts from 1-yr independent validation.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
The upper panels of Fig. 8 show the CRPSS of the CBPR hindcasts in reference to the MEFP from 1-yr independent validation for all 16 locations selected in section 3. Recall that the MEFP and CBPR parameters differ most significantly between the large event-including and excluding folds at these locations. Hence, one may expect the hindcast results from dependent and independent validation to differ most significantly at these locations. Collectively, Figs. 6 and 7 and the upper panels of Fig. 8 show that the comparative performance of the CBPR versus the MEFP is very similar between dependent and independent validation, an indication that one may expect the dependent validation results to hold for independent validation for other locations as well.
CRPSS of the CBPR hindcasts in reference to the MEFP from (top) 1-yr and (bottom) 20-yr independent validation for 16 locations in 13 RFCs.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
The lower panels of Fig. 8 show the CRPSS of the CBPR from 20-yr independent validation for all 16 locations in 13 RFCs selected in section 3. The hindcasts are from 10-fold cross validation over the 20-yr reforecast period but only for the 61-day window in which the large event occurred. Hence, the lower panels of Fig. 8 amount to independent validation at the respective locations for the 2-month season in which the large event occurred. The lower panels show that the CBPR significantly improves conditional performance over the MEFP but slightly deteriorates unconditional performance versus the MEFP, a consequence of trading type I error for reduced type II error as explained above and in agreement with the CE-specific quasi-analytical results (Kim et al. 2025).
b. Streamflow
The upper panels of Fig. 9 show the CRPSS of the streamflow hindcasts for 48 locations in 11 RFCs used in 1-yr dependent validation for streamflow. The CRPSS for the lowest probability threshold of 0.005 (i.e., the 0.5th percentile of observed QME) effectively represents the unconditional CRPSS.
CRPSS of the streamflow hindcasts forced by the CBPR in reference to those forced by the MEFP for (top) 48 locations in 11 RFCs used in 1-yr dependent validation and (bottom) 14 locations in 11 RFCs used in 1-yr independent validation.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
The upper panels show that the CBPR improves streamflow hindcasts both conditionally and unconditionally across all lead times for most locations for the large event-bearing 2-month periods and that the margin of improvement is larger for higher flows. The RMSESS results are similar to the CRPSS results and are more and less favorable to CBPR relative to MEFP at lower and higher thresholds, respectively, similarly to the precipitation results in Fig. 6. For streamflow, both 1- and 20-yr independent validation experiments were carried out for 14 locations in 11 RFCs which exclude AP/CHLA2 and MB/SLDM8 from the 16 locations used for precipitation based on hydrologic uncertainty assessment (Kim 2024). The lower panels of Fig. 9 show the CRPSS of the streamflow hindcasts forced by the CBPR precipitation hindcasts in reference to those forced by the MEFP from 1-yr independent validation. Recall in section 3 that these locations have the largest parametric uncertainty for precipitation. Hence, one may consider the lower panels of Fig. 9 representing the lower bound of the improvement by the CBPR for streamflow in large event-bearing periods. Though the sample size is small for the independent validation results, Fig. 9 shows that the dependent and independent validation results are qualitatively similar, that the improvement by the CBPR is significant not only conditionally but also unconditionally for most locations for large event-bearing periods, and that the improvement is larger for higher flows.
Figure 10 shows the 20-yr independent validation results for streamflow. They are qualitatively similar to the 20-yr independent validation results for precipitation (see the lower panels of Fig. 8) in that the margin of improvement by the CBPR is larger for higher flows, but the unconditional performance of the CBPR is somewhat lower than that of the MEFP as expected from the type I versus type II error trade-off. The shading scheme used in Fig. 10 is the same as that in Fig. 9. The conspicuously negative CRPSS at high thresholds in Fig. 10 is associated with SESC1 (Sespe Creek near Fillmore) in Southern California (see Fig. 1) and warrants attention. Its contributing basin, SESC1HOF, is the most poorly performing location for CBPR in 20-yr independent validation for precipitation. At this location, the ensemble mean forecasts and the verifying observation tend to form bifurcating upper tails resulting in extremely large heteroscedastic forecast errors (Kim 2024). This situation arises due to the atmospheric flow-dependent variations in predictability and predictive skill of precipitation in the region (Moore 2023). In CBPR, λ is optimized by minimizing conditional mean CRPS in which the conditioning threshold is increased incrementally until one of the two criteria for the type I versus type II error trade-off is violated (Kim et al. 2025). Such magnitude-dependent parameter estimation is necessarily more susceptible to magnitude-dependent heteroscedastic errors than OLSR which always uses all available pairs of forecast and observation (i.e., in the bivariate normal space). Signs of such increased susceptibility may be seen by revisiting Fig. 4. Note in Fig. 4 that λ in CBPR for the large event-excluding fold (in red) for SESC1HOF swings from the largest value of λ among the large event-including folds (in pink) to ρ value in OLSR for the large event-excluding fold (in blue) across different CEs (see Table 2 for the forecast subhorizons). Similar patterns are also observed at LLYC1 (Santa Ynez River—Los Laureles) in the same region. Not surprisingly, the quality of the MEFP precipitation hindcasts at these locations is sensitive to thinning, or importance sampling, of the reforecast data for estimation of the MEFP parameters with reduced sample size (Kim 2022).
CRPSS of the streamflow hindcasts forced by the CBPR for 14 locations in 11 RFCs in 20-yr independent validation. The shading scheme is the same as in Fig. 9.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
To assess the performance of the CBPR in the mean sense across all locations for 20-yr 24-h MAP and the resulting QME, we show in Fig. 11 the CRPSS of the CBPR hindcasts in reference to the MEFP for the largest 24-h MAP and the largest QME resulting from the largest 2-day MAP event at each location in the 20-yr reforecast period. The figure reflects all 82 and 48 locations used in dependent validation of precipitation and streamflow, respectively. In the figure, each location contributes a single data point to the mean CRPS for each variable. Each black solid circle in Fig. 11 hence averages 82 and 48 CRPS values for precipitation and streamflow, respectively. To provide a sense of the sampling uncertainty, each panel in Fig. 11 also shows the 99.9% confidence intervals obtained via bootstrapping. As described in section 2, the 24-h precipitation amounts used in Fig. 11 generally correspond to a range of precipitation frequency used for design of flood mitigation infrastructure. Hence, one may consider Fig. 11 to be indicative of the potential improvement by the CBPR for likely high-impact events. The CRPSS for 24-h precipitation (left panel) indicates that, on average, the CBPR improves over the MEFP up to 10 days of lead time and that the average margin of improvement is larger for shorter lead times, exceeding 20% for days 1 and 2. The CRPSS for mean daily streamflow (right panel) indicates that the CBPR improves over the MEFP up to 14 days of lead time and that the average margin of improvement is over 10% for days 1–7 exceeding 15% for days 3 and 4. That the peak CRPSS for streamflow occurs at day 4 is a reflection of the predictive skill in the memory of the ICs such as soil moisture and channel storage and the travel time of surface runoff to the catchment outlet.
CRPSS of the CBPR hindcasts for (left) the largest 24-h MAP and (right) the largest QME resulting from the largest 2-day MAP event at each location in the 20-yr reforecast period.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
Finally, the upper panels of Fig. 12 show the scatterplots of the 2-week (days 1–14) ensemble mean precipitation hindcasts from the MEFP (upper-left panel) and the CBPR (upper-right panel) versus the verifying observation for the 82 locations used in 1-yr dependent validation. The lower panels of Fig. 12 are for the ensemble mean streamflow hindcasts averaged over the 2-week forecast horizon versus the verifying observed flow for the 48 locations used in 1-yr dependent validation. Also shown in each panel (in cyan) is the corresponding quantile–quantile plot. The sample sizes in the upper and lower panels are 4971 and 2874, respectively, representing the total number of 14-day-ahead precipitation and streamflow hindcasts valid for the 2-month periods that include the largest 2-day MAP events at the 82 and 48 locations, respectively. Figure 12 condenses the comparative results of varying lead time–dependent future input uncertainty (upper panels) and hydrologic uncertainty (lower panels) across widely varying predictability and predictive skill regimes into single plots to illustrate the impact of CBPR on the precipitation and streamflow ensemble mean hindcasts. One may visually ascertain in Fig. 12 that the CBPR reduces mean bias, reduces CB, increases correlation, and improves accuracy over the MEFP for both precipitation and streamflow in periods of large events. In the upper panels of Fig. 12, the CBPR reduces the unconditional RMSE for 2-week precipitation (days 1–14) by about 6%. For 4-day precipitation (days 1–4), which is particularly important for reservoir operations, the reduction is about 15%. In the lower panels, the CBPR reduces the unconditional RMSE for 2-week streamflow (days 2–14) by about 7%. For 1-week streamflow (days 2–7), the reduction is about 11%. The margin of improvement is larger for heavier precipitation and higher streamflow as seen in the 1-yr validation results presented above.
Scatter- and quantile–quantile plots of the 2-week (days 1–14) ensemble mean precipitation hindcasts from (top left) the MEFP and (top right) the CBPR vs the common verifying observation for the 82 locations used in 1-yr dependent validation. Those for the ensemble mean streamflow hindcasts averaged over the 2-week forecast horizon forced by (bottom left) the MEFP and (bottom right) the CBPR vs the common verifying observed flow for the 48 locations used in 1-yr dependent validation.
Citation: Weather and Forecasting 40, 6; 10.1175/WAF-D-24-0142.1
5. Conclusions and future research recommendations
The CBPR-aided MEFP is comparatively evaluated with the RFC-operational MEFP via hindcasting experiments. The aim is to improve the accuracy of the MEFP-generated ensemble forecasts for heavy-to-extreme precipitation in support of the HEFS. The precipitation events considered are larger than 20-yr 24-h events in terms of precipitation frequency which generally require flood mitigation infrastructure and hence represent likely high-impact events. For 24-h precipitation, the CBPR-aided MEFP is shown to improve forecast skill over the RFC-operational MEFP up to 10 days of lead time. The average margin of improvement as measured by the CRPSS is larger for shorter lead times, exceeding 20% for days 1 and 2. For mean daily streamflow, the CBPR-aided MEFP is shown to improve forecast skill over the RFC-operational MEFP up to 2 weeks of lead time. The average margin of improvement as measured by the CRPSS is over 10% for days 1–7, exceeding 15% for days 3 and 4. The margin of improvement is larger for heavier precipitation and higher flows and varies significantly from location to location with the predictability of precipitation, predictive skill in the conditioning GEFS ensemble mean precipitation forecast, and hydrologic uncertainty. The above improvement comes from the reduction of CB and actively trading type I error for reduced type II error, i.e., accepting, to a tolerable degree, slightly deteriorated false positives in favor of greatly reduced false negatives.
Arguably, the most appealing performance attribute of the CBPR-aided MEFP is that the margin of improvement over the RFC-operational MEFP is larger for heavier precipitation. The above attribute is expected from the type I versus type II error trade-off employed in CBPR as demonstrated with the CE-specific quasi-analytical results (Kim et al. 2025). This work empirically validates that the above performance characteristic holds for the MEFP ensemble results as well. A negative consequence of the above trade-off is that, though very rare, the CBPR-aided MEFP may significantly exacerbate overforecasting of relatively small amounts of precipitation when the conditioning GEFS ensemble mean forecasts are significant overforecasts. If necessary, the above exacerbation of type I error may be reduced by adjusting the acceptance criteria for the trade-off (Kim et al. 2025) but at some expense of increased type II error.
As does OLSR, CBPR assumes that the predictor–predictand relationship is linear (i.e., in the bivariate normal space). If the GEFS ensemble mean precipitation forecasts have severely heteroscedastic or bifurcating errors due, e.g., to flow regime–dependent predictability of precipitation and predictive skill in the numerical weather prediction forecast, one may expect the quality of the CBPR regression coefficient λ to deteriorate. Additional research is needed to recognize such error patterns (Jozaghi 2021) before optimizing λ. Kim (2024) identifies additional areas of improvement for the CBPR–MEFP and for the MEFP as a whole.
Acknowledgments.
Sunghee Kim was supported by the Probabilistic Forecast and Evaluation Support Program of the NWS. The proto-CBPR is based upon work supported in part by the COMET Program under UCAR Subaward SUBAWD000020 and NOAA JTTI Program under Grants NA17OAR4590174, NA17OAR4590184, and NA16OAR4590232. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the NWS.
Data availability statement.
GEFSv12 is available at https://registry.opendata.aws/noaa-gefs-reforecast/. Details of the CHPS, which includes the MEFPPE and MEFP, and how to request access are available from the CHPS release manager Shaif Hussein (shaif.hussein@noaa.gov) at the Office of Water Prediction (OWP) in NOAA/NWS. The government has unlimited rights to the data and software developed in this work. Details of them and how to request access are available from Mark A. Fresch (mark.a.fresch@noaa.gov) at NOAA/NWS/OWP.
APPENDIX
List of Acronyms and Abbreviations
| CB |
Conditional bias |
| CBPR |
Conditional bias-penalized regression |
| CDF |
Cumulative distribution function |
| CE |
Canonical event |
| CHPS |
Community Hydrologic Prediction System |
| CRPS |
Continuous ranked probability score |
| CRPSS |
Continuous ranked probability skill score |
| GEFS |
Global Ensemble Forecast System |
| HEFS |
Hydrologic Ensemble Forecast Service |
| IC |
Initial condition |
| MAP |
Mean areal precipitation |
| ME |
Mean error |
| MEFP |
Meteorological Ensemble Forecast Processor |
| MEFPPE |
Meteorological Ensemble Forecast Processor Parameter Estimation Program |
| MSE |
Mean-square error |
| OLSR |
Ordinary least squares regression |
| QME |
Mean daily flow |
| QPF |
Quantitative precipitation forecast |
| RFC |
River Forecast Center |
| RMSE |
Root-mean-square error |
| RMSESS |
Root-mean-square error skill score |
| SAC |
Sacramento soil moisture accounting model |
| SS |
Schaake shuffle |
| UHG |
Unit hydrograph |
REFERENCES
Bras, R. L., and I. Rodriguez-Iturbe, 1985: Random Functions and Hydrology. Addison-Wesley, 559 pp.
Brown, J. D., J. Demargne, D.-J. Seo, and Y. Liu, 2010: The Ensemble Verification System (EVS): A software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Modell. Software, 25, 854–872, https://doi.org/10.1016/j.envsoft.2010.01.009.
Brown, J. D., L. Wu, M. He, S. Regonda, H. Lee, and D.-J. Seo, 2014: Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 1. Experimental design and forcing verification. J. Hydrol., 519, 2869–2889, https://doi.org/10.1016/j.jhydrol.2014.05.028.
Burnash, R. J. C., R. L. Ferral, and R. A. McGuire, 1973: A generalized streamflow simulation system – Conceptual modeling for digital computers. National Weather Service, NOAA, and the State of California Dept. of Water Resources Tech. Rep., Joint Federal-State River Forecast Center, 68 pp.
Chow, V. T., D. R. Maidment, and L. W. Mays, 1988: Applied Hydrology. McGraw-Hill Series in Water Resources and Environmental Engineering, McGraw-Hill, 572 pp.
Clark, M., S. Gangopadhyay, L. Hay, B. Rajagopalan, and R. Wilby, 2004: The Schaake Shuffle: A method for reconstructing space–time variability in forecasted precipitation and temperature fields. J. Hydrometeor., 5, 243–262, https://doi.org/10.1175/1525-7541(2004)005<0243:TSSAMF>2.0.CO;2.
Cui, B., Z. Toth, Y. Zhu, and D. Hou, 2012: Bias correction for global ensemble forecast. Wea. Forecasting, 27, 396–410, https://doi.org/10.1175/WAF-D-11-00011.1.
Demargne, J., and Coauthors, 2014: The science of NOAA’s operational hydrologic ensemble forecast service. Bull. Amer. Meteor. Soc., 95, 79–98, https://doi.org/10.1175/BAMS-D-12-00081.1.
Georgakakos, K. P., T. M. Modrick, D.-J. Seo, S. Kim, H. Sadeghi, and J. Brown, 2015: Contract Dg-133w-13-Cq-0042 Hydrologic/Hydrometeorological Modeling Final Report on Task 4: Methodology for extreme precipitation analysis using MEFP. Hydrologic Research Center, submitted to NWS., 113 pp.
Guan, H., and Coauthors, 2022: GEFSv12 reforecast dataset for supporting subseasonal and hydrometeorological applications. Mon. Wea. Rev., 150, 647–665, https://doi.org/10.1175/MWR-D-21-0245.1.
Hamill, T. M., and Coauthors, 2022: The reanalysis for the Global Ensemble Forecast System, version 12. Mon. Wea. Rev., 150, 59–79, https://doi.org/10.1175/MWR-D-21-0023.1.
Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559–570, https://doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2.
Jolliffe, I. T., and D. B. Stephenson, 2012: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons, 240 pp.
Jozaghi, A., 2021: Improving operational hydrologic forecasting via conditional bias-penalized multi-sensor precipitation estimation and multi-model streamflow prediction. Ph.D. dissertation, The University of Texas at Arlington, 137 pp., https://mavmatrix.uta.edu/civilengineering_dissertations/444.
Jozaghi, A., M. Nabatian, S. Noh, D.-J. Seo, L. Tang, and J. Zhang, 2019: Improving multisensor precipitation estimation via adaptive conditional bias–penalized merging of rain gauge data and remotely sensed quantitative precipitation estimates. J. Hydrometeor., 20, 2347–2365, https://doi.org/10.1175/JHM-D-19-0129.1.
Jozaghi, A., H. Shen, M. Ghazvinian, D.-J. Seo, Y. Zhang, E. Welles, and S. Reed, 2021: Multi-model streamflow prediction using conditional bias-penalized multiple linear regression. Stochastic Environ. Res. Risk Assess., 35, 2355–2373, https://doi.org/10.1007/s00477-021-02048-3.
Jozaghi, A., H. Shen, and D.-J. Seo, 2024: Adaptive conditional bias-penalized kriging for improved spatial estimation of extremes. Stochastic Environ. Res. Risk Assess., 38, 193–209, https://doi.org/10.1007/s00477-023-02563-5.
Kim, S., 2022: Assessment of potential impact of reduced-sample calibration of the meteorological ensemble forecast processor on precipitation and streamflow forecasts for the hydrologic ensemble forecast service. Lynker Technologies Supplemental Tech. Rep., 73 pp., https://doi.org/10.13140/RG.2.2.36509.52961.
Kim, S., 2024: Comparative evaluation of CBPR-aided MEFP: Summary, findings and technical recommendations. Lynker, 87 pp., https://doi.org/10.13140/RG.2.2.17006.01605.
Kim, S., and Coauthors, 2018: Assessing the skill of medium-range ensemble precipitation and streamflow forecasts from the Hydrologic Ensemble Forecast Service (HEFS) for the upper Trinity River basin in North Texas. J. Hydrometeor., 19, 1467–1483, https://doi.org/10.1175/JHM-D-18-0027.1.
Kim, S., A. Jozaghi, and D.-J. Seo, 2025: Improving ensemble forecast quality for heavy-to-extreme precipitation for the Meteorological Ensemble Forecast Processor via conditional bias-penalized regression. J. Hydrol., 647, 132363, https://doi.org/10.1016/j.jhydrol.2024.132363.
Krzysztofowicz, R., 1999: Bayesian theory of probabilistic forecasting via deterministic hydrologic model. Water Resour. Res., 35, 2739–2750, https://doi.org/10.1029/1999WR900099.
Lee, H. S., Y. Liu, J. Brown, J. Ward, A. Maestre, M. A. Fresch, H. Herr, and E. Wells, 2018: Validation of ensemble streamflow forecasts from the Hydrologic Ensemble Forecast Service (HEFS). 2018 Fall Meeting, Washington, D.C., Amer. Geophys. Union, Abstract #H31A-05.
Lee, H. S., and Coauthors, 2020: On the improvements in precipitation, temperature and streamflow forecasts from the Hydrologic Ensemble Forecast Service after upgrading from the GEFSv10 to the GEFSv12. 2020 Fall Meeting, Online, Amer. Geophys. Union, Abstract #H111-0034.
Moore, B. J., 2023: Flow dependence of medium-range precipitation forecast skill over California. Wea. Forecasting, 38, 699–720, https://doi.org/10.1175/WAF-D-22-0081.1.
North Central Texas Council of Governments, 2009: iSWM criterial manual for site development and construction, revised 2015. https://iswm.nctcog.org/criteria-manual.html.
NWS, 2016: MEFPPE configuration guide. NOAA/NWS/Office of Water Prediction, accessed 27 June 2019, https://vlab.ncep.noaa.gov/documents/207461/1893010/MEFPPEConfigurationGuide.pdf.
NWS, 2017: Meteorological Ensemble Forecast Processor (MEFP) user’s manual. NOAA/NWS/Office of Water Prediction, accessed 27 June 2019, https://vlab.ncep.noaa.gov/documents/207461/1893026/MEFP_Users_Manual.pdf.
Pavlovic, S., S. Perica, M. St Laurent, and A. Mejía, 2016: Intercomparison of selected fixed-area areal reduction factor methods. J. Hydrol., 537, 419–430, https://doi.org/10.1016/j.jhydrol.2016.03.027.
Perica, S., S. Pavlovic, M. St. Laurent, C. Trypaluk, D. Unruh, and O. Wilhite, 2018: Precipitation-frequency Atlas of the United States: Volume 11 Version 2.0: Texas. NOAA NWS Atlas 14, 283 pp., https://www.weather.gov/media/owp/oh/hdsc/docs/Atlas14_Volume11.pdf.
Roe, J., and Coauthors, 2010: NOAA’s community hydrologic prediction system. Second Joint Federal Interagency Conf., Las Vegas, NV, Advisory Committee on Water Information, 1–12, https://training.weather.gov/nwstc/CHPS/roe.pdf.
Roundy, J. K., X. Yuan, J. Schaake, and E. F. Wood, 2015: A framework for diagnosing seasonal prediction through canonical event analysis. Mon. Wea. Rev., 143, 2404–2418, https://doi.org/10.1175/MWR-D-14-00190.1.
Schaake, J., J. Demargne, M. Mullusky, E. Welles, L. Wu, H. Herr, X. Fan, and D.-J. Seo, 2007: Precipitation and temperature ensemble forecasts from single-value forecasts. Hydrol. Earth Syst. Sci. Discuss., 4, 655–717, https://doi.org/10.5194/hessd-4-655-2007.
Schweppe, F. C., 1973: Uncertain Dynamic Systems. Prentice-Hall, 563 pp.
Seo, D.-J., 2013: Conditional bias-penalized kriging (CBPK). Stochastic Environ. Res. Risk Assess., 27, 43–58, https://doi.org/10.1007/s00477-012-0567-z.
Seo, D.-J., H. Herr, and J. Schaake, 2006: A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrol. Earth Syst. Sci. Discuss., 3, 1987–2035, https://doi.org/10.5194/hessd-3-1987-2006.
Seo, D.-J., M. M. Saifuddin, and H. Lee, 2018: Conditional bias-penalized Kalman filter for improved estimation and prediction of extremes. Stochastic Environ. Res. Risk Assess., 32, 183–201, https://doi.org/10.1007/s00477-017-1442-8.
Seo, D.-J., S. Kim, B. Alizadeh, R. A. Limon, M. Ghazvinian, and H. Lee, 2019: Improving precipitation ensembles for heavy-to-extreme events and streamflow post-processing for short-to-long ranges. NOAA/NWS/Office of Water Prediction, Dept. of Civil Engineering, The University of Texas at Arlington Final Rep., 52 pp.
Seo, D.-J., H. Shen, and H. Lee, 2022: Adaptive conditional bias-penalized Kalman filter with minimization of degrees of freedom for noise for superior state estimation and prediction of extremes. Comput. Geosci., 166, 105193, https://doi.org/10.1016/j.cageo.2022.105193.
Shen, H., H. Lee, and D.-J. Seo, 2022a: Adaptive conditional bias-penalized Kalman filter for improved estimation of extremes and its approximation for reduced computation. Hydrology, 9, 35, https://doi.org/10.3390/hydrology9020035.
Shen, H., D.-J. Seo, H. Lee, Y. Liu, and S. Noh, 2022b: Improving flood forecasting using conditional bias-aware assimilation of streamflow observations and dynamic assessment of flow-dependent information content. J. Hydrol., 605, 127247, https://doi.org/10.1016/j.jhydrol.2021.127247.
Sukovich, E. M., F. M. Ralph, F. E. Barthold, D. W. Reynolds, and D. R. Novak, 2014: Extreme quantitative precipitation forecast performance at the Weather Prediction Center from 2001 to 2011. 29, 894–911, https://doi.org/10.1175/WAF-D-13-00061.1.
Toth, Z. E., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 3297–3319, https://doi.org/10.1175/1520-0493(1997)125<3297:EFANAT>2.0.CO;2.
Whitin, B., and K. He, 2015: MEFP large precipitation event analysis. California-Nevada River Forecast Center, NWS, 27 pp.
Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. Elsevier Academic Press, 648 pp.
Wu, L., D.-J. Seo, J. Demargne, J. D. Brown, S. Cong, and J. Schaake, 2011: Generation of ensemble precipitation forecast from single-valued quantitative precipitation forecast for hydrologic ensemble prediction. J. Hydrol., 399, 281–298, https://doi.org/10.1016/j.jhydrol.2011.01.013.
