1. Introduction
Numerical weather prediction (NWP) models are in widespread operational use for regional and global forecast applications. An ongoing challenge in mesoscale NWP is to determine the ideal method for verifying the performance of high-resolution, detailed forecasts based on the application. Traditional objective techniques that evaluate NWP model performance based on point error statistics and precipitation threat scores may not be positively correlated with the value of forecast information for certain users of mesoscale NWP guidance. In addition, subjective evaluation techniques can be very costly and time consuming. As a result, objective phenomenological-based verification methodologies are required in order to determine the added value of high-resolution NWP models.
A coordinated effort between personnel from Dynacs, Inc. (transferred to ASRC Aerospace Inc. during the project), the Applied Meteorology Unit (operated by ENSCO, Inc.), and the National Aeronautics and Space Administration Kennedy Space Center (KSC) was established in order to develop advanced techniques for objectively evaluating the performance of the Regional Atmospheric Modeling System (RAMS; Pielke et al. 1992) mesoscale NWP model, currently used operationally on the Eastern Range at Cape Canaveral Air Force Station (CCAFS). These techniques were applied to evaluate model performance in forecasting the sea-breeze (SB) phenomenon over east-central Florida. The verification of the SB was chosen because this phenomenon occurs quite frequently in east-central Florida, particularly in the spring and summer months. In addition, the SB can significantly impact space operations because of the sharp wind shifts and thunderstorm development often associated with SB transition zones.
Some recent studies have addressed the deficiencies in applying traditional objective verification statistics to high-resolution NWP model configurations and have presented alternative means for verifying phenomena in mesoscale models. Nutter and Manobianco (1999) and Manobianco and Nutter (1999) performed an objective point verification and subjective phenomenological verification, respectively, of the 29-km version of the National Centers for Environmental Prediction (NCEP) Meso Eta Model. Case et al. (2002) performed both an objective and subjective verification of RAMS during the 2000 Florida summer, including a validation of the model-predicted SB and daily thunderstorm initiation. These studies demonstrated that objective point error statistics alone (i.e., measures-oriented approach) cannot adequately define a mesoscale model's utility, and that phenomenological verification is also required as part of the validation process. While these studies helped to quantify the value of the mesoscale models in predicting specific meteorological phenomena, a manual subjective intervention was used to perform phenomenological verification, which can be quite expensive in terms of the required manpower resources.
Recent efforts presented by Baldwin et al. (2001, 2002) have demonstrated the need for an improved, events-oriented technique for precipitation verification rather than the traditional measures- or distributions-oriented approach. Baldwin et al. (2001) exemplified the problem by showing a theoretical distribution of observed precipitation with embedded heavy cells. The authors then verified two hypothetical forecast fields, one with a smooth precipitation pattern without any embedded cells and another with a very similar pattern to the simulated observed field, but slightly out of phase. The authors demonstrated the failure of traditional measures- and distributions-oriented approaches in that the smooth forecast field generated better statistics (e.g., root-mean-square error, bias, threat score, and correlation coefficient) despite appearing less realistic than the detailed forecast precipitation pattern. Baldwin et al. (2002) subsequently developed an events-oriented verification technique using a cluster analysis on different modes of precipitation patterns.
The primary goal of this project has been to develop algorithms and software from the best candidate methodology to demonstrate a proof-of-concept procedure that can automatically detect an SB from forecast and observed wind data, and then compare the two SB datasets in a meaningful, quantitative manner for verification. The development of a tool that can verify a wide range of phenomena, or a general technique that handles many types of meteorological boundaries and discontinuities, was beyond the scope of the present study. Such a tool is highly desired and certainly an ultimate goal. The technique developed here could be readily enhanced to cover a broader range of phenomena, as described in section 3e.
The initial strategy was based on processing a time sequence of images in the spatial domain, since the observed features of interest are typically evident in the spatial domain. This initial strategy was not entirely successful since the SB transition zone is not always clearly apparent in single spatial images. Therefore, a robust methodology was developed to treat both the spatial and temporal data together.
The remainder of this paper is organized as follows. Section 2 presents the observed data and RAMS model configuration used to develop the objective technique for verifying the SB phenomenon. Section 3 describes the methodology used to develop the objective SB verification technique and provides sample output. Section 4 presents a validation of the objective algorithm and the resulting verification results, and section 5 summarizes the paper.
2. Model and observational data
The three-dimensional, nonhydrostatic mode of RAMS (version 4a) was run on four nested grids with a horizontal grid spacing of 60, 15, 5, and 1.25 km (Fig. 1). RAMS uses a stretched vertical coordinate from near the surface up to 18 195 m, with additional vertical levels in grids 3 and 4 to provide enhanced vertical resolution near the ground. A summary of the horizontal and vertical grid parameters is provided in Table 1. The physical parameterization schemes used in RAMS include a microphysics scheme following Cotton et al. (1982), a modified Kuo cumulus convection scheme (Tremback 1990), the Chen and Cotton (1988) radiation scheme, a Mellor and Yamada (1982)–type turbulence closure, and an 11-layer soil–vegetation model (Tremback and Kessler 1985) with fixed soil moisture in the initial condition. The modified Kuo scheme is run on grids 1–3 whereas grid 4 utilizes explicit convection only. The mixed-phase microphysics scheme is run on all four grids.
RAMS was initialized twice daily at 0000 and 1200 UTC using the Eta 12-h forecast grids from its forecast cycle 12 h earlier (due to operational time constraints), as well as all available national and local observational data. Observations were analyzed onto hybrid coordinates using the RAMS Isentropic Analysis (ISAN) package (Tremback 1990) without any balancing or nudging techniques. The ISAN hybrid coordinate consists of a combination of isentropes and terrain-following surfaces on which data are analyzed within the RAMS model domain, similar to the NCEP Rapid Update Cycle model (Benjamin et al. 1998). For sea surface temperature initialization, RAMS used fixed monthly climatological means on grid 1, and these values were subsequently interpolated to the inner grids. The lateral boundary conditions were nudged (Davies 1983) by 12–36-h forecasts from the NCEP Eta Model, interpolated onto an 80-km grid. Output from the Eta Model was available every 6 h for boundary conditions to RAMS. Two-way interactive boundary conditions were used on the inner-three nested grids.
This study used the high-resolution network of 44 wind towers across KSC/CCAFS (Fig. 2) in conjunction with NWP forecasts from only the 1200 UTC RAMS simulations during July and August 2000. The KSC/ CCAFS tower network has an average station spacing of ∼5 km, and the data archive provided wind information every 5 min. The RAMS output from the innermost grid 4 centered on KSC/CCAFS was used for the development of the SB verification algorithm. The daily RAMS forecasts were run for all of July and August 2000 with output every 5 min, consistent with the time resolution of the observed tower data.
To ensure that the observed and forecast fields were compared over identical effective domains, the model forecasts were interpolated to the position and height of the KSC/CCAFS tower locations. This interpolation provided a common starting point for the forecast and observed data and eliminated any artificial land–water differences that would have been introduced by using the RAMS grids directly. The observations and interpolated forecasts were then objectively analyzed to identical analysis grids using the Barnes (1964) algorithm. Identical uniform rectangular grids were used for comparison because they were required by several of the candidate methodologies. Although the final technique that was developed does not have such a stringent requirement, the data were used in this format in order to complete the work within the time frame and budget of the project.
As described in Tustison et al. (2001), grid-to-point and point-to-grid interpolations can lead to scale representativeness errors in quantitative precipitation forecast verification. These representativeness errors also apply to the grid-to-point and point-to-grid interpolations of the u and υ wind components for this study. However, according to Tustison et al. (2001), these errors should be relatively small due to the high spatial resolutions of the KSC/CCAFS tower network (∼5 km) and the RAMS grid (1.25 km).
3. Sea-breeze detection and verification methodology
The predominant mesoscale feature across the Florida peninsula during the summer months is the land-/sea-breeze cycle. Intense solar heating during the day creates a thermal contrast between air over land (Ta) and air over the water (Tw). When Ta exceeds Tw sufficiently, a thermally induced direct circulation results in the lowest few kilometers of the atmosphere. Near the surface, air flows from the cooler air mass residing over water to the warmer air mass over land, whereas a return circulation aloft flows from land to water. A boundary interface or SB “front” typically develops at the leading edge of the SB circulation and advances inland as the day progresses. Air rises along this leading edge while air descends above the water, completing the circulation.
At night, the SB circulation dissipates after the thermal contrast between land and water weakens, and the Coriolis force rotates the wind field in a clockwise sense, resulting in surface winds blowing from land to water (i.e., the land breeze). The land-breeze circulation is generally weaker than the sea-breeze circulation in both velocity and height of development since the ocean-based heat source for the land breeze is much weaker than the land-based heat source for the SB circulation (Atkinson 1981).
To verify the daytime SB across east-central Florida, a technique named contour error map (CEM) was developed by using a binary threshold to distinguish between easterly (onshore) and westerly (offshore) wind directions. The CEM incorporates both spatial and temporal wind data at each grid point to identify observed and forecast SB transition times. The CEM features a filtering technique to identify the correct transition times from offshore to onshore wind flow at every grid point in the analysis domain. To ensure focus on the SB boundary only, an erosion technique was introduced to remove extraneous boundaries not associated with the primary SB front, such as river breezes (refer to geography in Fig. 2) and precipitation outflow boundaries. The various components of the CEM verification algorithm are described in more detail below.
a. Sea-breeze transition time estimation
The development and maintenance of a wind shift from an offshore to an onshore component was used as a means for determining and verifying the occurrence and timing of the SB passage. During prevailing easterly (onshore) flow, an increase in wind speed can occur during the morning hours, signifying a sea-breeze passage; however, these sea breezes tend to be weak and were not taken into account for this study. The offshore to onshore wind-direction criterion was applied to both the observed and RAMS wind fields. The coastline of east-central Florida is approximately oriented along a 335°–155° line (Fig. 2); however, for the purposes of simplifying the initial technique development, wind directions between 0° and 180° were considered onshore winds, while 180° to 360° wind directions were defined as offshore.
A time estimation filter was developed to identify the SB transition time in both the observed and forecast grids for each day during July and August 2000. Every grid point in the observed and forecast data was processed using the SB filter technique explained in the appendix. Recombining processed time domain data into spatial images resulted in two-dimensional grids of observed and forecast SB transition times. These grids of observed and forecast SB transition times were then converted into the General Meteorological Package visualization and display software for analysis and presentation purposes.
Table 2 shows sample output using 5-min data for the month of July 2000 at grid location x = 55, y = 42. Figure 3a displays the filter outputs for 23–27 July, corresponding to the entries in Table 2. The raw data plotted in Fig. 3a are the sine of the wind direction ϕ. In general, sinϕ could be replaced by sin(ϕ − ϕ0), where ϕ0 is the offset from true north (0°) as a result of the local orientation of the coastline (i.e., 335°–155°). For this study, ϕ0 = 0 was assumed.
To illustrate that the SB transition time estimation filter works well by examining time series at individual grid points, Table 3 shows a grid point with a separation in the x direction of 5 km (westward) and a separation in the y direction of 8.75 km (northward) from the grid point used in Table 2. Figure 3b displays the filter outputs for 23–27 July, corresponding to the entries in Table 3. As can be seen, there is a strong correlation in the time domain signals by comparing the adjacent grid output in Tables 2 and 3 and Figs. 3a and 3b. The location of the two grid points is shown in Fig. 3c.
b. Image erosion to suppress contamination by river breezes
Image erosion is a common processing technique used to shrink an image object in some predictable way (Gonzalez and Woods 1992). Image erosion was used to suppress the river-breeze part of the SB transition time images using the gradient of the transition times to trigger the erosion process. The river breeze can often develop in advance of the actual SB transition and moves from west to east (from the Indian River to KSC/CCAFS), opposite of the direction of the SB.
The inverse of the gradient of the SB transition time is proportional to the sea-breeze boundary velocity. If the east-to-west direction is taken as positive, then a positive value of the gradient in sea-breeze transition times indicates an SB boundary propagating inland from the coast. However, a negative gradient in the SB transition times indicates a west-to-east propagation of the wind-direction boundary. Since the SB time estimation filter suppresses most effects of outflow boundaries due to convective rainfall, a negative gradient in the SB transition time is a strong indicator of river-breeze effects. This characteristic was used to eliminate the river-breeze-contaminated portions of the CEM difference images and to isolate the SB transition times. By scanning east to west, if a negative gradient was detected, then all SB times to the west of that point were recoded as “no SB.” This simple technique resulted in a reasonable suppression of the river-breeze phenomenon that contaminated the primary SB boundary propagation. Figure 4 shows an example of the SB transition times from 18 August 2000, before (Fig. 4a) and after (Fig. 4b) image erosion.
c. Verification parameters and interpretation
The SB verification and overall quality of the RAMS-predicted SB is quantified by four parameters:
fO ≡ fractional grid area with only observed SB transition,
fR ≡ fractional grid area with only RAMS SB transition,
τ ≡ mean error or bias, and
σ ≡ error standard deviation.
The observed-only (fO) and RAMS-only (fR) fractional grid area can provide categorical and skill scores on how well the model predicts the occurrence of the SB phenomena over the entire analysis domain. For example, a domainwide forecast “hit” results in fO = fR = 0, whereas a complete forecast miss (or false alarm) results in fO = 1 (fR = 1). The rule of thumb is that smaller observed-only and forecast-only fractions translate into more accurate forecasts of the occurrence of the SB phenomenon.
In addition to these four parameters, the mean wind direction and wind speed on the seaward side of the SB transition were computed in order to determine the skillfulness of RAMS in predicting the characteristics of the post-SB wind environment. The mean wind calculations were made for all times and grid points following an SB transition up to 0000 UTC the following day.
The resulting CEM algorithm with the transition time estimation filter and image erosion can be summarized in four steps:
Point processing: Calculates the sine of wind direction at each point in x,y, and t space.
Temporal processing: Processes a continuous time series at each x,y grid point to determine a best estimate of the offshore to onshore SB transition time.
Spatial processing: Constructs spatial images of SB transition times at all grid points; performs image erosion by computing two-dimensional spatial gradients of the SB transition time and removing grid points with negative gradients (i.e., boundary motion opposite to that of the SB leading edge).
Comparison and analysis: Verifies RAMS to the observed fields by calculating the parameters defined in Eqs. (1) and (2), and by computing the mean post-SB wind direction and speed.
d. Sample output from 18 July 2000
This subsection presents sample CEM output from 18 July 2000, illustrating a day with a typical SB passage in both the observed and forecast wind fields. Figure 5 depicts an hourly sequence of the observed, gridded wind field at 16.5 m from 1600 to 1900 UTC on 18 July. Northwesterly winds were prevalent across much of the domain at 1600 UTC, with only a small portion of the grid near the coast experiencing onshore winds from the northeast (indicated by shading in Fig. 5a). By 1700 UTC, east-northeast winds advanced inland along the entire eastern portion of the domain (Fig. 5b). Over the next 2 h, the SB transition zone moved through much of the remainder of the domain with east-northeast winds prevailing behind the SB front (Figs. 5c,d).
The RAMS forecast wind fields interpolated to 16.5 m (Fig. 6) are quite similar to the hourly observed winds. At 1600 UTC, only a slight shift to an easterly component occurred along the extreme eastern portion of the grid domain (Fig. 6a). Over the next 3 h (Figs. 6b–d), the RAMS SB wind shift advanced inland at a very similar rate and orientation compared to the observed winds (Figs. 5b–d).
The CEM output at each grid point provides a basis for isochrones of the SB transition zone shown in Figs. 7a and 7b. The RAMS isochrones of SB transition time (Fig. 7a) illustrate the steady west-southwestward progress of the SB front from about 1600 to 1900 UTC. The observed pattern in Fig. 7b is quite similar, with only slight deviations from the forecast pattern. The observed field is subtracted from the RAMS forecast of SB transition times (forecast − observed), yielding the difference field of SB transition times in Fig. 7c. Most of the SB transition time differences are less than 0.5 h in magnitude, with little indication of a systematic error across the verification domain.
e. Application to other phenomena
The CEM algorithm provides a framework that could be modified or enhanced to verify other meteorological phenomena. Many meteorological phenomena that can lead to significant sensible weather contain surface convergence lines or wind shifts, which are the focus of the CEM algorithm in identifying the forecast SB errors. With some modifications, the CEM could be enhanced to include surface (potential) temperatures, dewpoints, wind speeds, and/or radar reflectivity to verify other propagating phenomena such as cold fronts, drylines, land breezes, and outflow boundaries. With the fundamental CEM technique in place to identify the transition times, determine regions of forecast misses and false alarms, and calculate timing errors, phenomenological error statistics and graphics could be generated for these other features as well.
4. Algorithm validation and results
This section presents the validation of the CEM algorithm and the RAMS verification results as generated by CEM. An interpretation of the CEM output is also provided to explain the significance of the objective verification parameters.
a. Validation of CEM
Tables 4 and 5 summarize the results of the CEM error statistics for July and August 2000 and present the subjective SB transition time for the observed and forecast wind fields. The subjective SB transition times were determined by examining animations of 5-min observed and RAMS forecast wind fields across KSC/CCAFS, similar to Figs. 5 and 6. The range of SB transition times were identified based on the presence and continuity of a landward-propagating wind shift to onshore in the two-dimensional wind field, as interpreted by an experienced meteorologist.
Additional archived data sources such as radar and satellite observations were examined for each day to ensure that no precipitation outflow boundaries caused the landward-moving wind shift line. The beginning and end times of the SB transition time ranges were recorded based on the first and last appearance of the landward-moving wind shift within the tower analysis domain. The presence of precipitation outflow boundaries and river breezes in the observed and forecast wind fields were noted during the analysis.
The CEM technique performed quite well when compared to these subjective meteorological assessments of the SB transition times. The algorithm correctly identified a forecast or observed SB occurrence or absence 93% of the time during the 2-month evaluation period from July and August 2000. Given the 104 possible validation events (52 days with nonmissing data for both observations and forecasts), there were 97 successes and 7 failures in identifying the correct SB occurrence or absence. All failures and their explanations are indicated by a footnote in Tables 4 and 5. Six of the seven failures were caused by precipitation outflow boundaries that generated easterly flow, resulting in a false identification of an SB occurrence (7, 15, 16, and 30 July and 22 August). The only CEM missed identification of an SB occurrence is found on 17 July when the observed SB transition briefly affected only the extreme eastern portion of the verification domain.
The CEM could be improved to recognize areas of precipitation by incorporating additional data sources such as radar reflectivity, stage IV precipitation products, and model-predicted rainfall rates. Image processing techniques could be used to denote areas of observed or forecast precipitation in relation to the transition times identified by CEM. The incorporation of these additional data sources is beyond the scope of the current effort; however, CEM could be improved by introducing more sophistication to account for wind transition zones associated with precipitation features.
A qualitative comparison between the CEM timing errors and the subjectively determined observed and forecast transition times indicates that the algorithm performed very well overall. In most instances, the mean SB transition time bias (τ) is comparable to the difference between the subjectively determined RAMS and observed transition times (forecast − observed). Actual CEM timing differences could occur across the domain because of the positioning and orientation errors of the forecast SB transition zone, which cannot be adequately depicted by simply differencing the subjective observed time ranges from the forecast time ranges.
Most substantial discrepancies between the CEM bias results and the subjective time ranges were again caused by observed or forecast areas of precipitation that led to complex wind patterns not handled well by CEM. Out of the 36 days correctly identified by CEM with both observed and forecast SB transitions (excluding the erroneous 30 July SB identification), only 4 days had substantial discrepancies between the subjectively determined SB time ranges and the CEM bias (τ): 13 and 26 July and 2 and 6 August (denoted by shaded cells and footnotes in Tables 4 and 5). Similar to the SB identification problem, three out of four of these discrepancies were caused by either observed or forecast precipitation outflow that contaminated the wind fields. The fourth discrepancy (2 August) was caused by prevailing onshore flow in RAMS near the initialization time, which led to an erroneously early identification of the SB transition time by CEM and thus, an erroneously large early (negative) bias.
b. Interpreting objective model verification results
The statistics of Tables 4 and 5 provide useful information about the forecast SB errors that could not be obtained from traditional verification methodologies. Zeros in the same row under both fO and fR indicate that neither a forecast nor observed SB occurred on that day (representing a forecast success). Blank rows indicate that forecast and/or observed data were missing for that day. A complete forecast miss or false prediction of an SB on a particular day is represented by a value of unity for fO (forecast failure) or fR (false alarm prediction).
The days with the best model skill in predicting the SB occurrence and timing are those with the smallest absolute values of the mean bias (τ) and the smallest standard deviation of the timing errors. Days that have a larger absolute value of τ indicate the greatest systematic timing errors in RAMS. An average early bias in the onset of the SB transition on a particular day is given by a negative τ, whereas a positive τ indicates a late bias in the onset of the SB transition.
The standard deviation (σ) denotes the amount of variation in the SB transition time error across the KSC/ CCAFS grid domain. If the overall timing bias τ is small, the RAMS SB forecast could still be in substantial error over portions of the domain because of a large σ. In these instances, a large σ combined with a nearly unbiased τ would indicate that the RAMS forecast SB boundary had a phase/orientation error or did not propagate in the correct manner. For example, the forecast SB could start too late along the coastal regions and then propagate too quickly across KSC/CCAFS, reaching the western portion of the domain too early. Such a scenario would yield a nearly unbiased domainwide timing error τ, but would also yield a relatively large σ because of the variation of the timing errors across the domain.
Based on the results of CEM in Tables 4 and 5, RAMS tended to predict the onset and movement of the SB transition too early and/or quickly. The domainwide timing biases provided by CEM indicated an early bias on 28 out of 37 days when both an observed and forecast sea breeze occurred over the same portions of the analysis domain. These results are consistent with a previous subjective verification of the RAMS sea-breeze predictions conducted during the same time of year (Case et al. 2002). In the Case et al. (2002) subjective verification, 12 towers were selected and examined daily for SB transitions during the 1999 and 2000 summer months. The authors found that RAMS had about a 0.3-h early (negative) timing bias in the SB onset at those 12 selected towers. However, these results cannot be directly compared to the CEM results since the current study takes into account all available KSC/CCAFS towers and only compares the SB times during July and August 2000. The CEM results are more cost effective and thorough than the subjective evaluation results of Case et al. (2002) for two reasons. First, all available observational and RAMS point forecast data were utilized in this verification as opposed to only 12 towers in the manual verification. Second, the CEM is fully automated, saving considerable manpower compared to the subjective analysis.
c. Mean post-sea-breeze wind comparisons
Using eroded SB transition times, the average of wind speed and direction for all days with a detected SB passage during July and August 2000 are shown in Fig. 8. Comparing the observed to forecast data, it can be seen that the post-SB wind direction is better predicted by RAMS than the post-SB wind speeds. The overall wind speed bias for all valid days is 2.0 m s−1, with an error standard deviation of 1.2 m s−1. Out of the 37 events shown in Fig. 8, all days experienced stronger post-SB winds in the model compared to observations. Meanwhile for wind direction, the overall bias is only 9°, with an error standard deviation of 26°, indicating that the post-SB forecast wind direction is relatively unbiased.
As noted earlier, the majority of events had a negative τ, or early timing biases in RAMS. The early timing bias in the RAMS SB transition may be caused by a number of factors in the model; however, in all SB events, RAMS overpredicted the strength of the post-SB wind speeds (Fig. 8). This overprediction could have been caused by the outdated version of RAMS and the deficiencies of the model configuration. The soil and vegetation models used in version 4a of RAMS has since been replaced with an improved scheme. Further, only climatology was used for sea surface temperatures, and soil moisture initialization was not available for version 4a.
Collectively, the CEM error statistics and displays (such as in Fig. 7) provide much more information about the SB phenomenon and associated wind field than can be obtained by traditional error statistics. For example, Fig. 9 shows a plot of the hourly u wind component mean, rms error, bias, and error standard deviation as a function of forecast hour from the 1200 UTC RAMS simulations during the summer of 2000 (Fig. C6 from Case 2001). The errors shown in Fig. 9 indicate that RAMS has an easterly bias throughout the tower network between forecast hours 3 and 15 (1500–0300 UTC), suggesting that the model may overforecast the strength of the easterlies associated with the SB circulation. However, these results cannot provide the same level of verification detail as CEM on the SB occurrence and timing throughout the tower domain.
d. Potential drawbacks of CEM and automated event verification
While the results presented in this paper appear favorable for objective verification of the SB, there are inevitably some concerns with CEM and event verification in general. First, the CEM technique is highly tuned to the SB phenomenon over a specific geographical region (east-central Florida). Very few organizations have access to high-resolution 5-min data, nor can they generate 5-min output of model data in real time because of practical considerations. Also, the 7% of SB events that CEM did not correctly identify (mostly due to precipitation and outflow boundaries) could negatively affect the overall verification results if applied to other time periods without subjective validation. Additional tuning and testing with more extensive databases than used in this study are probably needed to handle situations with precipitation and outflow boundaries and thus improve the overall utility provided by the verification tool. Care should be exercised when implementing and using such an automated feature-based verification tool, as supervision by an experienced meteorologist is probably required to ensure that the tool operates correctly.
5. Summary
This paper presented the CEM objective technique to verify RAMS predictions of the SB phenomenon over east-central Florida. The CEM technique identifies SB transition times in objectively analyzed grids of observed and forecast wind, verifies the RAMS SB transition times against the observed times, and computes the mean post-sea-breeze wind direction and speed to compare the observed and forecast winds behind the SB front. The CEM technique is more informative and efficient compared to traditional objective model validation statistics and previously used subjective verification methodologies because it is automated, provides spatial and temporal information that cannot be obtained by traditional error statistics, accurately identifies and verifies the sea-breeze transition times, and provides verification contour maps and simple statistical parameters for easy interpretation. The primary drawback of the CEM as implemented here is that it is highly tuned to the SB phenomena over east-central Florida.
The CEM uses filtering techniques to identify the SB transition times at each observed and RAMS grid point. Once the transition times are identified at all grid points, the CEM computes the fractional area of forecast misses (fO) and false alarms (fR) and calculates the bias and error standard deviation to explain the overall timing bias and variation of the SB timing errors across the comparison domain. In addition, the output from CEM allows for easy display of the SB transition times and the variation of timing errors across the domain. Finally, the CEM computes the mean post-SB wind direction and speed for all times and grid points following the SB transition.
The CEM technique performed quite well when compared to subjective assessments of the SB transition times. The algorithm correctly identified a forecast or observed SB occurrence or absence 93% of the time during the 2-month evaluation period. Nearly all failures in CEM were the result of complex precipitation features (observed or forecast) that contaminated the wind field, resulting in a false identification of an SB transition.
A qualitative comparison between the CEM timing errors and the subjectively determined observed and forecast transition times indicates that the algorithm performed very well overall. Most discrepancies between the CEM results and the subjective analysis were again caused by observed or forecast areas of precipitation that led to a contaminated SB signal.
Based on the results of CEM, the RAMS tended to predict the onset and movement of the sea-breeze transition too early and/or quickly. The domainwide timing biases provided by CEM indicated an early bias on 28 out of 37 days when both an observed and forecast SB occurred over the same portions of the analysis domain. A comparison of the mean post-SB winds indicates that RAMS has a positive wind speed bias for all days, physically consistent with the early bias in the SB transition time.
Objective error statistics for specific meteorological phenomena, such as those developed in this project for the SB, can provide forecasters and model developers with an important tool in diagnosing model errors and biases. The potential savings in time and resources, combined with the knowledge gained from such an evaluation, could prove invaluable for the use and future development of high-resolution NWP models. The CEM technique also provides a framework for verifying other phenomena involving boundaries and discontinuities such as frontal passages, drylines, land breezes, lake breezes, outflow boundaries, or any other phenomena involving a wind shift line. Additional datasets and meteorological variables could be incorporated into the technique to expand the verification to include additional phenomena in different geographical locations and to improve the overall utility of the verification results.
Acknowledgments
This work was performed with funding from the Kennedy Space Center Director's Discretionary Fund. The authors appreciate the constructive comments from three anonymous reviewers who helped improve the quality of the paper.
Mention of a copyrighted, trademarked, or proprietary product, service, or document does not constitute endorsement thereof by the authors, ENSCO, Inc., ASRC Aerospace Inc., Dynacs, Inc., the AMU, the National Aeronautics and Space Administration, or the U.S. government. Any such mention is solely for the purpose of fully informing the reader of the resources used to conduct the work reported herein.
REFERENCES
Atkinson, B. W., 1981: Sea/land breeze circulation. Mesoscale Atmospheric Circulations, Academic Press, 125–214.
Baldwin, M. E., Lakshmivarahan S. , and Kain J. S. , 2001: Verification of mesoscale features in NWP models. Preprints, Ninth Conf. on Mesoscale Processes, Fort Lauderdale, FL, Amer. Meteor. Soc., 255–258.
Baldwin, M. E., Lakshmivarahan S. , and Kain J. S. , 2002: Development of an “events-oriented” approach to forecast verification. Preprints, 19th Conf. on Weather Analysis and Forecasting and 15th Conf. on Numerical Weather Prediction, San Antonio, TX, Amer. Meteor. Soc., 210– 213.
Barnes, S. L., 1964: A technique for maximizing details in numerical weather map analysis. J. Appl. Meteor, 3 , 396–409.
Benjamin, S. G., Brown J. M. , Brundage K. J. , Schwartz B. , Smirnova T. , Smith T. L. , Morone L. L. , and DiMego G. J. , 1998: The operational RUC-2. Preprints, 16th Conf. on Weather Analysis and Forecasting, Phoenix, AZ, Amer. Meteor. Soc., 249–252.
Case, J. L., 2001: Final report on the evaluation of the Regional Atmospheric Modeling System in the Eastern Range Dispersion Assessment System. NASA Contractor Rep. CR-2001-210259, Kennedy Space Center, FL, 147 pp. [Available from ENSCO, Inc., 1980 N. Atlantic Ave., Suite 230, Cocoa Beach, FL 32931.].
Case, J. L., Manobianco J. , Dianic A. V. , Wheeler M. M. , Harms D. E. , and Parks C. R. , 2002: Verification of high-resolution RAMS forecasts over east-central Florida during the 1999 and 2000 summer months. Wea. Forecasting, 17 , 1133–1151.
Case, J. L., Manobianco J. , Lane J. E. , Immer C. D. , and Merceret F. J. , 2003: A new objective technique for verifying mesoscale numerical weather prediction models. NASA Contractor Rep. CR-2003-211187, Kennedy Space Center, FL, 40 pp. [Available from ENSCO, Inc., 1980 N. Atlantic Ave., Suite 230, Cocoa Beach, FL 32931.].
Chen, S., and Cotton W. R. , 1988: The sensitivity of a simulated extratropical mesoscale convective system to longwave radiation and ice-phase microphysics. J. Atmos. Sci, 45 , 3897–3910.
Cotton, W. R., Stephens M. A. , Nehrkorn T. , and Tripoli G. J. , 1982: The Colorado State University three-dimensional cloud/mesoscale model—1982. Part II: An ice phase parameterization. J. Rech. Atmos, 16 , 295–320.
Davies, H. C., 1983: Limitations of some common lateral boundary schemes used in regional NWP models. Mon. Wea. Rev, 111 , 1002–1012.
Gonzalez, R. C., and Woods R. E. , 1992: Digital Image Processing. Addison-Wesley, 716 pp.
Hillman, G. D., and Lane J. E. , 1989: Real-time determination of IIR coefficients for cascaded Butterworth filters. Preprints, Int. Conf. on Acoustics Speech and Signaling Processing, Glasgow, Scotland, IEEE, 1353–1356.
Manobianco, J., and Nutter P. A. , 1999: Evaluation of the 29-km Eta Model. Part II: Subjective verification over Florida. Wea. Forecasting, 14 , 18–37.
Mellor, G. L., and Yamada T. , 1982: Development of a turbulence closure model for geophysical fluid problems. Rev. Geophys. Space Phys, 20 , 851–875.
Nutter, P. A., and Manobianco J. , 1999: Evaluation of the 29-km Eta Model. Part I: Objective verification at three selected stations. Wea. Forecasting, 14 , 5–17.
Pielke, R. A., and Coauthors, 1992: A comprehensive meteorological modeling system—RAMS. Meteor. Atmos. Phys, 49 , 69–91.
Rabiner, L. R., and Gold B. , 1975: Theory and Application of Digital Signal Processing. Prentice-Hall, 752 pp.
Tremback, C. J., 1990: Numerical simulation of a mesoscale convective complex: Model development and numerical results. Ph.D. dissertation, Colorado State University, 247 pp.
Tremback, C. J., and Kessler R. , 1985: A surface temperature and moisture parameterization for use in mesoscale numerical models. Preprints, Seventh Conf. on Numerical Weather Prediction, Montreal, QC, Canada, Amer. Meteor. Soc., 355–358.
Tustison, B., Harris D. , and Foufoula-Georgiou E. , 2001: Scale issues in verification of precipitation forecasts. J. Geophys. Res, 106 , 11775–11784.
APPENDIX
Details of the Sea-Breeze Transition Time Estimation Filter
Each grid point was processed individually by an SB transition time detector composed of a parallel low-pass (LP) boxcar filter (Rabiner and Gold 1975) and a high-order bandpass (BP) filter (Hillman and Lane 1989) centered on a frequency of one per day. The LP filter was used to remove small-scale wind features with a frequency on the order of 1/3 h−1, whereas the BP filter was designed to simulate the land-/SB oscillation for a 24-h periodic cycle, as observed in nature. In the event of missing data in the time series, the SB filter algorithm performed a linear interpolation across missing data before the data were fed to the LP and BP filters. After each grid point was preprocessed by the SB filter, the spatial image was reconstructed.
The SB time estimation filter can be considered as an SB transition time detector and estimator made up of two filter sections: an LP filter in parallel with a BP filter. The BP filter is based on a recursive or infinite impulse response (IIR) filter, whereas the LP filter is a nonrecursive or finite impulse response (FIR) filter. The LP filter is implemented as a moving average of length L = 31 points, centered about the middle of the sliding window. The LP filter can also be considered as an FIR filter of length L where all coefficients are equal to unity.
The BP filter is based on an eighth-order, maximally flat, Butterworth, IIR filter design. This filter type is not zero or linear phase; however, summing the outputs of two identical IIR filter structures, where the input data in the second filter is read in reverse order, results in a zero-phase recursive structure. The center frequency f0 is set to match the 24-h diurnal period of the land-/SB cycle. The BP filter is implemented by summing the outputs of two eighth-order filters with identical characteristics, where one filter processes a block of data forward in time, and the other filter processes the data backward in time from the end of the block. The mathematical details of the LP and BP filters can be found in Case et al. (2003).
The SB filter structure is not designed to be a real-time process because of the need to implement zero-phase filters. If the current SB filter were implemented in a real-time meteorological system, the BP filter structure would be linear phase with a minimum time delay of about 36 h, based on this dual eighth-order IIR filter. In other words, based on current wind direction data, the result of the SB filter is to estimate the SB transition time no more recently than the previous day.
The signal processing strategy behind the SB transition time filter is summarized by Fig. A1. The BP filter provides an SB transition time predictor, which is compared to the LP filtered wind direction signal at every spatial grid point. If the time difference between the predicted BP–SB time and the LP–SB time exceeds 6 h, then no SB is recorded for that day.
The RAMS domains for the 60-km mesh grid (grid 1) covering much of the southeastern United States and adjacent coastal waters, the 15-km mesh grid (grid 2) covering the FL peninsula and adjacent coastal waters, the 5-km mesh grid (grid 3) covering east-central FL and adjacent coastal waters, and the 1.25-km mesh grid (grid 4) covering the area immediately surrounding KSC/CCAFS
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
The locations of the 44 KSC/CCAFS observational towers used to verify the RAMS forecast sea breezes over east-central FL.
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Results of SB transition time filter as applied to wind-direction data from 23–27 Jul 2000. The filter results are shown at (a) grid point (x = 55, y = 42) and (b) grid point (x = 51, y = 35), while (c) indicates the spatial locations of each grid point. In (a) and (b), the gray line is the sine of the wind direction, the thick black line is output from the LP filter, the dotted line is output from the BP filter, and the sticks with triangles mark the SB transition times
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Erosion based on the gradient of the SB transition time of the observed wind field on 18 Aug 2000: (a) original hourly SB transition times (UTC), and (b) eroded hourly SB transition times (UTC)
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Hourly sequence of objectively analyzed tower winds at 16.5 m, valid on 18 Jul 2000 at (a) 1600, (b) 1700, (c) 1800, and (d) 1900 UTC. Shading denotes areas with wind directions between 0° and 180° (i.e., onshore)
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Hourly sequence of objectively analyzed RAMS forecast winds (interpolated to 16.5 m) from the 1.25-km grid, initialized at 1200 UTC 18 Jul. Valid times are (a) 1600 UTC (4-h forecast), (b) 1700 UTC (5-h forecast), (c) 1800 UTC (6-h forecast), and (d) 1900 UTC (7-h forecast) on 18 Jul 2000. Shading denotes areas with wind directions between 0° and 180° (i.e., onshore)
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Sample output of the CEM algorithm for 18 Jul 2000. (a) RAMS hourly forecast SB transition times (UTC), (b) observed hourly SB transition times (UTC), and (c) the difference in the SB transition times (h)
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Mean post-SB winds based on eroded SB transition times: (a) mean wind speeds for Jul 2000, (b) mean wind speed for Aug 2000, (c) mean wind direction for Jul 2000, and (d) mean wind direction for Aug 2000
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
A meteogram plot of the u wind component errors (m s−1) from the 1200 UTC operational RAMS forecast cycle during the 2000 summer months, verified at the 16.5-m level of the KSC/CCAFS wind-tower network. Parameters plotted as a function of forecast hour are (a) mean observed (dashed) and forecast u winds (solid), (b) rms error, (c) bias, and (d) error standard deviation (SD)
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
Fig. A1. The SB filter signal processing block diagram
Citation: Weather and Forecasting 19, 4; 10.1175/1520-0434(2004)019<0690:AOTFVS>2.0.CO;2
A summary of the grid configuration parameters for all four RAMS grids. The model parameters include the number of grid points in the x, y, and z directions (nx, ny, and nz), horizontal grid spacing (dx), minimum and maximum vertical resolutions (dzmin and dzmax), and the heights of the minimum and maximum physical vertical levels (z-min and z-max), with all distances given in meters
Filter output showing the SB code* and SB transition time (if any) for observed data from Jul 2000 at grid coordinate x = 55, y = 42. The SB transition times are given in units of UTC days of the month and local daylight time (LDT). Note that SB transition times are determined based on a changeover from offshore (>180° wind direction) to onshore (<180° wind direction)
Filter output showing the SB code* and SB transition time (if any) for observed data from Jul 2000 at grid coordinate x = 51, y = 35. The SB transition times are given in units of UTC days of the month and LDT. Note that SB transition times are determined based on a changeover from offshore (>180° wind direction) to on shore (<180° wind direction)
Sea-breeze timing error statistics for eroded CEM results and subjectively determined range of observed and RAMS times of the SB transition (UTC) for Jul 2000. The parameters shown are the mean SB transition time bias (τ ), the std dev of the SB transition differences (σ), the fractional area of the domain with only an ob served SB transition ( fO ), and the fractional area of the domain with only a RAMS forecast SB transition ( fR ). Erroneous identifications of an SB occurrence/absence and inconsistent biases are indicated by cells set in bold face, along with a footnote explaining the reason(s) for the discrepancy
Sea-breeze timing error statistics for eroded CEM results and subjectively determined range of observed and RAMS times of the SB transition (UTC) for Aug 2000. The parameters shown are the mean SB transition time bias (τ ), the std dev of the SB transition differences (σ), the fractional area of the domain with only an ob served SB transition ( fO ), and the fractional area of the domain with only a RAMS forecast SB transition ( fR )
