Automatic detection of microbursts with Doppler radar data is an interesting challenge. Traditionally, manual detection is performed by trained meteorologists who scan through the volumetric radar data for appropriate signatures, bringing to bear human powers of pattern detection and analysis. More recently, automatic systems have been devised to perform this detection using computer techniques together with fuzzy logic. Here a system that attempts to emulate human detection using the technology of agent networks (i.e., networks of cooperating asynchronous software entities) is presented. In this approach, agents detect reflectivity cores and high divergent shear zones. Their output is integrated by higher-level agents to detect microbursts and to track microbursts through time. The system is implemented in the Java language and is successfully detecting microbursts in data from radars near Sydney and Darwin in Australia.
Many meteorological data types exist, including surface observations, upper wind and temperature data, radar images, lightning data, and satellite images. These data are ingested by various national weather services at a rate of gigabytes per day, and hence there is a pressing need to automate their processing and interpretation as much as possible. One approach we are exploring to achieve this is the use of agent networks. The work described in this paper is an application of this approach to microburst detection from Doppler and reflectivity radar data.
In section 2 we describe what a microburst is and how one can be recognized. In the next section we talk about agent networks in general, and how their use is relevant to the current application. This is followed, in section 4, by a description of the architecture of the agents we are employing here. We then describe in more detail the actual agents used for microburst detection. In section 6 we present the results of this work.
A microburst is a mass of relatively cold air that, due to its density, descends from cumulus clouds and spreads out horizontally when it reaches the ground (Fujita 1985; Wilson et al. 1984; Hjelmfelt 1988) (Fig. 1). It is often associated with intense thunderstorms and heavy precipitation (“wet” microbursts) but in very dry environments can also occur with high-based weak convection and virga, with no precipitation reaching the surface (“dry” microbursts). When associated with a line of convective cells, a microburst line may result (Hjelmfelt and Roberts 1985). The wind shear associated with the downdraft and horizontal outflow can be extremely hazardous for aircraft during takeoff and landing (National Research Council 1983). This is compounded by the fact that a microburst is typically small-scale and short-lived, and timely advice of its presence is not possible with traditional observing systems. For this reason, systems that automatically detect microbursts and provide timely advice of the associated wind shear are of considerable interest. Examples of such systems are those developed using a network of anemometers or Doppler radar. If a microburst occurs over a closely spaced network of anemometers then a divergent wind flow at the surface can be identified. Systems using networks designed to provide protection for the runway and its associated approach paths have been developed and are operating at many international airports (Stoll 1991). The microburst also has a characteristic divergence signature in Doppler radar data, and systems that identify this signature and provide appropriate warnings are also operating. Here we examine systems based on Doppler radar.
One such system, the Microburst Automatic Detection (MAD) algorithm (Albo 1994), performs this task using fuzzy logic sequentially on each pixel of the radar data to generate an “interest map” based on shear, reflectivity, and clutter maps. The wind shear prediction application within the Integrated Terminal Weather System (ITWS; Wolfson et al. 1994) also uses fuzzy logic and interest maps, but in this case predicts microbursts from vertically integrated liquid water (VIL), center of mass movements, echo bottom, and sounding data. These useful systems do have their limitations, however, as the interest map approach does not naturally allow the use of pattern matching (i.e., detecting shape) until the final step of interpreting the interest map. In the agent approach described in this paper, such processing can occur at every level.
The authors of the ITWS system point out that an advantage of the interest map approach is that it does not throw away low-level (or raw) information in the early processing stages but rather keeps it in the form of fuzzy interest maps, making it available to the higher-level machine intelligence techniques. Similarly, in our agent-based approach, low-level information is also retained and, in this case, is available to high-level processing through inquiry messages sent between agents.
There are a number of signatures attendant on microbursts that can be detected with radar (Potts 1989; Roberts and Wilson 1989). However, not all signatures are necessarily present simultaneously in the data. That is, a microburst could be inferred from a subset of the possible signatures. The most natural method of coping with this situation is to use an uncertainty calculus in the detector—for instance, fuzzy logic, Bayesian calculus, or Dempster–Shafer evidential reasoning (Dempster 1968; Shafer 1976). The latter has been adopted in this study.
The most reliable microburst signature is the divergent airflow formed on the ground by the microburst. Since Doppler radar only detects radial velocity, this usually shows up as a pair of “bull's-eye” patterns of radial inflow and outflow that may or may not be symmetrically located around the microburst axis, depending on the prevailing winds (Figs. 2, 3). This signature will be extended along one axis in the case of a microburst line.
As the microburst drops from the cloud, mass continuity implies that some convergent flow near the cloud base or in the cloud will be evident as a convergent region in the Doppler radar data. However, this signature may be weaker or not detectable, due to the complexity of the wind field in clouds and the fact that at the cloud level the convergence may take place over a much larger depth than at the surface.
The developing microburst consists of cold air and precipitation and is generally evident in radar data as a region of enhanced reflectivity or a “reflectivity core.” Moreover, the core is descending and, if the radar is sampling at sufficient frequency, say, every 2–5 min, the core may be detectable in more than one frame. If the core in each of the two frames was matched, and the resultant velocity fell within appropriate bounds, then this would provide a means of predicting (as opposed to detecting) microbursts. However, prediction falls outside the ambit of the current work.
Microbursts are produced by convective cells (thunderstorms or showers), which in turn occur in areas of atmospheric instability. If such instability is known from the meteorological context, or other microbursts have already been detected in the area, these facts can be fed into the detector to enhance the likelihood of detection.
Note that in the work described here we deal with wet microbursts (those with strong precipitation signals in the core), due to lack of data from dry microbursts. However, we believe our system could handle the latter with appropriate tuning.
3. Agent networks in image processing
In the current work, detecting microbursts relies on finding patterns or structures in two- or three-dimensional data. This activity thus falls within the field known as image processing (or computer vision). The use of agent networks in image processing is a relatively recent phenomena, spurred on partly by the increasing complexity required of the task. Its precedents lie in the ideas of Hewitt (1975), Minsky (1975), and Schank and Abelson (1977). However, 1990–99 was really the decade when agent networks came to the fore, with several workers involved in their application to images. For instance, Baujard and Garbay (1993) isolate muscle fibers in medical images using a multiagent segmentation system, Dance (1997) makes legal interpretations of traffic scenes, and Boissier and Demazeau (1994) explore control issues in vision-based agent networks. Agent networks are especially appropriate in real-time image processing and when the processing is attempting to emulate the cognitive abilities of humans. In support of this, cognitive scientists have recently been proposing models for human vision (and cognition in general) that resemble agent networks (Dennett 1991; Hofstadter 1998; Lakoff 1987).
Another argument for the use of agent networks in this context is that it has been found by practitioners that designing and implementing systems using this approach is easier than the traditional single thread-of-control programming paradigm. In programming large systems, there is a sort of hierarchy of approaches, starting with assembler language programming, compiled language programming like FORTRAN, block-structured language programming like Pascal and C, object-oriented programming as in C++ and Java, and agent programming. In this hierarchy, there is increasing “chunking” (or modularization) of the problem as one ascends.
Agent orientation is distinguished from object orientation in that each agent runs in its own thread of control and has the characteristics described in section 4. This means that, unlike objects, agents are not necessarily in the same memory space and may in fact be running on separate machines. Thus, the agent-oriented system, with its autonomous interacting “subsystems,” is better able to mirror the physical world, and can thus also mirror the human conceptualization of the world. Related to this argument is the ease with which the agent approach allows communication between system users (for instance, meteorologists) and computer systems people. Since the problem has been “chunked” in the same way that people break up the problem space, the mapping between the two “languages” is relatively straightforward.
4. Agent network architecture
According to Wooldridge and Jennings (1995), an agent network is a network of asynchronous software entities with the following properties.
Autonomy: Agents operate without the direct intervention of humans or others, and have some kind of control over their actions and internal state.
Social ability: Agents interact with other agents (and possibly humans) via some kind of agent–communication language.
Reactivity: Agents perceive their environment, (which may be the physical world, a user via a graphical user interface, a collection of other agents, the Internet, or perhaps all of these combined), and respond in a timely fashion to changes that occur in it.
Proactiveness: Agents do not simply act in response to their environment; they are able to exhibit goal-directed behavior by taking the initiative.
In our system, agents are constructed from Java-threaded objects (autonomy) that communicate with each other through a message-passing system based on queues attached to each object instance (social ability). Their reactivity corresponds to a “bottom-up” or “data-driven” mode of operation, in which agents react to changes in data derived from real-time weather observations (i.e., radar), or messages from upstream agents, and inform other agents of these events. Proactiveness (“top-down” mode) is manifested by agents that are actively seeking to detect interesting weather phenomena (such as the microburst detector described below), by sending inquiry messages to other agents. Agents typically have various mixes of these modes. For instance, in our system several agents will react to a message and respond by seeking information from other agents.
5. Microburst detector
Figure 4 shows the set of agents for detecting microbursts. Each agent runs autonomously at its task. Upon detection the lower-level agents pass the coordinates and characteristics of the detected signature, together with a confidence measure, on to a higher agent. Temporal continuity is an important constraint, so while some agents' interest is limited to a single radar image (the CoreDetector agent in Fig. 4), other agents are concerned with correlating signatures from a sequence of radar images (the BurstTracks agent in Fig. 4).
Since all the possible signatures of a microburst are not necessarily present, and there is usually noise and ground clutter in the signal, the system uses Dempster–Shafer (Dempster 1968; Shafer 1976) evidential reasoning to combine the signatures. In this scheme, each fact passed in a message is associated with a pair of numbers between 0 and 1 that corresponds to the degree of belief in the fact (the support) and the degree of belief in its inverse (plausibility). In this calculus there are techniques for combining belief via conjunction, disjunction, and the other logical operators. This is not the place to discuss evidential reasoning in detail (see Pearl 1988); suffice it to say that this calculus was chosen because it is a compact technique for carrying degrees of belief for and against a given proposition around the network. A microburst is detected if the support and plausibility, as calculated from the support of the evidence, exceeds a pair of thresholds. Below we discuss each agent in turn.
a. Read input radar data
This agent, labeled “ReadData” in Fig. 4, simply has the job of reading into the agent system the next available radar file and passing components of it to other interested agents. Each dataset contains reflectivity and radial velocity (Doppler) data for all available azimuth angles and elevation angles (scans). The full volume set of reflectivity data is sent on to the core detector, and the lowest-level velocity scan is sent to the shear detector.
b. Core-detector agent
This detector, labeled “CoreDetector” in Fig. 4, finds volumes of connected high reflectivity in the radar data. This is performed by first thresholding the reflectivity data at defined values, producing a binary (two-valued) volume (a stacked set of binary images). This binary volume is in turn passed through a labeling routine that labels connected regions using 6-connectedness (3D pixels are labeled with their neighbors if joined through any of their six faces in three dimensions), using a region-joining algorithm based on trees of labels (two regions are joined by creating a new common root for them both). That is, as one scans through the volume-labeling pixels, if one comes across neighboring pixels with different labels, those labels are identified. In this algorithm we could have used higher than 6-connectedness (i.e., 8-connectedness through each corner, 12-connectedness through each edge, or 26-connectedness for all neighbors), but this increases the cost of the algorithm without appreciably improving the outcome.
If a labeled region exceeds a size threshold (expressed in km3) and a maximum height threshold (in km), then it is regarded as a candidate reflectivity core and passed on to the microburst-detector agent, together with details of its size, maximum reflectivity, and maximum height.
The core-detector agent also reacts to inquiry messages from the microburst detector (which is responding to the shear detector). These messages are concerned with confirming hypotheses that a microburst exists based on a high shear event. If this event is associated with a region of high reflectivity over a threshold volume and height, then the message is confirmed, and this response is returned to the microburst detector for further processing.
c. Shear-detector agent
This agent, labeled “ShearDetector” in Fig. 4, is sent both the velocity data for the lowest-level scan by ReadData and queries from the microburst detector for confirmation of shear. This agent searches for highly divergent shear regions in the data by convolving with a differentiating kernel (Albo 1994). The gradient is returned as the shear for each point. Regions above a defined shear are then labeled using a similar technique to that for reflectivity cores (in this case using 2D 4-connectedness). Labeled regions over a threshold area are then stored locally in a database and also sent on to the microburst detector, together with the location and other details of the potential burst.
The inquiry query from the microburst detector contains the location of a reflectivity core. The stored high-shear regions are then scanned for a match between the core and the shear (based on location), and if the area of high shear exceeds a threshold then the potential candidates are returned to the microburst detector.
d. Microburst-detector agent
This detector, labeled “MBDetector” in Fig. 4, receives messages from the core detector and the shear detector, giving it the location and other parameters of possible microbursts. Upon receiving a core (shear) detector message, this agent sends an enquiry to the shear (core) detector agent with the potential burst's location. If it receives an affirmative reply, and the combined confidence measure (computed using Dempster–Shafer evidential reasoning) exceeds a threshold, then it sends messages to the display agent (which shows the burst as a symbol on images of the reflectivity and velocity fields) and to the microburst-tracking agent.
The center of a reflectivity core can be somewhat offset from the center of the corresponding high-shear region, due to several factors. The core's center can be several kilometers in altitude and can descend at an angle, resulting in a shear zone that is horizontally offset by some kilometers. Another important factor is the forcing of the downdraft, which is largely a result of precipitative loading and evaporation. These may not be collocated, and the downdraft may be displaced from the precipitation core. Finally, the radar can take five or more minutes to complete a volume scan. In a high wind, the weather system can move several kilometers between the lowest sampled tilt and the highest. For this reason, to identify a core with a shear zone, we have allowed a maximum separation dependent on the cube root of the core volume.
In this architecture, it is possible that one microburst event could be found from two pathways, shear-detected first or core-detected first. For this reason, microburst events are identified by their location (and time). Thus, if two events are separated by less than a characteristic distance (calculated from the cube root of the core volume), then they are identified and reported as one event.
The threshold in some cases is higher to start a detection than to confirm a detection; for example, if the shear region is above 9 km2 then a detection is started, but to confirm an inquiry the threshold is between 3 and 5 km2 (depending on the locality). This is to allow for cases where there is such strong evidence for a burst from one source that the confirming evidence can be correspondingly weaker. This threshold distinction takes place in the core- and shear-detector agents, but could just as easily have been made in this agent.
Below we list the thresholds used by the system. These have been set through a combination of empiricism and consultation with meteorologists. Note that these parameters are tuned for different regions, in this case the Darwin and Sydney regions in Australia.
Shear area: threshold 9 km2 for detection, 3 km2 for confirmation in the Sydney area, and 5 km2 for the Berrimah radar
Shear: threshold 0.0019 s−1
Core reflectivity: threshold 35 dBZ
Core volume: threshold 10 km3 for detection, 2.5 km3 for confirmation
Core minimum height: threshold 4150 m
Maximum distance from core center to shear center: cube root of core volume plus 1 km
Minimum velocity difference across burst: 10 m s−1
The belief values for the detections are calculated from the thresholds using an interpolation from 0.0 support to 1.0 support around each threshold; that is, the belief is zero below the threshold and rises from 0.5 at the threshold to 1.0 at 10% above the threshold. The plausibility is set to 1.0 in all cases.
It would be possible to adjust the sensitivity of the microburst detector so that it reacts to the “atmospheric instability” as derived from a history of microbursts on the day or from atmospheric modeling.
e. Microburst-tracking agent
This detector, labeled “BurstTracks” in Fig. 4, receives messages from the microburst detector agent with details about each microburst found. This agent's job is to find possible trajectories connecting bursts in temporally contiguous radar volumes that satisfy continuity criteria reasonably well.
This agent finds tracks by first finding the potential velocity between the current burst and all bursts in the previous volume. If this velocity is less than a defined maximum (set at 40 m s−1), then the potential track is extended back to the next previous volume, where a predicted burst location is calculated. If this lies within a threshold distance from a real burst, this track is retained. The track with the best confidence measure as computed from the thresholds is chosen. This track is then sent to the display agent.
If a microburst at one time connects to two bursts at another time, these are connected and displayed. There is no attempt at this stage to make the set of tracks globally consistent. In fact, it is possible for a burst event to split into two separate events at a later time and thus produce a forked track.
Unfortunately, at present, the radars in this study are sending volume scans at 10-min intervals, which is not frequent enough to tie microbursts together reliably. We hope soon to have the radars running every 5 min, which will facilitate tracking of bursts, not only horizontally but vertically, as well. Ideally, for microburst detection we should run the radar at 1–3-min intervals per volume.
6. Experimental results
The system described above was run on a series of radar volumes obtained from the Kurnell radar situated next to Botany Bay in Sydney (Fig. 5), the Badgerys Creek radar also near Sydney, and the Berrimah radar near Darwin (Fig. 6). The Kurnell radar is a 5.3-cm wavelength (C band) radar with a peak power of 500 kW, half-power beamwidth of 0.9°, nominal range of 150 km, and range resolution of 250 m. The Badgerys Creek radar has polarimetric capability (Keenan et al. 1998) but is similar to the Kurnell radar in this Doppler application, except that range resolution is 300 m and the half-power beamwidth is 1.0°. The Berrimah radar is also a C-band radar at 5.3-cm wavelength with peak power of 250 kW, half-power beamwidth of 1.0°, nominal range of 250 km, and range resolution of 500 m.
We have examined 53 datasets obtained from seven different days and three radars. Thunderstorms were present on each of the days and there were known microbursts based on a manual analysis of the Doppler radar data. The first day contains two series of 3 and 5 volumes from the Kurnell radar, the second day consisted of 2 volumes from Kurnell, the third day 2 volumes from Kurnell, the fourth day 8 volumes from Badgerys Creek, and the last three days 33 volumes from Berrimah near Darwin. Contiguous volumes are 10 min apart, which is not frequent enough for an operational system but is adequate for the purposes of this study. Typical output images are shown in Figs. 7 and 8.
The output from the runs was compared with the results of a manual analysis of the same radar volumes by an experienced meteorologist. In the analysis by Wilson et al. (1984) a microburst was declared if the lowest elevation of the Doppler data showed a divergence region with the distance between the maximum radial inflow and maximum radial outflow initially ≤4 km and the difference in radial wind speed ≥10 m s−1. For this study we also include manually detected events where the horizontal scale is >4 km, providing the shear was ≥0.0025 s−1. The reason for this is that the contiguous volume scans are 10 min apart, in which time a microburst may have developed and grown to a horizontal scale >4 km. Also, for an automated warning system the threshold shear of 0.0025 s−1 is considered sufficiently hazardous for aviation that such events should be detected. Only microburst events within 45 km of the radar were considered to minimize the effects of increasing beamwidth with range. A total of 73 microburst events were detected, and in all cases the microbursts were associated with high radar reflectivity (wet microbursts). In addition, false alarms that were identified by the agent network but not detected in the manual analysis were examined in order to determine the cause. To compare the human analysis with the agent system output, a hit was recorded if the human detected a microburst within 8 km (center to center) of an automatic detection. Our system scores are presented in Table 1.
The false alarms are in general initiated by the shear detector due to noise, with the reflectivity core detector having a much weaker signal for these events. If, however, we increase the threshold on the core detector, we lose genuine detections. In other words, there is more subtlety to the microbursts than our system is dealing with. This situation might be improved if we invoke more detectors, for instance a high-level convergent shear detector, and relate a core in one volume to shear in subsequent ones. Also, more frequent scans, particularly at lower elevations, would allow for testing of the persistence of a microburst over several volumes, which should reduce the false alarm rate.
The commonly used statistical measures for algorithm performance, namely probability of detection (POD), false alarm rate (FAR), and critical success index (CSI) are defined as follows (Marzden 1998):
where h is hits, fd is false detections, e is events, and md is missed detections.
The number of events is low for meaningful statistical analysis; nevertheless, the performance results for our system as shown in Table 1 are POD of 0.71, FAR of 0.21, and CSI of 0.60. These results are comparable with the system reported by Albo (1994), which achieved POD of between 0.58 and 0.76 (depending on the radar configuration), FAR between 0.09 and 0.20, and CSI between 0.54 and 0.67. It should be noted here that Albo's results were based on more frequent velocity scans, which will have some impact on statistical results. Also, our 8-km difference between human truth and algorithm burst locations would boost the PODs and reduce the FARs relative to Albo, whose hits were defined by an overlap between the human and algorithm microbursts.
In this paper we have demonstrated the feasibility of using agent technology in the meteorological context, especially for detecting microbursts. We set out to emulate the processing that meteorologists perform in detecting microbursts and succeeded to some degree in that objective. For emulating human abilities, the agent technology was found to be reasonably easy to modify and tune, although there is obviously much more that needs to be done to bring the system up to near-human standards. More pattern recognition at all levels is required.
Judging from the statistical measures of the algorithm's performance, our agent network proved comparable in efficiency to a more mature system (Albo 1994) performing the same task, although less resources were expended on building the system described here. In the future we would like to compare our system with other microburst-detection systems using the same data.
Further testing of the system is required to evaluate the performance for a much greater number of microburst events. It is also proposed that the performance be compared with the output from a low-level wind shear alert system that utilizes a network of anemometers. Furthermore, microbursts develop rapidly and have a relatively short lifetime, and the 10-min radar volume scans used in this study would be inadequate for an operational warning system. An appropriate scanning program for the radar needs to be developed that would enable the earliest detection of a developing microburst. This may include sector volume scans centered over the airport with more frequent low-level scans. The performance of the agent network in this environment needs to be evaluated.
It should be noted that the work described here is experimental; the agent approach is being tested with a fairly simple problem. If successful we intend to take this approach much further, applying it to our general “forecast database,” in which all meteorological data types will be processed at different levels and timescales and from different user perspectives, to provide our recipients with a variety of forecast products, especially automated alerting systems for forecasters.
Corresponding author address: Dr. Sandy Dance, Australian Bureau of Meteorology Research Centre, 150 Lonsdale St., Melbourne, VIC 3000, Australia. Email: email@example.com