Evaluation of Data Reduction Algorithms for Real-Time Analysis

Abstract

Data reduction tools are developed and evaluated using a data analysis framework. Simple (nonadaptive) and intelligent (adaptive) thinning algorithms are applied to both synthetic and real data and the thinned datasets are ingested into an analysis system. The approach is motivated by the desire to better represent high-impact weather features (e.g., fronts, jets, cyclones, etc.) that are often poorly resolved in coarse-resolution forecast models and to efficiently generate a set of initial conditions that best describes the current state of the atmosphere. As a precursor to real-data applications, the algorithms are applied to one- and two-dimensional synthetic datasets. Information gleaned from the synthetic experiments is used to create a thinning algorithm that combines the best aspects of the intelligent methods (i.e., their ability to detect regions of interest) while reducing the impacts of spatial irregularities in the data. Both simple and intelligent thinning algorithms are then applied to Atmospheric Infrared Sounder (AIRS) temperature and moisture profiles. For a given retention rate, background, and observation error, the optimal 1D analyses (i.e., lowest MSE) tend to have observations that are near regions of large curvature and gradients. Observation error leads to the selection of spurious data in homogeneous regions of the intelligent algorithms. In the 2D experiments, simple thinning tends to perform better within the homogeneous data regions. Analyses produced using AIRS data demonstrate that observations selected via a combination of the simple and intelligent approaches reduce clustering, provide a more even distribution along the satellite swath edges, and, in general, have lower error and comparable computational requirements compared to standard operational thinning methodologies.