The Promise of Load Balancing the Parameterization of Moist Convection Using a Model Data Load Index

Abstract

The parameterization of physical processes in atmospheric general circulation models contributes to load imbalances among individual processors of message-passing distributed-multiprocessor systems. Load imbalances increase the overall time to completion of a model run and should be eliminated or reduced as much as possible. Presented herein is a new technique that shows promise for load balancing the parameterization of moist convection found in the Community Climate System Model's (CCSM's) Community Atmosphere Model version 3 (CAM3). At the heart of this technique is a load index that is a marker for moist convection (called a model data load index). The marker for moist convection correlates directly to the amount of processing time per model grid cell and can therefore be used to effect a load balance. Spatial locality on the model grid and temporal locality between model time steps exist that allow a decomposition from a load-balancing step to be retained for multiple time steps. The analysis in this paper shows that the load balance does not need to be applied at every time step and that the number of steps in which the previous load balance remains effective is large enough for the overhead to be cost effective. Tests performed indicate that this technique is scalable to higher-resolution models as well as to higher processor counts than those presented. Through the use of the Load Balancing and Scheduling Framework (LBSF), this technique shows promise in reducing (by ?47%) the time of the unbalanced load of one particular subroutine in CAM3 at the T85 spectral truncation. A maximum of 3.75 s of total execution time is saved over a 2430-time-step simulation. When extrapolated to a 1000-yr simulation, this translates to a potential savings of ?22 h in that subroutine alone. Similar methods applied to remaining subroutines can add up to a significant savings. These results are encouraging in that a fine-grained load-balancing technique using the evolving characteristics of geophysical data paves the way for load balancing a broad range of physical calculations, both in CAM3 and other scientific applications, where more general techniques are not practical.