Exascale Computational Fluid Dynamics in Heterogeneous Systems

Abstract

Exascale computing has extended the reach of resolved flow simulations in complex, heterogeneous systems far beyond conventional computational fluid dynamics capabilities. As a result, unprecedented pore and microscale resolution have been achieved in domains that have been traditionally modeled by, and limited to, continuum, effective medium approaches. By making use of computational resources on the new exascale supercomputer, Frontier, at the Oak Ridge Leadership Computing Facility, we performed flow simulations that have pushed the limits of domain-to-resolution ratios by several orders of magnitude for heterogeneous media. Our approach is an incompressible, Navier–Stokes CFD solver based on adaptive, embedded boundary (EB) methods supported by the Chombo software framework for applied partial differential equations (PDEs). The computational workhorse in the CFD application code is an elliptic solver framework in Chombo for pressure-Poisson and viscous, Helmholtz terms that leverages a PETSc-hypre software interface tuned for accelerator-based platforms. We demonstrate scalability of the approach by replicating a unit cylinder packed with microspheres to achieve over 400 × 109 degrees-of-freedom simulated. These simulations model domain lengths of over 20 meters with channel volumes of over 400 cm3 and containing millions of packed spheres with 20 micron grid resolution, challenging current understanding of what it means to be a representative elementary volume (REV) of the continuum scale in heterogeneous media. We also simulate a range of Reynolds numbers to demonstrate wide applicability and robustness of the approach.