A Reduced-Order Model-Based Reinforcement Learning Approach to the Control of Nonlinear Partial Differential Equations

Abstract

Abstract. In this paper, we present a reduced-order model-based reinforcement learning method, leveraging the iterative linear quadratic regulator (ILQR) algorithm for the optimal control of nonlinear partial differential equations (PDEs). This approach introduces a novel modification to the ILQR technique: it employs the method of snapshots to construct a reduced-order linear time-varying (LTV) approximation of the nonlinear partial differential equation (PDE) dynamics around the current estimate of the optimal trajectory. The identified LTV model is then used to solve a time-varying reduced-order linear quadratic regulator (LQR) problem, yielding an improved estimate of the optimal trajectory and an updated reduced basis, with the process iterated until convergence. The convergence behavior of the reduced-order approach is analyzed and the algorithm is shown to converge to a limit set that is dependent on the truncation error in the reduction. The proposed method is evaluated on the viscous Burgers' equation and two phase-field models for microstructure evolution in materials, showcasing a substantial reduction in computational cost compared to the standard ILQR approach, with minimal impact on performance.