Autonomous Navigation for Cellular-Connected UAV in Highly Dynamic Environments: A Deep Reinforcement Learning Approach

Abstract

This study investigated the navigation problem for cellular-connected unmanned aerial vehicles (UAVs), particularly in highly dynamic urban environments. To address this problem, the UAV is required not only to evade high-speed obstacles in the airspace but also to avoid the coverage holes of cellular base stations (BS). Moreover, the UAV needs to reach the destination to complete the navigation task. Hence, it is imperative to design the trade-off in action selections between collision evasion and destination-approaching scenarios, while also considering the expected communication outage duration as a crucial reference. To overcome this multiobjective optimization challenge, we propose a deep reinforcement learning (DRL)-based algorithm aimed at enabling the UAV to acquire an optimal decision-making policy. Specifically, we formulated the navigation problem as a Markov decision process (MDP) and developed a layered recurrent soft actor–critic (RSAC)-based DRL framework, stimulating the UAV to resolve two fundamental subtasks of UAV navigation. Furthermore, we develop a multilayer perception (MLP)-based integrated evaluation network to select a particular action from the two subsolutions, satisfying the demands for the entire navigation problem. The layered architecture simplifies the navigation problem, thereby enhancing the convergence speed of the proposed algorithm. Numerical results indicate that the layered-RSAC-based UAV can autonomously perform scheduled navigation tasks in our designed simulated urban environments with superior effectiveness.