A Hierarchical Deep Reinforcement Learning Approach for Throughput Maximization in Reconfigurable Intelligent Surface-Aided Unmanned Aerial Vehicle–Integrated Sensing and Communication Network

Haitao Chen, Jiansong Miao, Ruisong Wang, Hao Li, Xiaodan Zhang

Integrated sensing and communication (ISAC) is considered a key technology supporting Beyond-5G/6G (B5G/6G) networks, which allows the spectrum resources to be used for both sensing and communication. In this paper, we investigate an unmanned aerial vehicle (UAV)-enabled ISAC scenario, where the UAV sends ISAC signals to communicate with multiple users (UEs) and senses potential targets simultaneously, and a reconfigurable intelligent surface (RIS) is deployed to enhance the communication performance. Aiming at maximizing the sum-rate throughput of the system, we formulate the joint optimization problem of the trajectory and the beamforming matrix of the UAV, the passive beamforming matrix of the RIS. Currently, many researchers are working on using deep reinforcement learning (DRL) to address such problems due to its non-convex nature; however, as the environment becomes increasingly complex, high-dimensional state space and action space lead to a decrease in the performance of DRL. To tackle this issue, we propose a novel hierarchical deep reinforcement learning (HDRL) framework to solve the optimization problem. Through decomposing the original problem into the trajectory optimization problem and the sum-rate throughput optimization problem, we adopt a hierarchical twin-delayed deep deterministic policy gradient (HTD3) structure to optimize them alternately. The experimental results demonstrate that the obtained system sum-rate throughputs of the proposed HDRL with an HTD3 structure are 33%, 50%, and 10% higher than those obtained by TD3, twin-TD3 (TTD3), and TD3 with hovering only (TD3HO), respectively.