MADDPG and P2P-VFRL for minimizing AoI in NTN network under CSI uncertainty

Non-terrestrial networks (NTNs) are emerging as a promising solution to handle the increasing of computation requirements. Due to the importance of latency in computation-intensive applications, the criterion of age of information (AoI) has been introduced, which gives a better view of the freshness of information. To this aim, in this paper we develop a hierarchical aerial computing framework composed of high altitude platform (HAP) and unmanned aerial vehicles (UAVs) to compute the full offloaded tasks of terrestrial mobile users who are connected by uplink non-orthogonal multiple access (UL-NOMA). In particular, the problem is formulated to minimize the AoI of all users with elastic tasks, by UAVs' trajectory and resource allocation on both UAVs and HAP, which is restricted by the channel state information (CSI) uncertainty and multiple resource constraints of UAVs and HAP. In order to solve this non-convex optimization problem, two methods of multi-agent deep deterministic policy gradient (MADDPG) and federated reinforcement learning (FRL) are proposed to design the UAVs' trajectory, channel, power and CPU allocation. The complexity of the algorithms and the impacts of different parameters are analyzed to verify the efficiency of the MADDPG approach.