Code of Paper: Federated Reinforcement Learning for Energy-Efficient D2D-IoT Networks with AoI Awareness

Abstract—This paper presents a novel approach to optimizing resource allocation in Internet of Things (IoT) networks, focusing on enhancing energy efficiency (EE) while maintaining age of information (AoI) awareness through device-to-device (D2D) communication. Our proposed solution integrates simultaneous wireless information and power transfer (SWIPT) with energy harvesting (EH) techniques. Specifically, D2D users employ time switching (TS) to harvest energy from the environment, while IoT users utilize power splitting (PS) to obtain energy from base stations (BS). The study aims to optimize energy efficiency by considering key constraints such as power transmission feasibility, minimum rate requirements, and simultaneous spectrum sharing and time allocation. The complex optimization problem is decomposed into three sub-problems: joint sub-channel allocation and PS, power control, and time allocation, while also considering the constraints of power budgets, harvested energy limitations, and maximum allowable AoI. Given the non-convex nature of the mixed-integer nonlinear programming (MINLP) problem, we propose a deep reinforcement learning (DRL) approach with a modified federated learning framework to tackle these challenges. Our method utilizes a software-defined network (SDN) controller as a central trainer to facilitate collaboration, efficiently manage radio resources, and maintain the desired AoI levels of D2D pairs. Simulation results demonstrate that our approach significantly outperforms existing methods, delivering substantial improvements in energy efficiency through optimized spectrum sharing and energy harvesting for IoT and D2D devices.