Benchmarking Symbolic Regression and Local Linear Modelling Methods for Reinforcement Learning

Reinforcement Learning (RL) agents can learn to control a nonlinear system without using a model of the system. However, having a model brings benefits, mainly in terms of a reduced number of unsuccessful trials before achieving acceptable control performance. Several modelling approaches have been used in the RL domain, such as neural networks, local linear regression, or Gaussian processes. In this article, we focus on a technique that has not been used much so far:\ symbolic regression, based on genetic programming. Using measured data, this approach yields a nonlinear, continuous-time analytic model. We benchmark two state-of-the-art  methods, SNGP -- Single Node Genetic Programming and MGGP -- Multi-Gene Genetic Programming, against a standard incremental local regression method called RFWR -- Receptive Field Weighted Regression. We have introduced slight modifications to the RFWR algorithm to better suit low-dimensional continuous-time systems. The benchmark is a highly nonlinear, dynamic magnetic manipulation system. The results show that using the RL framework and a proper approximation method, it is possible to design a stable controller of such a complex system without the necessity of any haphazard learning. While all of the approximation methods were successful, MGGP achieved the best results at the cost of higher computational complexity.