Simplified optimized control using reinforcement learning algorithm for a class of stochastic nonlinear systems

Abstract In this work, a reinforcement learning (RL) based optimized control approach is developed by implementing tracking control for a class of stochastic nonlinear systems with unknown dynamic. The RL is constructed in identifier-actor-critic architecture, where the identifier aims for determining the stochastic system in mean square, the actor aims for executing the control action and the critic aims for evaluating the control performance. In almost all of the published RL-based optimal control, since both actor and critic updating laws are yielded on the basis of implementing gradient descent method to the square of Bellman residual error, these methods are very complex and are performed difficultly. By contrast, the proposed optimized control is obviously simple because the RL algorithm is derived based on the negative gradient of a simple positive function. Furthermore, the proposed approach can remove the assumption of persistence excitation, which is required for most RL based adaptive optimal control. Finally, based on the adaptive identifier, the system stability is proven by using the quadratic Lyapunov function rather than quartic Lyapunov function, which is usually required for most stochastic systems. Simulation further demonstrates that the optimized stochastic approach can achieve the desired control objective.