COMPARATIVE ANALYSIS OF SYNCHRONOUS AND PROXIMAL METHODS REINFORCEMENT LEARNING WITH DIFFERENT NEURAL NETWORK ARCHITECTURES BASED ON AN EXAMPLE OF THE CARTPOLE ENVIRONMENT

Abstract

Proximal Strategy Optimization (or PPO) is an optimization method that solves the main problems inherent in deep strategy gradient algorithms. PPO is an add-on to methods such as REINFORCEMENT and performance critic, which allows for a steady monotonous increase in their productivity and sustainability. The algorithm of proximal optimization of the strategy is considered as the object of research. The subject of the study is the effect of PPO on the quality of the agent’s functioning, compared with the algorithm of the actor-critic with advantage (A2C), for different types of neural network architectures. The main purpose of the work is to demonstrate the effects of using the proximal optimization strategy in comparison with A2C agents, taking into account the architectural features of their neural network apparatus. The relevance of this study is to consider the strengths and weaknesses of the PPO algorithm compared to A2C using the CartPole-v0 environment as an example, with a discrete action space. In the original article, PPO is based on two neural networks that are not connected by common hidden layers. The originality of the work consists in determining the influence of the combined architecture of the actor-critic network on the quality of the agent’s functioning using the proximal optimization strategy. Four experimental stands were implemented, considered in pairs. Two A2C and PPO agents with a separate neural network architecture and a pair of A2C and PPO with combined hidden layers. The following metrics were used for comparison: graphs of the distribution of a random value of total income per episode, the average number of episodes before reaching the average reward, the average and standard deviation, and the average reward for the last 100 episodes. Based on the results obtained using the classical CartPole-v0 simulation environment, which simulates the task of stabilizing an inverse pendulum, qualitative changes in agent behavior using the proximal optimization strategy were demonstrated. A comparative analysis of the functioning of A2C and PPO with different types of architecture of the actor-critic network is performed.