Исследование и сравнительный анализ методов оптимизации, используемых при обучении нейронных сетей

Abstract

Modern methods of deep learning of neural networks consist in finding the minimum of some continuous error function. In recent years, various optimization algorithms have been proposed that use different approaches to update model parameters. This article is devoted to the analysis of the most common optimization methods used in the tasks of teaching neural networks and forming recommendations on the choice of an algorithm for setting up neural networks on different data sets based on the identified properties. In the process of analysis, various implementations of the gradient descent method, impulse methods, adaptive methods, quasi-Newtonian methods were considered, the problems of their use were generalized, and the main advantages of each method were identified.