Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis
2021
Stochastic gradient descent (SGD) is one of the most popular algorithms in
modern machine learning. The noise encountered in these applications is
different from that in many theoretical analyses of stochastic gradient
algorithms. In this article, we discuss some of the common properties of energy
landscapes and stochastic noise encountered in machine learning problems, and
how they affect SGD-based optimization. In particular, we show that the learning rate in SGD with machine learning
noise can be chosen to be small, but uniformly positive for all times if the
energy landscape resembles that of overparametrized deep learning problems. If
the objective function satisfies a Lojasiewicz inequality, SGD converges to the
global minimum exponentially fast, and even for functions which may have local
minima, we establish almost sure convergence to the global minimum at an
exponential rate from any finite energy initialization. The assumptions that we
make in this result concern the behavior where the objective function is either
small or large and the nature of the gradient noise, but the energy landscape
is fairly unconstrained on the domain where the objective function takes values
in an intermediate regime.
Keywords:
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
30
References
4
Citations
NaN
KQI