- This event has passed.
BIU AI and ML Learning Club – June 2, Do Stochastic, Feel Noiseless: Stable Optimization via a Double Momentum Mechanism
June 2 @ 12:00 pm - 1:00 pm IDT
On May 26, Dr. Kfir Levy from the Technion will give a talk titled: Do Stochastic, Feel Noiseless: Stable Optimization via a Double Momentum Mechanism
Abstract:
The tremendous success of the Machine Learning paradigm heavily relies on the development of powerful optimization methods, and the canonical algorithm for training learning models is SGD (Stochastic Gradient Descent). Nevertheless, the latter is quite different from Gradient Descent (GD) which is its noiseless counterpart. Concretely, SGD requires a careful choice of the learning rate, which relies on the properties of the noise as well as the quality of initialization.
It further requires the use of a test set to estimate the generalization error throughout its run. In this talk, we will present a new SGD variant that obtains the same optimal rates as SGD, while using noiseless machinery as in GD. Concretely, it enables to use the same fixed learning rate as GD and does not require to employ a test/validation set. Curiously, our results rely on a novel gradient estimate that combines two recent mechanisms which are related to the notion of momentum.
Finally, as much as time permits, I will discuss several applications where our method can be extended.
BIO:
Kfir Y. Levy is an Assistant Professor at the ECE Department of the Technion
His research is in Machine Learning and Optimization.
He is focused on the design and analysis of efficient algorithms for a wide class of Machine Learning and decision making scenarios.
Kfir did his post-doc at the Institute for Machine Learning at ETHZ working with Prof. Andreas Krause. Previously, he did his PhD at the IE&M Department of the Technion, working under the guidance of Prof. Elad Hazan. Before that, he completed his master’s at the ECE Department of the Technion under the guidance of Prof. Nahum Shimkin.