Second-order Optimization Made Practical – Talk by Tomer Koren, TAU

Title: Second-order Optimization Made Practical

Abstract: Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent.  Higher-order (preconditioned) optimization methods have become far less prevalent, despite compelling theoretical properties, due to their impractical computation, memory and communication costs.  I will present some recent theoretical, algorithmic and infrastructural advances that allow for overcoming these challenges and obtaining significant gains from higher-order methods in practice, at very large scale and on highly-parallel computing architectures.