Generalization in Deep Learning Through the Lens of Implicit Rank Minimization. Nadav Cohen (TAU).

CS colloquium lecture will be given by Dr. Nadav Cohen from Tel-Aviv University.

Title: Generalization in Deep Learning Through the Lens of Implicit Rank Minimization

Abstract:
Understanding deep learning calls for addressing three fundamental questions: expressiveness, optimization and generalization.  Expressiveness refers to the ability of compactly sized neural networks to represent functions capable of solving real-world problems.  Optimization concerns the effectiveness of simple gradient-based algorithms in solving non-convex neural network training programs.  Generalization involves the phenomenon of deep learning models not overfitting even when having much more parameters than examples to learn from.  This talk will describe a series of works aimed at unraveling some of the mysteries behind generalization.  Appealing to matrix and tensor factorizations, I will establish implicit regularization towards low (matrix and tensor) ranks.  I will then discuss implications of this finding to both theory and practice of modern neural networks, highlighting the potential of ranks to explain and improve generalization in deep learning.

Works covered in the talk were in collaboration with Sanjeev Arora, Wei Hu, Yuping Luo, Asaf Maman and Noam Razin.

Short Bio:
Nadav Cohen is an Asst. Prof. of Computer Science at Tel Aviv University.  His research focuses on the theoretical and algorithmic foundations of deep learning.  He earned a BSc in electrical engineering and a BSc in mathematics (both summa cum laude) at the Technion Excellence Program for Distinguished Undergraduates, followed by a PhD (direct track) in computer science at the Hebrew University of Jerusalem.  Subsequently, he was a postdoctoral research scholar at the Institute for Advanced Study in Princeton.  For his contributions to deep learning, Nadav received a number of awards, including the Google Doctoral Fellowship in Machine Learning, the Rothschild Postdoctoral Fellowship, the Zuckerman Postdoctoral Fellowship, and the Google Research Scholar Award.
Reminder – there is a calendar of the faculty colloquium in which you can see all our known future plans for the colloquium.