BIU learning club – Tom Tirer – Exploring Deep Neural Collapse via Extended and Controlled Unconstrained Features Models

Location:

Engineering building (1103), room 329

Title:

Exploring Deep Neural Collapse via Extended and Controlled Unconstrained Features Models

Abstract:

Training deep neural networks for classification often includes minimizing the training loss beyond the zero training error point. In this phase of training, a “neural collapse” (NC) behavior has been empirically observed: the variability of features (outputs of the penultimate layer) of within-class samples decreases and the mean features of different classes approach a certain tight frame structure. Recent papers have shown that minimizers with this structure emerge when optimizing a simplified “unconstrained features model” (UFM) with a regularized cross-entropy loss. In this talk, I will start with a review on the empirical findings on the NC phenomenon. Then, I will present some of our theoretical results. Namely, we show that the minimizers of a UFM exhibit (a somewhat more delicate) NC structure also with regularized MSE loss, and we analyze the gradient flow in this case, identifying the different dynamics of the features’ within- and between-class covariance matrices. Then, we extend the UFM in two different ways to provide mathematical reasoning for the depthwise empirical NC behavior and/or the effect of regularization hyperparameters on the closeness to collapse. We support our theory with experiments in practical deep learning settings

Short Bio:

Dr. Tom Tirer received his BSc in electrical engineering from Ben-Gurion University of the Negev in 2010, and his MSc and PhD in electrical engineering from Tel Aviv University in 2016 and 2020, where he also was a postdoctoral researcher during 2021. In 2022 he was a postdoctoral researcher at NYU Center for Data Science, until he joined the Faculty of Engineering at Bar-Ilan University in October 2022. His research interests are in the (often intersecting) fields of signal and image processing, machine learning and optimization. Alongside his academic endeavors, he also worked for several years in the industry in various engineering, algorithms and research roles.