BIU Learning Club 15.11 — Tomer Galanti — On the Modularity and Optimization Dynamics of Hypernetworks

The recording of Tomer’s presentation is available in this link. 

Title: On the Modularity and Optimization Dynamics of Hypernetworks.
Abstract:
Hypernetworks are architectures that produce the weights of a task-specific implicit network. A notable application of hypernetworks in the recent literature involves implicit neural representations. In these scenarios, the hypernetwork learns a representation corresponding to the weights of a shallow MLP, which typically encodes visual information.
Two aspects of hypernetworks are discussed. (i) Their Modularity, which is the ability to effectively learn distinct functions for each task separately. (ii) Their optimization dynamics. We show that unlike typical architectures, infinitely wide hypernetworks do not guarantee convergence to a global minima under gradient descent. We further show that convexity can be achieved by increasing the dimensionality of the hypernetwork’s output, to implicitly represent wide MLPs. In this dually infinite-width regime, we identify the functional priors of these architectures by deriving their corresponding GP and NT kernels.

Zoom:
https://us02web.zoom.us/j/83601731206?pwd=d0dtVy9LQmx2ZHdNcTl4NUl0S2lLQT09
Meeting ID: 836 0173 1206
Passcode: 062420

—————————————————————–
On Sunday 15.11.2020 12:00 we will host Tomer Galanti from TAU.
Tomer will talk about the very interesting topic of hypernetworks, and will present his recent work in this field.
The meeting will take place via zoom. Please find details below.