BIU learning club – Dan Rosenbaum – Functa: data as neural fields

Location:

Engineering building (1103), room 329

Title:

Functa: data as neural fields

Abstract:

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using a neural field, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification.

Short Bio:

Dan Rosenbaum is a senior lecturer in the Department of Computer Science at the University of Haifa. He is working on machine learning and computer vision, and specifically on 3D scene understanding and on using generative approaches that model vision as an inverse problem. Before joining the university of Haifa Dan completed his PhD in 2016 at the Hebrew University of Jerusalem, advised by Prof. Yair Weiss, and worked as a research scientist at DeepMind in London between 2016 and 2021. 

https://danrsm.github.io/