BIU AI and ML Learning Club – May 26, Real-to-Sim: Towards interpretable and controllable digital twins (Note the Venue)

Do we live in a simulation? Perhaps we should consider the possibility. Replicating real-world observations into a digital twin offers numerous potential benefits. For instance, in autonomous navigation, one could recreate safety-critical scenarios to test an agent’s behavior more efficiently and without risking human lives. True-to-life simulations can enable counterfactual analysis, aiding in the interpretability of AI decision-making. Furthermore, they allow us to relive captured experiences for immersive entertainment.

In my research, I develop tools that enable these capabilities through the reconstruction of scene properties, such as geometry and color, and through perception methods like 3D scene segmentation and 3D object detection. This also includes the controllable generation and manipulation of content. To ensure scalability, my focus is particularly on representations that respect the symmetries in the data, such as rotation equivariance, and that can leverage large datasets at scale by minimizing the need for supervision.

Among these topics, in this talk, I will specifically highlight several of my recent papers. These include studies on 3D object detection from single images [1], neural fields for dynamic outdoor scene reconstruction [2], and piecewise equivariant representations [3].

[1] 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features. Chenfeng Xu, Huan Ling, Sanja Fidler, Or Litany. CVPR 2024

[2] Approximately Piecewise E(3) Equivariant Point Networks. Matan Atzmon, Jiahui Huang, Francis Williams, Or Litany. ICLR 2024

[3] EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision. Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, Yue Wang. ICLR 2024