Learning Club talk by Yair Carmon

Learning Club – Yair Carmon

Zoom:

https://us02web.zoom.us/j/84847086771?pwd=K2QrMnZrQ21XU2JaUi9KS24wUHN2QT09

Meeting ID: 848 4708 6771
Passcode: 607548

Title:

Accuracy on the line: on the predictability of out-of-distribution generalization

Abstract:

To make machine learning reliable, we must understand generalization to out-of-distribution environments (unseen during training) in addition to the in-distribution generalization measured by standard test sets. We show empirically that, to very good approximation and to a large extent, out-of-distribution performance is in fact a simple function of in-distribution performance – our experiments span a wide range of models, computer vision datasets and types of distributions shifts. We also present a number of exceptions to this relationship and test hypotheses for their causes. Finally, we show how a simple Gaussian generative model exhibits several of the phenomena we observe, and discuss the scope and implications of our findings.

Joint work with John Miller, Rohan Tauri, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang and Ludwig Schmidt.