Detecting and Correcting for Label Shift with Black Box Predictors by Zachary Chase Lipton (DSI Learning Club)

June 11th, Mon. 11:00 , Zachary Chase Lipton (webpage).

Carnegie Mellon University (CMU).

Location: Gonda Building (901), Room 101.

Detecting and Correcting for Label Shift with Black Box Predictors

Abstract:

Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets), cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the conditional p(x|y) does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution p(y). BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE’s consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.

Bio:

Zachary Chase Lipton is an assistant professor at Carnegie Mellon University. His research spans both core machine learning methods and their social impact. concentrating on machine learning for healthcare, data-efficient deep learning, temporal structure, and learning under domain adaptation. This work addresses diverse application areas, including diagnosis, dialogue systems, and product recommendation. He is the founding editor of the Approximately Correct blog and the lead author of Deep Learning – The Straight Dope, an open-source interactive book teaching deep learning through Jupyter notebooks. Find on Twitter (@zacharylipton) or GitHub (@zackchase).