Generalization in RL – A Bayesian Perspective (Aviv Tamar, Technion)

On Sunday 19.12.2021, we will host Aviv Tamar from Technion.

Zoom
https://us02web.zoom.us/j/85352616833?pwd=ZDBQNVBESFg2cERlalRhbWNrbzIrQT09

Meeting ID: 853 5261 6833
Passcode: 897342

Title
Generalization in RL – A Bayesian Perspective

Abstract
How can an agent learn to quickly perform well in an unknown task? This is the basic question in reinforcement learning (RL), and is critical for any effective combination of planning and learning. We study this question under the Bayesian (a.k.a. Meta RL) setting, where the essence is generalization: given a training set of tasks from some task distribution, design an agent that quickly performs well in an unseen test task. In this talk, I will describe several approaches to generalization that we have investigated. The first approach is inductive bias – value iteration networks are neural networks that include a differentiable planning module, and generalize better by learning an explicit planning computation. The second approach is learning to explore – BOReL is an offline Bayesian RL algorithm that can learn effective exploration policies from offline data. Key to making this work is solving an identifiability problem, which we characterize, and devise effective data collection policies to mitigate. Finally, I will present recent work on the theory of generalization in Bayesian RL — given N MDPs sampled from some prior, can we guarantee near Bayes-optimal performance with high probability? We develop the first (to our knowledge) PAC bounds for this problem, where our key contribution is showing that by adding a suitable regularization term, the solution to the Bayesian RL optimization problem is uniformly stable. To show this, we prove that regularized MDPs, while not convex, satisfy a quadratic growth property, which is just enough to show stability, and thereby derive generalization bounds.