Learning Club talk by Joanathan Berant

The recording of Jonathan’s talk:

Thanks to Jonathan for presenting and everyone who attended,

On Sunday 20.6.2021 at 12:00, we will host Jonathan Berant from Tel-Aviv university.

——

Pre-trained language models combined with fine-tuning on labeled data have led to impressive results throughout natural language understanding. In this talk, I will describe some recent work that goes beyond this traditional paradigm. First, I will present a class of models with an inductive bias towards tree structures and show that such models exhibit much better compositional generalization, that is, they generalize better to new compositions. Second, I will describe Splinter, a pre-training technique that emulates question answering during pre-training and leads to large gains in the few-shot setup. For example, Splinter achieves 73 F1 on SQuAD with as few as 128 examples. Time permitting, I will also describe some recent work on improving the robustness of models through adversarial training, where we propose a new discrete attack and show it can substantially improve robustness through online augmentation.