Investigating fine-tuning of pre-trained language models. Marius Mosbach, from Saarland University

Title: 

Abstract:
Since the introduction of pre-trained masked language models in 2018,fine-tuning has become the de facto standard approach for both researchers and practitioners interested in “solving” NLP downstream tasks. As of today, the state-of-the-art on almost any NLP benchmark is held by a fine-tuned language model. Given the ubiquity and success of the pre-train then fine-tune paradigm, there is a strong need for research that provides insights about the fine-tuning process. In this talk, I will present our work on fine-tuning stability and the interplay between fine-tuning and probing for linguistic knowledge. Additionally, I will present ongoing research that investigates the generalization of few-shot pattern-based fine-tuning.