Learning Club BIU — NLP Student Session 26.1

Abstract: Contextualized word representations, such as ELMo and BERT, were shown to perform well on a various of semantic and structural (syntactic) task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in few-shot parsing setting.

Abstract: Gender bias has been shown to affect many applications in NLU. In the setting of machine translation (MT), research has primarily focused on measuring bias via synthetic datasets. We present an automatic method for identifying gender biases in MT using a novel-application of BERT-generated sentence perturbations. Using this method, we compile a dataset to serve as a benchmark for evaluating gender bias in MT across a diverse range of languages. Our dataset further serves to highlight the limitations of the current task definition which requires a single translation be produced, even in the presence of underspecified input.