Abstract

Pre-trained contextual embeddings like BERT have shown substantial improvement across a wide range of natural language processing tasks.

% Advent of BERT revolutionized the domain of NLP by beating state-of-the-art results in various tasks.

We proposed Sliding-BERT, which incorporates BERT with state-of-the-art conversational machine comprehension (MC) model, \textsc{FlowQA}, and supersedes its standing performance on the \daffy challenge. We designed a striding filter to overcome the sequence length limit of BERT model in the long conversation context. We also applied various aggregation methods to handle the incompatible tokenization between BERT and \textsc{FlowQA} models. Given the long conversation context, we used gradient accumulation to simulate batched training scenarios without extra memory cost during training. We also found that pretraining our Sliding-BERT on CoQA dataset helps improve its performance on \daffy dataset. Detailed analysis of the model performance considering the types of questions, lengths of questions and other metrics of QA datasets indicates that our Sliding-BERT exceeds \textsc{FlowQA} model in terms of F1, HEQ-Q, and HEQ-D scores by a significant margin.

Statistics