The application of pre-trained transformer models to UK Court of Appeal legal judgments

The emergence of Transformer-based Pre-trained Language Models (PLMs) has had a significant impact across a variety of Natural Language Processing (NLP) domains. Pre-training language models on curated legal corpora can assist researchers in developing models to improve performance on downstream legal NLP tasks. This paper reports experiments with pre-training and fine-tuning language models on British and Irish Legal Information Institute (BAILII) data, addressing specifically Appeal Court judgments (these being, in effect, meta-judgments on previous legal judgments). We pre-train BERT-based language models on this corpus and evaluate their effectiveness on BAILII-based domain-specific tasks such as named entity recognition (NER), multi-label classification (MLC), and question answering (QA). The performance of this is then compared to baseline RoBERTa (Robustly Optimized BERT Pretraining Approach) and DistilRoBERTa models with two publicly available PLMs specifically designed for legal text (i.e., LegalBERT and CaseLawBERT), all pre-trained on the BAILII dataset. Pre-training on BAILII improves the performance of the PLMs on downstream tasks, and domain-specific pre-training enables a relatively smaller model such as BERT to achieve performance at par with a larger model such as RoBERTa. The pre-trained PLMs are now publicly available for downstream tasks on BAILII.

For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission.