The 2017 Shared Task on Extrinsic Parser Evaluation. Towards a Reusable Community Infrastructure

Stephan Oepen1, Jari Björne2, Richard Johansson3, Emanuele Lapponi4, Filip Ginter2, Erik Velldal4, Lilja Øvrelid1
1University of Oslo and Center for Advanced Study at the Norwegian Academy of Science and Letters, 2University of Turku, 3University of Gothenburg, 4University of Oslo


Abstract

The 2017 Shared Task on Extrinsic Parser Evaluation (EPE 2017) seeks to provide better estimates of the relative utility of different types of dependency representations for a variety of downstream applications that depend heavily on the analysis of grammatical structure. EPE 2017 defines a generalized notion of lexicalized syntactico-semantic dependency representations and provides a common interchange format to three state-of-the-art downstream applications, viz. biomedical event extraction, negation resolution, and fine-grained opinion analysis. As a first step towards building a generic and extensible infrastructure for extrinsic parser evaluation, downstream applications have been generalized to support a broad range of diverse dependency representations (including divergent sentence and token boundaries) and to allow fully automated re-training and evaluation for a specific collection of parser outputs. Nine teams participated in EPE 2017, submitting 49 distinct runs that encompass many different families of dependency representations, distinct approaches to pre-processing and parsing, and various types and volumes of training data.