ConLL Shared Task on Universal Dependency Parsing

The 2018 edition of the Extrinsic Parser Evaluation Initiative (EPE 2018) runs in collaboration with the 2018 Shared Task on Multilingual Parsing from Raw Text to Universal Dependencies at the 2018 Conference on Computational Natural Language Learning (CoNLL 2018).  With minimal additional overhead, participants of the UD Parsing Shared Task will be enabled to have their parsers evaluated against the three EPE downstream systems.  Seeing as the focus of the UD Parsing Shared Task is on different evaluation perspective in 2018, the connection to the EPE framwork will provide new opportunities for correlating intrinsic metrics with downstream effects on three relevant applications.  However, to date the EPE infrastructure regrettably is only applicable to parsing systems for English.

Participation and Interfaces

Participation in the EPE 2018 campaign will be open to all participants in the UD Parsing Shared Task.  It will require parsing an additional 1.1 million tokens of running English text (from various domains) into basic UD trees.  The EPE parser inputs will be provided as a separate data package on the TIRA platform, using the same general interfaces as the core parsing task (e.g. via the metadata.json format).  Although Parser outputs will be collected in CoNLL-U format and will later be automatically transferred from the TIRA platform to the cluster that runs the EPE infrastructure.  For each individual parser output, the three EPE downstream systems will be retrained and optimized, in oder to avoid downstream bias towards one specific parsing system.  As the downstream systems interface to syntactic structure by way of character offsets (i.e. substring indices), CoNLL-U parser outputs will be automatically converted to the more general EPE interchange format for dependency graphs.  A basic conversion and validation script will be installed on TIRA, which all participants are encouraged to invoke upon completion of the parsing job.

To not interfere with the potentially busy last few days of the UD Parsing Shared Task, the 2018 EPE initiative closes one week after the official end of the (intrinsic) evaluation phase in the UD Parsing Shared Task.  Extrinsic, end-to-end results will be published in a format similar to the 2017 scores; these will be available (just) in time for inclusion in the submissions of system descriptions.  The EPE 2018 organizers will contribute a short summary of downstream results (and correlations to intrinsic evaluation results) to the proceedings volume of the UD Parsing Shared Task.

Tentative Schedule

June 15, 2018
Availability of EPE Parser Inputs on TIRA
June 29, 2018
EPE Parser Outputs are Due on TIRA
July 6, 2018
End-to-End Evaluation Results Available

2018 Co-Organizers

The organizers of the EPE 2018 initiative can be reached at the following address:

XHTML 1.0 | Last updated: 2018-06-11 (13:06)