Motivation & Goals

The 2020 Conference on Computational Natural Language Learning (CoNLL) hosts a shared task (or ‘system bake-off’) on Cross-Framework Meaning Representation Parsing (MRP 2020). The goal of the task is to advance data-driven parsing into graph-structured representations of sentence meaning. All things semantic are receiving heightened attention in recent years. And despite remarkable advances in vector-based (continuous and distributed) encodings of meaning, ‘classic’ (discrete and hierarchically structured) semantic representations will continue to play an important role in ‘making sense’ of natural language. While parsing has long been dominated by tree-structured target representations, there is now growing interest in general graphs as more expressive and arguably more adequate target structures for sentence-level analysis beyond surface syntax and in particular for the representation of semantic structure.

Following up from its 2019 predecessor shared task, the MRP shared tasks series combines for the first time formally and linguistically different approaches to meaning representation in graph form in a uniform training and evaluation setup. Participants are invited to develop parsing systems that support five distinct semantic graph frameworks—which all encode core predicate–argument structure, among other things—in the same implementation. Training and evaluation data will be provided for all five frameworks. Participants are asked to design and train a system that predicts sentence-level meaning representations in all frameworks in parallel. Architectures that utilize complementary knowledge sources (e.g. via parameter sharing) are encouraged (though not required). Learning from multiple flavors of meaning representation in tandem has hardly been explored (with notable exceptions, e.g. the parsers of Peng et al., 2017; 2018; Hershcovich et al., 2018; Stanovsky & Dagan, 2018; or Lindemann et al., 2019).

The MRP tasks seek to reduce framework-specific ‘balkanization’ in the field of meaning representation parsing.  Expected outcomes include (a) a unifying formal model over different semantic graph banks, (b) uniform representations and scoring, (c) systematic contrastive evaluation across frameworks, and (d) increased cross-fertilization via transfer and multi-task learning.  We hope to engage the combined community of parser developers for graph-structured output representations, including from six prior framework-specific tasks at the Semantic Evaluation (SemEval) exercises between 2014 and 2019.  Owing to scarcity of semantic annotations across frameworks, the shared task is sub-divided into two tracks; (a) cross-framework meaning representation parsing, regrettably limited to English for the time being, and (b) cross-lingual meaning representation parsing, introducing one additional language for each framework.

Some semi-formal definitions and a brief review of the five semantic graph frameworks represented in the shared task are available on separate pages.  The task will provide training data across frameworks in a uniform JSON serialization, as well as conversion and scoring software. If the task sounds potentially interesting to you, please follow the instructions for prospective participants.

Tentative Schedule

March 30, 2020
Initial Public Call for Participation
Availability of Sample Training Graphs
Re-Release of 2019 Training & Development Data
Definition of Cross-Framework Evaluation Metric
Availability of Evaluation Software
April 27, 2020
Initial Release of English Training & Development Data
Second Public Call for Participation
Updated Sample Training Graphs
June 1, 2020
Cross-Lingual Training Data for Four Frameworks
Availability of English Morpho-Syntactic Companion Trees
June 15, 2020
Closing Date for Additional Data Nominations
July 27–August 10, 2020
Evaluation Period (Held-Out Data)
August 17, 2020
Preliminary Evaluation Results (Download)
Gold-Standard Evaluation Graphs
Template for System Description Papers
September 13, 2020
Submission of System Descriptions
September 28, 2020
Reviewer Feedback Available
October 12, 2020
Camera-Ready Manuscripts
November 19–20, 2020
Presentation and Discussion of Results

Differences to MRP 2019

The shared task is a follow-up and extension of its predecessor, MRP 2019, which was hosted as part of the 2019 Conference on Computational Natural Language Learning.  The proceedings volume from 2019 provides a detailed task overview, as well as system descriptions for eighteen partcipating teams.  Key differences in the 2020 edition of the MRP shared task include the addition of a graph-based encoding of Discourse Representation Structures (dubbed DRG); a generalization of Prague Tectogrammatical Graphs (to include more information from the original annotations); and a separate cross-lingual track, introducing one extra language (beyond English) for each of the frameworks involved.  To reduce the threshold to participation, two of the target frameworks represented in MRP 2019 are not in focus this year, viz. the purely bi-lexical DELPH-IN MRS Bi-Lexical Dependencies (DM) and Prague Semantic Dependencies (PSD).  These graphs largely overlap with the corresponding (but richer) frameworks in MRP 2020, EDS and PTG, respectively, and the original bi-lexical semantic dependency graphs remain independently available.

XHTML 1.0 | Last updated: 2020-10-04 (20:10)