Motivation & Goals

The 2020 Conference on Computational Natural Language Learning (CoNLL) hosts a shared task (or ‘system bake-off’) on Cross-Framework Meaning Representation Parsing (MRP 2020). The goal of the task is to advance data-driven parsing into graph-structured representations of sentence meaning. All things semantic are receiving heightened attention in recent years. And despite remarkable advances in vector-based (continuous and distributed) encodings of meaning, ‘classic’ (discrete and hierarchically structured) semantic representations will continue to play an important role in ‘making sense’ of natural language. While parsing has long been dominated by tree-structured target representations, there is now growing interest in general graphs as more expressive and arguably more adequate target structures for sentence-level analysis beyond surface syntax and in particular for the representation of semantic structure.

Following up from its 2019 predecessor shared task, the MRP shared tasks series combines for the first time formally and linguistically different approaches to meaning representation in graph form in a uniform training and evaluation setup. Participants are invited to develop parsing systems that support five distinct semantic graph frameworks—which all encode core predicate–argument structure, among other things—in the same implementation. Training and evaluation data will be provided for all five frameworks. Participants are asked to design and train a system that predicts sentence-level meaning representations in all frameworks in parallel. Architectures that utilize complementary knowledge sources (e.g. via parameter sharing) are encouraged (though not required). Learning from multiple flavors of meaning representation in tandem has hardly been explored (with notable exceptions, e.g. the parsers of Peng et al., 2017; 2018; Hershcovich et al., 2018; Stanovsky & Dagan, 2018; or Lindemann et al., 2019).

The MRP tasks seek to reduce framework-specific ‘balkanization’ in the field of meaning representation parsing. Expected outcomes include (a) a unifying formal model over different semantic graph banks, (b) uniform representations and scoring, (c) systematic contrastive evaluation across frameworks, and (d) increased cross-fertilization via transfer and multi-task learning. We hope to engage the combined community of parser developers for graph-structured output representations, including from six prior framework-specific tasks at the Semantic Evaluation (SemEval) exercises between 2014 and 2019. Owing to scarcity of semantic annotations across frameworks, the shared task is sub-divided into two tracks; (a) cross-framework meaning representation parsing, regrettably limited to English for the time being, and (b) cross-lingual meaning representation parsing, introducing one additional language for each framework.

Some semi-formal definitions and a brief review of the five semantic graph frameworks represented in the shared task are available on separate pages. The task will provide training data across frameworks in a uniform JSON serialization, as well as conversion and scoring software. If the task sounds potentially interesting to you, please follow the instructions for prospective participants.

Tentative Schedule

March 30, 2020: Initial Public Call for Participation; Availability of Sample Training Graphs; Re-Release of 2019 Training & Development Data; Definition of Cross-Framework Evaluation Metric; Availability of Evaluation Software
April 27, 2020: Initial Release of English Training & Development Data; Second Public Call for Participation; Updated Sample Training Graphs
June 1, 2020: Cross-Lingual Training Data for Four Frameworks; Availability of English Morpho-Syntactic Companion Trees
June 15, 2020: Closing Date for Additional Data Nominations
July 27–August 10, 2020: Evaluation Period (Held-Out Data)
August 17, 2020: Preliminary Evaluation Results (Download); Gold-Standard Evaluation Graphs; Template for System Description Papers
September 13, 2020: Submission of System Descriptions
September 28, 2020: Reviewer Feedback Available
October 12, 2020: Camera-Ready Manuscripts
November 19–20, 2020: Presentation and Discussion of Results

Differences to MRP 2019

The shared task is a follow-up and extension of its predecessor, MRP 2019, which was hosted as part of the 2019 Conference on Computational Natural Language Learning. The proceedings volume from 2019 provides a detailed task overview, as well as system descriptions for eighteen partcipating teams. Key differences in the 2020 edition of the MRP shared task include the addition of a graph-based encoding of Discourse Representation Structures (dubbed DRG); a generalization of Prague Tectogrammatical Graphs (to include more information from the original annotations); and a separate cross-lingual track, introducing one extra language (beyond English) for each of the frameworks involved. To reduce the threshold to participation, two of the target frameworks represented in MRP 2019 are not in focus this year, viz. the purely bi-lexical DELPH-IN MRS Bi-Lexical Dependencies (DM) and Prague Semantic Dependencies (PSD). These graphs largely overlap with the corresponding (but richer) frameworks in MRP 2020, EDS and PTG, respectively, and the original bi-lexical semantic dependency graphs remain independently available.

Definitions	Frameworks	Objectives	Data & Tools	Evaluation	Results	News & Contact
Motivation & Goals The 2020 Conference on Computational Natural Language Learning (CoNLL) hosts a shared task (or ‘system bake-off’) on Cross-Framework Meaning Representation Parsing (MRP 2020). The goal of the task is to advance data-driven parsing into graph-structured representations of sentence meaning. All things semantic are receiving heightened attention in recent years. And despite remarkable advances in vector-based (continuous and distributed) encodings of meaning, ‘classic’ (discrete and hierarchically structured) semantic representations will continue to play an important role in ‘making sense’ of natural language. While parsing has long been dominated by tree-structured target representations, there is now growing interest in general graphs as more expressive and arguably more adequate target structures for sentence-level analysis beyond surface syntax and in particular for the representation of semantic structure. Following up from its 2019 predecessor shared task, the MRP shared tasks series combines for the first time formally and linguistically different approaches to meaning representation in graph form in a uniform training and evaluation setup. Participants are invited to develop parsing systems that support five distinct semantic graph frameworks—which all encode core predicate–argument structure, among other things—in the same implementation. Training and evaluation data will be provided for all five frameworks. Participants are asked to design and train a system that predicts sentence-level meaning representations in all frameworks in parallel. Architectures that utilize complementary knowledge sources (e.g. via parameter sharing) are encouraged (though not required). Learning from multiple flavors of meaning representation in tandem has hardly been explored (with notable exceptions, e.g. the parsers of Peng et al., 2017; 2018; Hershcovich et al., 2018; Stanovsky & Dagan, 2018; or Lindemann et al., 2019). The MRP tasks seek to reduce framework-specific ‘balkanization’ in the field of meaning representation parsing. Expected outcomes include (a) a unifying formal model over different semantic graph banks, (b) uniform representations and scoring, (c) systematic contrastive evaluation across frameworks, and (d) increased cross-fertilization via transfer and multi-task learning. We hope to engage the combined community of parser developers for graph-structured output representations, including from six prior framework-specific tasks at the Semantic Evaluation (SemEval) exercises between 2014 and 2019. Owing to scarcity of semantic annotations across frameworks, the shared task is sub-divided into two tracks; (a) cross-framework meaning representation parsing, regrettably limited to English for the time being, and (b) cross-lingual meaning representation parsing, introducing one additional language for each framework. Some semi-formal definitions and a brief review of the five semantic graph frameworks represented in the shared task are available on separate pages. The task will provide training data across frameworks in a uniform JSON serialization, as well as conversion and scoring software. If the task sounds potentially interesting to you, please follow the instructions for prospective participants. Tentative Schedule March 30, 2020 Initial Public Call for Participation Availability of Sample Training Graphs Re-Release of 2019 Training & Development Data Definition of Cross-Framework Evaluation Metric Availability of Evaluation Software April 27, 2020 Initial Release of English Training & Development Data Second Public Call for Participation Updated Sample Training Graphs June 1, 2020 Cross-Lingual Training Data for Four Frameworks Availability of English Morpho-Syntactic Companion Trees June 15, 2020 Closing Date for Additional Data Nominations July 27–August 10, 2020 Evaluation Period (Held-Out Data) August 17, 2020 Preliminary Evaluation Results (Download) Gold-Standard Evaluation Graphs Template for System Description Papers September 13, 2020 Submission of System Descriptions September 28, 2020 Reviewer Feedback Available October 12, 2020 Camera-Ready Manuscripts November 19–20, 2020 Presentation and Discussion of Results Differences to MRP 2019 The shared task is a follow-up and extension of its predecessor, MRP 2019, which was hosted as part of the 2019 Conference on Computational Natural Language Learning. The proceedings volume from 2019 provides a detailed task overview, as well as system descriptions for eighteen partcipating teams. Key differences in the 2020 edition of the MRP shared task include the addition of a graph-based encoding of Discourse Representation Structures (dubbed DRG); a generalization of Prague Tectogrammatical Graphs (to include more information from the original annotations); and a separate cross-lingual track, introducing one extra language (beyond English) for each of the frameworks involved. To reduce the threshold to participation, two of the target frameworks represented in MRP 2019 are not in focus this year, viz. the purely bi-lexical DELPH-IN MRS Bi-Lexical Dependencies (DM) and Prague Semantic Dependencies (PSD). These graphs largely overlap with the corresponding (but richer) frameworks in MRP 2020, EDS and PTG, respectively, and the original bi-lexical semantic dependency graphs remain independently available.
Definitions	Frameworks	Objectives	Data & Tools	Evaluation	Results	News & Contact
XHTML 1.0 \| Last updated: 2020-10-04 (20:10)