Word order judgement database

McCaffery, Martin (2018). Word order judgement database. [Data Collection]. Colchester, Essex: UK Data Service. 10.5255/UKDA-SN-853215

We present a multifaceted investigation into the relevance of word order in machine translation. We introduce two tools, DTED and DERP, each using dependency structure to detect differences between the structures of machine-produced translations and human-produced references. DTED applies the principle of Tree Edit Distance to calculate edit operations required to convert one structure into another. Four variants of DTED have been produced, differing in the importance they place on words which match between the two sentences. DERP represents a more detailed procedure, making use of the dependency relations between words when evaluating the disparities between paths connecting matching nodes. In order to empirically evaluate DTED and DERP, and as a standalone contribution, we have produced WOJ-DB, a database of human judgments. Containing scores relating to translation adequacy and more specifically to word order quality, this is intended to support investigations into a wide range of translation phenomena. We report an internal evaluation of the information in WOJ-DB, then use it to evaluate variants of DTED and DERP, both to determine their relative merit and their strength relative to third-party baselines. We present our conclusions about the importance of structure to the tools and their relevance to word order specifically, then propose further related avenues of research suggested or enabled by our work.

Data description (abstract)

The associated Ph.D thesis investigates the relevance of grammatical structure when using dependency parsing to evaluate multiple aspects of quality in machine-translated sentences. To this end, two tools were produced. In order to evaluate the performance of these and other tools, a body of native English speakers were presented with a series of sentences and asked to rate their quality on two five-point Likert scales. This dataset contains 1783 sets of scores provided by 36 participants and numerous automatic metrics for 1089 unique sentences.

Data creators:
Creator Name Affiliation ORCID (as URL)
McCaffery Martin University of St Andrews https://orcid.org/0000-0003-3902-5160
Contributors:
Name Affiliation ORCID (as URL)
Nederhof Mark-Jan University of St Andrews https://orcid.org/0000-0002-1845-6829
Sponsors: Engineering and Physical Sciences Research Council
Topic classification: Media, communication and language
Science and technology
Keywords: machine translation, sentence quality, surveys, automatic metrics, BLEU, translation evaluation
Project title: The mat sat on the cat: investigating structure in the evaluation of order in machine translation
Alternative title: WOJ-DB: a Word Order Judgement Database
Grant holders: Martin McCaffery
Project dates:
FromTo
27 May 201323 June 2017
Date published: 28 Jun 2018 15:12
Last modified: 03 Jul 2018 12:53

Available Files

Data

Read me

Downloads

data downloads and page views since this item was published

View more statistics

Altmetric

Edit item (login required)

Edit Item Edit Item