McCaffery, Martin
(2018).
Word order judgement database.
[Data Collection]. Colchester, Essex:
UK Data Service.
10.5255/UKDA-SN-853215
We present a multifaceted investigation into the relevance of word order in machine translation. We introduce two tools, DTED and DERP, each using dependency structure to detect differences between the structures of machine-produced translations and human-produced references.
DTED applies the principle of Tree Edit Distance to calculate edit operations required to convert one structure into another. Four variants of DTED have been produced, differing in the importance they place on words which match between the two sentences. DERP represents a more detailed procedure, making use of the dependency relations between words when evaluating the disparities between paths connecting matching nodes.
In order to empirically evaluate DTED and DERP, and as a standalone contribution, we have produced WOJ-DB, a database of human judgments. Containing scores relating to translation adequacy and more specifically to word order quality, this is intended to support investigations into a wide range of translation phenomena.
We report an internal evaluation of the information in WOJ-DB, then use it to evaluate variants of DTED and DERP, both to determine their relative merit and their strength relative to third-party baselines. We present our conclusions about the importance of structure to the tools and their relevance to word order specifically, then propose further related avenues of research suggested or enabled by our work.
Data description (abstract)
The associated Ph.D thesis investigates the relevance of grammatical structure when using dependency parsing to evaluate multiple aspects of quality in machine-translated sentences. To this end, two tools were produced.
In order to evaluate the performance of these and other tools, a body of native English speakers were presented with a series of sentences and asked to rate their quality on two five-point Likert scales.
This dataset contains 1783 sets of scores provided by 36 participants and numerous automatic metrics for 1089 unique sentences.
Data creators: |
|
Contributors: |
|
Sponsors: |
Engineering and Physical Sciences Research Council
|
Topic classification: |
Media, communication and language Science and technology
|
Keywords: |
machine translation, sentence quality, surveys, automatic metrics, BLEU, translation evaluation
|
Project title: |
The mat sat on the cat: investigating structure in the evaluation of order in machine translation
|
Alternative title: |
WOJ-DB: a Word Order Judgement Database
|
Grant holders: |
Martin McCaffery
|
Project dates: |
From | To |
---|
27 May 2013 | 23 June 2017 |
|
Date published: |
28 Jun 2018 15:12
|
Last modified: |
03 Jul 2018 12:53
|
Collection period: |
Date from: | Date to: |
---|
6 October 2016 | 10 November 2016 |
|
Geographical area: |
St Andrews, Scotland |
Country: |
United Kingdom |
Spatial unit: |
Administrative > Communities (Scotland) |
Data collection method: |
Survey: native English speakers were presented with a series of sentences and asked to rate their quality on two five-point Likert scales.
|
Observation unit: |
Individual |
Kind of data: |
Numeric, Numeric, Text |
Type of data: |
Experimental data
, Qualitative and mixed methods data |
Resource language: |
English |
|
Data sourcing, processing and preparation: |
See Ph.D. thesis, chapter 6.
|
Rights owners: |
|
Contact: |
|
Notes on access: |
The Data Collection is available to any user without the requirement for registration for download/access.
|
Publisher: |
UK Data Service
|
Last modified: |
03 Jul 2018 12:53
|
|
Available Files
Data
Read me
Edit item (login required)
|
Edit Item |