Large SMT Data-Sets Extracted from Wikipedia

From Wikipedia Quality
Jump to: navigation, search
Large SMT Data-Sets Extracted from Wikipedia
Authors
Dan Tufiš
Radu Ion
Ştefan Daniel Dumitrescu
Dan C. Ştefǎnescu
Publication date
2014
ISBN
978-295174088-4
Links

Large SMT Data-Sets Extracted from Wikipedia - scientific work about Wikipedia quality published in 2014, written by Dan Tufiš, Radu Ion, Ştefan Daniel Dumitrescu and Dan C. Ştefǎnescu.

Overview

The article presents experiments on mining Wikipedia for extracting SMT useful sentence pairs in three language pairs. Each extracted sentence pair is associated with a cross-lingual lexical similarity score based on which, several evaluations have been conducted to estimate the similarity thresholds which allow the extraction of the most useful data for training three-language pairs SMT systems. The experiments showed that for a similarity score higher than 0.7 all sentence pairs in the three language pairs were fully parallel. However, including in the training sets less parallel sentence pairs (that is with a lower similarity score) showed significant improvements in the translation quality (BLEU-based evaluations). The optimized SMT systems were evaluated on unseen test-sets also extracted from Wikipedia. As one of the main goals of their work was to help Wikipedia contributors to translate (with as little post editing as possible) new articles from major languages into less resourced languages and vice-versa, authors call this type of translation experiments "in-genre" translation. As in the case of "in-domain" translation, their evaluations showed that using only "in-genre" training data for translating same genre new texts is better than mixing the training data with "out-of-genre" (even) parallel texts.

Embed

Wikipedia Quality

Tufiš, Dan; Ion, Radu; Dumitrescu, Ştefan Daniel; Ştefǎnescu, Dan C.. (2014). "[[Large SMT Data-Sets Extracted from Wikipedia]]". ICIC Express Letters Volume 8, Issue 7, July 2014, pp. 1877-1882. ISBN: 978-295174088-4.

English Wikipedia

{{cite journal |last1=Tufiš |first1=Dan |last2=Ion |first2=Radu |last3=Dumitrescu |first3=Ştefan Daniel |last4=Ştefǎnescu |first4=Dan C. |title=Large SMT Data-Sets Extracted from Wikipedia |date=2014 |isbn=978-295174088-4 |url=https://wikipediaquality.com/wiki/Large_SMT_Data-Sets_Extracted_from_Wikipedia |journal=ICIC Express Letters Volume 8, Issue 7, July 2014, pp. 1877-1882}}

HTML

Tufiš, Dan; Ion, Radu; Dumitrescu, Ştefan Daniel; Ştefǎnescu, Dan C.. (2014). &quot;<a href="https://wikipediaquality.com/wiki/Large_SMT_Data-Sets_Extracted_from_Wikipedia">Large SMT Data-Sets Extracted from Wikipedia</a>&quot;. ICIC Express Letters Volume 8, Issue 7, July 2014, pp. 1877-1882. ISBN: 978-295174088-4.