Resolució anafòrica en traducció automàticael cas de l’espanyol i el català

  1. Alvarez-Vidal, Sergi 1
  1. 1 Universitat Pompeu Fabra
    info
    Universitat Pompeu Fabra

    Barcelona, España

    ROR https://ror.org/04n0g0b29

    Geographic location of the organization Universitat Pompeu Fabra
Journal:
Linguamática

ISSN: 1647-0818

Year of publication: 2024

Volume: 16

Issue: 1

Pages: 3-13

Type: Article

DOI: 10.21814/LM.16.1.424 DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Linguamática

Abstract

In the last decade, machine translation (MT) has increased its presence not only in the translation industry but also in society as a whole, in part due to the good results in quality produced by neural machine translation (NMT). Currently, large language models (LLMs) such as GPT (Generic Pre-trained Transformer) can generate text on endless topics, and also translate documents taking into account a larger context. Even so, for closely-related languages such as Spanish and Catalan rule-based machine translation (RBMT) is used daily to translate thousands of words. This article studies how RBMT, NMT and GPT perform translating from Spanish into Catalan, two Romance languages with very similar structure in which RBMT systems have shown to perform well. We use a challenge test set focusing on anaphora resolution, specifically weak pronouns, a group of pronouns which do not have a direct correlation between the two languages. As RBMT models only take into account sentence level information, we only study intra-sentential appearances. The goal is to assess a complex syntactic phenomenon which can help understand which system translates better contextual information. Results show the two GPT models tested are the ones with the less number of errors, followed by the NMT models. Even so, the number of errors in the model with the best results is 47\%, which does not correspond to general assessment results usually obtained for this language combination.

Bibliographic References

  • Alvarez, Sergi, Antoni Oliver & Toni Badia. 2019. Does NMT make a difference when post-editing closely related languages? The caseof Spanish-Catalan. En Machine Translation Summit XVII: Translator, Project and User Tracks, 49–56.
  • Aranberri, Nora, Gorka Labaka, Arantza Díaz de Ilarraza & Kepa Sarasola. 2017. Ebaluatoia: crowd evaluation for English–Basque machine translation. Language Resources and Evaluation 51(4). 1053–1084.10.1007/s10579-016-9335-x
  • Arnold, Doug, Dave Moffat, Louisa Sadler & Andrew Way. 1993. Automatic test suite genera-tion. Machine Translation 8 (1/2). 29–38.
  • Bahdanau, Dzmitry, Kyunghyun Cho & Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv[cs.CL]10.48550/arXiv.1409.0473
  • Barrault, Loïc, Magdalena Biesialska, Ondˇrej Bojar, Marta R. Costajussà, Christian Feder-mann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubeˇsi ́, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post & Marcos Zampieri. 2020. Fin-dings of the 2020 conference on machine translation (wmt). En 5th Conference on Machine Translation, 1–55.
  • Bawden, Rachel, Rico Sennrich, Alexandra Birch & Barry Haddow. 2018. Evaluating discour-se phenomena in neural machine translation. En Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1304–1313.10.18653/v1/N18-1118
  • Bayatli, Sevilay, Sefer Kurnaz, Ilnar Salimzianov, Jonathan North Washington & Francis M. Tyers. 2018. Rule-based machine translation from Kazakh to Turkish. En 21st Annual Con-ference of the European Association for Machine Translation, 49–58.
  • Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo & Marcello Federico. 2018. Neural versus phrase-based MT quality: An indepthanalysis on English–German and English–French. Computer Speech & Language 49. 52–70.10.1016/j.csl.2017.11.004
  • Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner,Sam McCandlish, Alec Radford, Ilya Sutskever & Dario Amodei. 2020. Language models are few-shot learners. arXiv [cs.CL]10.48550/arXiv.2005.14165
  • Brussel, Laura, Arda Tezcan & Lieve Macken. 2018. A fine-grained error analysis of NMT, PBMT and RBMT output for English-to-Dutch. En 11th International Conference on Language Resources and Evaluation (LREC),3799–3804.
  • Buysschaert, Joost, María Fernández-Parra, Koen Kerremans, Maarit Koponen & Gys-Walt van Egdom. 2018.L’acceptació dela disrupció digital en la formació en tra-ducció:la immersió tecnològica en simulacions de despatxos de traducció. Tra-dumàtica tecnologies de la traducció 16. 125–133.10.5565/rev/tradumatica.209
  • Castilho, Sheila, Clodagh Quinn Mallon, Rahel Meister & Shengya Yue. 2023. Do online machine translation systems care for context? what about a GPT model? En 24th Annual Conference of the European Association for Machine Translation, 393–417.
  • Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Yota Geor-gakopoulou, Pintu Lohar, Andy Way, AntonioValerio, Antonio Valerio Miceli Barone & Maria Gialama. 2017. A comparative quality evaluation of PBSMT and NMT using profesional translators. En Machine Translation Summit XVI, 116–131.
  • Costa-jussà, Marta R. 2017. Why Catalan–Spanish neural machine translation? analysis, comparison and combination with standard rule and phrase-based technologies. En 4th Workshop on NLP for Similar Languages, Varieties and Dialects, 55–62.10.18653/v1/W17-1207
  • van Deemter, Kees & Rodger Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics 26(4). 629–637.
  • Do Campo Bayón, María & Pilar Sánchez-Gijón.2019. Evaluating machine translation in alow-resource language combination: Spanish–Galician. En Machine Translation Summit XVII, 30–35.
  • España-Bonet, Cristina, Gorka Labaka, Arantza Díaz de Ilarraza & Lluís Màrquez. 2011. Hybrid machine translation guided by a rule–based system. En Machine Translation Summit XIII
  • Ferrando, Javier, Gerard I. Gállego, Belen Alastruey, Carlos Escolano & Marta R. Costa-jussà. 2022. Towards opening the blackbox of neural machine translation: Source and target interpretations of the transformer. En Conference on Empirical Methods in Natural Language Processing, 8756–8769.10.18653/v1/2022.emnlp-main.599
  • Fité Labaila, Ricard. 2001. La traducció automàtica aplicada a la premsa escrita. El casd’El Periódico en català. Treballs de Comunicació 21–25.
  • Fité Labaila, Ricard. 2007. Cas d’integració de la TA: El Periódico. Tradumàtica: traducció i tecnologies de la informació i la comunicación 4.
  • Forcada, Mikel L. 2017. Making sense of neuralmachine translation. Translation Spaces 6(2). 291–309. 10.1075/ts.6.2.06for
  • Forcada, Mikel L., Mireia Ginestí-Rosell, Jacob Nordfalk, Jim O’Regan, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Gema Ramírez-Sánchez & Francis M. Tyers. 2011. Apertium: a free/open-source platform for rule-based machine trans-lation.Machine Translation 25. 127–144.10.1007/s10590-011-9090-0
  • Ge, Niyu, John Hale & Eugene Charniak. 1998. A statistical approach to anaphora resolution. En 6th Workshop on Very Large Corpora, 161–170.
  • Guillou, Liane & Christian Hardmeier. 2016. PROTEST: A test suite for evaluating pro-nouns in machine translation. En 10th International Conference on Language Resources and Evaluation (LREC), 636–643.
  • Hendy, Amr, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify & Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? A comprehensive evaluation. arXiv [cs.CL]10.48550/arXiv.2302.09210
  • Hobbs, Jerry R. 1978. Resolving pronoun references. Lingua 44(4). 311–338. 10.1016/0024-3841(78)90006-2
  • Institut d’Estudis Catalans. 2016. Gramàtica dela llengua catalana. Institut d’Estudis Cata-lans
  • Isabelle, Pierre, Colin Cherry & George Foster.2017. A challenge set approach to evaluatingmachine translation. EnConference on Empi-rical Methods in Natural Language Processing,2486–2496.10.18653/v1/D17-1263
  • Islam, Md. Adnanul, Md. Saidul Hoque Anik & A. B. M. Alim Al Islam. 2022. An enhanced RBMT: When RBMT outperforms modern data–driven translators. IETE Technical Review 39(6). 1473–1484.10.1080/02564602.2022.2026828
  • Kehler, Andrew, Douglas Appelt, Lara Taylor & Aleksandr Simma. 2004. The (non)utility of predicate-argument frequencies for pronoun interpretation. En Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics,289–296.
  • Koponen, Maarit, Leena Salmi & Markku Nikulin. 2019. A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Machine Translation 33. 61–90.10.1007/s10590-019-09228-7
  • Lappin, Shalom & Herbert J. Leass. 1994. Analgorithm for pronominal anaphora resolution. Computational Linguistics 20(4). 535–561.
  • Mitkov, Ruslan. 1999. Introduction: Special issue on anaphora resolution in machine translation and multilingual NLP. Machine Translation 14(3). 159–161.10.1023/A:1011132522992
  • Mitkov, Ruslan, Sung-Kwon Choi & Randall Sharp. 1995. Anaphora resolution in machine translation. En 6th Conference on Theoretical and Methodological Issues in Machine Trans-lation of Natural Languages, 87–95.
  • Popovic, Maja & Sheila Castilho. 2019. Challenge test sets for MT evaluation. En Machine Translation Summit XVII, presentation. Rios, Annette, Mathias Müller & Rico Sennrich. 2018. The word sense disambiguation test suite at WMT18. En 3rd Conference on Machine Translation, 588–596. 10.18653/v1/W18-6437
  • Sennrich, Rico. 2017. How grammatical is character–level neural machine translation? Assessing MT quality with contrastive translation pairs. En 15th Conference of the European Chapter of the Association for Computational Linguistics, 376–382.
  • Sghaier, Mohamed Ali & Mounir Zrigui. 2020. Rule-based machine translation from Tuni-sian dialect to modern standard Arabic. Procedia Computer Science 176. 310–319. 10.1016/j.procs.2020.08.033
  • Stanovsky, Gabriel, Noah A. Smith & Luke Zettlemoyer. 2019. Evaluating gender bias in mac-hine translation. En 57th Annual Meeting of the Association for Computational Linguistics, 1679–1684.10.18653/v1/P19-1164
  • Tognini-Bonelli, Elena. 2001. Corpus linguistics at work. John Benjamins Publishing Company. 10.1075/scl.6
  • Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Go-mez, Lukasz Kaiser & Illia Polosukhin. 2017. Attention is all you need. arXiv [cs.CL]10.48550/arXiv.1706.03762
  • Voita, Elena, Pavel Serdyukov, Rico Sennrich & Ivan Titov. 2018.Context-aware neural machine translation learns anaphora resolution. En 56th Annual Meeting of the Associati-on for Computational Linguistics, 1264–1274.10.18653/v1/P18-1117
  • Wicks, Rachel & Matt Post. 2022. Does sentence segmentation matter for machine translati-on? En 7th Conference on Machine Translation, 843–854.
Data source: Dialnet