From BLEU to COMET: A Comparative Analysis of Traditional and Modern Evaluation Metrics in Arabic–Turkish Audiovisual Translation
Yasemin Yazal 
Abstract. Translation, one of the fundamental needs of the modern era, serves as a vital bridge of communication between people and cultures. The growing demand for translation has made the development of automated translation systems inevitable, and their use has rapidly expanded in the digital age. Although these systems enable faster translation processes, their outputs may still remain limited in terms of cultural transfer, semantic consistency, contextual adequacy, and overall coherence. Consequently, automatic evaluation metrics have become increasingly important for assessing the quality of machine-generated translations. This study provides a comparative theoretical and practical analysis of traditional metrics, namely BLEU, ROUGE, and METEOR, and modern metrics, namely BERTScore, LaBSE, and COMET, used in machine translation evaluation. Since traditional metrics mainly rely on surface-level lexical overlap between candidate and reference texts, they may remain limited in evaluating semantic and contextual equivalence in morphologically complex language pairs such as Turkish and Arabic. Therefore, the empirical section of this study focuses on multilingual analysis through modern evaluation metrics. The findings indicate that Language Reactor subtitle translations often achieve higher scores in modern metrics in terms of semantic alignment with the source text, whereas human translations yield more successful results in terms of cultural transfer, fluency, and contextual naturalness.
Keywords: machine translation; COMET; BERTScore; LaBSE; language reactor