BRAPT: A New Metric for Translation Evaluation Based on Psycholinguistic Perspectives

Abstract

There are some metrics to evaluate automatic text translations in the literature. However, the state-of-the-art of these metrics still has limitations. One of them is the dependence of an exact and ordered pairing of words for evaluating similarity among texts. Another, is the non-consideration of the semantics of the text in such comparison. Previous studies point out the need to analyze the semantics of words in the evaluation of translations. In this scenario, this paper presents a novel metric capable of evaluating the differences in automatic text translations that takes into account the semantics of the words presented in the texts. As a proof of concept, we selected ten journalistic texts written in English. These texts have been translated to Portuguese by a specialist and by three automatic text translation tools. Experimental results show the potential of the proposed metric in evaluating these translations, indicating it can perform better than the state-of-the-art metric.