A Comparison of Mainstream Large Language Models’ Performance in Chinese-to-Japanese Political Text Translation: An Empirical Analysis Based on BLEU and TER

Abstract

With the rapid development of generative artificial intelligence, Large Language Models (LLMs) are being increasingly applied in the field of machine translation. However, their performance in high-difficulty domains such as political text translation still requires systematic evaluation. Using the Report to the 20th National Congress of the Communist Party of China (CPC) as the research corpus, this study selects four mainstream LLMs—DeepSeek, Doubao, ChatGPT, and Gemini—as research subjects. Taking the official Japanese version translated by the Institute of Party History and Literature of the CPC Central Committee as the reference text, this study quantitatively evaluates the Chinese-to-Japanese translation results of each model using two automated evaluation metrics: BLEU (Bilingual Evaluation Understudy) and TER (Translation Edit Rate), supplemented by qualitative analysis through case comparisons. The results indicate that Gemini performed best across both BLEU and TER metrics, with its translations approaching human standards in terms of structural restoration, terminology handling, and stylistic conformity. ChatGPT and DeepSeek showed moderate overall performance, with differences that were not statistically significant. Doubao performed the worst in both metrics, with primary issues concentrated in the inappropriate use of honorifics (Keigo) and the mistranslation of specific technical terms. The conclusions of this paper provide empirical evidence for the application of generative AI in professional translation and offer references for the optimization of models for political text translation in the future.