TranslateGemma Technical Report

2026-01-21

TranslateGemma Technical Report

Google Translate Research Team

We present TranslateGemma, a suite of open machine translation models based on the Gemma 3 foundation models. To enhance the inherent multilingual capabilities of Gemma 3 for the translation task, we employ a two-stage fine-tuning process. First, supervised fine-tuning is performed using a rich mixture of high-quality large-scale synthetic parallel data generated via state-of-the-art models and human-translated parallel data. This is followed by a reinforcement learning phase, where we optimize translation quality using an ensemble of reward models, including MetricX-QE and AutoMQM, targeting translation quality. We demonstrate the effectiveness of TranslateGemma with human evaluation on the WMT25 test set across 10 language pairs and with automatic evaluation on the WMT24++ benchmark across 55 language pairs. Automatic metrics show consistent and substantial gains over the baseline Gemma 3 models across all sizes. Notably, smaller TranslateGemma models often achieve performance comparable to larger baseline models, offering improved efficiency. We also show that TranslateGemma models retain strong multimodal capabilities, with enhanced performance on the Vistra image translation benchmark. The release of the open TranslateGemma models aims to provide the research community with powerful and adaptable tools for machine translation.

1. Introduction

In an increasingly interconnected world, machine translation (MT) plays a pivotal role in bridg- ing language barriers, facilitating global commu- nication, and democratizing access to informa- tion. The development of large language mod- els (LLMs) has significantly advanced the state-of- the-art in MT. However, progress is greatly bene- fitted by the availability of strong, open models that allow for transparency, reproducibility, and community-driven innovation.

To this end, we present TranslateGemma, an open variant of the Gemma 3 foundation model (Gemma Team, 2025), specifically enhanced for machine translation. While Gemma 3 is already a potent multilingual LLM, TranslateGemma has been further refined to deliver superior trans- lation quality. This improvement is achieved through a two-stage process: Supervised Fine- tuning (SFT) on a diverse corpus of parallel data (Section 3) and Reinforcement Learning (RL) from human and model-based feedback (Sec- tion 4).

Our SFT approach leverages a blend of human- translated and synthetically-generated parallel texts, carefully curated to improve translation quality without compromising the model’s gen-

eral capabilities. The RL phase employs a com- bination of reward models designed to optimize translation quality. We demonstrate the efficacy of TranslateGemma on the WMT25 and WMT24++ datasets, showing substantial gains across 55 lan- guage pairs.

Furthermore, TranslateGemma retains the in- herent multimodal capabilities of the original Gemma 3 model. Our experiments on the Vis- tra corpus (Salesky et al., 2024) indicate that the enhancements in text translation also positively impact image translation performance, showcas- ing the model’s versatility. We believe the release of TranslateGemma will provide a valuable re- source for researchers and practitioners in the field of machine translation.

2. Training data

We use two types of data for the training of the models, most of it shared between the SFT and RL phases.

2.1. Synthetic Gemini-Generated Translation Data

Our goal is to generate high-quality synthetic data for each language, as this has been shown

Corresponding author(s): vilar@google.com. See Contributions section for full author list. © 2026 Google. All rights reserved

TranslateGemma Technical Report

to greatly improve translation quality (Finkel- stein et al., 2024). As the source of monolingual data we use the MADLAD-400 corpus (Kudugunta et al., 2023).

We aim to produce up to 10K synthetic exam- ples per language pair. In order to select the source sentences that potentially benefit more from the synthetic data generation, we first bucket the original segments by length. We then sample each bucket to obtain 1 million source seg- ments for each language pair we wish to generate synthetic data for. We then run a preliminary fil- tering step across these source segments where we take 2 samples from Gemini 2.5 Flash, once using greedy decoding and once sampled with a temperature of 1.0 and compare their scores ac- cording to MetricX 24-QE (Juraska et al., 2024). We select the sources where the sample achieves the largest improvement over the greedy decod- ing. The intuition behind this source filtering ap- proach is that we wish to select sources that will benefit the most from 128-sample QE decoding, so we use 2 samples as a low-cost approximation.

After this selection process, for each of the sources for each language pair we generate 128 samples from Gemini 2.5 Flash and then apply a MetricX 24-QE filter to select the best-performing examples. We generate translations of two dis- tinct lengths this way: individual sentences and text blobs of up to 512 tokens. This way we aim to support both translations of individual segments as well as longer texts. For generating these trans- lations we used the same prompt as we used for further training (see Section 5.2). In order to avoid formatting issues or erroneous translations, we apply an additional formatting filtering step, again based on Gemini 2.5 Flash. This method- ology was applied for all language pairs that are covered by WMT24++ (Deutsch et al., 2025) plus an additional set of 30 language pairs that are specified in Appendix B.

2.2. Human-Generated Translation Data

To increase the diversity and script coverage of the data we also include data for additional lower- resource languages. For these languages, we opt to use human-generated parallel data instead.

This data comes from the SMOL (Caswell et al., 2025) and GATITOS (Jones et al., 2023) datasets. SMOL covers 123 languages and GATITOS covers 170.

2.3. Language distribution

The final proportion of languages for the SFT and RL phases can be found in Figure 1. For RL we used the same translation data as for SFT, except for GATITOS and SMOL that were used in SFT only. We provide the full list of languages that were included in training in Appendix C.

2.4. Generic Instruction-Following Data

Our SFT mixture also includes 30% generic instruction-following data from the original Gemma 3 mixture. The purpose of including this data is to prevent the model from overfitting to the translation task and to maintain generic instruction-following capabilities.

3. Supervised Fine-Tuning

For supervised fine-tuning (SFT), we begin with the released Gemma 3 27B, 12B and 4B check- points. We use parallel data including both human-generated texts as well as synthetic data generated by Gemini (Gemini Team, 2025), as described in Section 2. In addition we use generic instruction-following data. We use the Kauldron SFT tooling1 to fine-tune the Gemma 3 check- points. For fine-tuning we use the AdaFactor opti- mizer (Shazeer and Stern, 2018) with a learning rate of 0.0001 and a batch size of 64, running for 200k steps. We update all model parameters, but freeze the embedding parameters, as preliminary experiments indicated this helped with transla- tion performance for languages and scripts not covered in the SFT data mix.

4. Reinforcement Learning

We performed reinforcement learning on top of the SFT checkpoint, using an ensemble of metrics

1https://kauldron.readthedocs.io/en/latest/

2

TranslateGemma Technical Report

English-Catalan 1.2%

Jamaican Patois 0.0% Chakma (Latin 0.1% English (en)- 0.2% English (en)- 0.4% English-Turkish 0.4% Portuguese- 0.5% French-English 0.5%

English-Serbian 0.6%

English-Norwegian 1.0%

Japanese-Chinese 7.6% English-Ukrainian 6.3% English-Czech 5.9% English-Swedish 2.5% English-Marathi 1.9% English-Croatian 1.8% English-Lithuanian 1.8% English-Serbian 1.8% English-Indonesian 1.8% English-Icelandic 1.7%

(a) SFT data mixture.

English-Gujarati 1.8%

other 0.8% English-Italian 0.6% Chinese-English 0.6% English-Kannada 0.8%

English-Finnish 1.1%

English-Malayalam 1.2%

English-French 1.3%

English-Swahili 1.4%

Japanese-Chinese 9.3%

English-Ukrainian 7.7%

English-Czech 7.2% English-Swedish 3.1% English-Marathi 2.3% English-Croatian 2.3% English-Lithuanian 2.2% English-Serbian 2.2% English-Indonesian 2.2% English-Icelandic 2.1%

(b) RL data mixture.

Figure 1 | Language distribution in the TranslateGemma data mixtures measured as model tokens.

as reward models, to further boost translation quality.

We used the following metrics as reward mod- els during RL:

• MetricX-24-XXL-QE (Juraska et al., 2024), a learned, regression-based translation metric producing a floating point score between 0 (best) and 25 (worst), matching the standard Multidimensional Quality Metrics (MQM) score range (Freitag et al., 2021). Met- ricX scores were linearly rescaled, using 5.0 − score, when computing rewards, so that higher scores indicate better quality. Al- though MetricX can take source, reference, and hypothesis as input, we used it as a QE metric by passing in an empty reference. • Gemma-AutoMQM-QE, a finetuned Au- toMQM model (Fernandes et al., 2023). This model was initialized from the Gemma 3- 27B-IT checkpoint (Gemma Team, 2025), and was trained on MQM ratings data from WMT 2020 - WMT 2023 (Freitag et al., 2021; Lommel et al., 2014). Default MQM weights (Freitag et al., 2021) were used in comput- ing (token-level) rewards from AutoMQM outputs. As with MetricX, it ignores the ref- erence translation. • ChrF (Popović, 2015), a lexical overlap-based translation metric. This was the only reward model for which the (synthetic) references were used. ChrF scores were scaled by a factor of two to be on approximately the same scale as the other rewards.

• Naturalness Autorater developed in-house, using the base RL policy model as a prompted LLM-as-a-Judge. As with AutoMQM, this Au- torater elicited span-level annotations. This Autorater was instructed to penalize spans in the machine-translated text which did not sound like they were produced by a native speaker (conditioned on the naturalness er- rors in the output not stemming from an un- natural source input). • Generalist reward model covering many tasks, including reasoning, instruction fol- lowing, and multilingual abilities, adapted from the general Gemma 3 post-training setup (Gemma Team, 2025).

We used RL algorithms extended to support token-level advantages, which were added to the advantages computed from sequence-level rewards. This allowed us to use fine-grained, span-level reward signals from AutoMQM and the Naturalness Autorater directly, for improved credit assignment and training efficiency in the spirit of Ramos et al. (2025). See Figure 2 for an illustration of how MetricX and AutoMQM rewards were (additively) combined during ad- vantage computation. The combined advantages were then batch-normalized.

5. Automatic Evaluation

5.1. Text translation

We evaluate TranslateGemma using MetricX 24 (Juraska et al., 2024) and Comet22 (Rei et al.,

3

TranslateGemma Technical Report

Figure 2 | Illustration of how sequence-level and token-level rewards are additively combined during advantage computation in RL. Note that advantage is computed from sequence-level rewards as ‘reward-to-go’, meaning that rewards are broadcast uniformly to every token.

Table 1 | Automatic evaluation results using Met- ricX and Comet22 (C22) on WMT24++.

Size System MetricX↓ C22↑

27B Gemma 3 4.04 83.1 TranslateGemma 3.09 84.4

12B Gemma 3 4.86 81.6 TranslateGemma 3.60 83.5

4B Gemma 3 6.97 77.2 TranslateGemma 5.32 80.1

2022). The TranslateGemma models consistently show improved translation quality compared to the baseline Gemma 3 models across all evaluated sizes and metrics, as detailed in Table 1.

For the 27B parameter model, the Trans- lateGemma version attains an average MetricX score of 3.09, a substantial reduction from the baseline Gemma 3’s score of 4.04. This repre- sents a relative decrease of approximately 23.5%, signaling a marked increase in translation fidelity. Similar trends are observed for the other model sizes. The 12B TranslateGemma model achieves a MetricX of 3.60, down from 4.86 for the base-

line (a 25.9% reduction), while the 4B Trans- lateGemma model scores 5.32, compared to 6.97 for the baseline (a 23.6% reduction).

Comet22 confirms the trend of improvements for the TranslateGemma model. In addition, this shows that improvements carry over to metrics not explicitly optimized for in the RL phase. For instance, the 12B TranslateGemma model shows a score of 83.5, up from 81.6. The 4B Trans- lateGemma model exhibits even larger increases, with Comet22 rising from 77.2 to 80.1.

The effect of model scale on performance is also apparent. As expected, larger models tend to produce better translations within both the base- line and TranslateGemma series. However, the en- hancements brought by the TranslateGemma fine- tuning are such that smaller TranslateGemma models can achieve performance levels compa- rable to or even exceeding those of larger base- line models. Notably, the 12B TranslateGemma model surpasses the performance of the larger 27B baseline Gemma 3 model. Similarly, the 4B TranslateGemma model achieves comparable re- sults to the 12B baseline Gemma 3 model. This efficiency gain allows for high-quality translation with reduced computational resources.

A more granular analysis of MetricX scores for

4

TranslateGemma Technical Report

each of the 55 language pairs, presented in Ap- pendix A, reveals that the improvements of Trans- lateGemma are consistent across all 55 language pairs evaluated. Some example improvements for specific languages are

• English to German: 1.63 down to 1.19, • English to Spanish: 2.54 down to 1.88, • English to Hebrew: 3.90 down to 2.72, • English to Swahili: 5.92 down to 4.45, • English to Lithuanian: 6.01 down to 4.39, • English to Estonian: 6.40 down to 4.61 and • English to Icelandic: 8.31 down to 5.69.

These examples highlight the model’s improved ability to handle a diverse range of languages, both for high-resource languages (e.g. German, English) as well as low-resource ones (e.g. Ice- landic, Swahili).

We also hypothesize that the 27B model, with its higher capacity, will have benefited more from the vast amount of languages seen during the SFT phase (detailed in Appendix C), although we do not have direct experimental confirmation of this.

5.2. Prompting the Model

The model has been trained using the prompt shown in Figure 3, which is also the prompt we used in our evaluations. We recommend using the same prompt for producing new translations. Tools for automatically wrapping the text with it are provided in the model repository.

5.3. Image Translation

We used the Vistra benchmark (Salesky et al., 2024) to assess whether the models retained their ability to translate text within images after our ad- ditional training steps. Note that no multimodal training data was used in the SFT or RL steps reported in this work. In order to simplify the evaluation protocol, we selected only images that, according to the reference, contained a single in- stance of text. This resulted in a set of 264 images. An example is shown in Figure 4. The input to the model was just the image together with a

Table 2 | Automatic evaluation results using Met- ricX and Comet22 (C22) for image translation performance, on the Vistra corpus. The scores are the average of translating from English into German, Spanish, Russian and Chinese.

Size System MetricX↓ C22↑

27B Gemma 3 2.03 76.1 TranslateGemma 1.58 77.7

12B Gemma 3 2.33 74.9 TranslateGemma 2.08 72.8

4B Gemma 3 2.60 69.1 TranslateGemma 2.58 70.7

prompt asking it to translate the text in it.2 In particular, we did not include any other informa- tion about the text, like its location in the image or a previous OCR pass.

The results, presented in Table 2, show that TranslateGemma retains the image processing capabilities of the base Gemma 3 models. The improvements in translation quality attained by TranslateGemma carry over for this task, with the exception of the 12B model measured in Comet22. We see MetricX score improvements of nearly 0.5 points in the case of the 27B model, or 0.25 for the 12B model.

The smaller 4B model obtains only small im- provements when compared to the baseline, prob- ably due to its limited capacity.

6. Human Evaluation

We conduct an additional human evaluation on a limited set of language directions to measure TranslateGemma’s translation performance. We do so using MQM (Freitag et al., 2021; Lom- mel et al., 2014), a human evaluation frame- work where professional translators highlight er- ror spans in translations, with document context, assigning a severity and category to each, with a score being automatically derived by counting

2The model release also includes an interface for image translation, similar to the one for text translation.

5

TranslateGemma Technical Report

You are a professional {source_lang} ({src_lang_code}) to {target_lang} ({tgt_lang_code}) translator. Your goal is to accurately convey the meaning and nuances of the original {source_lang} text while adhering to {target_lang} grammar, vocabulary, and cultural sensitivities. Produce only the {target_lang} translation, without any additional explanations or commentary. Please translate the following {source_lang} text into {target_lang}:\n\n\n{text}

Figure 3 | Preferred prompt when using the model. source_lang refers to the source language name, e.g. English, src_lang_code to the source language code, e.g. en-US, target_lang to the target language, e.g. German, and tgt_lang_code to the target language code, i.e. de-DE.

Figure 4 | Example of the pictures included in the Vistra benchmark.

the errors with a weighting scheme. We collected the annotations using the open-source Anthea tool.3 We evaluate the models in 10 language pairs, from 3 distinct source languages:

• English to German • English to Chinese (Simplified) • English to Italian • English to Serbian (Cyrillic) • English to Korean • English to Swahili (Kenyan) • English to Marathi • Czech to Ukrainian • Czech to German • Japanese to English

We selected this set to have a mix of high- and low-resource languages, in addition to having

3https://github.com/google-research/ google-research/tree/master/anthea

different language families and writing systems. The source data is all taken from the WMT25 translation task, using the literary, news, and so- cial domains. For all language pairs, we evalu- ated TranslateGemma 12B and 27B, as well as Gemma 3 27B.

To avoid issues with rater fatigue, each docu- ment in the dataset was truncated at paragraph boundaries to have no more than 12 source sen- tences, skipping documents with more than 12 sentences in the first paragraph. However, for the literary domain, where each document is an entire book chapter, documents were split into “chunks” of 1 or more paragraphs up to the 12-sentence limit, with each chunk being human-evaluated in isolation. Following Riley et al. (2024), we used a “pseudo-SxS” rater as- signment, where all system outputs for a particu- lar source document were evaluated by the same rater.

The results can be found in Table 3. For most language pairs, the human evaluation con- firms the trend we see on the automatic met- rics, with TranslateGemma clearly outperforming Gemma 3. There are two exceptions: when the target language is German, where both models are on par, and Japanese→English where Trans- lateGemma actually suffers a regression. Looking into the error categorization, we found that this is due to mistranslation of named entities, while other error categories did improve.

The improvements for TranslateGemma are es- pecially relevant for low-resource language pairs. E.g. for English to Marathi we obtain an improve- ment of 1.6 points, or 1.0 for English to Swahili or

6

TranslateGemma Technical Report

Table 3 | MQM results of the human evaluation for TranslateGemma and Gemma 3. Lower scores are better.

TranslateGemma Gemma 3

Language Pair 27B 12B 27B

English→Italian 1.8 2.0 2.5 English→German 2.3 3.2 2.2 English→Marathi 3.1 4.6 4.7 English→Korean 3.1 4.6 3.8 English→Swahili 4.2 5.2 5.2 Czech→Ukrainian 5.3 8.5 6.3 English→Chinese 6.3 8.4 7.4 English→Serbian 8.7 15.8 10.4 Czech→German 10.3 11.4 10.2 Japanese→English 13.4 15.7 11.6

Czech to Ukrainian. The human evaluation also confirms the performance difference between the 27B and 12B TranslateGemma models already demonstrated by the automatic metrics. That said, the 12B model still stays competitive with the bigger Gemma 3 model, especially for high- resource languages.

7. Conclusions

In this work, we introduced TranslateGemma, a series of open models based on Gemma 3, specifi- cally enhanced for machine translation. Through a combination of supervised fine-tuning on di- verse, high-quality parallel data—blending hu- man and synthetic sources—and a novel reinforce- ment learning approach utilizing an ensemble of reward models, we have improved translation per- formance across a wide spectrum of languages and model sizes (4B, 12B, and 27B parameters).

Our automatic evaluations on the WMT24++ dataset, encompassing 55 language pairs, show consistent gains for TranslateGemma models over the baseline Gemma 3 models both in MetricX and Comet22. We observed a strong perfor- mance increase across various language types, including high-resource languages like German and Spanish, and lower-resource languages such as Icelandic or Swahili. A key finding is the en-

hanced efficiency of the TranslateGemma models, where smaller fine-tuned models often match or exceed the performance of larger baseline models, offering a better trade-off between quality and computational cost.

Furthermore, we have shown that Trans- lateGemma models retain the multimodal capabil- ities of the original Gemma 3. Experiments on the Vistra benchmark indicate that the improvements in text translation extend to the image transla- tion task, particularly for the 12B and 27B models, without any specific multimodal fine-tuning.

The release of the TranslateGemma models con- tributes a valuable set of open-source tools for the machine translation community, fostering fur- ther research and application development. We believe these models will serve as a strong foun- dation for a variety of translation-related tasks and encourage their adoption and exploration.

References

Isaac Caswell, Elizabeth Nielsen, Jiaming Luo, Colin Cherry, Geza Kovacs, Hadar Shemtov, Partha Talukdar, Dinesh Tewari, Baba Ma- madi Diane, Djibrila Diane, Solo Farabado Cissé, Koulako Moussa Doumbouya, Edoardo Ferrante, Alessandro Guasoni, Christopher Homan, Mamadou K. Keita, Sudhamoy Deb-

7

TranslateGemma Technical Report

Barma, Ali Kuzhuget, David Anugraha, Muham- mad Ravi Shulthan Habibi, Sina Ahmadi, An- thony Munthali, Jonathan Mingfei Liu, and Jonathan Eng. 2025. SMOL: Profession- ally translated parallel data for 115 under- represented languages. In Proceedings of the Tenth Conference on Machine Translation, pages 1103–1123, Suzhou, China. Association for Computational Linguistics.

Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Eliz- abeth Salesky, Firas Trabelsi, Stephanie Win- kler, Biao Zhang, and Markus Freitag. 2025. WMT24++: Expanding the language cover- age of WMT24 to 55 languages & dialects. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12257–12284, Vi- enna, Austria. Association for Computational Linguistics.

Patrick Fernandes, Daniel Deutsch, Mara Finkel- stein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Ma- chine Translation, pages 1066–1083, Singapore. Association for Computational Linguistics.

Mara Finkelstein, David Vilar, and Markus Freitag. 2024. Introducing the NewsPaLM MBR and QE dataset: LLM-generated high-quality parallel data outperforms traditional web-crawled data. In Proceedings of the Ninth Conference on Ma- chine Translation, pages 1355–1372, Miami, Florida, USA. Association for Computational Linguistics.

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Asso- ciation for Computational Linguistics, 9:1460– 1474.

Gemini Team, Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen

Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gem- ini 2.5: Pushing the frontier with advanced rea- soning, multimodality, long context, and next generation agentic capabilities.

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. 2025. Gemma 3 technical report.

Alexander Jones, Isaac Caswell, Orhan Firat, and Ishank Saxena. 2023. GATITOS: Using a new multilingual lexicon for low-resource machine translation. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pages 371–405, Singapore. Associa- tion for Computational Linguistics.

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Confer- ence on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computa- tional Linguistics.

Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Fi- rat. 2023. Madlad-400: A multilingual and document-level large audited dataset. Ad- vances in Neural Information Processing Systems, 36:67284–67296.

Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional qual- ity metrics (mqm): A framework for declar- ing and describing translation quality met- rics. Tradumàtica: tecnologies de la traducció, (12):455–463.

Maja Popović. 2015. chrF: character n-gram F- score for automatic MT evaluation. In Pro- ceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Lin- guistics.

Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick

8

TranslateGemma Technical Report

Fernandes, and André F. T. Martins. 2025. Fine- grained reward optimization for machine trans- lation using error severity mappings.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and An- dré F. T. Martins. 2022. COMET-22: Unbabel- IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). As- sociation for Computational Linguistics.

Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. 2024. Finding replicable hu- man evaluations via stable ranking probability. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4908–4919, Mexico City, Mexico. Association for Computational Linguistics.

Elizabeth Salesky, Philipp Koehn, and Matt Post. 2024. Benchmarking visually-situated transla- tion of text in natural images. In Proceedings of the Ninth Conference on Machine Translation, pages 1167–1182, Miami, Florida, USA. Asso- ciation for Computational Linguistics.

Noam Shazeer and Mitchell Stern. 2018. Adafac- tor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR.

Contributions

Core Contributors

Mara Finkelstein Isaac Caswell Tobias Domhan Jan-Thorsten Peter Juraj Juraska Parker Riley Daniel Deutsch Geza Kovacs★

Lead

David Vilar Markus Freitag

Contributors

Cole Dilanni Colin Cherry Eleftheria Briakou Elizabeth Nielsen Jiaming Luo Kat Black Ryan Mullins Sweta Agrawal Wenda Xu

Support

Erin Kats Stephane Jaskiewicz

★Now at Anthropic.

9

TranslateGemma Technical Report

A. Automatic metrics per language

Table 4 | Comparison of performance of the TranslateGemma (GT) models with baseline Gemma models (G3) for each language pair in the WMT24++ set using MetricX.

TranslateGemma Gemma 3

27B 12B 4B 27B 12B 4B

en→ar_EG 2.54 2.78 3.57 3.32 3.70 4.60 en→ar_SA 2.42 2.66 3.43 3.19 3.64 4.49 en→bg_BG 2.80 3.25 4.30 3.90 4.47 5.81 en→bn_IN 1.88 2.12 2.92 2.56 2.87 3.86 en→ca_ES 3.18 3.58 5.20 4.07 4.89 6.97 en→cs_CZ 3.48 4.03 5.41 4.62 5.32 7.47 en→da_DK 2.11 2.45 3.25 3.00 3.36 4.40 en→de_DE 1.19 1.36 1.93 1.63 1.93 2.72 en→el_GR 2.57 3.34 4.66 3.73 4.34 6.31 en→es_MX 1.88 2.06 2.51 2.54 2.75 3.35 en→et_EE 4.61 6.15 11.03 6.40 8.89 14.78 en→fa_IR 1.99 2.28 3.34 2.98 3.41 4.77 en→fi_FI 3.19 3.77 5.68 4.19 5.11 7.54 en→fil_PH 2.98 3.17 4.20 3.62 4.03 5.22 en→fr_CA 2.21 2.37 2.92 2.78 2.97 3.76 en→fr_FR 2.19 2.44 2.97 2.78 3.01 3.90 en→gu_IN 4.69 4.93 6.32 5.27 5.79 7.67 en→he_IL 2.72 3.12 4.99 3.90 4.41 6.70 en→hi_IN 3.52 3.69 4.33 4.11 4.36 5.03 en→hr_HR 2.05 2.31 3.17 2.62 3.08 4.26 en→hu_HU 4.24 5.00 7.84 5.51 6.79 10.75 en→id_ID 2.07 2.17 2.63 2.72 2.84 3.27 en→is_IS 5.69 7.93 15.54 8.31 12.16 19.22 en→it_IT 1.88 2.17 2.64 2.60 2.84 3.60 en→ja_JP 3.53 3.82 4.44 4.11 4.30 5.09 en→kn_IN 4.18 4.78 7.11 5.09 6.82 10.48 en→ko_KR 2.81 2.97 3.93 3.43 3.79 4.72 en→lt_LT 4.39 5.41 9.58 6.01 7.71 13.39 en→lv_LV 5.69 7.22 12.12 7.55 9.90 15.75 en→ml_IN 3.64 4.30 7.33 4.77 6.84 11.89 en→mr_IN 3.17 3.47 4.30 4.11 4.60 5.64 en→nl_NL 1.67 2.01 2.84 2.48 2.82 3.87 en→no_NO 2.09 2.38 3.26 2.94 3.17 4.23 en→pa_IN 3.67 4.44 5.53 4.40 5.99 11.20 en→pl_PL 4.14 4.58 5.64 5.17 5.64 7.07 en→pt_BR 2.13 2.36 2.93 2.90 3.15 3.77 en→pt_PT 2.55 2.68 3.09 3.39 3.78 4.13 en→ro_RO 2.86 3.25 4.18 3.99 4.39 5.70 en→ru_RU 2.18 2.48 3.25 3.01 3.39 4.54 en→sk_SK 3.81 4.54 6.70 5.04 5.86 9.17 en→sl_SI 3.55 4.24 7.12 4.56 5.73 9.39 en→sr_RS 2.78 2.68 5.66 3.75 3.76 6.94 en→sv_SE 2.00 2.31 3.14 2.73 3.06 4.14 en→sw_KE 4.45 5.36 10.65 5.92 7.90 14.05 en→sw_TZ 4.30 5.25 10.30 5.73 7.85 13.89 en→ta_IN 2.87 2.98 3.90 3.53 3.83 5.04 en→te_IN 3.76 3.97 4.83 4.41 4.74 5.76 en→th_TH 2.33 2.66 3.49 2.96 3.19 4.14 en→tr_TR 4.18 4.64 6.17 5.32 6.02 8.03 en→uk_UA 2.98 3.29 4.16 3.79 4.28 5.40 en→ur_PK 3.12 3.59 5.67 3.86 4.86 7.80 en→vi_VN 1.97 2.20 2.87 2.56 2.88 3.62 en→zh_CN 1.86 2.07 2.66 2.47 2.61 3.27 en→zh_TW 2.04 2.21 2.77 2.63 2.80 3.62 en→zu_ZA 6.99 10.73 18.29 9.05 14.80 21.52

27B 12B 4B 27B 12B 4B

TranslateGemma Gemma 3

10

TranslateGemma Technical Report

B. Additional synthetic data languages

For the following languages we created synthetic data, in addition to the languages covered by WMT24++:

English-Armenian, English-Hawaiian, English-Western Frisian, English-Corsican, English-Hmong, English-Maltese, English-Tajik, English-Samoan, English-Macedonian, English-Mongolian, English- Galician, English-Albanian, English-Uzbek, English-Uyghur, English-Belarusian, English-Sinhala, English-Basque, English-Haitian Creole, English-Bosnian, English-Kyrgyz, English-Kazakh, English- Khmer, English-Scottish Gaelic, English-Lao, English-Irish, English-Luxembourgish, English-Burmese, English-Sundanese, English-Javanese, English-Malay.

C. Full list of languages for SFT

Table 5 shows the languages paired with English in both directions, Table 6 the languages paired with English as source language and Table 7 the languages pairs not involving English. Together these three tables give the full language coverage of the SFT data used for TranslateGemma.

11

TranslateGemma Technical Report

Table 5 | Languages paired with English in both directions.

Abkhaz (ab) Acehnese (ace) Acholi (ach) Afar (aa) Afrikaans (af) Ahirani (ahr) Alur (alz) Amharic (am) Assamese (as) Assyrian Neo-Aramaic (aii) Avar (av) Awadhi (awa) Aymara (ay) Badaga (bfq) Bagheli (bfy) Bagri (bgq) Balinese (ban) Baluchi (bal) Bambara (bm) Banjar (Arabic script) (bjn-Arab) Banjar (bjn) Baoul˘00e9 (bci) Bashkir (ba) Batak Karo (btx) Batak Simalungun (bts) Batak Toba (bbc) Bemba (Zambia) (bem) Betawi (bew) Bhojpuri (bho) Bikol (bik) Bodo (India) (brx) Braj (bra) Breton (br) Buginese (bug) Bundeli (bns) Buryat (bua) Cantonese (yue) Chakma (Latin script) (ccp-Latn) Chamorro (ch) Chechen (ce) Chhattisgarhi (hne) Chichewa (ny) Chinese (zh-CN) Chittagonian (ctg) Chuukese (chk) Chuvash (cv) Crimean Tatar (Cyrillic script) (crh) Crimean Tatar (Latin script) (crh-Latn) Dari (fa-AF) Dhivehi (dv) Dhundari (dhd) Dinka (din) Dogri (doi) Dombe (dov) Dutch (nl) Dyula (dyu) Dzongkha (dz) East Circassian (kbd) Eastern Huasteca Nahuatl (nhe) Efik (efi) Egyptian Arabic (arz) Ewe (ee) Faroese (fo) Fijian (fj) Fon (fon) French (fr) Friulian (fur) Fulani (ff) Ga (gaa) Garo (Latin script) (grt-Latn) German (de) Goan Konkani (gom) Guarani (gn) Hakha Chin (cnh) Hausa (ha) Hiligaynon (hil) Hindi (hi) Ho (Warang Chiti script) (hoc-Wara) Hunsrik (hrx) Iban (iba) Igbo (ig) Ilocano (ilo) Indonesian (id) Inuktut (Syllabics) (iu) Isoko (iso) Italian (it) Jamaican Patois (jam) Japanese (ja) Jingpo (kac) K’iche’ (quc) Kalaallisut (kl) Kangri (xnr) Kanuri (kr) Kapampangan (pam) Karakalpak (kaa) Kashmiri (Devanagari script) (ks-Deva) Kashmiri (ks) Kedah Malay (meo) Khasi (kha) Kiga (cgg) Kikuyu (ki) Kiluba (Luba-Katanga) (lu) Kinyarwanda (rw) Kituba (DRC) (ktu) Kokborok (trp) Komi (kv) Kongo (kg) Korean (ko) Krio (kri) Kumaoni (kfy) Kurdish (Sorani) (ckb) Kurukh (kru) Lahnda Punjabi (Pakistan) (pa-Arab) Latgalian (ltg) Lepcha (lep) Libyan Arabic (ayl) Ligurian (lij) Limbu (Limbu script) (lif-Limb) Limburgish (li) Lingala (ln) Lombard (lmo) Luganda (lg) Luo (luo) Madurese (mad) Magahi (mag) Maithili (mai) Makassar (mak) Malagasy (mg) Malay (Jawi Script) (ms-Arab) Mam (mam) Mandeali (mjl) Manx (gv) Mapudungun (arn) Marshallese (mh) Marwadi (mwr) Mauritian Creole (mfe) Meadow Mari (chm) Meiteilon (Manipuri) (mni-Mtei) Mewari (mtr) Minang (min) Mizo (lus) Modern Standard Arabic (ar) Moor˘00e9 (mos) Morrocan Arabic (ar-MA) Mundari (Devanagari script) (unr-Deva) NKo (bm-Nkoo) Navajo (nv) Ndau (ndc-ZW) Nepalbhasa (Newari) (new) Nepali (ne) Nigerian Pidgin (pcm) Nimadi (noe) North Levantine Arabic (apc) North Ndebele (nd) Northern Sami (se) Nuer (nus) Occitan (oc) Oromo (om) Ossetian (os) Pangasinan (pag) Papiamento (pap) Polish (pl) Q’eqchi’ (kek) Quechua (qu) Rohingya (Latin script) (rhg-Latn) Romani (rom) Rundi (rn) Russian (ru) Sambalpuri (spv) Sango (sg) Sanskrit (sa) Santali (Latin Script) (sat-Latn) Saraiki (skr) Sepedi (nso) Sesotho (st) Seychellois Creole (crs) Shan (shn) Sherpa (Tibetan script) (xsr-Tibt) Shina (scl) Shona (sn) Sicilian (scn) Silesian (szl) Sindhi (Devanagari script) (sd-Deva) Somali (so) South Ndebele (nr) Spanish (es) Sudanese Arabic (Deprecated BCP) (apd) Surgujia (sgj) Surjapuri (sjp) Susu (sus) Swahili (sw) Swati (ss) Sylheti (syl) Tahitian (ty) Tamazight (Latin Script) (ber-Latn) Tamazight (Tifinagh Script) (ber) Tetum (tet) Thai (th) Tibetan (bo) Tigrinya (ti) Tiv (tiv) Tok Pisin (tpi) Tonga (Tonga Islands) (to) Tsonga (ts) Tswana (tn) Tulu (tcy) Tumbuka (tum) Tunisian Arabic (aeb) Turkish (tr) Tuvan (tyv) Twi (ak) Udmurt (udm) Venda (ve) Venetian (vec) Vietnamese (vi) Wagdi (wbr) Waray (Philippines) (war) West Circassian (ady) Wolof (wo) Xhosa (xh) Yakut (sah) Yoruba (yo) Yucatec Maya (yua) Zapotec (zap)

Table 6 | Languages from English

Albanian (sq) Arabic (Egypt) (ar-EG) Armenian (hy) Bangla (bn) Basque (eu) Belarusian (be) Bosnian (bs) Bulgarian (bg) Burmese (my) Catalan (ca) Chinese (Taiwan) (zh-TW) Corsican (co) Croatian (hr) Czech (cs) Danish (da) Estonian (et) Filipino (fil) Finnish (fi) French (Canada) (fr-CA) Galician (gl) Greek (el) Gujarati (gu) Haitian Creole (ht) Hawaiian (haw) Hebrew (he) Hmong (hmn) Hungarian (hu) Icelandic (is) Inuktut (Latin) (iu-Latn) Irish (ga) Javanese (jv) Kannada (kn) Kazakh (kk) Khmer (km) Kyrgyz (ky) Lao (lo) Latvian (lv) Lithuanian (lt) Luxembourgish (lb) Macedonian (mk) Malay (ms) Malayalam (ml) Maltese (mt) Marathi (mr) Mongolian (mn) Norwegian (no) Persian (fa) Portuguese (Brazil) (pt-BR) Portuguese (Portugal) (pt-PT) Punjabi (pa) Romanian (ro) Samoan (sm) Santali (Ol Chiki script) (sat) Scottish Gaelic (gd) Serbian (Cyrillic) (sr-Cyrl) Serbian (Latin) (sr-Latn) Sinhala (si) Slovak (sk) Slovenian (sl) Sundanese (su) Swahili (Kenya) (sw-KE) Swahili (Tanzania) (sw-TZ) Swedish (sv) Tajik (tg) Tamil (ta) Telugu (te) Tshiluba (Luba-Lulua) (lua) Ukrainian (uk) Urdu (ur) Uyghur (ug)

Table 7 | Non-English language pairs

Amharic (am)↔Arabic (ar) Amharic (am)↔Mandarin Chinese (zh) Arabic (ar)↔Swahili (sw) Cantonese (yue)↔Mandarin Chinese (zh) Cantonese (yue)↔Taiwanese Mandarin (zh-Hant) Chinese (zh-CN)→Japanese (ja) Czech (cs)→German (de) Czech (cs)→Ukrainian (uk) Mandarin Chinese (zh)↔Swahili (sw)

12