We are currently building SynthCommonCorpus, an open synthetic pretraining corpus for agentic models, with the target of 20 trillion tokens across all 24 official languages of the European Union. 

Nine are done: 830 billion tokens written by Synthotron-8B, a mixture-of-experts model with 1.5 billion active parameters. The remaining fifteen come next, and the first release is planned for November 2026. With these volumes, the evaluation of the generated corpus' quality largely rests on automated judges. Which language to generate next, which seeds to drop, when a language is good enough to ship? 

Those judges are language models, and their competence varies by language. Our new paper, Which Model Should Judge That Language?, measures how much. Across 25 European languages, the best judge we tested catches only slightly more than half of controlled losses of fluency, and almost none of the subtle ones.

Full paper available here, and the main points follow below.

Why fluency is hard to judge at scale

The failure we expect most from a multilingual synthetic corpus is translationese. Synthotron-8B writes an English reasoning trace beside each conversation, and its teachers, like most large models, have seen far more English than Maltese or Latvian. The likely result is text that is grammatical and grounded in its seed, yet reads as translated: English clause order, literal calques, collocations a native speaker would not choose. It is also the kind of error a reader notices last.

Our own pipeline shows how little we know about it. We use a model to evaluate the quality of our outputs across EU languages, scoring each answer from 0 to 5. These scores help us identify which languages need improvement before we generate at scale, but they raise an obvious question: how reliable is the judge itself? If it gives Finnish a high score, is that because the output is genuinely great, or because the model isn't particularly good at evaluating Finnish? Without independent validation, it's difficult to know how much we can trust these scores, especially when comparing languages.

Traditionally, we would rely on expert judgements here. However, expert human ratings are slow to collect, expensive, and often noisy. They are thus out of reach at corpus scale. Agreement with a second automated judge can reproduce the same blind spots. It’s difficult to judge the judges, because existing multilingual judge benchmarks mix correctness, helpfulness, safety and style, and these attributes are often correlated, meaning that sensitivity to naturalness alone stays difficult to evaluate. Still, these benchmarks already find that judges weaken as language resources decrease (Son et al., 2024), which is not ideal.

Cost is another constraint. The strongest LLM judge we tested cost about $49 per 1,000 evaluations. At that rate, evaluating all 893.7 million conversations in our corpus just once would cost roughly $44 million. We therefore need a cheaper way to evaluate quality, while making sure the results are reliable for each language.

A benchmark of controlled translationese

Since our objective is primarily to detect unwanted English influence on generated texts, we need a judge that can provide a reliable signal for that. We therefore ask: can this judge detect a controlled loss of naturalness in a Finnish text whose content does not change?

We started with learner essays and their teacher corrections, from the MultiGEC dataset. Judges usually preferred the corrected version: from 57.3% of pairs for Nemotron 3.5 Lightning to 84.3% for Gemini 3.7 Flash. This is a good sign, LLMs as a judge seem to work. However, teachers edit very differently from one corpus to the next, so the quality of the judgments could not be compared across languages.

The Translationese Detection Benchmark, TDBench, fixes that variable. We sampled 50 original-language web passages from the original side of FineTranslations for each of 25 European language groups, 1,250 passages in all, from Spanish and German to Basque, Lower Sorbian, Limburgish and Scottish Gaelic. GPT-5.6 Luna rewrote each passage three times, in the same language, trying to mimic what would have happened if the passage had been translated from English instead of written by a native speaker, while keeping meaning, facts, names, numbers and length. The subtle version allows a few literal phrases, the mild one also adopts English structure more broadly, and the heavy one uses a very English-inspired syntax.

Each judge scores every text alone, on a 0 to 5 rubric where 3 means understandable but awkward and 5 means written by a fluent speaker. They never see multiple versions of the same passage at the same time, so they cannot perform comparisons. A degradation counts as detected only when the judge scores the original strictly higher: a tie is a miss. We compared fifteen judges: nine hosted general-purpose LLMs; Jev, a specialized pretrained classifier; and five open judges run locally, including the three released M-Prometheus checkpoints and an adapted CE-Judge pipeline.

What the judges catch

No judge detects much more than half of the degradations. Claude Fable 5.1 leads with an estimated macro detection rate of 53.5%, measured on the 1,015 passages it completed within budget. Gemini 3.7 Flash follows at 48.7%, then Jev at 47.7% and GPT-6 Astra at 45.8%. On the passages all three LLMs completed, paired bootstrap tests support both gaps: 4.9 points between Fable and Gemini, 3.1 between Gemini and Astra.

The strength of the degradation matters more than the choice among leading judges. Fable detects 79.5% of heavy translationese, 65.5% of mild and 15.5% of subtle, and on average rates subtle variants slightly above their sources. Jev is the best of the leading judges on subtle changes, at 18.9%. Its near-continuous scores also give it the best accuracy over all six pairs in each set of four texts, 67.9% against 67.4% for Fable. The open judges built for multilingual evaluation score between 12.9% and 17.7%, largely because their integer scales produce ties on 62 to 74% of pairs.

The language matters as much. On the same passages, Fable ranges from 63.6% in Spanish to 14.7% in Scottish Gaelic, and its ranking of languages follows how much data each one has, with a Spearman correlation of 0.74 against our resource index. The best judge changes with the language: Jev beats Fable on Lower Sorbian and Galician, and Nemotron 3 Ultra has the highest rate on Maltese while reaching only 12.7% on Welsh. Being on a provider's list of supported languages guarantees little.

Cost reorders the field. Fable's 81.2% coverage cost about $200, retries included. Jev scored the full benchmark for $0.38, which would put a pass over the corpus in the tens of thousands of dollars rather than tens of millions.

What we still cannot measure

TDBench measures agreement with an ordering that a model wrote. GPT-5.6 Luna produced every degradation, so the benchmark checks whether judges respond to the changes one generator was asked to make. A rewrite can drift in meaning, add an artifact unrelated to translationese, or smooth a flaw already present in the web source. Fable rates subtle variants above their sources on average, and we cannot yet say how often it is wrong and how often the source was simply worse. Until native speakers rerank the texts, the ground truth is itself a model's output.

The scores are not calibrated either. Hosted models answer in steps of 0.25, Jev returns a near-continuous expected score, and the open judges use integers that produce ties. Paired comparisons let us rank judges whose 4 does not mean the same thing, but none of these scores can serve as an absolute quality threshold without a calibration study against human ratings. What matters is that this can now happen for a few languages only, and generalize well to others.

That said, fluency cannot be reduced to one gold standard. A rubric that rewards text reading like an original by a fluent speaker can penalize dialects, learner usage and code-switching. For Limburgish, Luxembourgish or Western Frisian, it is an open question whose standard of naturalness a judge applies. Native-speaker validation will have to cover those varieties, document who the annotators are, and pay them for their expertise.

One interesting question still remains open: does a model that recognizes translationese avoid producing it? If judging competence predicts generation fluency, the same benchmark would tell us which generator to trust in which language. Testing it means pairing judge scores with native-speaker ratings of text the same models write, and we leave this for future work.

What comes next

Now that we have validated our judges in each language before trusting their scores, we would like to test a router that sends texts written in well-supported languages to a low-cost classifier and more difficult cases to a larger model. Given Jev orders variants against each other better than any LLM we tested, this makes it a reasonable first pass. But for some languages, it just isn’t discriminative enough.

For TDBench itself, the next steps are native-speaker validation, several corruption models instead of one, more languages and varieties, and interference from source languages other than English. We will release the benchmark data, prompts, score caches and evaluation scripts.

The first release of SynthCommonCorpus is planned for November 2026, with the next fifteen EU languages, seed text from Common Corpus and longer reasoning traces to follow.