A new benchmark finds that coding agents remain unreliable on software tasks shaped by non-English languages and regional conventions. Terminal-Bench-LILT contains 300 tasks written in Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish and Chinese.
Native-speaker programmers created the tasks around problems that do not have direct English equivalents, including character encoding, text normalization, internationalization and culturally specific formatting. The authors evaluated six frontier models and report that the strongest achieved a 63.1% pass rate. Many tasks were not solved by any tested model, and results varied substantially between languages.
Model rankings on this suite did not follow their rankings on general coding benchmarks, according to the paper. That makes multilingual coding a separate capability to test rather than something teams can infer from a strong English score. The work is a newly announced arXiv preprint and has not yet established how closely the task set predicts performance inside production codebases. The authors have released sample tasks publicly, allowing developers to inspect the benchmark's scope before applying its result to their own languages and workflows.