Evaluation

MTOB (Machine Translation from One Book)

MTOB benchmark (Tanzer et al., 2023): learn to translate English-Kalamang (ISO kgv, <200 speakers) from a ~500-page grammar and ~2000-entry wordlist in context; human evaluation with 0-6 quality scale; compared to GPT-4 Turbo and Claude 3 (half-grammar) and human learner

Multilingual Google DeepMind — Gemini

Ask about this evaluation

How it was run

Open a release to see the setup, scoring, and other details that the lab published. Only fields the lab actually disclosed are shown.

Gemini 1.5 Feb 2024

Google DeepMind — Gemini

rich detail 13 method fields published

See how this test was run