Gemma 2
Google DeepMind — Gemma
partial detail 15 method fields published
Evaluation
Distillation versus from scratch ablation
impact of knowledge distillation on small language models: a 2B model trained over 500B tokens either from scratch or with distillation from a 7B teacher (10x the compute-optimal token count for a 2B model)
How it was run
Open a release to see the setup, scoring, and other details that the lab published. Only fields the lab actually disclosed are shown.