Evaluation

Memorization rate evaluation (exact and approximate)

memorization rate defined as the ratio of generations that match training data to all generations; methodology of Gemma Team (2024b): subsample a large portion of training data distributed uniformly across corpora and test for discoverable extraction (Nasr et al. 2023) with a prefix of length 50 and suffix of length 50; 'exactly memorized' if all tokens in the continuation match the source suffix, 'approximately memorized' if within 10% edit distance