Evaluation

Text needle-in-a-haystack retrieval (long-context)

Needle-in-a-haystack evaluation (Kamradt 2023): retrieve a 'magic number' needle inserted at varying depths in Paul Graham essay haystack up to 1M tokens, extended to 10M; compared to GPT-4 Turbo

Core capabilities Google DeepMind — Gemini

Ask about this evaluation

How it was run

Open a release to see the setup, scoring, and other details that the lab published. Only fields the lab actually disclosed are shown.

Gemini 1.5 Feb 2024

Google DeepMind — Gemini

rich detail 8 method fields published

See how this test was run