Beyond MMLU-Pro: celeris-1 on GSM8K, IFEval and DROP
Today we're adding three diverse additional benchmarks: GSM8K (grade-school math word problems, free-form answers), IFEval (verifiable instruction following, checked by Google's official program) and DROP (reading comprehension over passages). Same head-to-head field as before: gpt-5, gpt-5-mini, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-2.5-flash and mercury-2 - and this time we ran the complete datasets, 11,395 questions per model.
Results
Accuracy is the percentage of questions answered correctly under each benchmark's standard metric (details below). Response time is the median wall time to a complete answer on that benchmark's workload.
| Model | GSM8K | IFEval | DROP | Mean | Resp p50 (range) | Timing basis |
|---|---|---|---|---|---|---|
| gemini-3.5-flash | 97.6 | 87.6 | 84.6 | 89.9 | 1.28-3.05 s | e2e, colocated client |
| gemini-3.5-flash-lite | 96.8 | 88.4 | 83.6 | 89.6 | 1.16-1.77 s | e2e, colocated client |
| gemini-2.5-flash | 96.3 | 87.6 | 82.7 | 88.9 | 1.17-2.67 s | e2e, colocated client |
| gpt-5 | 95.8 | 86.3 | 79.3 | 87.2 | 0.74-5.54 s | server-reported |
| celeris-1 | 93.7 | 80.8 | 81.4 | 85.3 | 55-133 ms | server-reported |
| gpt-5-mini | 95.8 | 84.8 | 74.4 | 85.0 | 0.67-3.76 s | server-reported |
| mercury-2 | 93.6 | 77.3 | 67.7 | 79.5 | 122-832 ms | server-reported |
On GSM8K the field is tightly packed: every model lands between 93.6 and 97.6, which says grade-school math no longer separates frontier APIs even with reasoning disabled. IFEval and DROP spread the field. On IFEval celeris-1 scores 80.8 - behind the Gemini and GPT-5 family (84.8-88.4), ahead of mercury-2 (77.3). On DROP - at 9,535 questions the largest and most precise of the three - it scores 81.4, within 1.3-3.2 points of the Gemini models (82.7-84.6) and ahead of gpt-5 (79.3), gpt-5-mini (74.4) and mercury-2 (67.7).
What it adds up to
Read together, the two charts make the trade concrete: the accuracy gap between celeris-1 and the strongest model we measured is 4.6 points on these benchmarks - and 1.9 points against gpt-5; the latency gap is 15-19×. A product that can absorb two-second responses should take the points. A product that needs answers inside a page load has, on this evidence, a two-to-five-point quality cost for a fifteen-fold speed gain - and celeris-1 is the only model on the chart that fits inside one.
How we measured
- Datasets. The complete benchmarks, not samples: the full GSM8K test split (1,319 questions), all 541 IFEval prompts, and the full DROP validation split (9,535 questions) - 11,395 questions per model. Every model saw byte-identical prompts.
- Scoring. GSM8K: final-number match (a strict
#### answerparse plus a last-number fallback, so format-noncompliant models aren't marked wrong on arithmetic they got right). IFEval: Google's official checker code, prompt-level "loose" tier. DROP: normalized exact match against any gold answer, or token-F1 ≥ 0.8. Strict-tier numbers for every cell are in the released data. - Reasoning off, verified. We recorded reported reasoning tokens on every request rather than trusting the request parameter:
| Model | Setting | Verified from usage data |
|---|---|---|
| gpt-5, gpt-5-mini | reasoning_effort=minimal (their floor) | 0 reasoning tokens on 100% of GSM8K + DROP rows; 13/541 (gpt-5) and 11/541 (mini) IFEval rows leaked a little |
| gemini-3.5-flash(-lite) | thinkingBudget=0 | 0 thinking tokens everywhere |
| gemini-2.5-flash | thinkingBudget=0 | honored on GSM8K + IFEval; ignored on DROP (~203 hidden tokens/row) |
| mercury-2 | reasoning_effort=none (instant) | constant ~6-token floor |
| celeris-1 | raw completions, thought channel closed | no reasoning capability |
- Latency. Non-streaming requests over persistent, pre-warmed
connections. Where the provider reports server-side processing time we use it
(OpenAI
openai-processing-ms, CelerisServer-Timing, Inceptionx-process-time). Google's API exposes no server timing, so Gemini rows are end-to-end wall time; we measured that connection's floor at ~0.45-0.6 s with 1-token probes, so Gemini's true serving latency is roughly that much lower than the table shows. - Caps and errors. 2,048-token output cap for every model (1-62 truncations per model across the full run, concentrated in IFEval essays and long DROP answers). 79,765 scored requests; exactly one failed and was scored incorrect - a gpt-5-mini request refused by OpenAI's content filter on an IFEval prompt. Transient provider rate-limits were retried through to completion.
Methodology notes
- Running the complete datasets puts binomial standard errors at ±0.5-0.7 points on GSM8K, ±1.5 on IFEval and ±0.4 on DROP. Most gaps in the table are real at that precision; the exception is celeris-1 vs gpt-5-mini on the three-benchmark mean (85.3 vs 85.0), which is within noise.
- gemini-2.5-flash's DROP score benefits from ~203 hidden reasoning tokens per question that the API produced despite a zero thinking budget (on 9,445 of 9,535 requests); no other model got equivalent computation there.
- "Minimal" is the lowest reasoning setting OpenAI offers for gpt-5; it is not a hard zero, though in practice it produced zero reasoning tokens on >99% of requests.
- One sample per question at temperature 0; we did not measure run-to-run variance.
- celeris-1 was not trained or finetuned on any of these benchmarks.
Speed only matters if the answers hold up, and quality only matters at a latency your product can ship. Measuring both on the same footing - same prompts, same caps, reasoning off everywhere - is how we test Celeris, and we'll keep publishing as the coverage grows. For why we believe diffusion is the way to get both, see why we're starting with diffusion.