How we test Celeris-1: speed and quality on the same footing

Celeris-1 is designed to be fast and accurate at the same time, not one at the cost of the other. We ran it against GPT-5, GPT-5 mini, Google’s Gemini 3.5 Flash Lite and Gemini 2.5 Flash, and Inception’s Mercury 2 on MMLU-Pro, through one harness with identical prompts, scoring, and dataset. Every model’s quality and latency come from the same run.

What we compared

Six models:

• Celeris-1
• GPT-5
• GPT-5 mini
• Gemini 3.5 Flash Lite
• Gemini 2.5 Flash
• Inception / Mercury 2

Reasoning mode intentionally trades latency for accuracy. The trade is not subtle: with reasoning left on at just its default budget, GPT-5 mini's p50 on MMLU-Pro is 18 seconds - not a meaningful measurement if you are optimizing for speed, and unusable for many agentic applications. To obtain the fastest possible configuration, we set the reasoning budget to 0 for Celeris-1, GPT-5, GPT-5 mini, Gemini 3.5 Flash Lite, and Gemini 2.5 Flash, and ran Mercury 2 in its instant mode. That same setting produces both the accuracy and the latency numbers; every number for a model comes from one run.

MMLU-Pro: reasoning under load

Full test set · 5-shot chain-of-thought · strict scoring

MMLU-Pro is a harder MMLU: ten options per question instead of four, across fourteen subjects, weighted toward reasoning over recall. Guessing scores ~10%.

We ran the full MMLU-Pro test set - every question across all fourteen categories - five-shot with chain-of-thought exemplars, scored strictly: the model has to state its answer as the answer is (X) and match the gold letter.

Up and to the left wins - more accurate, faster. Celeris-1 sits far to the left at 75.9% and 158 ms. Gemini 3.5 Flash Lite leads on accuracy at 83.0% but takes 1.2 seconds, about 8× longer. GPT-5 reaches 81.9% at 2.0 s and GPT-5 mini 78.5% at 2.5 s - 13× to 16× slower than Celeris-1 for a few points of accuracy. Gemini 2.5 Flash lands at 73.0% at 2.6 s, behind Celeris-1 on both accuracy and speed. Inception’s Mercury 2 - the other diffusion model, and the only one in the same latency class - lands at 63.7% and 257 ms, behind Celeris-1 on both accuracy and speed.

We didn’t count the network

How much of each latency is the model, and how much is the network round trip? We minimized the question: where a provider exposes a server-side response time, that is the number we report (openai-processing-ms and its equivalents) - no network is counted against those models. The Gemini endpoints expose no server timing, so their figures are end-to-end wall clock measured from a client colocated in their serving region, which keeps the network term to tens of milliseconds against multi-second responses. Either way, the gap is the model.

What it adds up to

Optimizing for speed and intelligence together is the whole point. Celeris-1 comes within a few points of the frontier on reasoning, beats every model in its latency class, and runs 8-16× faster than the frontier on model time. That puts a frontier-level answer inside the hot path - every keystroke, every turn of a conversation, every step of an agent loop.

Methodology notes

• MMLU-Pro: TIGER-Lab/MMLU-Pro, the full test set across all fourteen categories, five-shot chain-of-thought, strict the answer is (X) scoring against the gold label.
• Configurations: the reasoning budget was set to 0 for Celeris-1, GPT-5, GPT-5 mini, Gemini 3.5 Flash Lite, and Gemini 2.5 Flash, and Mercury 2 ran in its instant mode (fastest configuration for every model). Latency is single-request (no batching), reported as the median (p50).
• Latency attribution: latencies are the provider’s own server-side response time (openai-processing-ms and its equivalents) for every model except the Gemini models, which expose no server timing and were measured end-to-end from a colocated client.
• No contamination: Celeris-1 was not trained or finetuned on MMLU-Pro. Its questions and answers were never in the training data; the score reflects general reasoning, not memorization of the benchmark.
• Gemini 2.5 Flash scoring: Gemini 2.5 Flash struggled to state its final answer in the required format, which scored it at just 29.1% on our run. We instead report its last known TIGER-Lab MMLU-Pro score, since it is no longer listed on the leaderboard.
• Mercury 2 configuration: under the standard 5-shot chain-of-thought setup, Mercury 2 performed poorly and often failed to format its responses correctly - 34.8% under strict scoring, and just 57.9% even when we accepted the answer in any format. Out of fairness we moved it to 0-shot forced chain-of-thought, where it reached the 63.7% we report.