Building the world's fastest LLMs: why we're starting with diffusion

At Celeris, our objective is to maximize useful intelligence delivered per unit of time. In this post, we explain how we're pursuing that goal - starting with diffusion - and what becomes possible when it's reached.

A different starting point

Today the work is split: labs design and train models, and inference providers make them faster after the fact. No one is designing for speed from the ground up. That is what we are doing: new techniques for both training and architecture, built for extreme throughput from the start.

Our current focus is diffusion.

Diffusion models have already demonstrated remarkable parallel-generation properties in other domains. In language, the sequential nature of autoregressive decoding has historically dominated. We see an opportunity to invert that assumption. Our approach draws on the strengths of both paradigms - the coherence of sequential generation and the speed of parallel decoding - and pushes well beyond existing hybrid methods.

This is not simply "making diffusion work for text." It is about creating training and inference paradigms in which parallelism is a first-class citizen.

The evidence for each of these properties is already public. On raw speed, parallelism gives diffusion decoding headroom that serial generation cannot reach: Fast-dLLM, an NVIDIA-led study, showed that decoding many tokens per step delivers up to a 27.6× throughput improvement on open diffusion models with minimal accuracy loss - from training-free acceleration alone.

Bidirectional context also closes a significant capability gap. Autoregressive models suffer from the reversal curse: trained that "A is B", they fail to infer "B is A" - GPT-4 answers 79% of forward questions about celebrity relationships but only 33% of the reversed ones. LLaDA, an 8B diffusion model that attends to the whole sequence at every step, breaks the pattern, surpassing GPT-4o on reversal reasoning.

And refinement changes what inference itself can do. An autoregressive model commits to every token permanently; a diffusion model can revise earlier predictions as the rest of the answer takes shape. Diffusion-of-Thought (NeurIPS 2024) showed a small diffusion model self-correcting its reasoning mid-generation and beating a much larger autoregressive model on math tasks in both accuracy and efficiency, with the number of refinement steps acting as a natural dial between latency and reasoning quality.

What's unique about our approach

We are working on hybrid approaches to diffusion - models that combine sequential and parallel generation inside a single system, spending sequential compute only where language demands it and decoding the rest in parallel. Building this into the architecture and the training objective, rather than bolting one paradigm onto the other at inference time, is the core of our research agenda - and it is what lets a model keep the coherence of autoregression while decoding at diffusion speeds.

Architecture search as a core capability

Hand-designing these systems is insufficient at the frontier. We are investing heavily in neural architecture search and automated research loops that let us systematically explore the space of architectures, training objectives, and inference strategies optimized for latency and intelligence density.

The goal is not just a single fast model. It is a research engine capable of continually discovering better ways to deliver frontier-level capability at microsecond-to-millisecond timescales.

What this enables

When intelligence arrives at true real-time speeds, entire categories of software become possible that are awkward or impossible today:

• Fluid, interruptible voice interfaces that feel instantaneous
• Agents that reason and act in tight feedback loops
• Scientific and engineering tools that keep pace with human thought
• Systems where latency no longer forces a compromise on model quality

We are still early. Our first models are coming soon. What we can say now is that we are deliberately choosing a harder path - rethinking both training and architecture rather than only optimizing the status quo - because we believe the payoff in capability-per-second will be decisive.

If you care about the systems that become possible when language models stop being the bottleneck, we'd love to hear from you.