blog on performance
6 min readWhy we measure TTFT, not tokens per second
Almost every inference provider brags about tokens per second. It is a big number, it looks great in a table and it is not what your user feels. What a person in front of a chat or an agent perceives is how long it takes for the first word to appear: TTFT, time to first token.
At GPU Flow we optimise and publish TTFT as the primary metric. Here is the reasoning, and what we do at the hardware and software level to lower it.
TTFT, what your user feels
The silence until the first word. At GPU Flow, under 100 ms from Spain.
tokens/s, the brochure number
The pace once it is already answering. It matters in batches, not in chats.
Throughput is for brochures, TTFT is for users
Tokens per second (throughput) measures the generation pace once the model is already answering. It genuinely matters for batch loads: processing a thousand documents overnight, indexing an entire archive.
But in an interactive experience (a chat, a copilot, a reasoning agent) the user mostly notices the initial silence: the time until the first token appears. With streaming, once text starts flowing the perception of speed is already won, even if throughput is moderate. High TTFT feels like a broken app; medium throughput with low TTFT feels fluid.
What makes the first token slow
TTFT is dominated by three things: network distance to the datacenter, queue time if the GPU is saturated, and prefill, which is processing your full prompt before emitting the first token. The longer the context, the heavier the prefill.
Optimising throughput (large batches, more parallel sequences) often worsens TTFT, because your request waits to fill a batch. These goals are in tension: choosing which to prioritise is a product decision, not just engineering.
What we do to lower it
We work the three layers where the first token is lost:
Hardware
Dedicated NVIDIA B200 GPUs, with no other tenant stealing latency at peak, and a datacenter in Spain, which shortens network distance for Western Europe.
Software
In the engine that serves the models (vLLM) we reuse the computation for the chunks of prompt that repeat, which in an assistant is nearly all of them, and we split prefill so the first word comes out sooner on long contexts. On most of the catalogue, not all of it: the vision models and the hybrid-architecture one do not support that caching, so we do not enable it there.
Network
If you combine Token Factory with a Sandbox, calls travel over the cluster internal network and never hit the public internet: fewer hops, lower latency and your data never leaves the European perimeter.
When throughput does matter
We are not saying tokens per second do not matter. For very long generation (reports, mass translation) or batch pipelines, throughput rules and is worth measuring. What we avoid is leading with the number that looks best in a table instead of the one that best describes the real experience.
That is why we publish TTFT and care for it at the hardware level: it is the honest metric for interactive inference.
Measure it yourself
The Token Factory demo shows the real TTFT of each response. Try it, or connect with 50 € of free credit and measure with your own prompt.