blog · performance
Why we measure TTFT, not tokens per second
Almost every inference provider brags about tokens per second. It is a big number, it looks great in a table and… it is not what your user feels. What a person in front of a chat or an agent perceives is how long it takes for the first word to appear: TTFT (time to first token).
At GPU Flow we optimise (and publish) TTFT as the primary metric. Here is the reasoning, and what we do at the hardware and software level to lower it.
Throughput is for brochures; TTFT is for users
Tokens per second (throughput) measures the generation pace once the model is already answering. It genuinely matters for batch loads: processing a thousand documents overnight, generating embeddings at scale, and so on.
But in an interactive experience (a chat, a copilot, a reasoning agent) the user mostly notices the initial silence: the time until the first token appears. With streaming, once text starts flowing the perception of speed is already won, even if throughput is moderate. High TTFT feels like a broken app; medium throughput with low TTFT feels fluid.
What makes the first token slow
TTFT is dominated by three things: network distance to the datacenter, queue time if the GPU is saturated, and "prefill", processing your full prompt before emitting the first token. The longer the context, the heavier the prefill.
Optimising throughput (large batches, more parallel sequences) often worsens TTFT, because your request waits to fill a batch. These goals are in tension: choosing which to prioritise is a product decision, not just engineering.
What we do to lower it
Hardware: dedicated NVIDIA B200 (Blackwell) GPUs, with no shared tenant stealing latency at peak. Datacenter in Spain, minimising network distance for users in Western Europe.
Software: in the vLLM engine we enable prefix caching (reusing the compute of repeated prefixes, typical of system prompts and RAG) and chunked prefill (splitting prefill so the first token comes out sooner on long contexts).
Network: if you combine Token Factory with a Sandbox, calls travel over the cluster’s internal RDMA network and never hit the public internet, fewer hops, lower latency, and your data never leaves the perimeter.
When throughput does matter
We are not saying tokens per second do not matter. For very long generation (reports, mass translation) or batch pipelines, throughput rules and is worth measuring. What we avoid is leading with the number that looks best in a table instead of the one that best describes the real experience.
That is why we publish TTFT and care for it at the hardware level: it is the honest metric for interactive inference.
Measure it yourself
The Token Factory demo shows the real TTFT of each response. Try it, or connect with 5 € of free credit and measure with your own prompt.
see Token Factory