Time to first token explained: how to measure TTFT and what it means for your users

What is time to first token (TTFT)?

Time to first token is the time between your application sending a request and receiving the first token of the response. NVIDIA’s documentation on LLM benchmarking defines it that way and notes that it generally includes three things: queueing time, prefill time and network latency.

Prefill is the phase where the model reads your whole prompt (system instructions, conversation history, attached documents) before it can write anything. Only then does decoding start, generating the answer one token at a time. That is why TTFT depends so much on how long your input is.

With streaming, TTFT is exactly the time your user spends looking at a frozen screen. Once text starts flowing the perception of speed is already won, even at an ordinary pace. Without streaming the user waits for the whole answer, and end-to-end latency becomes the number that matters instead.

TTFT vs TPOT vs ITL: the LLM latency metrics

LLM latency usually means one of four different numbers. They are worth keeping apart, because each one is fixed by different things:

What makes time to first token go up

Almost all of TTFT comes from four places. Knowing which one dominates for you tells you what to fix:

Why TTFT and throughput pull in opposite directions

An inference engine serves many users at once by putting their requests into batches. Bigger batches get more tokens per second out of the same GPU, which is what makes tokens cheap, but each request shares the compute with more neighbours.

vLLM, the open-source engine behind a large share of open-model serving, describes this through its per-batch token budget: according to its documentation, a larger budget improves TTFT because more prefill fits in each step, and a smaller one improves ITL because fewer prefills slow down everyone else’s decoding. Its chunked prefill splits long prompts so they can be mixed with that decoding, and a huge document does not freeze every other stream.

Another part of the same engine, prefix caching, keeps the computation for prefixes it has already processed. When two requests start the same way, which is normal for an assistant (same system prompt, same history), the second one skips that part of prefill.

Definitions and engine behaviour from the official NVIDIA and vLLM documentation, checked on 9 November 2026. Sources: docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html, docs.vllm.ai/en/latest/design/prefix_caching.html, docs.vllm.ai/en/latest/configuration/optimization.html

How to measure TTFT for an LLM API

The honest way to know your TTFT is to measure it on your side, with streaming, because then it includes everything your user feels: network, TLS, proxies, queueing and prefill. Server-side metrics usually start counting once the request has reached the engine, and leave part of the path out.

This script uses the official OpenAI SDK against Token Factory. It works the same with any compatible API: change the base URL, the key variable and the model. It sends one warm-up request that does not count, then twenty measured ones, and prints each TTFT and pace plus the percentiles.

# ttft.py: measure TTFT and pace for a streamed model
# pip install openai
import math
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gpuflow.ai/v1",
    api_key=os.environ["GPUFLOW_API_KEY"],
)

MODEL = "qwen-3.8-27b"
PROMPT = "Explain in five sentences what a model context window is."
RUNS = 20


def one_request():
    start = time.perf_counter()
    first = None
    pieces = 0
    usage = None
    stream = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": PROMPT}],
        max_tokens=300,
        stream=True,
        stream_options={"include_usage": True},
    )
    for chunk in stream:
        if chunk.usage:
            usage = chunk.usage
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta
        # Reasoning models stream their thinking in a separate field.
        text = (
            delta.content
            or getattr(delta, "reasoning_content", None)
            or getattr(delta, "reasoning", None)
        )
        if text:
            if first is None:
                first = time.perf_counter()
            pieces += 1
    end = time.perf_counter()
    if first is None:
        raise RuntimeError("empty response")
    tokens = usage.completion_tokens if usage else pieces
    decode = end - first
    speed = (tokens - 1) / decode if tokens > 1 and decode > 0 else 0.0
    return first - start, speed


def percentile(values, p):
    ordered = sorted(values)
    rank = max(1, math.ceil(p / 100 * len(ordered)))
    return ordered[rank - 1]


one_request()  # warm-up: opens the connection, not counted
ttfts, speeds = [], []
for i in range(RUNS):
    ttft, speed = one_request()
    ttfts.append(ttft)
    speeds.append(speed)
    print(f"#{i + 1:<3} TTFT {ttft * 1000:7.0f} ms   {speed:6.1f} tokens/s")

print(f"TTFT      p50 {percentile(ttfts, 50) * 1000:.0f} ms   p95 {percentile(ttfts, 95) * 1000:.0f} ms")
print(f"tokens/s  p50 {percentile(speeds, 50):.1f}   p5 {percentile(speeds, 5):.1f}")

The tokens per second it prints are the pace of that request once it started answering (the inverse of TPOT), not the server’s total capacity. If the response carries no token count, the script counts the chunks it receives, which is an approximation.

With reasoning models, the script counts the first reasoning token as the first token too. If your interface only shows the final answer, measure the first content token instead: remove the two getattr lines.

Five rules for a measurement you can use

1. 1Use your real prompt at its real length. A “hello” measures the network, not your use case.
2. 2Measure from where your users or your servers are, not from a laptop on another network.
3. 3Fix max_tokens and the model, and never change two things between runs.
4. 4Repeat at different times of day: the queue changes with load, and TTFT with it.
5. 5Also test the long context your conversation will have by turn twenty, not just turn one.

p50 and p95 latency: why the average lies

The average of twenty runs hides what hurts most: the request that takes three times as long. That is why latency is reported in percentiles. The p50 (the median) is what half of your users experience; the p95 is the value that only one request in twenty exceeds.

The p95 is the one to watch, because a user makes many requests in a session and eventually hits the slow tail. With twenty runs, the p95 is in practice your second worst result; to pin it down you need hundreds of samples, spread over time.

When you compare providers, compare the same percentile, with the same prompt, at the same time of day. A latency figure with no percentile, no prompt length and no word on where it was measured from cannot be compared with anything.

What is a good time to first token for your product?

It depends on what your user is doing while they wait. In a streamed chat, TTFT is nearly all of the perceived speed. In a voice agent it adds up with speech recognition and speech synthesis, so each piece gets a smaller share of the budget. In an agent that chains ten calls, the end-to-end latency of each step is what counts. In an overnight batch job, TTFT is irrelevant and throughput and price rule.

The useful move is to set your own p95 target from your interface, measure it with the script, and measure again whenever you change model, prompt length or provider.

How this works on GPU Flow

We do not publish a TTFT figure. A number that holds for your use case has to be measured with real traffic, from outside, with prompts like yours, and we have not done that yet: a figure without that measurement would be a brochure number. That is why we give you the script.

What we do have in place, and what works in TTFT’s favour:

Related

• Migrate from the OpenAI API to an open-source alternative in 15 minutes (code included)Move from the OpenAI SDK to GPU Flow in five steps: change the base URL, pick the equivalent model and keep streaming, tools and reasoning.
• An OpenAI API alternative: GPU Flow vs OpenAIPrice per million tokens, audio, migration and when to stay on OpenAI, with their prices checked on their own site.