LLM API and inference

7 min readofficial docs cited

vLLM vs Ollama vs a managed API: how to serve open-source models without owning a GPU

Three ways to put an open-source model behind an API. They are not competitors so much as tools for different jobs: one for your laptop, one for a GPU server under real traffic, and one for when you would rather not run either.

GPU Flow team. Published on 9 November 2026.

If you have searched for "vllm vs ollama" you have probably seen benchmark charts with one bar towering over the other. Those charts measure throughput under many simultaneous requests, which is what vLLM is designed for and Ollama is not. The more useful question is what you are trying to serve, to whom, and on which hardware. This article goes through each option using its own documentation, then puts costs and operations side by side.

What is vLLM?

vLLM describes itself as "a fast and easy-to-use library for LLM inference and serving". Its documentation lists what makes it fast under load: PagedAttention to manage the attention key and value memory efficiently, continuous batching of incoming requests, chunked prefill and prefix caching.

It also supports a long list of quantization formats (FP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF and more), tensor, pipeline, data, expert and context parallelism to spread one model over several GPUs, and an OpenAI-compatible API server that also speaks the Anthropic Messages API.

The quickstart is short, but read its requirements: Linux and Python 3.11 to 3.14. Hardware support covers NVIDIA and AMD GPUs and x86, ARM and PowerPC CPUs, although the whole point of vLLM is getting the most out of a GPU.

bash

# vLLM (Linux, Python 3.11 to 3.14)
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct
# OpenAI-compatible server on http://localhost:8000

What is Ollama?

Ollama presents itself as a way to "use open models in your desktop apps and coding agents, or build them into your application". It is the shortest path from zero to a model answering on your own machine: you install it, pull a model and it is running, with an API that OpenAI and Anthropic clients can talk to.

Its defaults tell you who it is built for. It binds to 127.0.0.1 on port 11434, so only your own machine can reach it. Each model handles one request at a time unless you raise OLLAMA_NUM_PARALLEL. The default context window is 4,096 tokens, and a model is unloaded from memory after five minutes without use. All sensible choices for a single developer, and all things you would change before putting it in front of users.

On hardware, its GPU documentation covers NVIDIA cards with compute capability 5.0 or higher, AMD cards through ROCm and Apple devices through Metal. When you run it locally, Ollama states that it does not see your prompts or data. It also offers cloud-hosted models, which is a different service with its own terms.

bash

# Ollama listens on 127.0.0.1:11434 by default
# To expose it on your network, change the bind address:
OLLAMA_HOST=0.0.0.0:11434 ollama serve

vLLM vs Ollama: the differences that matter

Both give you an HTTP API in front of an open-source model, and both work with OpenAI-style clients. Where they part ways is in what happens when more than one person uses it.

	vLLM	Ollama
Built for	Serving many concurrent requests on GPUs	Running models on your own computer
Concurrency	Continuous batching across all incoming requests	One request per model at a time by default
Where it runs	Linux, with NVIDIA or AMD GPUs (CPUs too)	Your desktop or a server, with NVIDIA, AMD or Apple GPUs
Default endpoint	localhost:8000, OpenAI-compatible	127.0.0.1:11434, OpenAI and Anthropic clients
Effort to start	Python environment, GPU drivers, model weights	One installer and one command

The short version: Ollama optimises for your time as a developer, vLLM optimises for the GPU’s time. If a handful of people use the model, Ollama is often enough. If an application with real users sends requests in parallel, vLLM is the tool designed for it.

Can you self-host an LLM without owning a GPU?

Yes, in three ways, and they trade money against effort differently.

On a CPU

Ollama runs small models on a laptop or a CPU server. It is fine for experiments and private tools; large models on a CPU are slow.

On a rented GPU

Rent a GPU by the hour or by the month and install vLLM or Ollama on it yourself. You own the stack without buying the hardware.

Through a managed API

Someone else runs the GPUs and the serving engine; you call an endpoint and pay per token. Nothing to install, nothing idle.

What each option really costs

The cost structures are opposite. A GPU you rent costs the same per hour whether it serves one request or a thousand: you pay for capacity. An API costs nothing while idle and grows with every token: you pay for use.

To put numbers on it with our own live prices: the smallest NVIDIA B200 slice we rent (23 GB) costs €1.10 an hour, or €723 a month reserved, and the whole card €6.50 an hour. That monthly figure, spent on Token Factory instead, buys between 599 million input tokens on GLM 5.3 NVFP4 (€1.206 per million), the most expensive model in the catalogue, and 12,045 million on Nemotron Nano 12B VL (€0.06 per million), the cheapest. Output tokens cost more, so a real workload buys fewer.

Which side wins depends on how many tokens your GPU would actually push through in a month, and that is a number only a benchmark of your own model, prompts and traffic pattern can give you. Be wary of anyone, us included, quoting a throughput figure for your workload without measuring it.

Two costs rarely appear in these comparisons. The first is idle time: a self-hosted GPU at three in the morning still bills. The second is people: the hours someone spends keeping the server patched, monitored and up.

What you take on when you self-host

Running the engine is the easy part. Running it as a service is where the work goes. Before you commit, check you have an answer for each of these:

1Access control. Neither tool should face the internet bare: put an authenticated proxy in front before you change the bind address.
2Capacity. A model must fit in GPU memory with room for the context of every concurrent request; plan the GPU size around your longest prompts.
3Upgrades. New model releases often need a newer engine version, and engine upgrades can change behaviour. Someone has to test them.
4Monitoring. Latency, errors, GPU memory and queue length need dashboards and alerts, or you find out from your users.
5Billing your own users. If several teams share the server, you need per-key usage tracking and limits.

How this works on GPU Flow

We offer the last two options from the list above, so you can pick without changing provider. Token Factory is the managed API: 9 open-source models behind one OpenAI and Anthropic compatible endpoint, billed per token in euros from a prepaid balance. When the balance runs out the API returns a 402 error; nothing is charged to your card automatically.

The language models that run on our own cluster are served with vLLM, the same engine described above, so the trade-offs in this article are the ones we live with every day. Not every model in the catalogue runs on our cluster, and each model page shows its own details.

If you need your own model, a fine-tune or full control of the engine, dedicated GPU is the other option: an NVIDIA B200 in our data centre in Spain, from a 23 GB slice at €1.10 an hour to the whole card at €6.50, with SSH access and drivers ready. You install vLLM, Ollama or anything else, and nobody else runs on your slice.

One thing that is not an option: our Sandboxes are CPU machines with no GPU inside. They are made for coding agents that call Token Factory over the internal network, not for running a model locally.

Which one should you pick?

A rule of thumb that holds for most teams:

1Trying models on your laptop, or a private tool for yourself: Ollama.
2An application with users, a model from a public catalogue, and no wish to run servers: a managed API.
3Your own fine-tuned model, a model nobody serves, or strict control of the engine: vLLM on a GPU you rent or own.
4Not sure yet: start on the API, measure your real token volume for a month, then do the GPU arithmetic with your own numbers.

About benchmarks

This article quotes no throughput or latency figures on purpose. Results depend on the model, the quantization, the prompt lengths and the number of concurrent users, and a chart measured on someone else’s workload says little about yours. Measure on your own traffic before you decide.

Skip the serving layer

Call open-source models through one OpenAI and Anthropic compatible API, billed per token in euros. If you need the GPU itself, rent a B200 by the hour or by the month.

see Token Factory rent a GPU