Performance and Hardware Tuning

Where Speed Comes From

Token speed depends on the model size, the device it runs on, the context length and the number of concurrent requests. A GPU beats CPU by a wide margin for large models.

Memory Budget

Model plus context must fit in RAM or VRAM. If a model exceeds memory, Ollama offloads layers to CPU, which slows prompts sharply. Choose a q4 quantized tag to stay in budget.

Useful Environment Variables

export OLLAMA_NUM_PARALLEL=4         # concurrent requests per model
export OLLAMA_MAX_LOADED_MODELS=2    # models kept in RAM
export OLLAMA_KEEP_ALIVE=5m          # unload idle models sooner

Set parallel requests high only when the hardware can serve them; the queue hides in slow generated tokens.

Context Length Is Memory

A long num_ctx costs roughly linear extra memory. Keep num_ctx as low as your task allows.

Key Points

• Q4 quantization is the best default for local speed.
• RAM and VRAM, not CPU cores, decide the ceiling.
• OLLAMA_NUM_PARALLEL trades memory for throughput.
• Cap context length to protect memory.