Experimenting with different Ollama-sourced AI models, I discovered that sometimes, in a very strange way, my CPU or GPU resources are not used efficiently. Out of the box, Ollama prioritizes system stability over raw hardware saturation, often leaving high-end multi-core CPUs and multi-gigabyte VRAM cards severely underutilized.
By understanding how Ollama packages models and routes workloads through its underlying llama.cpp inference engine, you can take full control over your hardware allocation.
There are three ways to accomplish this: 1. Via Custom Modelfiles (Bake hardware parameters directly into the model). 2. Via Daemon Environment Variables (Set fleet-wide defaults across all models). 3. Via API / Runtime Options (Override on the fly without rebuilding anything).
Method 1: The Modelfile Approach (Baking Hardware Parameters)
Ollama models are packaged very similarly to Docker container images, where base weights, system prompts, templates, and runtime parameters are explicitly declared.
For example, we can inspect the exact configuration file for qwen2.5-coder:32b:
✦ ❯ ollama show qwen2.5-coder:32b --modelfile
This outputs the complete Modelfile definition, starting with a base blob hash:
`dockerfile # Modelfile generated by “ollama show” # To build a new Modelfile based on this, replace FROM with: # FROM qwen2.5-coder:32b
FROM /usr/share/ollama/.ollama/models/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 TEMPLATE “””…””” SYSTEM You are Qwen, created by Alibaba Cloud. You are a helpful assistant. LICENSE “””…””” `
We can easily extend and specialize this model by dumping its configuration, referencing the clean model name in the FROM directive, and adding explicit hardware PARAMETER directives.
Step 1: Dump the Modelfile
ollama show qwen2.5-coder:32b --modelfile >> custom-qwen2.5-coder
Step 2: Edit Hardware Directives
Open custom-qwen2.5-coder in your favorite editor and configure your allocation:
`dockerfile FROM qwen2.5-coder:32b
SYSTEM You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Hardware Allocation Parameters
PARAMETER num_gpu 99
PARAMETER num_thread 16
Context & Memory
PARAMETER num_ctx 8192
`
Understanding What num_gpu and num_thread Actually Do
PARAMETER num_gpu(GPU Layer Offloading):- In Ollama,
num_gpudoes not mean the number of physical graphics cards. It specifies the number of neural network layers offloaded into GPU VRAM. - A 32B or 70B model consists of dozens of discrete layers (typically 32 to 80 layers).
- Setting
PARAMETER num_gpu 999tells Ollama: “Offload every single layer into VRAM if it fits.” - Setting
PARAMETER num_gpu 0tells Ollama: “Do not use GPU acceleration; run 100% on CPU and system RAM.” This is especially useful on machines with weak integrated GPUs where partial offloading slows down inference due to PCIe bandwidth bottlenecks. PARAMETER num_thread(CPU Worker Allocation):- Sets the number of CPU threads used for matrix calculations during prompt processing and token generation.
- The Core Allocation Caveat: Ollama’s default is often conservative (e.g. 4–6 threads). If you have a modern 20-core CPU, bumping this to 16 or 18 significantly accelerates generation.
- Pro Tip: Do not exceed your physical CPU performance cores. In
llama.cpp, setting threads equal to logical hyperthreads (e.g. 32 threads on 16 physical cores) causes cache thrashing and context-switching overhead, which actually reduces tokens per second.
Full Modelfile Parameters Reference
Beyond hardware allocation, you can fine-tune model behavior in the same file:
num_ctx: Context window size in tokens (e.g.PARAMETER num_ctx 8192). Larger windows remember more conversation history but require more VRAM/RAM.num_predict: Maximum number of tokens to generate in response (e.g.PARAMETER num_predict 2048).temperature: Creativity control (0.0for deterministic code/facts,0.7–1.0for creative writing).top_k/top_p: Limits the token selection pool, balancing predictability and variety.repeat_penalty: Penalizes repetitive phrases (e.g.PARAMETER repeat_penalty 1.1).seed: Deterministic random seed (e.g.PARAMETER seed 42).stop: Custom stop sequences (e.g.PARAMETER stop "User:").
Step 3: Build the Custom Model Image
Build the optimized model image using ollama create:
ollama create custom-qwen2.5-coder:32b -f ./custom-qwen2.5-coder
Verify that your newly compiled model is available:
✦ ❯ ollama list
NAME ID SIZE MODIFIED
custom-qwen2.5-coder:32b 353407281410 19 GB Just now
qwen2.5-coder:32b 4bd6cbf2d094 19 GB 15 hours ago
Now, when you run ollama run custom-qwen2.5-coder:32b, it will immediately saturate your designated CPU threads or GPU layers without throttling.
Method 2: The Service Level (Fleet-Wide CPU/GPU Defaults)
If you download and test dozens of models and do not want to rebuild a custom Modelfile for each one, you can configure system-wide hardware allocation at the systemd service daemon level.
On Linux systems, edit the Ollama systemd override:
sudo systemctl edit ollama.service
Add your environment variables inside the [Service] block:
`ini [Service] # Force specific GPU device (e.g. GPU 0, or “0,1” for multi-GPU) Environment=”CUDA_VISIBLE_DEVICES=0″
Number of parallel requests/conversations to handle concurrently
Environment=”OLLAMA_NUM_PARALLEL=4″
Keep multiple models loaded in VRAM simultaneously
Environment=”OLLAMA_MAX_LOADED_MODELS=2″
Keep models warm in memory for longer (default: 5m, or -1 for indefinitely)
Environment=”OLLAMA_KEEP_ALIVE=24h”
`
Save and restart the daemon:
sudo systemctl daemon-reload
sudo systemctl restart ollama
Method 3: On-The-Fly Runtime Overrides (API & WebUI)
If you use tools like Open WebUI, LangChain, LiteLLM, or call Ollama directly via its REST API, you don’t even need to touch the Modelfile or restart the daemon. You can inject options directly into your inference payload:
curl http://localhost:11434/api/generate -d '{
"model": "qwen2.5-coder:32b",
"prompt": "Write a high-performance HTTP server in Rust",
"options": {
"num_thread": 16,
"num_gpu": 99,
"num_ctx": 8192,
"temperature": 0.2
}
}'
Summary & Performance Checklist
| Goal | Best Method | Directive / Parameter | | :— | :— | :— | | Maximize GPU VRAM | Modelfile or API | PARAMETER num_gpu 99 (or options.num_gpu = 99) | | Force Pure CPU Inference | Modelfile or API | PARAMETER num_gpu 0 (avoids PCIe bottleneck) | | Maximize CPU Cores | Modelfile or API | PARAMETER num_thread (set to physical cores) | | Multi-Query Concurrency | Systemd daemon | Environment="OLLAMA_NUM_PARALLEL=4" | | Keep Warm in VRAM | Systemd daemon | Environment="OLLAMA_KEEP_ALIVE=-1" |
Taking 5 minutes to tune these parameters ensures your local AI stack runs at maximum throughput, whether you are crunching local embeddings or streaming 100+ tokens per second on custom hardware.
Just use llama.cpp and dont mess with offloading since that ruins performance. If putting all the layers on your gpu is slow get a better gpu with more vram.
Ollama is hot garbage and you’re wasting your time.
You are wrong for PARAMETER num_gpu 0. Unless you wanted to offload all layers to the CPU, the PARAMETER num_gpu 0 (even though misleading) is for the layers to be loaded to the GPU. Ollama is very conservative on vram management hence the use of that parameter. If more than enough resources, you set this to 999. What is directly related to the GPU ID is, as an example for Nvidia, NVIDIA_VISIBLE_DEVICES=0 (GPU 0 which you get from nvidia-smi). Again, even that one doesn’t make any sense because it says “visible” instead of something like NVIDIA_GPU_ID.
“Unless you wanted to offload all layers to the CPU” – this is exactly what I wanted. Basically I wanted for Ollama not to load any layers in the GPU of that machine that is very slow. The second intention was to use then all the available CPU cores.