Advanced Ollama Setup

Modelfile, environment variables, API tuning, running multiple models, speculative decoding.

🇷🇺 Russian version: advanced-setup.ru.md

← Local models · Tools →

Contents

1. Environment variables
2. Modelfile — custom models
3. HTTP API — access from programs
4. OpenAI-compatible endpoints
5. Parallel requests and multiple models
6. Speculative decoding (speedup)
7. Whats next

1. Environment variables

Environment variables are the simplest way to control how Ollama uses memory, CPU, and network. They must be set before starting the server.

How to set a variable

Method 1 — temporary (single run):

OLLAMA_KV_CACHE_TYPE=q4_0 ollama serve

Method 2 — session-wide:

export OLLAMA_KV_CACHE_TYPE=q4_0
ollama serve

Method 3 — permanent (recommended):

Add lines to ~/.zshrc (executed every time you open a terminal):

echo 'export OLLAMA_KV_CACHE_TYPE=q4_0' >> ~/.zshrc
echo 'export OLLAMA_FLASH_ATTENTION=1' >> ~/.zshrc

Then apply changes (or open a new terminal):

source ~/.zshrc

Variable reference

Recommended config for Mac 16 GB

cat >> ~/.zshrc << 'EOF'

# Ollama — optimization for Mac 16 GB
export OLLAMA_KV_CACHE_TYPE=q4_0    # 4× memory savings
export OLLAMA_FLASH_ATTENTION=1      # memory + speed
export OLLAMA_NUM_PARALLEL=2         # 2 parallel requests
export OLLAMA_KEEP_ALIVE=10m         # keep model 10 min
export OLLAMA_GPU_OVERHEAD=536870912 # 512 MB system reserve
EOF

Then apply:

source ~/.zshrc

Verify variables are applied

# Show all Ollama environment variables
env | grep OLLAMA

Restart ollama serve (or the Ollama application) after changing variables.

2. Modelfile — custom models

Modelfile is like a Dockerfile but for AI models. With it you can:

• Change the system prompt (how the model behaves)
• Tune generation parameters (temperature, context window)
• Create a specialized version of a model for your specific task

Basic Modelfile

Create a file named Modelfile (no extension):

FROM qwen3.5:4b
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
SYSTEM "You are a friendly AI assistant. Answer concisely."

What this does:

• FROM — base model qwen3.5:4b
• PARAMETER temperature 0.7 — moderate creativity (0 = deterministic, 2 = chaos)
• PARAMETER num_ctx 8192 — 8K token context window
• SYSTEM — system prompt (how the model should behave)

# Build the custom model
ollama create my-assistant -f ./Modelfile

# Run it
ollama run my-assistant

Parameter reference

Example: Translation model

FROM qwen3.5:4b
PARAMETER temperature 0.3
SYSTEM "You are a professional translator. Translate between Russian and English. Output only the translation, no comments."

Example: Coding model

FROM qwen2.5-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
SYSTEM "You are a senior Python developer. Write code with type hints, docstrings, and error handling."

Example: Brainstorming

FROM qwen3.5:4b
PARAMETER temperature 1.5
PARAMETER top_p 0.95
SYSTEM "You are a creative copywriter. Generate unusual ideas, think outside the box."

TEMPLATE — prompt format

Some models require a special message format:

FROM llama3.3:8b

TEMPLATE """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

<|eot_id|><|start_header_id|>user<|end_header_id|>

<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""

You usually don’t need to change the template — it’s already built into the model. But if a model behaves oddly, you can override it.

Inspect existing models

# Show Modelfile of installed model
ollama show qwen3.5:4b --modelfile

# Show parameters
ollama show qwen3.5:4b --parameters

# Show system prompt
ollama show qwen3.5:4b --system

3. HTTP API — access from programs

Ollama runs an HTTP server on port 11434. Any program can talk to it.

Basic chat

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [
    {"role": "user", "content": "Hello! How are you?"}
  ],
  "stream": false
}'

Text generation

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:4b",
  "prompt": "Write a haiku about programming",
  "stream": false
}'

JSON mode

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [
    {"role": "user", "content": "Extract name, age, city from: My name is Ivan, Im 25, from Moscow"}
  ],
  "format": "json",
  "stream": false
}'

Result: the model will infer the JSON format:

{"name": "Ivan", "age": 25, "city": "Moscow"}

JSON Schema (structured output)

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [
    {"role": "user", "content": "Extract data from: Vasya, 30, engineer"}
  ],
  "format": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "age": {"type": "integer"},
      "occupation": {"type": "string"}
    },
    "required": ["name", "age"]
  },
  "stream": false
}'

Embeddings

curl http://localhost:11434/api/embed -d '{
  "model": "all-minilm",
  "input": "Text to vectorize"
}'

Memory management

View loaded models:

curl http://localhost:11434/api/ps

Unload a model from memory:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:4b",
  "keep_alive": 0
}'

A value of keep_alive: 0 means “unload immediately”. By default, a model stays in memory for 5 minutes.

Streaming (for chat apps)

Without "stream": false, Ollama returns the response token by token. This creates a “live typing” effect in chat applications:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Tell a story about a robot"}]
}'

List models

curl http://localhost:11434/api/tags

Python (requests library)

import requests
import json

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "qwen3.5:4b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "stream": False
    }
)
print(response.json()["message"]["content"])

4. OpenAI-compatible endpoints

Ollama supports OpenAI API format. Any program that works with OpenAI can work with Ollama — just change the server address.

Endpoints

curl

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # any value works, Ollama ignores it
)

response = client.chat.completions.create(
    model="qwen3.5:4b",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

JavaScript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',
});

const response = await client.chat.completions.create({
  model: 'qwen3.5:4b',
  messages: [{ role: 'user', content: 'Hello!' }],
});

console.log(response.choices[0].message.content);

Using with Continue.dev

In ~/.continue/config.json:

{
  "models": [
    {
      "title": "Local chat",
      "provider": "openai",
      "model": "qwen3.5:4b",
      "apiBase": "http://localhost:11434/v1",
      "apiKey": "ollama"
    },
    {
      "title": "Autocomplete",
      "provider": "openai",
      "model": "qwen2.5-coder:1.5b",
      "apiBase": "http://localhost:11434/v1",
      "apiKey": "ollama"
    }
  ]
}

Using with Aider

export OLLAMA_API_BASE=http://localhost:11434
aider --model ollama/qwen2.5-coder:7b

Important differences from real OpenAI API

5. Parallel requests and multiple models

By default, Ollama processes requests sequentially: one request — one response. But you can enable parallel processing.

Parallel requests to one model

Set OLLAMA_NUM_PARALLEL before starting the server:

export OLLAMA_NUM_PARALLEL=2
ollama serve

Now two apps (e.g., Continue.dev in VS Code + Open WebUI chat) can use the same model simultaneously.

How many can you set? Depends on your RAM. If a model weighs 4 GB and you have 16 GB, 2–3 is fine. If a model weighs 8 GB on a 16 GB machine — only 1.

Check how many requests are currently being processed:

curl http://localhost:11434/api/ps

Multiple models in memory

export OLLAMA_MAX_LOADED_MODELS=3
ollama serve

Now you can keep up to 3 different models in RAM simultaneously. Switching between them is instant (no reloading).

Example scenario:

1. qwen3.5:4b — fast chat (3.4 GB)
2. qwen2.5-coder:7b — coding (4.7 GB)
3. all-minilm — embeddings (0.1 GB)

Total: ~8 GB used, which is fine for a 16 GB Mac.

Manually unload a model

# Via API
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:4b",
  "keep_alive": 0
}'

Or use ollama ps to see what’s loaded first:

ollama ps

How to tell a model has been unloaded

Run ollama ps — it shows an empty list if no model is loaded.

Important: If a model is not in use, Ollama unloads it automatically after OLLAMA_KEEP_ALIVE (default 5 minutes). You can reduce this:

export OLLAMA_KEEP_ALIVE=30s   # unload after 30 seconds of inactivity

6. Speculative decoding (speedup)

A small fast model (draft model) drafts tokens, a large model validates them.

Result: generation speeds up 1.5–2× without quality loss.

How it works

1. The draft model (small) quickly guesses the next 5–10 tokens
2. The large model checks them all at once (in parallel)
3. If the draft guessed correctly — accept; if wrong — the large model corrects

Setup via Modelfile

FROM qwen2.5-coder:7b
DRAFT qwen2.5-coder:1.5b

ollama create my-coder-sd -f ./Modelfile
ollama run my-coder-sd

Setup via API

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "Write a sorting function in Python",
  "options": {
    "draft": "qwen2.5-coder:1.5b"
  },
  "stream": false
}'

Expected speedup

Rule: draft model should be from the same family (Qwen + Qwen, Llama + Llama) and 4–10× smaller.

Download draft models

ollama pull qwen2.5-coder:1.5b
ollama pull llama3.2:3b

Performance example (M4 Pro 24 GB, Qwen3-Coder-30B)

When NOT to use:

• Low RAM — the draft model takes extra space
• Model is already fast (>50 tok/s) — speedup is negligible
• Draft model from a different family — guess quality drops significantly

7. Whats next

In section: getting-started · running-models · models · catalog · quantization · memory-and-context · tools · advanced-setup · troubleshooting · apple-silicon
Related sections: Zero Level · AI Agents · Use Cases
Navigation: ← Local Models · ↑ Back to main · 🇷🇺 Русский