Tools Comparison

Complete comparison of local inference tools: Ollama, LM Studio, MLX, llama.cpp, vLLM, GPT4All, and more.

🇷🇺 Russian version: tools.ru.md

← Local models · Setup →

Contents

1. Overview
2. Scenarios: which tool to choose
3. Ollama
4. LM Studio
5. llama.cpp
6. MLX (Apple Silicon)
7. vLLM
8. GPT4All
9. Jan
10. Open WebUI
11. Aider
12. Continue.dev
13. Enchanted
14. LocalAI
15. MindWork AI Studio
16. Comparison table
17. Benchmarks on Apple Silicon
18. Scaling and Production
19. Decision Tree
20. Installation guides
21. Terminology glossary
22. What’s next

1. Overview

There are many tools for running local LLMs. They differ in:

• Supported formats (GGUF, MLX, AWQ)
• Hardware support (CPU, CUDA, Metal)
• Quantization options (Q4, Q5, Q8, etc.)
• API compatibility (OpenAI, custom)
• Ease of setup

2. Scenarios: which tool to choose

3. Ollama

The most user-friendly option. One command to download and run any model.

brew install ollama
ollama run qwen3.5:4b

Parameter	Value
Interface	CLI + HTTP API
Open Source	MIT
Stars	148K+
Apple Silicon	Metal (native) + MLX (v0.19+, 32GB+ Mac)
GPU support	Metal (Mac), CUDA, ROCm, Vulkan
GitHub	ollama/ollama

How it works under the hood:

• Wrapper around llama.cpp with automatic model downloading
• Models stored in ~/.ollama/models/blobs/ in GGUF format
• On run: checks locally → downloads if missing → loads into RAM → starts llama.cpp backend
• HTTP API on port 11434, compatible with OpenAI

GPU Support Matrix:

Platform	Backend	Status
Apple Silicon	Metal	Native
Apple Silicon	MLX	(v0.19+, 32GB+ RAM only, limited models)
NVIDIA Linux	CUDA
NVIDIA Windows	CUDA
AMD Linux	ROCm
AMD Windows	ROCm	Experimental
Intel GPU	Vulkan	Limited
CPU only	llama.cpp	Always fallback

Key features:

• OpenAI-compatible API (/v1/chat/completions)
• Tool calling (function calling) for agents
• Modelfile — custom models (temperature, system prompt, template)
• Automatic GPU detection (CUDA + Metal)
• Docker support
• Parallel requests via OLLAMA_NUM_PARALLEL
• Multimodal (vision models)
• OLLAMA_KV_CACHE_TYPE — KV cache quantization control
• OLLAMA_MAX_LOADED_MODELS — multiple models in memory

When to choose Ollama:

• Need a simple API for integration (Continue.dev, Aider, Open WebUI)
• Want one command ollama run <model> with no config
• Need cross-platform (Mac + Linux + Windows)
• Ollama 0.19+ — automatic MLX backend for compatible models
• Need maximum speed (MLX is 20–40% faster)
• Need fine-grained inference control (llama.cpp gives more)

Best for: General use, agents, beginners, production

API Endpoints

Ollama provides an HTTP API on port 11434. All requests are POST, responses in JSON.

Basic examples:

# Generate (no streaming)
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:4b",
  "prompt": "What is the meaning of life?",
  "stream": false
}'

# Chat
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

# JSON mode
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Extract the name from: My name is John"}],
  "format": "json",
  "stream": false
}'

# List installed models
curl http://localhost:11434/api/tags

# Loaded models (memory, CPU)
curl http://localhost:11434/api/ps

# Unload model from memory
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:4b",
  "keep_alive": 0
}'

# Embeddings
curl http://localhost:11434/api/embed -d '{
  "model": "all-minilm",
  "input": ["text for vectorization"]
}'

Structured output (JSON Schema):

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Extract data: John, 25, New York"}],
  "format": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "age": {"type": "integer"},
      "city": {"type": "string"}
    },
    "required": ["name", "age", "city"]
  },
  "stream": false
}'

OpenAI-compatible endpoints

Any OpenAI library can connect to Ollama by changing base_url:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # any value
)

response = client.chat.completions.create(
    model="qwen3.5:4b",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

# curl version
curl http://localhost:11434/v1/chat/completions -d '{
  "model": "qwen3.5:4b",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Modelfile Parameters

Modelfile — model configuration in Ollama (analogous to Dockerfile for LLMs):

FROM qwen3.5:4b
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
SYSTEM "You are a professional Python developer."

All PARAMETER instructions:

*num_ctx in Modelfile = 2048, but via API auto-selected based on VRAM. On M1 16GB → 4096 (see README).

Other Modelfile instructions:

Environment Variables

Key OLLAMA_* variables for configuration:

4. LM Studio

GUI application. Download, load, and chat with models without terminal.

Parameter	Value
Interface	GUI + Headless daemon
Open Source	Closed source
Apple Silicon	MLX (auto-detection) + llama.cpp
OS	macOS, Windows, Linux
Website	lmstudio.ai

Key features:

• Built-in model search on HuggingFace (by category, size)
• One-click download and load
• Automatic backend selection: MLX for Mac, CUDA for NVIDIA
• Drag-and-drop GGUF files
• Built-in chat with system prompt, template, parameters
• Local API server (OpenAI compatible)
• Headless daemon since v0.3 (runs in background)
• Model comparison mode

LM Studio vs Ollama on M4 Pro 24GB (Qwen3-Coder-30B MoE):

(Source: asiai.dev)

When to choose LM Studio:

• Just starting out — don’t want to use CLI
• Want to quickly test different models
• Need maximum speed on Mac (MLX)
• Need automation / CI (no CLI)
• Need full open source

Best for: Beginners who prefer GUI, quick testing, visual model comparison

5. llama.cpp

The engine behind Ollama. Lower level, more control.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make -j

# Run a model
./main -m model.gguf -p "Hello" -n 128

Parameter	Value
Interface	CLI + Server
Open Source	MIT
Stars	70K+
Apple Silicon	Metal (native)
OS	All platforms
GitHub	ggml-org/llama.cpp

Unique features:

• GGUF format: K-quants (Q4_K_M, Q5_K_M) and I-quants (IQ2_XXS, IQ3_XXS)
• All GGUF quantizations (Q2_K through Q8_0)
• Speculative decoding (1.5–2× faster)
• KV cache quantization (q8_0, q4_0)
• Built-in embedding endpoint
• Flash attention
• Batch processing
• LoRA adapters on the fly
• Extensive parameter control

Speculative decoding (M3 Ultra 192GB, Llama 3.1 70B Q4):

When to choose llama.cpp:

• Need cross-platform (GGUF everywhere)
• Want full control over inference
• Use non-standard quantizations
• Want simplicity of one command

Best for: Power users, fine-tuning, benchmarking, embedding in custom apps

6. MLX (Apple Silicon)

Apple’s ML framework. Optimized for M series chips.

pip install mlx-lm
mlx_lm.generate --model Qwen/Qwen3.5-4B --prompt "Hello"

Parameter	Value
Interface	Python API + CLI
Open Source	Apache 2.0
Stars	18K+
Apple Silicon	Native (built by Apple for Apple Silicon)
OS	macOS only
GitHub	ml-explore/mlx

What is MLX: MLX is Apple’s machine learning framework for Apple Silicon. It includes:

• mlx — core (arrays, autograd, optimizers)
• mlx-lm — LLM layer (inference + fine-tuning)
• mlx-examples — examples (LoRA, QLoRA, adapters)

Unlike llama.cpp:

• Not a wrapper — native framework for Apple Silicon
• Supports training (fine-tuning, LoRA, QLoRA), not just inference
• Up to 2× faster than llama.cpp on Mac (native optimization)
• Uses uniform quantization (4-bit, 8-bit), not K-quants

Key features:

• Up to 46% faster than llama.cpp on Apple Silicon
• 49% less memory usage
• Native Metal acceleration
• 2/3/4/6 bit quantization
• Good for fine-tuning (LoRA)

Fine-tuning with MLX:

from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Qwen3-8B-4bit")

# LoRA fine-tuning
from mlx_lm import lora
lora.train_lora(
    model=model,
    tokenizer=tokenizer,
    train_set="data.jsonl",
    num_lora_layers=16,
    lora_rank=8,
)

Performance (M5 Max 128GB, Llama 3.1 70B Q4):

When to choose MLX:

• Maximum speed on Mac
• Need fine-tuning / LoRA / QLoRA
• Python ecosystem
• Mac only (not cross-platform)
• Need a GUI

Best for: Mac users wanting maximum speed, fine-tuning on Apple Silicon

7. vLLM

Production serving engine. For high-throughput, multi-user scenarios.

pip install vllm
vllm serve Qwen/Qwen3.5-4B

Parameter	Value
Interface	HTTP API (OpenAI-compatible)
Open Source	Apache 2.0
Stars	45K+
Apple Silicon	Not supported (NVIDIA/AMD only)
GitHub	vllm-project/vllm

Key technologies:

• PagedAttention — efficient KV cache management (eliminates fragmentation)
• Continuous batching — dynamic add/remove requests from batch
• Prefix caching — cache common prompt prefixes
• Tensor parallelism — distribute model across multiple GPUs
• Speculative decoding — built-in
• AWQ/GPTQ quantization
• Kubernetes ready

When to use vLLM:

• Production server with high load (10+ concurrent users)
• Batch processing thousands of prompts
• Multi-GPU cluster
• Not for Mac (NVIDIA/AMD Linux only)

Best for: Production serving, multiple users, high throughput

8. GPT4All

Privacy-focused. Runs entirely on CPU, no GPU needed.

pip install gpt4all

Parameter	Value
Interface	GUI + Python API + CLI
Open Source	MIT
Stars	72K+
Apple Silicon	Metal
GitHub	nomic-ai/gpt4all

LocalDocs (RAG):

• Built-in document indexing (PDF, TXT, MD, DOCX)
• Vector database (custom Nomic implementation)
• Local file search — no data sent to the cloud
• Folder attachments (up to 10 collections)

GPT4All vs Open WebUI for RAG:

Criteria	GPT4All	Open WebUI
Setup	One .dmg	Docker / pip
Documents	LocalDocs	9 vector databases
Multi-user
API	Local	OpenAI-compatible
Resources	Lightweight	Heavier (Python)
Formats	PDF, TXT, MD, DOCX	Any via document loaders

Key features:

• No GPU required
• Built-in RAG
• Local plugin ecosystem
• Very easy setup

When to choose GPT4All:

• Only need RAG on local documents
• Minimal setup (one click)
• Need an API server / integrations
• Multi-user access

Best for: CPU-only machines, privacy-critical applications

9. Jan

Desktop AI client. Beautiful GUI for running models locally with optional cloud model support.

Parameter	Value
Interface	GUI (Electron)
Open Source	AGPL (core), GUI closed
Stars	35K+
Apple Silicon	Metal
GitHub	janhq/jan

Key features:

• Model Hub — search, browse, and download models
• Vision model support (via llama.cpp)
• Remote model support (OpenAI, Anthropic, local servers)
• Thread management (conversation history)
• Local server mode

When to choose Jan:

• Need a polished desktop client
• Want to combine local and cloud models in one app
• Need lightweight (Electron uses significant RAM)

10. Open WebUI

Web interface for Ollama. Feature-rich web UI with RAG, multi-user, and extensive backend support.

Parameter	Value
Interface	Web (Python + Svelte)
Open Source	MIT
Stars	70K+
Install	docker run
GitHub	open-webui/open-webui

Key features:

• Connect to Ollama, OpenAI, Anthropic, Google, AWS Bedrock
• 9 vector databases: Chroma, Milvus, Qdrant, Weaviate, PGVector, Elastic, Meilisearch, Pinecone, Supabase
• Agentic RAG — agent decides when and how to search documents
• Multi-user mode (teams)
• Web search, image generation, audio input
• Themes and customization

When to choose Open WebUI:

• Need a web interface for a team
• Need advanced RAG with agentic search
• Need support for 9+ vector databases
• Only need single-user (overkill)

11. Aider

CLI coding agent. AI pair programming in your terminal — connects to local models via Ollama.

Parameter	Value
Interface	CLI
Open Source	Apache 2.0
Stars	25K+
Local models	Yes (via Ollama)
GitHub	paul-gauthier/aider

Architecture:

• Architect/Editor split — one plans, the other writes code
• Repo map — automatic repository map for context
• Map-refine — refines map as changes are made
• Lint & test integration — automatic code verification

Quality with local models:

• Qwen 2.5 Coder 7B — ~50% of GPT-4o quality on Aider tasks
• DeepSeek R1 7B — good for refactoring
• Qwen3-Coder-30B — ~70% of GPT-4o

When to choose Aider:

• Need an autonomous coding agent in CLI
• Need refactoring via prompts
• Need a GUI / visual interface

12. Continue.dev

IDE plugin. AI assistant for VS Code and JetBrains with local model support.

Parameter	Value
Interface	IDE plugin (VS Code, JetBrains)
Open Source	Apache 2.0
Stars	25K+
Local models	Yes (via Ollama)
GitHub	continuedev/continue

How it works:

• Connects to any OpenAI-compatible API (Ollama, LM Studio, vLLM)
• Chat — conversation with model inside IDE
• Inline editing — select code → prompt → model modifies
• Tab autocomplete — completions as you type (via separate model)
• @codebase — full repository context via embeddings

Per-role model configuration:

{
  "models": [
    {
      "title": "Quick chat",
      "provider": "ollama",
      "model": "qwen3:4b"
    },
    {
      "title": "Autocomplete",
      "provider": "ollama",
      "model": "qwen2.5-coder:1.5b"
    },
    {
      "title": "Complex refactoring",
      "provider": "ollama",
      "model": "qwen2.5-coder:14b"
    }
  ]
}

When to choose Continue.dev:

• Work in VS Code / JetBrains
• Want an AI assistant without Copilot subscription
• Need tab autocomplete with local models
• Don’t use an IDE

13. Enchanted

Native macOS/iOS client. Minimalistic SwiftUI frontend for Ollama.

Parameter	Value
Interface	macOS + iOS GUI (SwiftUI)
Open Source	MIT
Apple Silicon	Native, connects to Ollama
GitHub	gluonfield/enchanted

Key features:

• Native macOS and iOS apps (SwiftUI)
• Connects to any Ollama server via URI (http://localhost:11434)
• Access from iPhone to Mac over local network
• Minimalistic Apple-style interface
• Markdown rendering, image support

When to choose Enchanted:

• Need an iPhone/iPad client in addition to Mac
• Like minimalistic design
• Don’t need iOS — Ollama CLI or Open WebUI is simpler

14. LocalAI

OpenAI API drop-in replacement. Full OpenAI API including TTS, STT, and image generation.

Parameter	Value
Interface	HTTP API (OpenAI-compatible)
Open Source	MIT
Stars	30K+
Install	Docker, binaries
GitHub	mudler/LocalAI

Backend support (60+):

• llama.cpp, transformers, diffusers, whisper, piper-tts, stable-audio
• Image generation, text-to-speech, speech-to-text
• Embeddings, reranking
• Model gallery (YAML configs)

When to choose LocalAI:

• Need full OpenAI API (including TTS, STT, images)
• Already have Docker infrastructure
• Only need LLM (overkill)

15. MindWork AI Studio

Universal multi-provider GUI. One interface for local and cloud models.

Parameter	Value
Interface	GUI (macOS, Windows, Linux)
Open Source	FSL-1.1-MIT (→ MIT in 2 years)
Stars	529
Apple Silicon	Native
GitHub	MindWorkAI/AI-Studio

Provider support:

Key features:

• Unified interface for local and cloud models — switch without restart
• Assistants for common tasks (translation, document analysis, slide generation)
• Image generation (DALL-E, Stable Diffusion)
• RAG (Qdrant, external data via ERI)
• Lua plugins, i18n, enterprise configurations
• Free for commercial use

When to choose MindWork AI Studio:

• Need one GUI for all providers (local + cloud)
• Want to quickly switch between models without configuration
• Need assistants for business tasks
• Only need local inference — Ollama or LM Studio is simpler
• Need CLI/API for automation — not a replacement for Ollama

16. Comparison table

Feature	Ollama	LM Studio	llama.cpp	MLX	vLLM	GPT4All	Jan	Open WebUI	Aider	Continue.dev	Enchanted	LocalAI	MindWork AI Studio
Type	Engine	Engine+GUI	Engine	Engine	Engine	Engine+GUI	Desktop UI	Web UI	CLI Agent	IDE Plugin	Mobile UI	Engine+API	Multi-GUI
Setup	1 command	Download	Build from source	pip install	pip/Docker	pip install	Download	docker run	pip install	IDE install	Download	Docker	Download
Platform	Mac/Lin/Win	Mac/Lin/Win	Mac/Lin/Win	Mac only	Linux	Mac/Lin/Win	Mac/Lin/Win	Web	Mac/Lin/Win	IDE	macOS/iOS	Mac/Lin/Win	Mac/Lin/Win
GPU support	CUDA+Metal	CUDA+Metal	CUDA+Metal	(M only)	(CUDA)	(CPU only)	Metal	Via backend	Via backend	Via backend	Via backend		Via backend
Open Source	MIT		MIT	Apache 2.0	Apache 2.0	MIT	AGPL (core)	MIT	Apache 2.0	Apache 2.0	MIT	MIT	FSL-1.1
Quantizations	K-quants	K-quants	K-quants + I-quants	2–6 bit	AWQ/GPTQ	K-quants	Via backend	N/A	N/A	N/A	N/A	All formats	N/A
Tool calling
OpenAI API
Docker				(Mac only)					Community
Parallel req.
Multi-model			Manual	Manual
RAG support	Manual		Manual	Manual		(LocalDocs)		(9 VDBs)		(@codebase)			(Qdrant)
Speed (M1 7B)	22–25 t/s	22–28 t/s	20–25 t/s	28–35 t/s	N/A	10–15 t/s	Via backend	N/A	N/A	N/A	N/A	Via backend	N/A

17. Benchmarks on Apple Silicon

17.1 MLX vs llama.cpp vs Ollama (M5 Max 128GB, Llama 3.1 70B Q4)

Backend	tok/s	Memory	Difference
MLX	18	~39 GB
llama.cpp Metal	14	~41 GB	–22%
Ollama (CPU)	12	~41 GB	–33%

(Source: CraftRigs)

17.2 MLX vs llama.cpp (M4 Max 36GB, Llama 3 8B Q4)

(Source: Contra Collective)

17.3 MLX vs llama.cpp (M3 Ultra 192GB, Llama 3.1 70B Q4)

(Source: Contra Collective)

17.4 Systematic comparison (arXiv, M2 Ultra 192GB, Qwen-2.5)

(Source: arXiv:2511.05502)

17.5 Speculative decoding acceleration

17.6 Context window impact on speed

Gemma 4 26B MoE on M3 Max 128GB:

(Source: PubliVault)

17.7 M4 Pro 24 GB, Qwen3-Coder-30B MoE

17.8 MLX backend in Ollama

Since March 2026, Ollama can use MLX backend on Mac with 32 GB+ RAM:

• Qwen 3.5-35B-A3B: 58 → 112 tok/s on M5 Max (+93%)
• Currently works with Qwen models, Llama/Mistral coming

18. Scaling and Production

18.1 Concurrent users

18.2 Docker support

Tool	Docker	Official image
Ollama		ollama/ollama
LM Studio		—
MLX	(Mac only)	mlx-community
llama.cpp		ghcr.io/ggml-org/llama.cpp
vLLM		vllm/vllm-openai
GPT4All		—
Jan		—
Open WebUI		ghcr.io/open-webui/open-webui
LocalAI		localai/localai
Aider		Community

18.3 GPU memory management

Tool	Unload	Offload	KV cache control	Multi-GPU
Ollama	On idle	OLLAMA_GPU_LAYERS	OLLAMA_KV_CACHE_TYPE
MLX	(manual)	N/A (always GPU)
llama.cpp		-ngl	--cache-type-k, --cache-type-v
vLLM		--gpu-memory-utilization	--kv-cache-dtype

19. Decision Tree

Want to run LLMs locally?
│
├─ Complete beginner, no terminal
│  └─ LM Studio ─────────────────────────────── GUI, auto-MLX, model browser
│
├─ Need one command and API
│  └─ Ollama ─────────────────────────────────── brew install + ollama run
│     │
│     ├─ Want a web UI → Open WebUI ─────────── Docker, RAG, multi-user
│     ├─ Want IDE assistant → Continue.dev ──── VS Code, tab autocomplete
│     └─ Want CLI agent → Aider ─────────────── architect/editor, repo map
│
├─ Mac, need maximum speed
│  └─ MLX (via LM Studio or mlx-lm) ─────────── +20–40%, –50% RAM
│
├─ Need production server on Linux
│  └─ vLLM ───────────────────────────────────── Continuous batching, multi-GPU
│
├─ Need RAG on documents
│  ├─ Single user → GPT4All
│  └─ Team → Open WebUI
│
├─ Need training / fine-tuning
│  └─ MLX (mlx-lm) ──────────────────────────── LoRA/QLoRA on Apple Silicon
│
├─ Need full control and flexibility
│  └─ llama.cpp (raw) ───────────────────────── GGUF, speculative decoding
│
├─ Need full OpenAI API (TTS, STT, images)
│  └─ LocalAI ───────────────────────────────── 60+ backends, Docker
│
├─ Need a desktop client for local + cloud
│  └─ Jan ───────────────────────────────────── Model Hub, remote support
│
└─ Need one GUI for all providers
   └─ MindWork AI Studio ────────────────────── Local + cloud, plugins, RAG

20. Installation guides

Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run qwen3:8b

# API
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Hello!",
  "stream": false
}'

# Modelfile
cat > Modelfile << 'EOF'
FROM qwen3:8b
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
SYSTEM "You are a professional Python developer."
EOF
ollama create my-coder -f Modelfile

LM Studio

# 1. Download .dmg from lmstudio.ai
# 2. Open → Search → find a model
# 3. Download → Load → Chat
# Server mode:
# Settings → Server → Enable → Port 1234

MLX

# Install
pip install mlx-lm

# Inference
python -m mlx_lm.generate \
  --model mlx-community/Qwen3-8B-4bit \
  --prompt "Hello, how are you?" \
  --max-tokens 256

# Chat
python -m mlx_lm.chat \
  --model mlx-community/Qwen3-8B-4bit

# HTTP server
python -m mlx_lm.server \
  --model mlx-community/Qwen3-8B-4bit

# Fine-tuning
python -m mlx_lm.lora \
  --model mlx-community/Qwen3-8B-4bit \
  --data data.jsonl \
  --num-layers 16 \
  --lora-rank 8

llama.cpp

# Build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && make -j

# Download GGUF
wget https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/qwen3-8b-q4_k_m.gguf

# Run
./main -m qwen3-8b-q4_k_m.gguf \
  -p "Hello!" \
  -n 256 \
  -ngl 99  # all layers on GPU

# Server
./server -m qwen3-8b-q4_k_m.gguf \
  --port 8080 \
  -ngl 99

# Embeddings
./embedding -m qwen3-8b-q4_k_m.gguf \
  -p "Some text for embedding"

vLLM

# Install (Linux only)
pip install vllm

# Server
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-8B \
  --dtype auto \
  --gpu-memory-utilization 0.9

# API
curl http://localhost:8000/v1/chat/completions -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Aider + Ollama

# Install
python -m pip install aider-chat

# Run with local model
export OLLAMA_API_BASE=http://localhost:11434
aider --model ollama/qwen2.5-coder:7b

# Architect mode (2 models)
aider --model ollama/qwen2.5-coder:7b \
  --editor-model ollama/qwen3:4b

# Modes
aider --chat-mode ask      # questions about code
aider --chat-mode code     # writing code
aider --chat-mode architect # architect + editor

Open WebUI

# Docker
docker run -d -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

# Connect to Ollama
# Settings → Connections → Ollama API URL: http://host.docker.internal:11434

GPT4All

# macOS
brew install --cask gpt4all

# Or download from https://gpt4all.io/

# Python API
pip install gpt4all

Jan

# Download from https://jan.ai/
# Or via Homebrew:
brew install --cask jan

# Open → Model Hub → Download → Start chatting

LocalAI

# Docker
docker run -p 8080:8080 \
  -v $PWD/models:/models \
  localai/localai:latest

# LLM
curl http://localhost:8080/v1/chat/completions -d '{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

# TTS
curl http://localhost:8080/v1/audio/speech -d '{
  "model": "tts-1",
  "input": "Hello, world!",
  "voice": "en_US-amy-medium"
}'

Enchanted

# Download from App Store (macOS and iOS)
# Or build from source:
git clone https://github.com/gluonfield/enchanted.git
# Open in Xcode → Build → Run

# Make sure Ollama is running on your Mac
# Configure the server URL in settings
# Start chatting from any device on your network

MindWork AI Studio

# Download from https://mindwork.ai/
# Available for macOS, Windows, Linux

# Or build from source:
git clone https://github.com/MindWorkAI/AI-Studio.git
# Follow build instructions in the repository

21. Terminology glossary

22. What’s next

← Back to navigation

In section: getting-started · running-models · models · catalog · quantization · memory-and-context · tools · advanced-setup · troubleshooting · apple-silicon
Related sections: Zero Level · AI Agents · Use Cases
Navigation: ← Local Models · ↑ Back to main · 🇷🇺 Русский