Local & Self-Hosting Stack
Table of Contents
Link copied!
Running models on your own machine or servers. It wins when you need privacy/data control (nothing leaves your box), offline operation, zero marginal cost at high volume, no rate limits, or full control over versions and sampling. It loses on frontier quality (the best models aren't open-weight or are too big for local hardware), on ops burden, and on cost-efficiency at low/spiky volume (an idle GPU still costs money).
The tools
| Tool | Best at | Watch out |
|---|---|---|
| Ollama | Developers/local apps who want "it just works": one command pulls and runs a quantized model with a built-in server and OpenAI-compatible API | Convenience defaults (context length, quant) can silently cap quality, so check them; built on llama.cpp under the hood |
| LM Studio | Non-terminal users and quick experimentation: desktop GUI to find, download, and chat with local models, plus a local server (strong on Apple Silicon MLX and Windows) | GUI-first, so less suited to headless servers |
| llama.cpp | Portability, quantization control, embedded/edge: the foundational C/C++ engine that defines GGUF and powers much of the ecosystem | Lower-level, so you manage flags and builds |
| vLLM / SGLang (see Inference, Serving & Optimization) | Production self-hosting on your own GPUs, when "local" means a real GPU server serving many users | High-throughput server engines, not a laptop on-ramp |
Also common: Open WebUI (the dominant self-hosted web frontend for Ollama/OpenAI-compatible backends), Jan and GPT4All (offline desktop apps like LM Studio), and KoboldCpp / text-generation-webui (enthusiast runtimes, the latter a natural home for EXL2/EXL3).
Hardware guidance (local)
- Apple Silicon (M-series, unified memory) is a strong fit for many local users: a 64 to 128 GB Mac can run sizable quantized models because CPU and GPU share memory. Use MLX-optimized builds for best speed.
- NVIDIA consumer GPUs (e.g., 24 GB-class like a 4090/5090-tier card) run 7B to 14B comfortably and 30B-class in 4-bit. Two cards or a 48 GB card open up ~70B in 4-bit.
- CPU/RAM-only works via llama.cpp for smaller models but is slow for decode. Fine for batch/offline.
Quantization formats
- GGUF. llama.cpp's format. A spectrum of quant levels (e.g., Q4_K_M, Q5_K_M, Q8_0) that trade size/speed for accuracy. Q4_K_M is the popular "good balance" default. Runs on CPU+GPU, any platform. The most common format for local models.
- MLX. Apple's array framework and model format optimized for Apple Silicon's unified memory. The fastest path on Macs.
- AWQ / GPTQ. GPU-oriented 4-bit weight quantization formats used with vLLM/TensorRT-LLM/Transformers for server inference.
- EXL2/EXL3 (ExLlama). Flexible bit-rate GPU quantization popular with enthusiasts for NVIDIA cards.