> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

---
title: "Local & Self-Hosting Stack"
url: https://daily.dev/agentic-ai-hub/local-self-hosting-stack/
description: "Running models on your own machine or servers."
lastUpdated: "2026-07-22"
---

Running models on your own machine or servers. It wins when you need **privacy/data control** (nothing leaves your box), **offline** operation, **zero marginal cost** at high volume, **no rate limits**, or **full control** over versions and sampling. It loses on frontier quality (the best models aren't open-weight or are too big for local hardware), on ops burden, and on cost-efficiency at low/spiky volume (an idle GPU still costs money).

## The tools

| Tool | Best at | Watch out |
| :---- | :---- | :---- |
| [Ollama](https://ollama.com/) | Developers/local apps who want "it just works": one command pulls and runs a quantized model with a built-in server and OpenAI-compatible API | Convenience defaults (context length, quant) can silently cap quality, so check them; built on llama.cpp under the hood |
| [LM Studio](https://lmstudio.ai/) | Non-terminal users and quick experimentation: desktop GUI to find, download, and chat with local models, plus a local server (strong on Apple Silicon MLX and Windows) | GUI-first, so less suited to headless servers |
| [llama.cpp](https://github.com/ggml-org/llama.cpp) | Portability, quantization control, embedded/edge: the foundational C/C++ engine that defines **GGUF** and powers much of the ecosystem | Lower-level, so you manage flags and builds |
| vLLM / SGLang (see [Inference, Serving & Optimization](/agentic-ai-hub/inference-serving-optimization/)) | Production self-hosting on your own GPUs, when "local" means a real GPU server serving many users | High-throughput server engines, not a laptop on-ramp |

*Also common:* [**Open WebUI**](https://github.com/open-webui/open-webui) (the dominant self-hosted web frontend for Ollama/OpenAI-compatible backends), [**Jan**](https://jan.ai/) and [**GPT4All**](https://www.nomic.ai/gpt4all) (offline desktop apps like LM Studio), and [**KoboldCpp**](https://github.com/LostRuins/koboldcpp) / [**text-generation-webui**](https://github.com/oobabooga/text-generation-webui) (enthusiast runtimes, the latter a natural home for EXL2/EXL3).

## Hardware guidance (local)

- **Apple Silicon** (M-series, unified memory) is a strong fit for many local users: a 64 to 128 GB Mac can run sizable quantized models because CPU and GPU share memory. Use **MLX**-optimized builds for best speed.  
- **NVIDIA consumer GPUs** (e.g., 24 GB-class like a 4090/5090-tier card) run 7B to 14B comfortably and 30B-class in 4-bit. Two cards or a 48 GB card open up ~70B in 4-bit.  
- **CPU/RAM-only** works via llama.cpp for smaller models but is slow for decode. Fine for batch/offline.

## Quantization formats

- **GGUF.** llama.cpp's format. A spectrum of quant levels (e.g., Q4_K_M, Q5_K_M, Q8_0) that trade size/speed for accuracy. **Q4_K_M is the popular "good balance" default.** Runs on CPU+GPU, any platform. The most common format for local models.  
- **MLX.** Apple's array framework and model format optimized for Apple Silicon's unified memory. The fastest path on Macs.  
- **AWQ / GPTQ.** GPU-oriented 4-bit weight quantization formats used with vLLM/TensorRT-LLM/Transformers for server inference.  
- **EXL2/EXL3** (ExLlama). Flexible bit-rate GPU quantization popular with enthusiasts for NVIDIA cards.