Why I Switched from Ollama to llama.cpp — and What I Learned
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A hands-on account of switching a local LLM setup from Ollama to llama.cpp on a dual-GPU AMD APU handheld (Ryzen 7840U with integrated Radeon 780M and discrete RX 7600M XT). Covers determining available VRAM via free/glxinfo, picking the right GPU device with --list-devices and visibility env vars, symlinking Ollama-downloaded GGUF blobs for use with llama-server, tuning -ngl layer offload and context size, wiring llama-server's OpenAI-compatible endpoint into Obsidian plugins, and reading server logs to diagnose CPU-offload bottlenecks. Ends with a two-week verdict: llama.cpp trades convenience for granular hardware control versus Ollama's simplicity.
Table of contents
The Reason of Switching to llama.cpp from ollamaDetermine How Much vRAMDownload LLM using OllamaConfigure llama.cpp to use LLMGet j3ffyang ’s stories in your inboxConfigure Obsidian using llama.cppTuningAfter 2 Weeks of TestingQuestions this post answers
How do I calculate the right -ngl value for llama-server on a GPU with limited VRAM?
Calculate the total quantized model weight size, divide by the number of layers to get per-layer size, then multiply by the number of layers you want on GPU and add the KV cache size for your context length. For example, with Qwen2.5-14B Q4_K_M (~9.04GB total, 48 layers), offloading 32 layers to GPU uses about 6.0GB of weights plus 1.1GB KV cache at -c 8192, fitting under a 7079MB VRAM budget. Track hardware-tuning tips like this alongside other llama.cpp workflows on daily.dev.
Why does llama.cpp pick the wrong GPU on a laptop with both an integrated and a discrete GPU?
llama.cpp defaults to the first device the backend enumerates, and on Vulkan the integrated GPU is often listed first, leaving the discrete GPU idle even though it would run models several times faster. Use --list-devices and --device to check and override the choice, or set GGML_VK_VISIBLE_DEVICES (Vulkan), HIP_VISIBLE_DEVICES (ROCm), or CUDA_VISIBLE_DEVICES (CUDA) to force the correct GPU. Developers juggling multi-GPU inference setups can follow tuning details like these on daily.dev.
Is it safe to run llama-server with CORS allowing all origins and no API key?
It is acceptable only because llama-server binds to 127.0.0.1 by default, making it reachable solely from the local machine despite the warning about cross-origin risk. If the server is ever exposed with --host 0.0.0.0, an --api-key should be set first to avoid the security risk the CORS warning flags. Keep local-LLM security defaults like this in view by following llama.cpp coverage on daily.dev.