--- title: "Run LLMs on Kubernetes with LLMKube" url: https://daily.dev/posts/run-llms-on-kubernetes-with-llmkube-qxeumbpd1 source_url: https://www.youtube.com/watch?v=xdMtc8jm88Q type: video:youtube source: "That DevOps Guy" published: 2026-05-31T07:40:52.627Z updated: 2026-05-31T07:52:01.196Z tags: ["kubernetes", "llm", "llama-cpp"] reading_time: 12 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Run LLMs on Kubernetes with LLMKube **[That DevOps Guy](https://daily.dev/sources/marcel-dempers)** · 12 min read · 0 upvotes · 0 comments ## Summary LLMKube is a Kubernetes operator that automates hosting LLMs on your own hardware using runtimes like llama.cpp and vLLM. It introduces two custom resource definitions (CRDs): a Model (describing what to run, where to download it, and hardware requirements) and an InferenceService (describing how to run it, replicas, runtime, and endpoint exposure). The tutorial walks through setting up a local kind cluster, installing LLMKube via Helm, deploying a Gemma 2B model from Hugging Face, and querying it via an OpenAI-compatible API endpoint. The resulting service can be connected to AI coding tools like OpenCode, used by agents inside the cluster, or routed through an AI gateway. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Running a self-hosted LLM in Kubernetes with vLLM](https://daily.dev/posts/running-a-self-hosted-llm-in-kubernetes-with-vllm-znxvebe4x) · CNCF · 3 upvotes · 1 comments - [Running LLM Inference on Kubernetes: What It Actually Takes](https://daily.dev/posts/running-llm-inference-on-kubernetes-what-it-actually-takes-opl6iozrg) · Fairwinds Blog · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#llm](https://daily.dev/tags/llm), [#llama-cpp](https://daily.dev/tags/llama-cpp) [View this post on daily.dev](https://daily.dev/posts/run-llms-on-kubernetes-with-llmkube-qxeumbpd1)