LLMKube is a Kubernetes operator that automates hosting LLMs on your own hardware using runtimes like llama.cpp and vLLM. It introduces two custom resource definitions (CRDs): a Model (describing what to run, where to download it, and hardware requirements) and an InferenceService (describing how to run it, replicas, runtime, and endpoint exposure). The tutorial walks through setting up a local kind cluster, installing LLMKube via Helm, deploying a Gemma 2B model from Hugging Face, and querying it via an OpenAI-compatible API endpoint. The resulting service can be connected to AI coding tools like OpenCode, used by agents inside the cluster, or routed through an AI gateway.
•12m watch time