EvalHub is a Kubernetes-based service for running LLM evaluation benchmarks. Without centralized resource management, shared clusters suffer from unmanaged GPU/CPU consumption, lack of job prioritization, and cluster instability. Kueue, a Kubernetes-native workload queueing system, addresses these issues by enforcing resource quotas per team, enabling priority-based preemption, supporting cohort-based resource borrowing, and automatically queuing jobs when quota is exhausted. The guide walks through setup for three personas — cluster administrator, namespace owner, and ML engineer — covering ClusterQueue and LocalQueue configuration, job submission via the EvalHub API, and monitoring/troubleshooting preemption scenarios using kubectl commands. Key operational notes include that preempted jobs restart from scratch (no checkpointing), and the EvalHub API only exposes high-level states, requiring direct kubectl queries for detailed preemption status.
