OpenCost 1.121.0 introduces Kubernetes-native inference cost tracking by integrating with llm-d and vLLM. The release exposes two distinct cost metrics — allocation-based (full hosting cost including idle GPU time) and usage-based (active inference compute only) — published to Prometheus and accessible via REST API. The distinction is critical: using only usage-based cost to justify self-hosting versus SaaS APIs is a common mistake, since idle GPU time can make self-hosting 4x more expensive than it appears. The integration also accounts for KV cache hits, disaggregated prefill/decode costs, and shared infrastructure components like the inference scheduler and gateway proxy. A proof-of-concept was validated on a 109-GPU, 30-model cluster. Platform and FinOps teams can use these metrics for chargeback reporting, utilization analysis, and build-vs-buy decisions.
