Autoscaling LLM inference deployments requires different thinking than traditional web services. GPU utilization metrics can appear healthy while request queues back up, and cold starts take minutes rather than seconds. Together AI's Dedicated Model Inference platform exposes inference-native autoscaling metrics including in-flight requests, TTFT, GPU utilization, and token throughput. The post explains how to choose the right metric, tune scale-up and scale-down windows asymmetrically (eager up, patient down), and budget for cold starts. An experiment replaying identical load under three policies (inflight_requests, ttft, gpu_utilization) shows that only the concurrency-based signal correctly detected saturation — TTFT stayed low due to continuous batching, and GPU utilization stayed under threshold for bursty short requests. The recommended default is inflight_requests with a target of 8, with latency or utilization metrics added only after observing real traffic patterns.
