Moving an ML model from a notebook to a production cluster exposes hidden assumptions about environment, hardware, and human oversight. The notebook is a controlled, forgiving environment — one machine, fixed libraries, a human watching every cell. The cluster is the opposite: mixed hardware, containerized dependencies, remote data, and unattended jobs. Common failure modes include non-deterministic GPU kernels producing floating-point drift, library version mismatches in container images, and data loader ordering changes when streaming from object storage. GPU memory fragmentation under concurrent jobs is often the real bottleneck, not compute utilization. Four habits close the gap: treat the environment as a fixed artifact by pinning all versions and seeds; test under contention rather than on an empty node; build automated guardrails for every failure a human would have caught manually; and promote models based on behavior under production-like conditions, not just a clean log.
