--- title: Distributed training on OpenShift AI 3.4 with Kubeflow... description: Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, replacing framework-specific CRDs like PyTorchJob, TFJob, and MPIJob with a unified TrainJob resource... canonical: https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2 | daily.dev og:description: Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, replacing framework-specific CRDs like PyTorchJob, TFJob, and MPIJob with a unified TrainJob resource... og:url: https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam og:image: https://api.daily.dev/og/posts/ZkGnOHkam.png og:image:alt: Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2 **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 14 min read · 0 upvotes · 0 comments ## Summary Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, replacing framework-specific CRDs like PyTorchJob, TFJob, and MPIJob with a unified TrainJob resource that translates into a JobSet under the hood. A walkthrough validates the full stack on NVIDIA GPUs: manual operator validation, then real-world fraud detection training across 1, 2, and 4 A10G GPUs showing near-linear DDP scaling (95% efficiency at 4 GPUs). It also covers suspend/resume with checkpoint recovery, GPU telemetry via DCGM Exporter, and advanced raw JobSet features like dependsOn/successPolicy, coordinator-based rendezvous, failure policies, and volumeClaimPolicies for multi-stage pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Kubeflow Trainer v2's TrainJob API differ from the old PyTorchJob, TFJob, and MPIJob CRDs? TrainJob replaces the separate framework-specific CRDs (PyTorchJob, TFJob, MPIJob) with a single unified resource. Instead of manually wiring MASTER_ADDR, environment variables, and torchrun commands for each framework, you reference a prebuilt ClusterTrainingRuntime and just specify training code, node count, and GPUs per node; Kubeflow Trainer v2 translates the TrainJob into a JobSet automatically. _daily.dev helps engineers evaluating Kubernetes training APIs track how tools like Kubeflow Trainer evolve._ ### What DDP scaling efficiency did PyTorch distributed training achieve going from 1 to 4 GPUs on OpenShift AI 3.4? Training a fraud-detection embedding model with PyTorch DDP across NVIDIA A10G GPUs achieved 1.94x speedup on 2 GPUs and 3.80x on 4 GPUs, roughly 95% scaling efficiency, cutting training time from over 15 minutes on one GPU to about 4 minutes on four, with model AUC-ROC quality remaining stable across all splits. _Developers benchmarking multi-GPU DDP setups can compare real efficiency numbers like these on daily.dev._ ### How do JobSet's dependsOn and successPolicy fields prevent GPU training pods from starting before data is ready? A JobSet's training group can declare dependsOn with a status of Complete against a data-prep group, so training pods are not created until data-prep succeeds, avoiding wasted GPU allocation on missing data. A successPolicy with operator All targeting the training group ensures the JobSet is only marked complete once training finishes, not when data-prep alone succeeds. _Teams designing multi-stage GPU pipelines can dig into JobSet orchestration patterns like this via daily.dev._ ## Similar posts on daily.dev - [A guide to JIT checkpointing with Kubeflow Trainer on OpenShift AI](https://daily.dev/posts/a-guide-to-jit-checkpointing-with-kubeflow-trainer-on-openshift-ai-y8behxml8) · Red Hat Developer · 0 upvotes · 0 comments - [Kubeflow Trainer v2.2: JAX & XGBoost Runtimes, Flux for HPC Support, and TrainJob progress and metrics observability](https://daily.dev/posts/kubeflow-trainer-v2-2-jax-xgboost-runtimes-flux-for-hpc-support-and-trainjob-progress-and-metri-q10hneoea) · Kubeflow · 1 upvotes · 0 comments - [Kubeflow unveils new cloud native innovations to supercharge AI](https://daily.dev/posts/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai-j6rwmavmb) · CNCF · 0 upvotes · 0 comments - [Fine-tune AI pipelines in Red Hat OpenShift AI 3.3](https://daily.dev/posts/fine-tune-ai-pipelines-in-red-hat-openshift-ai-3-3-kdpwnunnd) · Red Hat Developer · 0 upvotes · 0 comments - [Red Hat OpenShift 4.20 boosts AI workloads, security](https://daily.dev/posts/red-hat-openshift-4-20-boosts-ai-workloads-security-fne0alrnp) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#kubernetes](https://daily.dev/tags/kubernetes), [#pytorch](https://daily.dev/tags/pytorch) [View this post on daily.dev](https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2","url":"https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam"},"datePublished":"2026-09-29T13:33:08.947Z","dateModified":"2026-09-29T13:33:31.563Z","description":"Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, replacing framework-specific CRDs like PyTorchJob, TFJob, and MPIJob with a unified TrainJob resource...","image":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","thumbnailUrl":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,kubernetes,pytorch","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/distributed-training-on-openshift-ai-3-4-with-kubeflow-trainer-v2-zkgnohkam#faq","mainEntity":[{"@type":"Question","name":"How does Kubeflow Trainer v2's TrainJob API differ from the old PyTorchJob, TFJob, and MPIJob CRDs?","acceptedAnswer":{"@type":"Answer","text":"TrainJob replaces the separate framework-specific CRDs (PyTorchJob, TFJob, MPIJob) with a single unified resource. Instead of manually wiring MASTER_ADDR, environment variables, and torchrun commands for each framework, you reference a prebuilt ClusterTrainingRuntime and just specify training code, node count, and GPUs per node; Kubeflow Trainer v2 translates the TrainJob into a JobSet automatically. daily.dev helps engineers evaluating Kubernetes training APIs track how tools like Kubeflow Trainer evolve."}},{"@type":"Question","name":"What DDP scaling efficiency did PyTorch distributed training achieve going from 1 to 4 GPUs on OpenShift AI 3.4?","acceptedAnswer":{"@type":"Answer","text":"Training a fraud-detection embedding model with PyTorch DDP across NVIDIA A10G GPUs achieved 1.94x speedup on 2 GPUs and 3.80x on 4 GPUs, roughly 95% scaling efficiency, cutting training time from over 15 minutes on one GPU to about 4 minutes on four, with model AUC-ROC quality remaining stable across all splits. Developers benchmarking multi-GPU DDP setups can compare real efficiency numbers like these on daily.dev."}},{"@type":"Question","name":"How do JobSet's dependsOn and successPolicy fields prevent GPU training pods from starting before data is ready?","acceptedAnswer":{"@type":"Answer","text":"A JobSet's training group can declare dependsOn with a status of Complete against a data-prep group, so training pods are not created until data-prep succeeds, avoiding wasted GPU allocation on missing data. A successPolicy with operator All targeting the training group ensures the JobSet is only marked complete once training finishes, not when data-prep alone succeeds. Teams designing multi-stage GPU pipelines can dig into JobSet orchestration patterns like this via daily.dev."}}]} ```