A Red Hat IT team upgraded OpenShift AI from 2.25.5 to 3.3.2 — the most architecturally significant release, replacing Service Mesh, authentication, and serverless components — using an AI coding assistant as a primary engineering partner. The approach compressed an estimated 6-month, 3-4 engineer project into 8 weeks with a single engineer, achieving a 60% reduction in engineering effort. The team built a custom Go CLI called rhoai-upgrade (9,384 lines across 28 files) implementing a gate-based orchestration pipeline with hexagonal architecture. Nine undocumented issues were discovered on the dev cluster, encoded into automation, and resolved automatically on pre-production and production clusters. The production upgrade completed in a single day with zero downtime for 297 users and 71 InferenceServices. Key troubleshooting patterns include NetworkPolicy label mismatches causing HTTP 503s and PVC naming convention changes causing HTTP 502s on workbenches.
Table of contents
PrerequisitesStep-by-step instructionsCommon issues and troubleshootingTips and best practicesNext steps for your OpenShift AI environmentQuestions this post answers
What changed architecturally in Red Hat OpenShift AI 3.3.2 compared to 2.x?
OpenShift AI 3.3.2 replaces three core components: Gateway API replaces OpenShift Service Mesh 2, kube-rbac-proxy replaces oauth-proxy, and RawDeployment replaces Serverless and ModelMesh. The dashboard URL also changes from rhods-dashboard-redhat-ods-applications.$APPS_DOMAIN to data-science-gateway.$APPS_DOMAIN, affecting bookmarks and hard-coded references across an organization. Engineers planning the 2.x to 3.3 migration track breaking changes like these on daily.dev before they hit production.
Why are my InferenceService routes returning HTTP 503 after upgrading OpenShift AI to 3.3.2?
The cause is a NetworkPolicy with a podSelector still referencing the old HAProxy IngressController label (ingresscontroller.operator.openshift.io/deployment-ingresscontroller: internal-router-shard). After the upgrade, the Istio Gateway Envoy pod uses a different label (gateway.networking.k8s.io/gateway-name: data-science-gateway), so the unchanged policy silently drops traffic. Fix it with an oc patch command updating the NetworkPolicy podSelector to match the new gateway label. Teams running OpenShift AI upgrades catch silent networking regressions like this faster on daily.dev.
Why do workbenches show HTTP 502 EACCES permission denied errors after upgrading to OpenShift AI 3.x?
In OpenShift AI 3.x, the persistent volume is only auto-mounted to /opt/app-root/src when the PVC name matches the notebook name exactly. The older 2.x convention appended a -pvc suffix, which causes the volume to appear in the pod spec but never actually mount. The code-server process then cannot write to its expected path. The fix requires two oc patch commands on the Notebook custom resource: one to rename the PVC reference and one to update the volume mount entry. Platform engineers upgrading OpenShift AI workbench environments find undocumented edge cases like this on daily.dev.