--- title: Enabling MLflow OpenAI Autolog on PySpark Workers description: When distributing LLM calls across PySpark workers using mapInPandas, MLflow's openai.autolog() silently fails to emit traces. Three separate issues must be... canonical: https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Enabling MLflow OpenAI Autolog on PySpark Workers | daily.dev og:description: When distributing LLM calls across PySpark workers using mapInPandas, MLflow's openai.autolog() silently fails to emit traces. Three separate issues must be... og:url: https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu og:image: https://api.daily.dev/og/posts/GcEK5npMu.png og:image:alt: Enabling MLflow OpenAI Autolog on PySpark Workers og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Enabling MLflow OpenAI Autolog on PySpark Workers **[DevBlogs](https://daily.dev/sources/devblogs)** · 5 min read · 1 upvotes · 0 comments ## Summary When distributing LLM calls across PySpark workers using mapInPandas, MLflow's openai.autolog() silently fails to emit traces. Three separate issues must be resolved: workers need explicit MLflow tracking URI and experiment name (not inherited from the driver), async trace export must be disabled to prevent daemon thread races against process termination, and parent-child trace linking is currently unsupported. The fix involves setting tracking URI, experiment name, and MLFLOW_ENABLE_ASYNC_TRACE_LOGGING=false inside the worker function. Once working, per-call tracing revealed hidden cost multipliers caused by Spark lazy evaluation re-executing LLM calls multiple times. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does mlflow.openai.autolog() produce no traces when I call it inside a PySpark mapInPandas worker function? Calling autolog() on workers is necessary but not sufficient because workers don't inherit the MLflow tracking URI or experiment name from the driver. Without explicitly calling mlflow.set_tracking_uri() and mlflow.set_experiment() inside the worker function before mlflow.openai.autolog(), the tracking URI defaults to an empty local path and traces are silently discarded with no errors. _daily.dev surfaces practical fixes like this for engineers debugging distributed MLflow tracing setups._ ### Why are some of my MLflow trace span artifacts missing from the detailed trace view even though trace metadata shows up correctly? MLflow's AsyncTraceExportQueue writes span artifacts via a background daemon thread that relies on atexit to flush on shutdown, so when running as a Databricks job task the Python process can exit before the flush completes, dropping roughly 5 out of 6 span artifacts. Setting MLFLOW_ENABLE_ASYNC_TRACE_LOGGING=false forces synchronous export, adding only about 100-500ms per trace versus 5-20 second LLM call latency. _Engineers chasing flaky observability gaps track fixes like this one on daily.dev._ ### How can I link multiple MLflow autolog traces from OpenAI chat.completions.create calls into a single parent trace? There is currently no built-in mechanism because MLflow's openai autolog uses start_span_no_context() which always creates root spans with no parent context, so each call produces an independent, disconnected trace. Workarounds include tagging traces with a shared batch_id via mlflow.set_span_attribute(), or dropping autolog entirely and using mlflow.start_trace() manually for full hierarchy control at the cost of autolog's structured parsing. _daily.dev helps teams weighing tracing trade-offs like this keep up with MLflow's evolving internals._ ## Similar posts on daily.dev - [From Black Box to Observability: Tracing OpenClaw with MLflow](https://daily.dev/posts/from-black-box-to-observability-tracing-openclaw-with-mlflow-4aomzss9g) · mlflow · 1 upvotes · 1 comments - [Full OpenTelemetry Support in MLflow Tracing](https://daily.dev/posts/full-opentelemetry-support-in-mlflow-tracing-urso8nc6k) · mlflow · 1 upvotes · 0 comments - [What a Useful AI Trace Should Actually Contain \+ Build One](https://daily.dev/posts/what-a-useful-ai-trace-should-actually-contain-build-one-4avkgyjsk) · Telerik · 0 upvotes · 0 comments - [AI Observability for Every TypeScript LLM Stack](https://daily.dev/posts/ai-observability-for-every-typescript-llm-stack-dczsq7dnz) · mlflow · 35 upvotes · 0 comments - [How AI observability works with MLflow](https://daily.dev/posts/how-ai-observability-works-with-mlflow-7nadkipz3) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#azure](https://daily.dev/tags/azure), [#openai](https://daily.dev/tags/openai), [#observability](https://daily.dev/tags/observability), [#pyspark](https://daily.dev/tags/pyspark) [View this post on daily.dev](https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Enabling MLflow OpenAI Autolog on PySpark Workers","url":"https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu"},"datePublished":"2026-07-03T12:51:59.985Z","dateModified":"2026-09-14T07:36:15.952Z","description":"When distributing LLM calls across PySpark workers using mapInPandas, MLflow's openai.autolog() silently fails to emit traces. Three separate issues must be...","image":"https://media.daily.dev/image/upload/s--58gMhC4P--/f_auto/v1722860399/public/Placeholder%2012","thumbnailUrl":"https://media.daily.dev/image/upload/s--58gMhC4P--/f_auto/v1722860399/public/Placeholder%2012","isAccessibleForFree":true,"articleSection":"DevBlogs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DevBlogs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f68b453325f482ebeb73fb780092713","url":"https://daily.dev/sources/devblogs"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,azure,openai,observability,pyspark","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DevBlogs","item":"https://daily.dev/sources/devblogs"},{"@type":"ListItem","position":3,"name":"Enabling MLflow OpenAI Autolog on PySpark Workers"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/enabling-mlflow-openai-autolog-on-pyspark-workers-gcek5npmu#faq","mainEntity":[{"@type":"Question","name":"Why does mlflow.openai.autolog() produce no traces when I call it inside a PySpark mapInPandas worker function?","acceptedAnswer":{"@type":"Answer","text":"Calling autolog() on workers is necessary but not sufficient because workers don't inherit the MLflow tracking URI or experiment name from the driver. Without explicitly calling mlflow.set_tracking_uri() and mlflow.set_experiment() inside the worker function before mlflow.openai.autolog(), the tracking URI defaults to an empty local path and traces are silently discarded with no errors. daily.dev surfaces practical fixes like this for engineers debugging distributed MLflow tracing setups."}},{"@type":"Question","name":"Why are some of my MLflow trace span artifacts missing from the detailed trace view even though trace metadata shows up correctly?","acceptedAnswer":{"@type":"Answer","text":"MLflow's AsyncTraceExportQueue writes span artifacts via a background daemon thread that relies on atexit to flush on shutdown, so when running as a Databricks job task the Python process can exit before the flush completes, dropping roughly 5 out of 6 span artifacts. Setting MLFLOW_ENABLE_ASYNC_TRACE_LOGGING=false forces synchronous export, adding only about 100-500ms per trace versus 5-20 second LLM call latency. Engineers chasing flaky observability gaps track fixes like this one on daily.dev."}},{"@type":"Question","name":"How can I link multiple MLflow autolog traces from OpenAI chat.completions.create calls into a single parent trace?","acceptedAnswer":{"@type":"Answer","text":"There is currently no built-in mechanism because MLflow's openai autolog uses start_span_no_context() which always creates root spans with no parent context, so each call produces an independent, disconnected trace. Workarounds include tagging traces with a shared batch_id via mlflow.set_span_attribute(), or dropping autolog entirely and using mlflow.start_trace() manually for full hierarchy control at the cost of autolog's structured parsing. daily.dev helps teams weighing tracing trade-offs like this keep up with MLflow's evolving internals."}}]} ```