Pulumi describes how its Pulumi HCL runtime is compatibility-tested against OpenTofu, built on the premise that any program valid for tofu apply must behave identically under pulumi up. The tfcompat test framework runs equivalent Terraform and Pulumi programs in parallel, recording each provider's gRPC calls and comparing outputs to define correctness as producing identical provider steps. The team also describes using LLM sub-agents (Claude, Codex) to autonomously hunt for compatibility bugs, since tests only need to encode a scenario rather than expected behavior, making false positives rare and enabling scaled, automated bug discovery via draft PRs.
Questions this post answers
How does Pulumi verify that Pulumi HCL behaves the same as OpenTofu's tofu apply?
Pulumi uses a test framework called tfcompat that runs the same HCL program through both pulumi up/preview and tofu apply/plan against in-memory Terraform providers, recording every provider gRPC call and stack output. A test passes only if both tools call identical provider RPCs and produce identical outputs, defining correctness as matching provider steps rather than matching an expected behavior written by the test author. daily.dev surfaces engineering deep dives like this for teams comparing IaC tool compatibility strategies.
How can LLM agents be used to find bugs in a compatibility test suite without producing false positives?
Because tfcompat tests only require specifying a scenario (the HCL files and providers used) rather than the expected correct behavior, an LLM cannot easily fabricate a false failure since the pass/fail criteria is defined by comparing Pulumi's and OpenTofu's actual provider calls, not a hardcoded expectation. Pulumi runs prompts instructing Claude or Codex sub-agents to find and prove bugs by writing new failing tfcompat tests, then opening draft PRs for each real divergence found. developers exploring agentic testing workflows can track emerging LLM bug-hunting patterns on daily.dev.
15.6K Impressions1 Comment