Supabase has open-sourced supabase/evals, a benchmark and framework for measuring how well AI coding agents (Claude Code, Codex, OpenCode) perform on real Supabase tasks like schema building, Edge Function debugging, and RLS policy fixes. The framework runs agents against actual Supabase environments using containerized stacks, scoring results with deterministic checks and LLM-as-a-judge. Key findings include: agents perform reasonably well without skills loaded, but skills help with edge cases and outdated knowledge; agents tend to avoid declarative schema workflows; newer Supabase libraries like @supabase/server are underused; skill activation is uneven across models; and Claude Code checks docs far less frequently than Codex-based agents. The benchmark results are publicly viewable and the regression suite runs daily internally.
