A practical pattern for LLM-based classification at scale: instead of constraining the LLM to a large enum of legal vocabulary values (which hits API limits and is expensive), let the LLM freely hallucinate plausible-but-fake category labels for a given input. Then use embedding similarity (e.g., MiniLM dot product) to map the hallucinated label to the closest real entry in your taxonomy. This avoids shipping large schemas to the model, works with cheaper/smaller LLMs, and sidesteps OpenAI's structured output enum limits.
Questions this post answers
How do I classify text into a large taxonomy with an LLM without hitting structured output enum limits?
Instead of constraining the LLM to a list of legal values, prompt it to freely generate a plausible-but-fake label for the input. Then embed that hallucinated label and dot-product it against pre-computed embeddings of your real taxonomy entries to find the closest match. This avoids shipping large schemas to the model, works with cheaper models, and sidesteps OpenAI's structured output enum size limits. Developers solving LLM classification at scale share patterns like this on daily.dev.
What is the upper limit on enums in OpenAI structured outputs?
OpenAI imposes an upper limit on the number of enum values you can send in a structured output schema. When a taxonomy has hundreds of categories, this constraint makes the standard Pydantic Literal approach impractical, which is one motivation for the hallucination-then-embedding-retrieval pattern as an alternative. Teams hitting OpenAI API constraints on taxonomy classification discuss workarounds on daily.dev.
2.8K Impressions1 Comment