Microsoft.Extensions.AI version 10.9.0 ships four new experimental IChatClient primitives for routing and failover: RoutingChatClient (base class for per-request client selection), SemanticRoutingChatClient (routes by embedding similarity to example utterances), FailoverChatClient (abstract retry-loop base class), and OrderedFailoverChatClient (walks a ranked client list on failure). All types carry the [Experimental] diagnostic MEAI001. The post covers option shaping across request and route levels, sticky session routing via IDistributedCache, staying on one model while varying reasoning effort, and explicit limitations such as no support for model cascading, ensemble routing, or hedging without additional fan-out logic.
Table of contents
RoutingChatClient Copy linkSemanticRoutingChatClient Copy linkFailoverChatClient Copy linkOrderedFailoverChatClient Copy linkBuilding on the primitives Copy linkLimitations Copy linkGetting Started Copy linkQuestions this post answers
What new routing and failover types were added to Microsoft.Extensions.AI 10.9.0?
Microsoft.Extensions.AI 10.9.0 ships four new experimental IChatClient primitives: RoutingChatClient (base class that selects and forwards to another client per request), SemanticRoutingChatClient (routes by embedding similarity to example utterances), FailoverChatClient (abstract base with a retry loop), and OrderedFailoverChatClient (walks a ranked list of clients in order). All are marked [Experimental] with diagnostic ID MEAI001. Teams wiring up multi-provider LLM failover can track these Microsoft.Extensions.AI primitives as they mature on daily.dev.
How does SemanticRoutingChatClient decide which chat client to use for a request?
SemanticRoutingChatClient embeds the last user message and compares it against app-provided example utterances per client using embedding similarity; the client whose profile scores highest above scoreThreshold wins, otherwise the configured defaultClient is used. Profile embeddings are generated lazily and cached after the first request. Key tunables are scoreThreshold, topK (default 1), and scoreAggregation (Mean or Sum), with topK: 5 plus Mean cited as a steadier starting point. Developers choosing between rule-based and semantic routing for LLM requests can compare approaches like this via daily.dev.
Why can't I just use ChatOptions.ConversationId to implement sticky routing across a multi-turn conversation?
ConversationId belongs to a provider's own stateful conversation and may not transfer if the session gets routed to a different client, so it is unreliable as a sticky-routing key. Instead, an application-owned session ID should be passed through ChatOptions.AdditionalProperties and the chosen route name stored in an external cache such as IDistributedCache (e.g., Redis), pinning only after a response completes successfully. Anyone designing session-sticky LLM routing can weigh these tradeoffs alongside similar architecture patterns on daily.dev.