The IChatClient Middleware Pipeline: Caching, Telemetry and Custom Guards

Learning outcome

You can compose an IChatClient pipeline with ChatClientBuilder: response caching, OpenTelemetry, function invocation and your own DelegatingChatClient (for example a budget guard). You can predict how the order of Use* calls changes behavior, and know which spans and tags a dashboard gets.

Intuition

It works like ASP.NET Core middleware. Each decorator wraps the next client, can act before and after the call, and can short-circuit. The order matters: a cache placed outside telemetry hides cache hits from your traces, and a budget guard placed outside the cache rejects requests the cache could have answered for free.

flowchart LR
    C["caller"] --> G["BudgetGuard<br/>(custom)"] --> K["DistributedCachingChatClient"] --> O["OpenTelemetryChatClient"] --> M["model (FAKE here)"]
    K -. "cache hit: returns early" .-> C

Deep dive

• Building: services.AddChatClient(inner).Use(...).UseDistributedCache(...).UseOpenTelemetry(...) or new ChatClientBuilder(inner)...Build(). The first Use is the outermost layer.
• Built-in middleware:
  • UseFunctionInvocation (Lesson 4);
  • UseDistributedCache(IDistributedCache), keyed on messages and options;
  • UseOpenTelemetry, with GenAI semantic-convention spans and metrics;
  • UseLogging;
  • ConfigureOptions, for defaults such as model id and temperature.
• Custom middleware: derive from DelegatingChatClient and override GetResponseAsync and GetStreamingResponseAsync, or use the Use(...) delegate overloads. Typical uses:
  • token budgets per user or tenant;
  • PII redaction before the call;
  • tenant-specific options;
  • retries and fallback (Lesson 11);
  • content checks (Lesson 12).
• Caching: exact-match caching only helps when the identical prompt repeats (FAQ bots, deterministic extraction over the same document, evaluation reruns). It does not catch paraphrases; semantic caching does, at the risk of serving a wrong answer for a similar-looking question. Include the tenant and user scope in what you cache, or you leak answers across tenants.
• Telemetry:
  • Spans are named chat {model} and carry gen_ai.* tags: operation, provider, request and response model, finish reasons and token usage.
  • Prompt and response text is only recorded when EnableSensitiveData is on. Keep it off in production unless you have a data-handling decision for it.
  • Export with the normal OpenTelemetry SDK (OTLP, Azure Monitor).

Code example: a four-layer pipeline

// A middleware pipeline around IChatClient: custom budget guard, response cache, OpenTelemetry and function
// invocation are all decorators, composed with ChatClientBuilder. The inner client is the FAKE scripted model.
using System.Diagnostics;
using FakeAI;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;

var fake = new FakeChatClient((m, o) => new ChatMessage(ChatRole.Assistant, $"Answer to: {m[^1].Text}"));
var cache = new MemoryDistributedCache(Options.Create(new MemoryDistributedCacheOptions()));

// Capture the spans the OpenTelemetry middleware emits (a real app exports them with the OTel SDK).
var spans = new List<Activity>();
using var listener = new ActivityListener
{
    ShouldListenTo = s => s.Name == "demo-ai",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = spans.Add,
};
ActivitySource.AddActivityListener(listener);

var services = new ServiceCollection();
services.AddChatClient(fake)                               // the order of Use* calls = outermost first
    .Use(inner => new BudgetGuard(inner, maxInputTokens: 60))
    .UseDistributedCache(cache)
    .UseOpenTelemetry(sourceName: "demo-ai", configure: c => c.EnableSensitiveData = false);
using var sp = services.BuildServiceProvider();
var chat = sp.GetRequiredService<IChatClient>();

foreach (var q in new[] { "What is HybridCache?", "What is HybridCache?", "What is output caching?" })
{
    var r = await chat.GetResponseAsync(q);
    Console.WriteLine($"{q,-26} -> {r.Text,-36} model calls so far: {fake.Calls}");
}
try { await chat.GetResponseAsync(string.Join(' ', Enumerable.Repeat("very long question", 30))); }
catch (InvalidOperationException e) { Console.WriteLine($"budget guard: {e.Message}; model calls so far: {fake.Calls}"); }

Console.WriteLine($"spans recorded: {spans.Count}");
foreach (var a in spans.Take(1))
{
    Console.WriteLine($"span name: {a.DisplayName}");
    foreach (var tag in a.TagObjects.OrderBy(t => t.Key)) Console.WriteLine($"  {tag.Key} = {tag.Value}");
}
Console.WriteLine("pipeline (outer -> inner): " + string.Join(" -> ", Chain(chat)));

static IEnumerable<string> Chain(IChatClient c)
{
    for (IChatClient? x = c; x is not null; x = (x as DelegatingChatClient)?.GetType().GetProperty("InnerClient", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)?.GetValue(x) as IChatClient)
        yield return x.GetType().Name;
}

/// <summary>Non-streaming demo: reject over-budget text estimates before calling the next client; not a hard billing limit.</summary>
sealed class BudgetGuard(IChatClient inner, int maxInputTokens) : DelegatingChatClient(inner)
{
    public override Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken ct = default)
    {
        var snapshot = messages.ToList();
        int estimate = Tokens.Estimate(snapshot);
        if (estimate > maxInputTokens) throw new InvalidOperationException($"estimated {estimate} input tokens > budget {maxInputTokens}");
        return base.GetResponseAsync(snapshot, options, ct);
    }
}

Output (.NET 10, Microsoft.Extensions.AI 10.10.0, FAKE model)

What is HybridCache?       -> Answer to: What is HybridCache?      model calls so far: 1
What is HybridCache?       -> Answer to: What is HybridCache?      model calls so far: 1
What is output caching?    -> Answer to: What is output caching?   model calls so far: 2
budget guard: estimated 147 input tokens > budget 60; model calls so far: 2
spans recorded: 2
span name: chat fake-model-1
  gen_ai.operation.name = chat
  gen_ai.provider.name = fake
  gen_ai.request.model = fake-model-1
  gen_ai.response.finish_reasons = ["stop"]
  gen_ai.response.model = fake-model-1
  gen_ai.usage.input_tokens = 9
  gen_ai.usage.output_tokens = 12
pipeline (outer -> inner): BudgetGuard -> DistributedCachingChatClient -> OpenTelemetryChatClient -> FakeChatClient

Code walkthrough

• The repeated question "What is HybridCache?" was served from the distributed cache: model calls stayed at 1. The new question raised them to 2.
• The long question was rejected by BudgetGuard ("estimated 147 input tokens > budget 60") before any model call, so model calls stayed at 2. The guard is outermost, so the rejection costs nothing.
• Only 2 spans were recorded for 3 successful requests. The cache sits outside the OpenTelemetry layer, so the cache hit never reached it. To see cache hits in traces, put telemetry outside the cache (or emit your own metric).
• The span chat fake-model-1 carried gen_ai.operation.name, gen_ai.provider.name, the request and response model, finish_reasons ["stop"], and usage tags input_tokens = 9 and output_tokens = 12 (fake estimates).
• The printed chain BudgetGuard -> DistributedCachingChatClient -> OpenTelemetryChatClient -> FakeChatClient confirms that the first Use is outermost.

Failure modes

• A cache key without tenant or user scope: one customer's answer is served to another. Add scope to the messages/options, or partition the cache.
• Telemetry with EnableSensitiveData = true in production: prompts containing personal data end up in your APM.
• Budget checks after the cache or after retries, so they count the wrong thing.
• Caching responses that depend on time or on tool results (prices, order status).
• Forgetting to override GetStreamingResponseAsync in custom middleware, so streaming calls skip your guard.

Interview angle

"How do you control cost and observe LLM calls across 20 endpoints?" Answer: "One IChatClient pipeline registered in DI. An outer budget and tenant guard, exact-match caching where prompts repeat, OpenTelemetry with GenAI conventions for latency, tokens and finish reasons, and sensitive data off. The order is deliberate: guard first, then cache, then telemetry, then the provider. That way traces show real model calls, and a separate cache-hit metric shows savings."

Practice questions

1. In builder.Use(A).Use(B), which runs first on a request?

Answer: A. The first Use is the outermost layer.

2. Why did only 2 spans appear for 3 requests?

Answer: The cache layer was outside OpenTelemetry and returned the cached response early, so the inner telemetry layer never ran for the hit.

3. What does EnableSensitiveData control?

Answer: Whether message content (prompts, responses, tool arguments) is recorded on spans. Off by default.

4. Name a risk of response caching in a multi-tenant app.

Answer: Cross-tenant leakage when the cache key does not include tenant or user scope.

Revision checklist

• ChatClientBuilder / AddChatClient(...).Use...: the first Use is outermost.
• Built-ins: function invocation, distributed cache, OpenTelemetry, logging, options.
• Custom: DelegatingChatClient; override both streaming and non-streaming.
• Order: guard, then cache, then telemetry, then provider (decide deliberately).
• Sensitive data off in telemetry.

Related on TechInterviewNotes

• Lesson: Output caching, HybridCache and rate limiting (same decorator idea for HTTP)
• Lesson: Cache keys, tenant boundaries and cache-aside reads
• Course: Prompt Engineering for .NET Developers

References

• Microsoft.Extensions.AI: build a pipeline
• OpenTelemetry semantic conventions for generative AI