Observability and Cost Control for LLM Apps in .NET

Q: Does response caching work with streaming?

UseDistributedCache caches a complete ChatResponse , so a cache hit is returned as a complete response rather than replayed as a stream. If your UI always expects a stream, wrap the cached text in a single synthetic update rather than skipping the streaming code path.

A feature built on a language model can pass every test and still fail in production for reasons ordinary APM was never built to catch: a silent model upgrade changes output quality, one tenant's prompts quietly triple the monthly bill, or a burst of traffic trips a provider's rate limit and every request after it starts failing. Observability and cost control for LLM applications in .NET means instrumenting every model call with the OpenTelemetry GenAI semantic conventions, tracking token usage and spend by feature and tenant, caching what is safe to cache, routing work to the right model, and handling throttling before it becomes an outage. This guide covers all of that for applications built on Microsoft.Extensions.AI, with an eye on what is genuinely production-ready today versus what is still evolving.

What Observability and Cost Control Mean for LLM Apps#

Three properties make LLM calls different from a typical downstream HTTP dependency. First, they are non-deterministic: the same prompt can produce different output on different calls, so quality has to be watched continuously, not verified once. Second, they are billed by token count, which varies with prompt length, retrieved context, conversation history and the model chosen, so cost is a function of how you prompt, not just how often you call. Third, providers throttle aggressively under load and return 429 Too Many Requests, which a naive retry loop can turn into a self-inflicted outage. An observability strategy for LLM apps has to cover all three: is the model still answering well, what is this feature costing, and is the traffic pattern about to get throttled.

How LLM Observability Works: The OpenTelemetry GenAI Conventions#

OpenTelemetry defines a dedicated set of semantic conventions for generative AI, published from the open-telemetry/semantic-conventions-genai repository. They standardize the span, metric and attribute names that every instrumented client, gateway and dashboard can agree on, so a trace produced by a .NET service and one produced by a Python service look the same in your backend. As of September 2026 these conventions are still marked Development status, not Stable, which means attribute names can still change between releases; pin the OpenTelemetry package versions you build on and re-check the spec before a major upgrade.

A GenAI inference span represents one call to a model and carries attributes such as these:

Attribute	Meaning
`gen_ai.operation.name`	The kind of operation, for example `chat` or `embeddings`
`gen_ai.provider.name`	The provider, for example `openai` or `azure.ai.openai`
`gen_ai.request.model`	The model or deployment requested
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`	Token counts for the request and the completion
`gen_ai.usage.reasoning.output_tokens`	Output tokens spent on internal reasoning, where a provider reports it
`gen_ai.usage.cache_read.input_tokens`	Input tokens served from a provider-managed prompt cache
`gen_ai.response.finish_reasons`	Why generation stopped, for example `stop` or `length`
`gen_ai.conversation.id`	A correlation id for a multi-turn conversation, when one is available
`gen_ai.prompt.name` / `gen_ai.prompt.version`	The name and version of a named prompt template, when you supply one

Notably, the attributes that carry actual prompt and completion text, gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions, are defined at Opt-In requirement level. They are not captured by default anywhere in the spec, precisely because prompts and completions routinely contain personal or sensitive data. Keep that opt-in default in mind for the logging section later in this guide.

Getting Started: UseOpenTelemetry on IChatClient#

Microsoft.Extensions.AI implements these conventions directly. Adding UseOpenTelemetry to a ChatClientBuilder wraps the client in a middleware that emits a span per model call, with the attributes above populated automatically:

using Microsoft.Extensions.AI;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

const string SourceName = "Contoso.Support.Chat";

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource(SourceName).AddOtlpExporter())
    .WithMetrics(m => m.AddMeter(SourceName).AddOtlpExporter());

builder.Services.AddChatClient(sp => sp.GetRequiredKeyedService<IChatClient>("inner"))
    .UseOpenTelemetry(sourceName: SourceName, configure: o =>
    {
        // Off by default: only enable after you have redaction and retention in place.
        bool captureContent = builder.Configuration.GetValue<bool>("Telemetry:CaptureContent");
        o.EnableSensitiveData = captureContent;
    })
    .UseDistributedCache()
    .UseFunctionInvocation();

EnableSensitiveData is the switch for the opt-in content attributes described above: leave it false in production until you have decided how prompt content will be redacted, retained and access-controlled. With it on, every Activity for a call also gets the full message and response content, which is invaluable in a staging environment and a liability if flipped on carelessly in production.

Viewing GenAI Traces in the .NET Aspire Dashboard#

The .NET Aspire dashboard (13.x as of September 2026) recognizes spans that carry gen_ai.* attributes in its trace views. A span that matches gets a dedicated GenAI action that opens a visualizer dialog rendering the conversation, the tool calls and the token usage as structured data instead of a flat attribute list, which is far faster to scan while developing a prompt locally. Point the dashboard's OTLP endpoint at your app while developing, as you would for any OpenTelemetry source, and the GenAI-aware view appears automatically for qualifying spans; no extra instrumentation is required beyond UseOpenTelemetry. The .NET Aspire guide covers wiring the dashboard into a full distributed application. The dashboard is a development-time tool: production traffic still needs a real backend, such as Azure Monitor or any OTLP-compatible vendor, for retention, alerting and cross-service correlation.

Tracking Token Usage and Cost per Feature and Tenant#

ChatResponse.Usage gives you token counts for a single call (InputTokenCount, OutputTokenCount, TotalTokenCount), and the OpenTelemetry attributes give you the same counts per span. Neither one knows which feature or which tenant made the call, because that context is specific to your application, not the model. Attach it yourself, consistently, at the point where you already know it:

using System.Diagnostics;
using Microsoft.Extensions.AI;

public sealed class CostTaggingChatClient(IChatClient innerClient, IMeterFactory meterFactory)
    : DelegatingChatClient(innerClient)
{
    private readonly Counter<double> _cost =
        meterFactory.Create("Contoso.Support.Chat").CreateCounter<double>("genai.cost.usd");

    public override async Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages,
        ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        ChatResponse response = await base.GetResponseAsync(messages, options, cancellationToken);

        string feature = Activity.Current?.GetTagItem("app.feature") as string ?? "unknown";
        string tenant = Activity.Current?.GetTagItem("app.tenant_id") as string ?? "unknown";
        double cost = EstimateCostUsd(response.ModelId, response.Usage);

        Activity.Current?.SetTag("app.cost_usd", cost);
        _cost.Add(cost,
            new KeyValuePair<string, object?>("app.feature", feature),
            new KeyValuePair<string, object?>("app.tenant_id", tenant),
            new KeyValuePair<string, object?>("gen_ai.request.model", response.ModelId));

        return response;
    }

    private static double EstimateCostUsd(string? modelId, UsageDetails? usage) =>
        usage is null || modelId is null ? 0
            : ModelPricing.InputPerMillion(modelId) * (usage.InputTokenCount ?? 0) / 1_000_000d
            + ModelPricing.OutputPerMillion(modelId) * (usage.OutputTokenCount ?? 0) / 1_000_000d;
}

Set app.feature and app.tenant_id as tags on the ambient Activity at the start of the request, for example in middleware, so every downstream client call, including tool calls that trigger their own model calls, inherits them. Emitting genai.cost.usd as a metric, not only as a span attribute, means your dashboard can aggregate spend by feature or tenant over time without re-scanning trace data. Keep ModelPricing as a small, explicitly maintained table you update when a provider changes prices; do not try to derive prices from the API at request time.

Response Caching and Prompt Caching#

Two different kinds of caching apply to LLM calls, and they solve different problems.

Response caching in your own application avoids paying for an identical call twice. UseDistributedCache on a ChatClientBuilder caches the full response, keyed on the messages and options, in any IDistributedCache:

builder.Services.AddStackExchangeRedisCache(o =>
    o.Configuration = builder.Configuration.GetConnectionString("redis"));

builder.Services.AddChatClient(innerClient)
    .UseDistributedCache()   // exact-match cache; skips the model entirely on a hit
    .UseOpenTelemetry(sourceName: SourceName);

This only helps for calls that are genuinely repeated with identical input, such as a fixed classification prompt applied to the same short strings, or a cached FAQ-style answer. It does nothing for open-ended chat, where the input is different every time.

Prompt caching is a provider-side feature that reuses previously processed prefix tokens, such as a long, unchanging system prompt or a large retrieved document, across requests that share that prefix, billed at a reduced rate for the cached portion. You do not implement it; you structure prompts to take advantage of it, by keeping the stable, shared part of a prompt first and the variable, per-request part last, and you observe it through gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens on the span. Support and exact thresholds vary by provider and model, so verify the current behavior for the specific model you deploy rather than assuming it applies uniformly.

A third option, semantic caching that matches on embedding similarity instead of an exact key, can catch near-duplicate questions that UseDistributedCache would miss. Treat it as an advanced, higher-risk technique: a similarity threshold that is too loose returns a plausible-sounding but wrong cached answer, silently. Only apply it to low-stakes, easily-verified answers, and log cache hits distinctly from fresh model calls so you can audit them.

Model Routing: Match the Model to the Task#

Not every call needs your largest, most expensive model. Classification, short extraction and simple rewrites are usually handled just as correctly, and far more cheaply, by a small model, while multi-step reasoning and open-ended generation justify a larger one. Register both behind keyed chat clients and route explicitly:

builder.Services.AddKeyedChatClient("small", sp => smallModelClient)
    .UseOpenTelemetry(sourceName: SourceName);
builder.Services.AddKeyedChatClient("large", sp => largeModelClient)
    .UseOpenTelemetry(sourceName: SourceName);

app.MapPost("/tickets/classify", async (
    TicketRequest request,
    [FromKeyedServices("small")] IChatClient smallClient,
    CancellationToken ct) =>
{
    ChatResponse<TicketCategory> result = await smallClient.GetResponseAsync<TicketCategory>(
        request.Body, cancellationToken: ct);
    return Results.Ok(result.Result);
});

app.MapPost("/tickets/draft-reply", async (
    TicketRequest request,
    [FromKeyedServices("large")] IChatClient largeClient,
    CancellationToken ct) =>
    Results.Ok((await largeClient.GetResponseAsync(request.Body, cancellationToken: ct)).Text));

This static routing by endpoint is the simplest and most predictable option, and it is enough for most applications: you already know, at the point you write the code, which features need which tier. A dynamic router that classifies task difficulty at runtime and picks a model per request adds real complexity and its own failure modes, so reserve it for cases with high volume and a wide difficulty spread. If you are already on Azure AI Foundry, its model router deployment type does this dynamic selection for you; see Azure OpenAI and Azure AI Foundry for .NET Developers for how it fits into a Foundry deployment.

Retries, Rate Limits and 429s#

Every provider throttles. Azure OpenAI and OpenAI both return 429 with a retry-after hint when a deployment exceeds its quota, and most official client libraries already retry transient status codes a few times with backoff by default. That default is a safety net, not a strategy: under sustained load, naive retries just add more load to an already-throttled endpoint. Wrap the chat client in an explicit, observable resilience layer instead of relying only on library defaults:

using Microsoft.Extensions.AI;
using Polly;
using Polly.Retry;

public sealed class ResilientChatClient(IChatClient inner, ILogger<ResilientChatClient> logger)
    : DelegatingChatClient(inner)
{
    private readonly ResiliencePipeline _pipeline = new ResiliencePipelineBuilder()
        .AddRetry(new RetryStrategyOptions
        {
            ShouldHandle = new PredicateBuilder().Handle<ClientResultException>(
                ex => ex.Status is 429 or 500 or 502 or 503),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            MaxRetryAttempts = 3,
            OnRetry = args =>
            {
                logger.LogWarning("Chat call retry {Attempt} after {Status}",
                    args.AttemptNumber, (args.Outcome.Exception as ClientResultException)?.Status);
                return ValueTask.CompletedTask;
            },
        })
        .Build();

    public override Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages,
        ChatOptions? options = null, CancellationToken cancellationToken = default) =>
        _pipeline.ExecuteAsync(async ct => await base.GetResponseAsync(messages, options, ct),
            cancellationToken).AsTask();
}

Beyond retries, treat sustained 429s as a capacity problem, not a bug: spread traffic across deployments or regions, shed or queue low-priority requests first, and apply your own per-user or per-tenant rate limit ahead of the provider's so one noisy caller cannot exhaust the whole application's quota. The resilience with Polly guide covers building pipelines like this one in more depth, and the Azure OpenAI and Foundry guide covers Azure's specific quota model and deployment types for sustained scaling.

Budgets and Alerts#

Cost control needs both a slow feedback loop, dashboards you check periodically, and a fast one, a gate that stops a runaway loop before it drains a monthly budget. A minimal fast gate checks accumulated spend against a limit before a costly call:

public sealed class BudgetGate(IBudgetStore budgets)
{
    public async Task<bool> TryChargeAsync(
        string tenantId, decimal estimatedCostUsd, CancellationToken ct)
    {
        DateOnly today = DateOnly.FromDateTime(DateTime.UtcNow);
        decimal spentToday = await budgets.GetSpendAsync(tenantId, today, ct);
        decimal limit = await budgets.GetDailyLimitAsync(tenantId, ct);

        if (spentToday + estimatedCostUsd > limit)
        {
            // Caller should degrade gracefully: smaller model, cached answer, or a clear error.
            return false;
        }

        await budgets.RecordSpendAsync(tenantId, estimatedCostUsd, ct);
        return true;
    }
}

Pair the fast gate with alerts on the metric you already emit: an alert rule on genai.cost.usd, aggregated by tenant or feature over a rolling window, catches gradual drift, such as a prompt template growing over several deploys, that a hard per-call budget would miss. A tool loop that runs away, covered in the function calling guide, is one of the most common causes of a cost spike, so make sure MaximumIterationsPerRequest is bounded before you rely on budgets alone to catch it.

Logging Prompts and Responses Safely#

Prompts and completions are some of the most sensitive data your application handles: they can contain customer PII, business-confidential context pulled in by retrieval, and, if a prompt injection attempt succeeded, attacker-controlled text. Log them deliberately, not by default:

Leave EnableSensitiveData and any equivalent content-capture flag off until you have a specific, reviewed reason to turn it on, and scope it to a short-lived debug environment rather than production by default.
Redact or tokenize obvious PII, such as emails and account numbers, before content reaches a log sink; the responsible AI guide covers detection and redaction techniques for exactly this.
Truncate long content in logs, and store full transcripts, if you need them for support or audits, in a separate, access-controlled store with its own retention policy rather than in general-purpose application logs.
Log the prompt's name and version, from the prompt engineering guide, alongside the response, so a quality issue can be traced back to the exact template that produced it without needing the raw content every time.

Best Practices#

Instrument every IChatClient with UseOpenTelemetry, and give each logical feature its own Activity source or consistent tags so traces are easy to filter.
Emit cost as a metric, not just as a log line, tagged by feature and tenant, so it can be dashboarded and alerted on without reprocessing traces.
Cache what is safe to cache: exact-match responses for repeated inputs, provider prompt caching for stable prefixes, and semantic caching only with a documented, audited risk tolerance.
Route by task, not by habit. Default every call to your largest model and cost scales with usage regardless of whether the task needed it.
Treat 429 as a capacity signal. Retries buy you time; the real fix is quota, spread load or a rate limit of your own.
Keep prompt content out of logs by default, and make turning it on an explicit, reviewed decision with a redaction and retention plan attached.

Common Pitfalls#

Measuring cost only from a provider's monthly invoice. By the time it arrives, the feature or tenant that caused a spike is hard to reconstruct without your own per-call attribution.
Enabling EnableSensitiveData everywhere "just for debugging" and forgetting to turn it off. It is easy to leave sensitive content flowing into a general logging pipeline indefinitely.
Retrying 429s aggressively without backoff. This amplifies load on an already-throttled endpoint and can extend an outage instead of recovering from it.
Relying on UseDistributedCache for open-ended chat. Two different users rarely send identical messages, so the hit rate is close to zero outside repeated, templated calls.
Assuming GenAI semantic convention attribute names are final. They are still in Development status; a minor OpenTelemetry package upgrade can rename an attribute you query on in a dashboard.
Building a dynamic model router before you have the data to justify it. Static routing by endpoint is simpler, cheaper to reason about, and sufficient for most applications.

Where to Track Cost and Latency#

Approach	Granularity	Effort	Best for
Provider billing dashboard	Account or deployment-wide	None	A sanity check against your own numbers
`ChatResponse.Usage` in application logs	Per call, if you tag it yourself	Low	Small apps, early development
OpenTelemetry traces and metrics with custom tags	Per call, per feature, per tenant	Medium	Production apps that need dashboards and alerts
Dedicated LLM observability or gateway platform	Per call, with built-in cost and quality analytics	Medium to high (new dependency)	Multi-team or multi-product platforms standardizing on one tool

Frequently Asked Questions#

Does the Aspire dashboard replace a production observability backend?#

No. It renders GenAI-aware trace views for spans it receives over OTLP, which is excellent for local development and debugging a prompt or tool loop, but it is not meant to retain data, alert, or correlate across a fleet of production instances. Export the same traces to Azure Monitor or another OTLP-compatible backend for production.

Are the OpenTelemetry GenAI semantic conventions stable enough to build dashboards on?#

They are usable today and are what Microsoft.Extensions.AI emits, but as of September 2026 they remain in Development status, meaning attribute names can still change in a future release of the conventions. Isolate the attribute names behind your own dashboard queries and metric definitions where practical, so a rename is a one-place fix.

Which token count should I bill against, input or output?#

Bill against both, since most providers price them separately and output tokens are usually more expensive per token than input tokens. UsageDetails exposes InputTokenCount and OutputTokenCount separately for exactly this reason; do not collapse them into TotalTokenCount if you need accurate per-model cost.

Does response caching work with streaming?#

UseDistributedCache caches a complete ChatResponse, so a cache hit is returned as a complete response rather than replayed as a stream. If your UI always expects a stream, wrap the cached text in a single synthetic update rather than skipping the streaming code path.

How do I stop a burst of 429s from cascading into an outage?#

Combine bounded retries with backoff and jitter, a circuit breaker that stops calling a consistently failing deployment for a short cool-down period, and your own per-tenant rate limit set below the provider's quota. The resilience with Polly guide covers composing these strategies into one pipeline.

Summary#

Instrument model calls with UseOpenTelemetry, which emits the OpenTelemetry GenAI conventions; keep prompt and response content capture opt-in.
The .NET Aspire dashboard renders a dedicated view for GenAI spans during local development; production still needs a real telemetry backend.
Tag every call with your own feature and tenant dimensions and emit cost as a metric, since the model provider has no concept of either.
Combine exact-match response caching, provider-side prompt caching and, cautiously, semantic caching to cut repeat spend.
Route routine work to smaller models, retry 429s with backoff and jitter, and enforce your own rate limits and budgets ahead of the provider's.