LLM Integration Interview Questions for .NET Developers

Q1 What problem does Microsoft.Extensions.AI solve, and what do IChatClient and IEmbeddingGenerator actually abstract?#

Short answer: Microsoft.Extensions.AI is a thin, provider-neutral set of abstractions — chiefly IChatClient for chat and completion-style calls and IEmbeddingGenerator<TInput, TEmbedding> for embeddings, shipped in the Microsoft.Extensions.AI and Microsoft.Extensions.AI.Abstractions NuGet packages — so application code, middleware and libraries can depend on one interface instead of a specific vendor SDK.

Before this library existed, every provider (an Azure OpenAI SDK, an OpenAI SDK, a local Ollama client) had its own request and response types, its own streaming model, and its own way of expressing tools. Any code that wanted to stay provider-agnostic had to hand-roll an abstraction, and every team did it slightly differently. Microsoft.Extensions.AI standardizes that shape: ChatMessage, ChatResponse, ChatOptions and ChatResponseUpdate for chat, and Embedding<T>/GeneratedEmbeddings<TEmbedding> for embeddings, with concrete provider implementations plugging into the same interfaces. Because it is an interface rather than a concrete client, it composes with the standard .NET patterns teams already use for HttpClient and DbContext: register it once with dependency injection, decorate it with cross-cutting behavior, and swap the underlying provider without touching call sites.

The detail candidates often miss is that IChatClient is deliberately built for decoration. ChatClientBuilder lets you wrap a base client with layers for function invocation, OpenTelemetry and logging, each layer itself an IChatClient, so the call site never needs to know how many layers of middleware sit underneath it — the same delegating-handler idea HttpClient uses, applied to model calls.

var chatClient = new ChatClientBuilder(baseClient)
    .UseFunctionInvocation()
    .UseOpenTelemetry(sourceName: "MyApp.Chat")
    .Build();

builder.Services.AddChatClient(chatClient);

What interviewers look for: that you understand this as an abstraction over a capability — chat, embeddings — not a single vendor, and that you can name the decorator/builder pattern without being prompted; it signals hands-on use rather than a skimmed blog post.

Common mistakes: describing Microsoft.Extensions.AI as "a wrapper around one specific SDK," and not knowing that IChatClient and IEmbeddingGenerator are separate interfaces because a provider can support one without the other.

Q2 Walk through GetResponseAsync versus GetStreamingResponseAsync. When do you choose streaming, and what does it cost you in complexity?#

Short answer: GetResponseAsync returns one materialized ChatResponse once generation finishes; GetStreamingResponseAsync returns IAsyncEnumerable<ChatResponseUpdate> and yields incremental fragments as the model produces them — you choose streaming whenever perceived latency matters more than having a single, easy-to-validate final object.

For a chat UI, a non-streamed response feels slow even when total generation time is identical, because the user stares at nothing until the whole answer lands; streaming turns that into a typing effect that starts within a few hundred milliseconds. The trade-off is that you now own reassembly and partial-state handling: update fragments have to be concatenated to reconstruct the final message, tool-call arguments can arrive split across several updates, and anything downstream that needs the whole answer — logging, structured-output parsing, moderation — either waits for the stream to finish or works incrementally, which most JSON parsers cannot do safely. In an ASP.NET Core minimal API, streaming to the browser usually means proxying updates over server-sent events or a chunked response, and the request's CancellationToken must be threaded through to the underlying call so a client disconnect actually cancels upstream generation instead of silently burning tokens for an answer nobody reads.

app.MapPost("/chat/stream", (ChatRequest req, IChatClient chatClient, CancellationToken ct) =>
    Results.Stream(async stream =>
    {
        await foreach (var update in chatClient.GetStreamingResponseAsync(req.Messages, ct: ct))
        {
            var bytes = Encoding.UTF8.GetBytes(update.Text ?? string.Empty);
            await stream.WriteAsync(bytes, ct);
            await stream.FlushAsync(ct);
        }
    }, "text/event-stream"));

What interviewers look for: awareness that streaming is a UX and cancellation-propagation decision, not a free performance win, and that reconstructing structured data from a stream is materially harder than from a single response.

Follow-up questions:

• How would you show partial function-call arguments to a user without exposing malformed JSON?
• What happens to token billing if a client disconnects mid-stream and cancellation isn't propagated?

Q3 Walk through how automatic function calling works end to end, and what can go wrong when you let a model call your code?#

Short answer: You expose .NET methods as AIFunction instances, typically via AIFunctionFactory.Create, attach them to ChatOptions.Tools, wrap the client with .UseFunctionInvocation(), and the middleware runs the loop of sending the tool schema, receiving a tool-call request, invoking your method, and feeding the result back to the model automatically.

The model does not execute code — it returns a structured message saying, in effect, "call function X with arguments Y." The function-invocation middleware intercepts that, deserializes the arguments against the function's parameter schema (derived from the method signature and [Description] attributes), invokes the actual delegate, serializes the return value, and sends it back as a tool-result message, repeating until the model returns a normal answer instead of another tool call. That hands a surprising amount of control to the model, which decides which tools to call, with what arguments, and how many times. The failure modes that matter at the senior level are an unbounded call loop — the model keeps calling a tool because it keeps getting an error it doesn't understand, so you need a maximum-iteration guard — argument hallucination, where the model invents a plausible-looking identifier instead of asking for clarification, and treating tool descriptions as trusted content when they may originate from user-controlled data, which is an indirect prompt-injection vector. A tool should never carry more authority than the calling user actually has; it must re-check authorization inside the method body rather than assume the model will behave.

[Description("Looks up the shipping status for an order the caller owns.")]
static async Task<string> GetOrderStatusAsync(
    [Description("The order ID")] string orderId, IOrderService orders, ClaimsPrincipal user)
{
    var order = await orders.GetForUserAsync(orderId, user.GetUserId());
    return order is null ? "Order not found or not accessible." : order.Status;
}

var options = new ChatOptions { Tools = [AIFunctionFactory.Create(GetOrderStatusAsync)] };

What interviewers look for: treating the model as an untrusted caller of your API surface rather than a trusted orchestrator — strong answers mention authorization, input validation and loop bounds unprompted.

Common mistakes: giving one tool too much power, such as a generic "run a query" tool instead of narrow, purpose-built ones, and not handling a tool call with missing or malformed arguments.

Q4 How do you get reliable structured output from an LLM, and how is that different from asking the model to "return JSON" in the prompt?#

Short answer: Microsoft.Extensions.AI exposes a generic GetResponseAsync<T> extension that sends your type's JSON schema to the model as part of the request and parses the result back into T, which is materially more reliable than a prompt instruction, because the schema constrains or strongly guides generation instead of relying on the model choosing to comply.

Asking nicely in the prompt works most of the time with a capable model, but "most of the time" is not a contract — eventually you get a response wrapped in prose, missing a required field, or with a string where a number was expected, and that failure surfaces as an unhandled exception in production, usually under load when you can least afford it. The structured-output extension instead attaches a JSON schema derived from T to the request, using native structured-output support where a provider offers it or schema-guided prompting where it does not, then deserializes the result into a strongly typed object instead of a string you parse yourself.

public record TicketAnalysis(string Summary, string Sentiment, bool NeedsFollowUp);

var response = await chatClient.GetResponseAsync<TicketAnalysis>(
    $"Analyze this support ticket:\n{ticketText}");

TicketAnalysis result = response.Result;

Even with schema-guided generation, you still need application-level validation for business rules a schema cannot express — a Sentiment value outside the set you expect, or a NeedsFollowUp flag that contradicts the summary — and you should keep result types small and flat, since deeply nested or highly polymorphic shapes are where schema-guided generation degrades the most.

What interviewers look for: the distinction between "the model was told to produce JSON" and "the request constrained the model's output," plus awareness that structured output reduces, but does not eliminate, the need for validation.

Common mistakes: trusting a typed result without validating business invariants, and using large, deeply nested response types where a flatter shape would parse far more reliably.

Q5 You need to support multiple model providers without a rewrite. What does IChatClient actually buy you, and where does the abstraction leak?#

Short answer: IChatClient buys you a single call-site shape and one dependency-injection registration across providers, which makes swapping a default provider mechanically cheap; it does not buy you identical behavior, identical feature sets or identical failure modes, so a serious multi-provider design needs a capability model layered on top of the interface, not just the interface itself.

The common surface — messages in, a response or stream of updates out, optional tools, optional structured output — covers most of what a typical chat feature needs, and for that slice IChatClient genuinely lets you swap providers behind a feature flag with no call-site changes. Where it leaks: context windows differ by an order of magnitude between models, so a prompt sized for a large-window model can silently truncate on a smaller one; not every provider supports parallel tool calls, vision input or the same structured-output guarantees, which is why ChatOptions has provider-specific extension points for exactly this reason; and error semantics differ — rate-limit responses, content-filter refusals and timeout behavior are not uniform, so resilience code tuned against one provider's quirks can quietly stop firing against another's. A production multi-provider architecture typically layers a capability descriptor over IChatClient and routes or degrades gracefully — a smaller context, disabled tool use, a different model — rather than assuming the interface alone makes providers interchangeable.

What interviewers look for: treating the interface as necessary but not sufficient; real fluency sounds like "here's what still breaks when you swap providers," not "just inject a different IChatClient."

Follow-up questions:

• How would you design a fallback chain across two providers for availability?
• How do you verify a prompt still performs acceptably after swapping the underlying model?

Q6 How do you choose which model to use for a given feature, and how do you defend that choice in an architecture review?#

Short answer: Treat it as a constrained optimization over quality, latency, cost and context window for the specific task, not a single "best model" decision: benchmark a small set of candidates against a task-representative evaluation set you control, and pick the cheapest, fastest model that clears your quality bar rather than the most capable model available.

Teams new to this default to the flagship model for everything, which is expensive and often unnecessary — a ticket-classification task or a short summarization step frequently performs just as well on a smaller, faster model, while a multi-step reasoning or code-generation task genuinely needs a stronger one. The right process defines the task narrowly, builds a small labeled evaluation set — even a few dozen realistic examples beats intuition — runs the candidate models against it with the same prompt, and compares on task-specific metrics alongside p50/p95 latency and per-request cost. Context window matters independently of raw capability: a model with a huge window but weaker instruction-following can perform worse on a retrieval-augmented task than a smaller-window model that handles the same context more faithfully. Because model rankings shift with nearly every release, the evaluation set — not a specific model name — is the durable asset; keeping it lets you re-run the comparison later and upgrade with evidence instead of intuition.

What interviewers look for: a repeatable, evidence-based selection process rather than a static opinion about which model is best; this is a fast-moving space, and interviewers are testing your process, not your current leaderboard knowledge.

Common mistakes: picking a model once at kickoff and never re-evaluating, and comparing models only on quality while ignoring that a slower or pricier model can be the wrong choice even when it scores marginally higher.

Q7 How do you build resilient LLM calls that survive rate limits and transient failures without cascading into an outage?#

Short answer: Layer retries with exponential backoff and jitter for transient errors, a circuit breaker to stop hammering a provider that is already failing, a timeout tuned to the feature's latency budget, and a concurrency or rate limiter on the outbound side so your own traffic doesn't trigger 429s in the first place — in .NET this is a natural fit for Microsoft.Extensions.Http.Resilience, which packages Polly-based strategies as a standard resilience pipeline.

A naive "retry on any exception" policy is actually risky with LLM calls, because retrying a request that already generated most of a long response wastes money and the user's remaining patience; retries should target genuinely transient conditions — 429s, 5xxs, timeouts — and respect a Retry-After header when a provider sends one instead of guessing a backoff. A circuit breaker matters more here than in a typical CRUD backend, because LLM providers have real, sustained incident windows, and continuing to retry into a broken endpoint just burns your own request budget; opening the circuit and failing fast, with a cached or degraded response, protects both sides. Rate-limiting your own outbound calls with a token-bucket limiter sized under the provider's documented limit is what actually prevents most 429s rather than only reacting to them, which matters most in multi-tenant systems where one noisy tenant can exhaust a shared quota.

services.AddHttpClient("llm").AddResilienceHandler("llm-pipeline", builder =>
{
    builder.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
    });
    builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions());
    builder.AddTimeout(TimeSpan.FromSeconds(30));
});

What interviewers look for: distinguishing retryable from non-retryable failures, and naming the outbound rate limiter as a prevention mechanism rather than talking only about reactive retry and backoff.

Common mistakes: blindly retrying non-idempotent or already-partially-billed calls, and using one global timeout that's wrong for both a fast classification call and a slow multi-step generation in the same codebase.

Q8 How do you unit test and integration test code that calls an LLM, given that model output is non-deterministic?#

Short answer: Test your code, not the model: because IChatClient and IEmbeddingGenerator are interfaces, unit tests substitute a fake or scripted implementation that returns fixed responses so you can assert on orchestration logic deterministically, while a smaller, separately budgeted suite of evaluation-style tests checks actual model quality against a labeled dataset with tolerance for variation.

The mistake most teams make early on is testing against the real model, which produces flaky, slow, expensive tests that assert on exact output — any change to the provider's model breaks them for reasons unrelated to a bug. Split the problem: deterministic tests use a fake IChatClient to verify that your code builds the right prompt, passes the right tools, handles a tool-call response correctly, and surfaces provider errors the way you intend, which is ordinary interface mocking with nothing LLM-specific about it. Quality and regression tests are a different discipline: run a small, versioned golden dataset through the real pipeline on a schedule rather than on every commit, given cost and latency, score outputs with an automatic metric or an LLM-as-judge rubric, and fail the build only on a meaningful regression against a baseline, not on every minor wording change. Integration tests that hit a real provider should be isolated, rate-limited, and tagged separately so they don't block a normal pull-request pipeline.

public sealed class FakeChatClient(ChatResponse canned) : IChatClient
{
    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        CancellationToken cancellationToken = default) => Task.FromResult(canned);

    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        CancellationToken cancellationToken = default) =>
        throw new NotSupportedException();

    public object? GetService(Type serviceType, object? serviceKey = null) => null;
    public void Dispose() { }
}

What interviewers look for: a clear split between deterministic orchestration tests and probabilistic quality evaluation, and treating a mocked IChatClient as exactly as normal as mocking any other dependency.

Common mistakes: asserting on exact model output text in a CI test, and never testing failure paths such as malformed tool arguments or a schema-violating structured response.

Q9 A feature's prompt — system instructions, conversation history and retrieved context — keeps exceeding the model's context window. How do you design around that?#

Short answer: Budget the context window explicitly across its consumers — system prompt, history, retrieved content, and room for the response — measure actual token counts rather than guessing from character length, and apply a deliberate trimming strategy before you hit the limit, rather than letting the provider reject or truncate the request for you.

The naive approach, appending messages until a call fails, produces a confusing production bug: the feature works fine in testing with short conversations and breaks for the most engaged users with long ones, exactly the wrong failure curve. A better design treats the window as a fixed budget split into a small, reserved slice for system instructions and tool schemas, a slice for conversation history, a slice for retrieved content, and a reserved slice for the response itself, sized from the model's known maximum output. For history, the two common strategies are a sliding window that keeps the last several turns verbatim and drops older ones, and rolling summarization that periodically collapses older turns into a short summary message preserving the gist at a fraction of the tokens; summarization keeps more continuity but costs an extra model call and can lose specific details a user later references. Token counting should use the real tokenizer for the model in use rather than a character heuristic, since the ratio varies across languages and content types, and the budget should leave real headroom rather than target the limit exactly.

What interviewers look for: an explicit budgeting mental model with reserved slices, not just "trim when it breaks," and awareness that a truncation strategy is a product decision about what the user loses as much as an engineering one.

Follow-up questions:

• How would you decide between summarizing old turns and simply dropping them?
• What happens to a tool-call and tool-result pair if you truncate only one half of it out of history?

Q10 You need multi-turn conversation memory in a stateless ASP.NET Core API that scales across many replicas. Where does chat history actually live?#

Short answer: It lives outside the process, in an external store keyed by a conversation or session ID — a database, a distributed cache, or a dedicated conversation store — never in in-memory server state, because a stateless, horizontally scaled API can route the next request from the same user to any replica.

The instinct to keep a list of messages in a singleton or a static dictionary works in a demo and fails the moment more than one instance is running, because the next request can land on a replica that has never seen that conversation. The standard fix is to treat conversation history like any other piece of user state in a stateless web tier: persist it, keyed by conversation ID, load the relevant slice at the start of a request, append the new turns, and write it back — the same read-modify-write pattern used for any stateful resource accessed from a stateless service. This is also where context-window budgeting and history storage intersect: most designs store the full transcript for audit and analytics but only load and send a bounded, budgeted window to the model on each turn. Concurrency matters too — if a user can have two tabs open, or a client retries a request, the design needs either optimistic concurrency on the stored conversation or an idempotency key on appended turns, or history ends up duplicated or interleaved under load.

What interviewers look for: immediately reaching for "state doesn't live on the server" the way any senior engineer would for session state generally, plus the added nuance of budgeting what actually gets sent to the model versus what gets persisted.

Common mistakes: conflating "the API is stateless" with "there is no conversation state anywhere," and forgetting concurrent-write handling when a client can issue overlapping requests for the same conversation.