Productionizing AI Applications Interview Questions

Q1 How do you build a continuous evaluation pipeline for an AI feature, both before and after deployment?#

Short answer: Maintain a versioned, labeled golden dataset representative of real usage, run it through the pipeline automatically on every meaningful change — prompt, model, retrieval logic — to catch regressions before deploy, and complement that offline signal with an online loop that samples live traffic, scores it, and tracks trend lines, because an offline dataset alone can't catch drift in traffic it was never built to represent.

The offline half is structurally similar to a regression test suite: a golden dataset of realistic inputs with either reference answers or a scoring rubric, run through the actual pipeline, not a stub, whenever something that could change behavior changes, with a clear pass/fail or score-delta gate against a baseline.

# illustrative CI gate: run the golden set through the real pipeline and fail on regression
dotnet run --project tools/Eval -- --dataset golden-v12.jsonl --baseline main --fail-below 0.90

Microsoft.Extensions.AI.Evaluation provides reusable evaluators for common dimensions such as relevance, groundedness and coherence, so teams don't hand-write a judge prompt for every metric from scratch, and it's designed to plug into a CI-style workflow rather than being a one-off notebook exercise. The online half exists because production traffic is never fully represented by any golden set: sampling a percentage of real requests, scoring them automatically and periodically with human review for calibration, and tracking the score trend over time catches the slow degradation a static suite structurally cannot — a subtle shift in query patterns, a provider's model quietly changing behavior on the same version string, a knowledge base drifting out of date. The two loops feed each other: a production failure spotted through online sampling gets added to the golden dataset so the same regression is caught offline next time, which turns evaluation into a system that improves over time instead of a static gate.

What interviewers look for: the offline-versus-online split stated explicitly, with a concrete mechanism for each, rather than "we test the prompts."

Common mistakes: treating evaluation as a one-time pre-launch activity instead of a running pipeline, and never feeding production failures back into the golden dataset.

Q2 How do you instrument an LLM application for observability, and what does OpenTelemetry's GenAI semantic conventions standardize?#

Short answer: Wrap the IChatClient with .UseOpenTelemetry() so every call emits a span and metrics automatically, and rely on the OpenTelemetry GenAI semantic conventions — standardized attribute names such as gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — so dashboards, alerts and cross-service correlation work the same way regardless of which provider or team instrumented the call.

Before a shared convention existed, every team invented its own attribute names for which model, how many tokens, and how long generation took, which made it impossible to build one dashboard across services owned by different teams, let alone correlate an LLM call with the rest of a distributed trace. The GenAI semantic conventions fix that the same way HTTP conventions standardized attributes like method and status code years earlier: gen_ai.system identifies the provider, request and response model attributes reveal when a provider silently serves a different model version than requested, and token-usage attributes flow into cost dashboards without custom parsing per provider. Wiring this up in Microsoft.Extensions.AI is a single decorator, instrumenting every call the same way an ASP.NET Core request already gets instrumented, so an LLM call shows up as a span nested correctly inside the rest of a request's trace instead of an opaque gap. The conventions also cover agent- and tool-related spans and events, not just raw chat calls, which matters once a feature involves multi-step tool use — you want the whole chain of retrieval, tool calls and generation as one connected trace, not disconnected log lines stitched together by timestamp after the fact.

var chatClient = new ChatClientBuilder(baseClient)
    .UseOpenTelemetry(sourceName: "MyApp.Chat", configure: o => o.EnableSensitiveData = false)
    .Build();

What interviewers look for: naming specific gen_ai.* attributes and explaining why a shared convention matters — cross-team dashboards, trace correlation — not just "we added logging."

Common mistakes: inventing ad hoc attribute names instead of the standard gen_ai.* set, and enabling verbose prompt and response capture in telemetry without considering it may contain sensitive user data.

Q3 How do you control and forecast LLM cost at the architecture level, not just by picking a cheaper model?#

Short answer: Meter token usage per request, per tenant and per feature so cost is attributable rather than a single opaque provider bill; route requests to the cheapest model that clears the quality bar for that specific task instead of using one model everywhere; and cut the volume of tokens sent and generated per request through trimmed context, capped output length and caching, since token volume, not model choice alone, is usually the biggest lever.

Cost visibility has to exist before cost control does — without per-request, per-tenant attribution using the gen_ai.usage.* telemetry as the raw signal, a rising bill is just a number with no actionable next step, and a single noisy tenant or a runaway agent loop can inflate the whole bill invisibly. Model routing, sending simple, high-volume tasks to a small, cheap model and reserving a stronger model for genuinely complex requests, is the highest-leverage single decision, because the price gap between model tiers is typically much larger than any other optimization, but it only works safely if the evaluation pipeline actually validates the cheaper model on that task class rather than assuming it's good enough. Beyond model choice, volume-side levers compound: trimming unnecessary context, capping maximum output tokens for tasks that don't need long-form answers, and caching identical or near-identical requests — exact-match caching is simple, semantic caching catches more but risks serving a stale or subtly wrong cached answer for a "similar but not identical" query, so it needs a conservative similarity threshold. Budgets and alerts, a per-tenant or per-feature spend ceiling with an alert well before it's reached, are the operational backstop that catches failure modes no architectural control fully prevents, like a bug that causes an agent to loop.

What interviewers look for: naming attribution as a prerequisite to control, and treating token volume and model routing as separate, compounding levers rather than one generic "make it cheaper" answer.

Follow-up questions:

• How would you design per-tenant cost attribution in a shared multi-tenant AI feature?
• What's the risk profile of semantic caching versus exact-match caching for cost savings?

Q4 How do you reduce and manage latency in an LLM-backed feature?#

Short answer: Attack perceived and actual latency separately: stream responses so the user sees output within a few hundred milliseconds even if total generation takes seconds, choose the smallest model that meets the quality bar since model size is usually the biggest latency factor, parallelize independent steps instead of running them sequentially, and set a hard timeout with a graceful fallback rather than letting a slow call hang the whole request.

Streaming is the cheapest latency win available, because it changes perceived latency without changing total generation time at all — a five-second response that starts streaming at a few hundred milliseconds feels categorically different from one that shows nothing for five seconds. Actual latency is dominated by model size and output length far more than by network overhead in most architectures, so the same model-routing decision that controls cost also controls latency, and the two should be evaluated together rather than as separate trade-offs. Sequential dependency is the other common self-inflicted latency problem: a pipeline that retrieves context, then calls a tool, then generates, when the tool call and retrieval don't actually depend on each other, pays for artificial serialization — running independent steps concurrently is a straightforward win that's frequently missed because code gets written in the order someone thought of the steps rather than the order they actually depend on each other. Every external call also needs a timeout with an explicit fallback — a cached or degraded response, a clear retry message — because an LLM call with no timeout is a latent availability bug waiting for the one slow provider incident that hangs every in-flight request behind it.

var contextTask = retriever.GetContextAsync(query, ct);
var historyTask = conversationStore.LoadAsync(conversationId, ct);
await Task.WhenAll(contextTask, historyTask);

What interviewers look for: separating perceived from actual latency explicitly, and catching unnecessary sequential dependencies as a concrete, checkable pattern rather than a vague "make it faster."

Common mistakes: treating streaming as a latency fix when total generation time is what actually matters for a use case, and not setting timeouts on LLM calls the same way every other external dependency gets one.

Q5 What is prompt injection, and how do you defend against it in an application that reads untrusted content or calls tools?#

Short answer: Prompt injection is untrusted content — a user message, a retrieved document, a tool's output — containing text crafted to make the model deviate from its original instructions; direct injection comes straight from the user, indirect injection is smuggled inside content the model reads on the user's behalf, and defense is layered: privilege separation so the model never has more authority than the request warrants, treating external content as data rather than instructions, output-side validation, and narrow, allow-listed tool scopes rather than trusting a single delimiter to hold.

No single prompt-level defense reliably holds against a determined adversary — telling the model to ignore instructions found in following content reduces but doesn't eliminate the risk, because the model still processes that content as language it can be influenced by, not as inert data with a hard boundary. This is why the practical defense is architectural, not linguistic: privilege separation means the model's tools and outputs are scoped to exactly what the current, authenticated user is allowed to do, so even a successful injection can only make the model attempt actions within that already-narrow boundary, and the tool itself enforces authorization independently rather than trusting the model's judgment. Indirect injection through retrieved content or tool output is the higher-risk case specifically because the content wasn't authored by the interacting user and can come from anywhere the pipeline reads from, so any pipeline that lets a model read external content and then take actions needs an explicit trust boundary between content the model reasons over and instructions the model executes, ideally with high-risk actions gated behind human approval regardless of what the model was told. Output-side validation catches what slips through: if a tool call's arguments or a generated response fall outside an expected, allow-listed shape, reject it rather than executing it, the same way you'd validate any other untrusted input at a system boundary.

[Description("Sends a password reset email to the address on the current user's account.")]
static Task SendPasswordResetAsync(ClaimsPrincipal user, IEmailService email) =>
    email.SendResetLinkAsync(user.GetVerifiedEmail()); // no attacker-controlled address input

What interviewers look for: rejecting "a good system prompt prevents injection" as a complete answer, and describing privilege separation and tool-level authorization as the real defense, with prompt-level instructions as a weak, supplementary layer at best.

Common mistakes: relying on delimiters or "ignore untrusted instructions" prompt wording as the primary defense, and giving a tool an attacker-reachable parameter instead of deriving sensitive values from an authenticated context.

Q6 What do "guardrails" mean concretely in a production AI system, and where do you put them in the pipeline?#

Short answer: Guardrails are layered checks at each boundary of the pipeline — input validation and moderation before the prompt is built, scope and grounding constraints in the system prompt itself, and output validation, schema checks and content moderation after generation but before anything is shown to a user or acted on — and no single layer is sufficient alone; the value comes from the combination.

Input-side guardrails catch what shouldn't reach the model at all: obviously malicious input, requests clearly outside the feature's intended scope, personal data that shouldn't be forwarded to a third-party provider — services like Azure AI Content Safety exist specifically to classify content for this kind of gate without every team building a classifier from scratch. Prompt-level guardrails, explicit scope and behavior instructions, are real but, as the injection question covers, the weakest layer on their own, since they're just more text the model can be argued out of; treat them as a helpful default, not a control. Output-side guardrails are where enforcement actually has teeth: schema validation on structured output rejects a malformed response before it reaches a caller, content moderation on generated text catches what a system prompt failed to prevent, and business-rule checks catch outputs that are well-formed but wrong, such as a discount outside an allowed range or a recommendation referencing something that doesn't exist. The architectural point interviewers want is that guardrails are a pipeline of independent checks at different stages, so a single bypass at one layer doesn't mean the request reaches the user unchecked; a system with only a good system prompt and nothing on the output side isn't actually guarded, it's just asked nicely.

What interviewers look for: describing guardrails as layered, independent checks across input, prompt and output stages, and correctly ranking prompt-level instructions as the weakest layer rather than the primary control.

Common mistakes: relying on the system prompt as the only guardrail, and validating output format without validating business rules the format can't express.

Q7 What compliance and responsible-AI concerns come up when shipping an LLM feature in a regulated environment, and how do you address them architecturally?#

Short answer: The recurring concerns are data residency and third-party data sharing, PII exposure in prompts and logs, auditability of what the system saw and said, and human oversight for consequential decisions — addressed architecturally with provider and region selection matching your data-handling requirements, systematic redaction of sensitive fields before they reach a model or a log, structured audit logging with retention policies, and explicit human review for decisions the model shouldn't make unilaterally.

Data residency and third-party sharing questions come first in most compliance reviews, because sending customer data to a model provider is a data-processing relationship like any other vendor integration and needs the same evaluation: which region processes the data, what the provider's retention and training-use terms are, and whether a regional or private deployment is required for a given dataset. PII exposure is a narrower but very concrete risk: prompts and telemetry can easily end up containing sensitive fields unless redaction happens deliberately, which is exactly the gap packages like Microsoft.Extensions.Compliance.Redaction are built for, classifying and automatically redacting sensitive data at the logging and telemetry boundary rather than relying on every developer to remember to scrub it by hand. Auditability matters because a regulated environment frequently needs to answer what the system saw and said for a specific past interaction, months later, which means prompts, retrieved context, tool calls and final responses need structured, retained logging with a defined retention period, not best-effort application logs that roll off after a few days. Human oversight ties back to the human-in-the-loop pattern used for agents: a regulated decision, a credit determination, a medical triage suggestion, an account action, usually cannot be made unilaterally by a model regardless of confidence, and the architecture needs to make that a structural guarantee rather than a prompt instruction the model could be talked out of.

What interviewers look for: treating the model provider as a vendor relationship subject to data-handling review, and naming concrete architectural controls — redaction, retained structured audit logs, mandatory human review — rather than a general "we follow responsible AI principles" answer.

Common mistakes: logging full prompts and responses for debugging without considering they may contain PII, and treating "the model is usually right" as sufficient justification to skip human review on a consequential decision.

Q8 How do you safely roll out a model upgrade without breaking behavior in production?#

Short answer: Run the new model against the same golden evaluation dataset and compare scores against the current production model before rollout, deploy behind a flag so a small percentage of real traffic can be routed to the new model and compared on live outcomes, and version prompts alongside evaluation results so a regression can be attributed to the model change specifically — with IChatClient's provider abstraction making the actual routing mechanically simple, which is exactly why the evaluation discipline matters more than the plumbing.

The trap is assuming a newer model version is a strict upgrade; in practice a new version can improve on some task classes and regress on others, including ones your evaluation set doesn't happen to cover well, so "it's newer" is not evidence of "it's better for this feature." The evaluation-first step reuses the golden dataset and scoring pipeline from the continuous evaluation question, run against both models side by side, which surfaces regressions before any real user sees them; a canary rollout — a small percentage of live traffic on the new model, monitored with the same observability and quality-tracking signals as the rest of the feature — catches what the offline set missed, since production traffic is always broader than any curated dataset. Prompts should be versioned alongside the model, because a prompt tuned against one model's specific behavior can perform noticeably differently against another, which means a model upgrade is frequently also a prompt-tuning exercise, not just a configuration flip, and treating it as "just change the model string" is how upgrades introduce silent regressions. Because IChatClient abstracts the provider call itself, the traffic-routing mechanism is simple; the discipline that actually prevents incidents lives entirely in the evaluation and canary process around that routing, not in the routing code itself.

What interviewers look for: rejecting "newer model equals strictly better" and describing a concrete side-by-side evaluation plus canary process, with prompt re-tuning named as part of the upgrade, not an afterthought.

Common mistakes: flipping a model version in configuration without re-running evaluation, and assuming prompts are model-agnostic when they were actually tuned against the old model's specific behavior.

Q9 LLM output is inherently non-deterministic. How do you monitor quality in production and catch degradation a one-time evaluation wouldn't?#

Short answer: Track evaluation-style scores continuously on sampled production traffic rather than only at deploy time, monitor proxy signals that correlate with quality — refusal rate, output-length distribution, user-facing feedback, error and timeout rates — and alert on trend changes in those signals rather than trying to catch every individual bad response, since some variance is expected and not every low-scoring response is a real incident.

A single evaluation run at deploy time is a snapshot, and production quality drifts for reasons that snapshot can't see: a provider updates a model behind the same version identifier, the mix of real user queries shifts away from what the golden dataset represents, or an upstream data source in a retrieval-based feature gradually degrades. Continuous sampled evaluation, scoring some percentage of live traffic on an ongoing basis rather than only at release time, turns quality monitoring from a point-in-time check into an operational signal, the same shift observability already made for uptime and latency. Proxy signals matter because scoring every request in full is often too expensive at full volume: a rising refusal rate can mean the input mix shifted toward genuinely unanswerable questions, or it can mean a regression; a sudden change in typical output length can indicate truncation, rambling, or a different stopping behavior than before; explicit user feedback, sampled and aggregated, is a weaker but real-world signal automated scoring can miss entirely. The alerting discipline that works is trend-based rather than threshold-per-response — a handful of low-scoring individual responses is expected noise, but a sustained shift in the trend line across any of these signals is worth paging someone for.

What interviewers look for: naming concrete proxy signals beyond "run evaluation again," and understanding that alerting has to be trend-based given inherent per-response variance, not a hair-trigger on individual low scores.

Follow-up questions:

• What sampling rate would you choose for continuous production evaluation, and what drives that trade-off?
• How would you distinguish "the query mix changed" from "the model actually got worse"?

Q10 Walk through designing safety layers for an agent that can take real-world actions, not just chat. How do you bound the blast radius of a mistake?#

Short answer: Scope every tool to the least privilege it needs, run high-risk actions through a dry-run or simulation mode before allowing live execution, gate irreversible or costly actions behind human approval, cap the agent's rate and budget so a malfunction can't scale unbounded, keep a complete audit trail of every action and why it was taken, and have a kill switch that can halt the agent's ability to act without taking down the whole system.

Least-privilege tool scoping is the foundational layer — each tool should do exactly one narrow thing with an authorization check it enforces itself, so even a fully successful injection or a confidently wrong model decision is bounded by what that narrow tool set can actually do, never by what's theoretically reachable through a broader credential. A dry-run or simulate mode, where a tool call returns what would happen without executing it, is worth building for any high-risk action during development and rollout, and in some designs stays available as a permanent middle option between full automation and full human execution. Rate and budget caps — a maximum number of actions per time window, a maximum spend, a maximum blast radius per invocation — turn a malfunctioning agent from an open-ended incident into a bounded one, the same principle as a circuit breaker applied to autonomous decision-making rather than network calls. Every action needs a structured audit log, what was attempted, with what arguments, under whose authority, and what the outcome was, both for the compliance case covered earlier and because debugging an agent that took a wrong real-world action without a trail of why it decided to is close to impossible after the fact. Finally, a kill switch that can disable an agent's ability to take actions, while optionally leaving it able to still respond conversationally, needs to exist and be tested before it's needed, not designed for the first time during an actual incident.

What interviewers look for: naming multiple independent containment layers — privilege scoping, dry-run, budgets, audit, kill switch — rather than one, and treating "bound the blast radius" as the actual design goal instead of "make the agent smarter so it doesn't make mistakes."

Common mistakes: relying on the model's own judgment as the only safety control, and building an audit log that records the final action but not the reasoning or authorization context behind it.