Multimodal AI in .NET: Vision, Audio and Speech

Q: Can I send an image directly through Microsoft.Extensions.AI, or do I need the OpenAI library?

You can do it either way. DataContent and UriContent on a ChatMessage work through the provider-neutral IChatClient , and ChatMessageContentPart.CreateImagePart works if you are calling the OpenAI .NET library directly. Prefer the IChatClient path when the rest of your code is already provider-neutral.

Multimodal AI in .NET means sending and receiving more than text: images, scanned documents, speech and generated audio, alongside the chat models most .NET developers already use. This guide is for developers who need to move past plain-text prompts and handle a screenshot, a PDF invoice, a support call recording or a voice interface from C#. You will learn how to send images to chat models with Microsoft.Extensions.AI, extract text and structure from documents, transcribe and synthesize speech, use OpenAI's realtime voice API, generate images, and reason about the cost and privacy trade-offs each modality brings.

What Is Multimodal AI?#

A multimodal model accepts, and sometimes produces, more than one kind of content in a single request: text plus images is the most common combination today, with audio and, increasingly, video following the same pattern. In .NET, "multimodal" spans two different kinds of API. Chat models such as GPT-5.1 accept images (and, on some models, audio) as part of an ordinary chat message, so vision is really just another content type inside the same IChatClient call you already use for text. Dedicated services, such as speech recognition, speech synthesis and document analysis, are separate, purpose-built APIs that are not chat completions at all, and usually cheaper and more accurate for their specific job than asking a general chat model to do the same task.

Knowing which category a task falls into is the first design decision. Asking a chat model to describe a photo or read a screenshot's error message plays to its strength: general, flexible understanding. Transcribing an hour of call center audio or extracting every field from a standardized invoice plays to a dedicated speech or document service's strength: purpose-built accuracy and predictable, usage-based pricing.

How Multimodal Content Works in Microsoft.Extensions.AI#

Microsoft.Extensions.AI represents non-text content as AIContent subclasses inside a ChatMessage, alongside plain TextContent. The two you will use for images and files are DataContent, which embeds raw bytes plus a media type, and UriContent, which references content by URL instead of sending the bytes inline:

using Microsoft.Extensions.AI;

byte[] screenshotBytes = await File.ReadAllBytesAsync("error-dialog.png", cancellationToken);

ChatMessage fromBytes = new(ChatRole.User,
[
    new TextContent("What is the likely root cause of this error?"),
    new DataContent(screenshotBytes, "image/png"),
]);

ChatMessage fromUrl = new(ChatRole.User,
[
    new TextContent("Summarize what this architecture diagram shows."),
    new UriContent(new Uri("https://cdn.contoso.com/docs/architecture.png"), "image/png"),
]);

ChatResponse answer = await chatClient.GetResponseAsync([fromBytes], cancellationToken: cancellationToken);

DataContent is the safer default for anything not already hosted somewhere the model provider can fetch, including user uploads. UriContent avoids sending large files through your own service when the content is already on a public or presigned URL, but it only works when the model provider's servers can reach that URL directly. Because this is an ordinary ChatMessage, everything from Microsoft.Extensions.AI, streaming, middleware, dependency injection, applies unchanged; multimodal input does not need a different client or pipeline.

Getting Started: Sending an Image to a Chat Model#

A minimal image-understanding call needs a multimodal-capable model, an image, and a question:

using Microsoft.Extensions.AI;

IChatClient client = new OpenAI.Chat.ChatClient("gpt-5.1",
    Environment.GetEnvironmentVariable("OPENAI_API_KEY")!).AsIChatClient();

byte[] receipt = await File.ReadAllBytesAsync("receipt.jpg");

ChatResponse response = await client.GetResponseAsync(
[
    new ChatMessage(ChatRole.User,
    [
        new TextContent("List each line item and its price as a JSON array."),
        new DataContent(receipt, "image/jpeg"),
    ]),
]);

Console.WriteLine(response.Text);

For a typed, schema-constrained result instead of prose, combine this with GetResponseAsync<T>, covered in Structured Outputs: Reliable JSON from LLMs in C#.

Vision: Screenshots, Diagrams and Photos#

Vision is the most broadly useful modality because it needs no new infrastructure: any code already calling IChatClient can add an image to the next message. Common uses include triaging bug reports that include a screenshot, reading a whiteboard photo from a design session, checking a generated UI against a mockup, and moderating user-uploaded images before they are stored. If you are working directly against the OpenAI .NET library instead of the abstraction, the equivalent call uses ChatMessageContentPart.CreateImagePart, covered alongside the rest of that library in Using the Official OpenAI .NET Library for C#:

using OpenAI.Chat;

List<ChatMessage> messages =
[
    new UserChatMessage(
        ChatMessageContentPart.CreateTextPart("Does this diagram show a single point of failure?"),
        ChatMessageContentPart.CreateImagePart(BinaryData.FromBytes(imageBytes), "image/png")),
];

ChatCompletion completion = await chatClient.CompleteChatAsync(messages);

Image quality and size affect both cost and accuracy: a very high-resolution photo costs more to process than a compressed screenshot and rarely improves the answer for tasks like reading dialog text or classifying a scene, so resize images to what the task actually needs before sending them.

Document Understanding and OCR#

Chat model vision is convenient for a handful of documents, but it does not scale to high-volume, structured extraction, and it gives no guarantee about layout fidelity, such as which value belongs to which table cell. Azure AI Document Intelligence is the purpose-built service for that job: it ships prebuilt models for common document types, including invoices, receipts and ID documents, plus a general layout model that extracts text, tables and key-value pairs from arbitrary forms with their positions on the page.

using Azure;
using Azure.AI.DocumentIntelligence;

DocumentIntelligenceClient client = new(
    new Uri(endpoint), new AzureKeyCredential(apiKey));

Operation<AnalyzeResult> operation = await client.AnalyzeDocumentAsync(
    WaitUntil.Completed, modelId: "prebuilt-invoice", BinaryData.FromStream(invoiceStream));

AnalyzeResult result = operation.Value;
foreach (DocumentField field in result.Documents[0].Fields.Values)
{
    Console.WriteLine($"{field.FieldType}: {field.Content}");
}

Azure AI Content Understanding is the newer, broader sibling of Document Intelligence: instead of a fixed set of prebuilt document models, you define an analyzer with a field schema and point it at documents, images, audio or video, and it extracts structured fields across all of them through one consistent pattern. Reach for Document Intelligence when a prebuilt model already matches your document type, such as invoices or receipts, and reach for Content Understanding when you need a custom schema or need to extract structured fields from audio or video rather than documents. Either way, run OCR and extraction as part of ingestion, well before the content reaches a chat model; see Retrieval-Augmented Generation (RAG) in .NET for fitting extracted document text into a retrieval pipeline.

Speech-to-Text: Transcription#

Two solid paths cover speech-to-text in .NET. Azure AI Speech, through the long-standing Microsoft.CognitiveServices.Speech SDK, excels at continuous, low-latency recognition, custom vocabulary for domain terms, and speaker diarization for multi-party calls:

using Microsoft.CognitiveServices.Speech;
using Microsoft.CognitiveServices.Speech.Audio;

var speechConfig = SpeechConfig.FromSubscription(apiKey, region);
using var audioConfig = AudioConfig.FromWavFileInput("support-call.wav");
using var recognizer = new SpeechRecognizer(speechConfig, audioConfig);

SpeechRecognitionResult result = await recognizer.RecognizeOnceAsync();
if (result.Reason == ResultReason.RecognizedSpeech)
{
    Console.WriteLine(result.Text);
}

The OpenAI .NET library's AudioClient is the other path, and it is often simpler when you are already calling OpenAI for chat: TranscribeAudio runs Whisper-family transcription, and TranscribeAudioDiarized labels which speaker said each segment, both covered in Using the Official OpenAI .NET Library for C#. Choose Azure AI Speech when you need continuous streaming recognition, custom pronunciation or tight Azure compliance boundaries; choose the OpenAI audio endpoints when a single OpenAI account and API surface for both chat and transcription is simpler to operate.

Text-to-Speech: Synthesizing Voice Output#

Text-to-speech mirrors the same choice. Azure AI Speech offers a large catalog of neural voices, fine-grained control over prosody through SSML, and custom voice creation for a brand-specific voice:

using Microsoft.CognitiveServices.Speech;

var speechConfig = SpeechConfig.FromSubscription(apiKey, region);
speechConfig.SpeechSynthesisVoiceName = "en-US-AvaMultilingualNeural";

using var synthesizer = new SpeechSynthesizer(speechConfig);
using SpeechSynthesisResult result = await synthesizer.SpeakTextAsync(
    "Your support ticket has been resolved.");

The OpenAI .NET library's AudioClient.GenerateSpeech produces natural-sounding speech from a small set of built-in voices with a single call, which is convenient when you already have an OpenAI-based pipeline and do not need SSML-level prosody control or a custom brand voice.

Realtime Voice: Low-Latency Speech-to-Speech#

Transcribe-then-generate-then-synthesize works, but it stacks several round trips of latency, which is noticeable in a live conversation. Realtime APIs collapse that into a single persistent, bidirectional connection that streams audio in and out while the model reasons and, where configured, calls tools mid-conversation. The OpenAI .NET library exposes this through OpenAI.Realtime, currently marked experimental behind the OPENAI002 diagnostic:

#pragma warning disable OPENAI002
using OpenAI.Realtime;

RealtimeClient client = new(Environment.GetEnvironmentVariable("OPENAI_API_KEY"));
using RealtimeSessionClient session = await client.StartConversationSessionAsync(model: "gpt-realtime");

await session.ConfigureConversationSessionAsync(new RealtimeConversationSessionOptions
{
    Instructions = "You are a calm, concise phone support agent.",
    AudioOptions = new()
    {
        InputAudioOptions = new() { TurnDetection = new RealtimeServerVadTurnDetection() },
        OutputAudioOptions = new() { Voice = RealtimeVoice.Alloy },
    },
});

await session.SendInputAudioAsync(microphoneStream);

await foreach (RealtimeServerUpdate update in session.ReceiveUpdatesAsync())
{
    // Handle RealtimeServerUpdateInputAudioBufferSpeechStarted, output audio deltas, and so on.
}

Server-side voice activity detection (RealtimeServerVadTurnDetection) decides when the caller has finished speaking, so you do not have to implement push-to-talk logic yourself. Azure OpenAI in Microsoft Foundry offers an equivalent realtime endpoint; see Azure OpenAI and Microsoft Foundry for .NET Developers for the Azure connection details. Reach for a realtime API only for genuinely live, spoken interactions, such as a phone agent; a chat UI with a microphone button is usually better served by plain transcription, because it is far simpler to log, moderate and debug.

Image Generation#

ImageClient in the OpenAI .NET library generates images from a text prompt and can edit an existing image with a mask, both detailed in Using the Official OpenAI .NET Library for C#:

using OpenAI.Images;

ImageClient imageClient = new("gpt-image-1", apiKey);

GeneratedImage edited = await imageClient.GenerateImageEditAsync(
    imageFilePath: "product-mockup.png",
    prompt: "Change the background to a plain white studio background.",
    options: new ImageEditOptions { Quality = GeneratedImageQuality.High });

Typical .NET use cases include generating placeholder or marketing imagery, producing on-brand icon sets from a text description, and editing product photos for a catalog. Always run generated or edited images intended for public use through a moderation and human-review step before publishing; image models can still produce content unsuitable for your audience even with safety settings enabled.

Practical Use Cases and Costs#

The table below maps common scenarios to the modality that fits them best, since reaching for the wrong one is the most common source of both wasted cost and disappointing accuracy.

Scenario	Best-fit capability	Why
Triaging a bug report with a screenshot	Chat model vision	Flexible, no extra infrastructure
Extracting fields from thousands of invoices	Document Intelligence	Purpose-built, structured, predictable cost
Live phone support agent	Realtime voice API	Sub-second, bidirectional audio
Transcribing recorded support calls for QA	Speech-to-text (batch)	Cheaper and more accurate than chat-model audio for bulk work
Reading text aloud in an accessibility feature	Text-to-speech	Purpose-built voices and SSML control
Generating marketing or placeholder imagery	Image generation	Purpose-built for image synthesis, not chat

Cost scales with modality in different ways: image input is usually priced by resolution, audio by duration, and document analysis by page, so a workload's true cost depends on which modality you chose, not just how many requests you send. Privacy deserves equal attention: images and audio are far more likely to contain personally identifiable or biometric information than typical text prompts, so apply the same data minimization, retention and regional hosting controls you would to any other sensitive data, and prefer Azure AI services when your compliance posture requires contractual data-handling guarantees. Responsible AI and LLM Security for .NET Applications covers the broader governance picture.

Best Practices#

Match the modality to the task. Use chat model vision for flexible, low-volume understanding, and a dedicated service for high-volume, structured or latency-sensitive work.
Resize and compress images before sending them. Most tasks do not need full resolution, and smaller images cost less and often return faster.
Run document and image extraction during ingestion, not at query time, so a slow OCR call never sits in a user's request path.
Treat realtime voice as a distinct architecture, not "chat with audio." Design for interruption, silence and turn-taking from the start.
Moderate generated content before it reaches end users, especially images and synthesized voice that could be mistaken for a real recording.
Pick Azure AI services when you need contractual data residency or retention guarantees, and OpenAI's own APIs when operational simplicity matters more.

Common Pitfalls#

Sending full-resolution images for simple classification tasks. This inflates cost for no accuracy gain on tasks like "is this a receipt or an invoice."
Using a chat model for bulk transcription. It works, but a dedicated speech service is typically cheaper and more accurate at scale, and supports streaming.
Ignoring turn detection in realtime sessions. Without server-side voice activity detection or an equivalent client-side strategy, the model talks over the caller.
Treating OCR output as ground truth. Extracted fields, especially from Document Intelligence's confidence scores, should be validated before driving an automated decision such as a payment.
Publishing generated images or synthesized voice without review. Both can produce unexpected or unsuitable output even with default safety settings on.

Frequently Asked Questions#

Can I send an image directly through Microsoft.Extensions.AI, or do I need the OpenAI library?#

You can do it either way. DataContent and UriContent on a ChatMessage work through the provider-neutral IChatClient, and ChatMessageContentPart.CreateImagePart works if you are calling the OpenAI .NET library directly. Prefer the IChatClient path when the rest of your code is already provider-neutral.

Should I use Azure AI Speech or the OpenAI audio endpoints?#

Use Azure AI Speech for continuous streaming recognition, custom vocabulary, SSML-level synthesis control or when your compliance requirements call for Azure's data-handling guarantees. Use the OpenAI audio endpoints when you already call OpenAI for chat and want one account and API surface for text, transcription and speech.

What is the difference between Document Intelligence and Content Understanding?#

Document Intelligence offers prebuilt models tuned for specific document types, such as invoices and receipts, plus a general layout model. Content Understanding is a newer, schema-driven service where you define the fields you want and it extracts them consistently across documents, images, audio and video. Start with Document Intelligence when a prebuilt model fits; reach for Content Understanding for custom schemas or non-document media.

Is a realtime voice API always the right choice for a voice feature?#

No. Realtime APIs add real infrastructure and design complexity, including turn detection and interruption handling, that a simple "record, transcribe, respond, synthesize" flow does not need. Reserve realtime voice for genuinely live, low-latency conversations such as phone support, and use batch transcription and text-to-speech for everything else.

How should I handle the privacy risk of sending images or audio to a model provider?#

Minimize what you send: crop or redact images to the relevant region, and consider on-device or Azure-hosted transcription for audio that may contain sensitive conversations. Review your provider's data retention and training-use policies, prefer Azure AI services when contractual guarantees matter, and apply the same consent and retention rules you would to any other biometric or personal data.

Summary#

Multimodal AI in .NET splits into chat model content, images and sometimes audio inside an ordinary IChatClient call, and dedicated services for speech and document extraction.
DataContent and UriContent carry images and files through Microsoft.Extensions.AI without a separate client or pipeline.
Azure AI Document Intelligence and Content Understanding handle structured extraction at a scale and accuracy chat model vision cannot match.
Azure AI Speech and the OpenAI audio endpoints cover transcription and synthesis; realtime APIs add low-latency, bidirectional voice for live conversations.
Match the modality to the task, review generated content before publishing, and treat images and audio as more privacy-sensitive than plain text.