A service whose memory climbs until it gets restarted is one of the most common escalations a lead engineer owns, because it rarely announces itself with a stack trace, only with a graph that trends the wrong way. This is a diagnostic discipline more than a trivia topic: it tests whether you can build a case from live metrics, capture the right artifact without making the incident worse, read a heap dump with SOS, and tell a genuine managed leak apart from normal generational GC behavior or a leak that lives entirely outside the managed heap. Interviewers at the lead and architect level use this topic to see whether you reach for a debugger by instinct or work the problem like an investigation, with a clear sequence of cheap checks before expensive ones. This page covers the scenario end to end: counters, dotnet-gcdump, SOS commands such as dumpheap -stat and gcroot, the root causes that show up again and again, unmanaged leaks that dumps alone cannot see, and how to fix and monitor for regressions.

Q1 A service's memory grows steadily until it has to be restarted. Walk through your diagnosis process from the first alert to a fix.#

Short answer: Confirm the growth is real and correlate it with load, capture lightweight dotnet-gcdump snapshots spaced apart to see which type is accumulating, escalate to a full dotnet-dump with SOS's gcroot when you need the exact reference chain, then fix the root cause and verify the metric flattens under sustained load before calling it done.

The sequence matters because each step is more expensive and more disruptive than the last, and skipping straight to the most expensive one wastes an incident's worth of time. Start with dotnet-counters against the live process to see whether GC Heap Size climbs independently of request volume, which rules out "it's just proportional to load." Next, take two dotnet-gcdump snapshots separated by real traffic and diff the type statistics between them; a type whose count and total size keep growing across both snapshots, while everything else stays flat, is your suspect. Only then take a full dotnet-dump, because it is heavier and pauses the process longer, and use SOS to find exactly what is holding the suspect type alive. Fix the root cause, then re-run the same load pattern in staging and confirm the counter trend is flat, not just "better," before treating the incident as closed.

Bash
dotnet-counters monitor --counters System.Runtime -p 4807

dotnet-gcdump collect -p 4807 -o before.gcdump
# ...let representative traffic run for a while...
dotnet-gcdump collect -p 4807 -o after.gcdump

dotnet-gcdump report after.gcdump

What interviewers look for: an ordered, cost-aware process rather than "I'd take a dump and look at it." Candidates who mention correlating with load and verifying the fix under the same conditions that reproduced the bug stand out.

Common mistakes: jumping straight to a full dump before confirming the problem is a real leak; declaring victory after one clean-looking snapshot instead of proving the trend is flat over time.

Follow-up questions:

  • How would you do this investigation against a pod you cannot get a shell into?
  • What would you check first if the growth started right after a specific deployment?

Q2 How do you tell a genuine managed memory leak apart from growth that isn't actually a leak?#

Short answer: A real leak keeps growing indefinitely and does not shrink after full garbage collections; benign growth plateaus at a level proportional to load, and some of what looks like growth is the GC's own sizing behavior, particularly Server GC and DATAS adapting heap size to demand rather than a bug.

Server GC deliberately sizes its heap segments for throughput, and on a machine with many cores that can mean a working set far larger than the live data would suggest, simply because the GC is trading memory for fewer, cheaper collections. Since .NET 9, DATAS (Dynamic Adaptation To Application Sizes) changed the default behavior to size the heap closer to the live data instead of the core count, which removed a lot of false alarms that used to come from classic Server GC on high-core-count machines, but it is still worth confirming which mode a suspect service is running before treating a plateau as a bug. The other frequent false positive is timing: gen2 and large-object-heap collections are intentionally infrequent, so a heap that has simply not been collected yet can look alarming on a short graph and look completely normal an hour later. The reliable signal is trend, not snapshot: watch GC Heap Size and, specifically, the gen2 portion of it across several full collections under steady, repeatable load. If it keeps climbing after multiple gen2 collections have had the chance to run, that is real; if it climbs and then holds steady, it was sizing behavior, not a leak.

What interviewers look for: knowledge that GC sizing behavior, not just application bugs, can produce a rising graph, and a concrete criterion (climbing across multiple full collections under steady load) for telling the two apart rather than a gut feeling.

Common mistakes: calling GC.Collect() in production code to "prove" a leak instead of using it only as a diagnostic step against a non-production instance; treating any rising graph as an incident without checking whether it tracks load.

Q3 How do you use dotnet-counters to build the case that this is a leak before you take a disruptive dump?#

Short answer: Record GC Heap Size, Gen 0/1/2 GC Count and Allocation Rate over an extended, load-correlated window, and look for a heap size that keeps climbing across multiple gen2 collections while allocation rate stays roughly proportional to traffic, not for a single alarming number.

A live monitor session is good for a quick look, but building a case for an incident review needs a record you can chart and share, which is what the collect subcommand is for: it writes the same counters to a file instead of a console view.

Bash
dotnet-counters collect --counters System.Runtime -p 4807 --format csv -o metrics.csv

Run it across a window long enough to see several gen2 collections happen naturally, ideally spanning a period with a known, repeatable traffic pattern so you can distinguish "heap size tracks requests per second" from "heap size only goes up." If you also have OS-level memory metrics for the same process, chart the container or process working set alongside the CLR's own GC Heap Size; a managed leak shows both climbing together, while a native leak shows the working set climbing while the CLR-reported heap size stays flat, which is the first clue that SOS alone will not find the cause.

What interviewers look for: the specific counters and the reasoning for why a time series beats a single reading, plus awareness that comparing the managed heap counter against OS-level memory is itself a diagnostic step, not just a nice chart.

Q4 When do you reach for dotnet-gcdump instead of a full dotnet-dump, and what's the practical difference?#

Short answer: dotnet-gcdump captures only the managed heap graph, objects, their types, sizes and references, cheaply enough to take repeatedly against a live production process; a full dotnet-dump captures the entire process image, including native memory and thread stacks, which is heavier, pauses the process for longer, and is what SOS's gcroot and thread-level commands need.

Bash
dotnet tool install --global dotnet-gcdump
dotnet-gcdump collect -p 4807 -o 20260924_1200.gcdump
dotnet-gcdump report 20260924_1200.gcdump

Because a .gcdump is comparatively cheap and safe to take repeatedly, it is the right tool for the "what type keeps growing" triage step: take one, wait, take another, and diff the type counts, either through dotnet-gcdump report or by opening both files in Visual Studio or PerfView, which can show the growth directly. A full dotnet-dump collect is the escalation once you know which type is suspect and need to answer "what is actually rooting these instances," because that requires walking the complete object graph and, for gcroot, correlating it against thread stacks and static handles that a gcdump alone does not include. The trade-off is real: a full dump of a large-memory process can take a long time to write, temporarily doubles the disk footprint, and pauses the process for the duration, so reserve it for when a gcdump has already narrowed the search.

What interviewers look for: the cost-versus-capability trade-off stated plainly, and the workflow of using gcdumps to triage before a full dump to confirm, rather than treating the two tools as interchangeable.

Common mistakes: taking a full dump as the first diagnostic step on a large production process without first narrowing the suspect type; forgetting that a gcdump alone cannot show why an object is rooted, only that it exists.

Q5 Walk through analyzing a full dump with SOS: dumpheap -stat, then gcroot. What are you looking for at each step?#

Short answer: dumpheap -stat ranks live object types by count and total size to identify the suspect; gcroot on an instance of that type walks backward from the object to whatever GC root is keeping it reachable, and that chain almost always points directly at the bug.

Text
> dumpheap -stat
...
Statistics:
      MT    Count    TotalSize Class Name
00007f6c1dc00f90   842,113   40,421,424 MyApp.Orders.OrderEvent
...

> dumpheap -mt 00007f6c1dc00f90
      Address               MT     Size
00007f6ad09421f8 00007f6c1dc00f90       48
...

> gcroot 00007f6ad09421f8
Thread 0:
    ...
    -> 00007f6a10002340 MyApp.Orders.OrderNotifier
    -> 00007f6a10002360 System.EventHandler`1[[MyApp.Orders.OrderEvent, MyApp]]
    -> 00007f6ad09421f8 MyApp.Orders.OrderEvent
Found 1 unique root.

Read dumpheap -stat by total size first, not just count: a million small strings can matter less than a few thousand large domain objects, but an unreasonably high count of an ordinary-looking type, one your own code defines rather than a BCL type, is usually the more actionable lead. Once a type looks suspect, dumpheap -mt <method-table-address> lists its live instances so you can pick one, and gcroot on that instance's address prints the reference chain from a genuine GC root, a static field, a thread's stack, or a handle, down to the object. In the example above, the chain leads through a static OrderNotifier type's event field, which is exactly the classic "forgot to unsubscribe" pattern: the publisher is static, so its invocation list lives for the process's whole lifetime, and every subscriber it still references is unreachable to garbage collection no matter how long ago the code that created the subscriber finished running.

What interviewers look for: fluency with the actual SOS command sequence, and the ability to read a gcroot chain and translate it back into a source-level bug rather than just describing the commands abstractly.

Common mistakes: sorting by count instead of total size and chasing a type that does not actually matter; stopping at "found the type" without running gcroot to confirm why it is rooted.

Q6 What are the most common root causes of managed memory leaks you've seen in production?#

Short answer: Event subscriptions that are never removed, caches with no eviction policy, and scoped services captured by something longer-lived, most often a singleton or a static field, account for the large majority of managed leaks.

C#
public static class OrderNotifier
{
    public static event EventHandler<OrderEvent>? OrderPlaced;
    public static void Raise(OrderEvent e) => OrderPlaced?.Invoke(null, e);
}

public sealed class OrderAuditPanel
{
    public OrderAuditPanel()
    {
        // The static event's invocation list now holds a reference to this
        // instance for the lifetime of the process, unless it unsubscribes.
        OrderNotifier.OrderPlaced += OnOrderPlaced;
    }

    private void OnOrderPlaced(object? sender, OrderEvent e) { /* ... */ }
}

Beyond the static-event pattern above, the same shape recurs in a handful of forms: a Dictionary<TKey, TValue> or ConcurrentDictionary<TKey, TValue> used as an ad hoc cache with no size limit and no expiration, which grows for as long as new keys appear; a singleton that captures a scoped dependency, such as a DbContext, through constructor injection, which ASP.NET Core's dependency injection container flags with Cannot consume scoped service ... from singleton when scope validation is enabled in Development, but which can slip through silently if that validation is off; IHostedService or timer callbacks that capture a per-request object and keep it alive for the life of the timer; and ThreadStatic or ThreadLocal<T> state on thread-pool threads that never gets cleared, which grows as the thread pool cycles through work over the life of the process. All of these share the same underlying shape: something with a long, often unbounded lifetime holds a reference to something that was only ever meant to be short-lived.

What interviewers look for: more than a list; the best answers explain why each pattern leaks (a long-lived owner reaching a short-lived object) instead of reciting examples from memory.

Follow-up questions:

  • How would dumpheap -stat look different for a cache leak versus an event-subscription leak?
  • What ASP.NET Core dependency injection setting catches the scoped-into-singleton mistake automatically?

Q7 How do unmanaged memory leaks show up differently, and how do you diagnose one that dumpheap can't see?#

Short answer: dumpheap only walks the managed GC heap, so a leak in native memory, an unmanaged buffer from a P/Invoke call, a native image or crypto library, an undisposed native handle, is invisible to it; the signature is a process working set that keeps climbing while the CLR's own GC Heap Size stays flat, and diagnosing it means reaching for OS-level tools instead of SOS.

The first check is exactly that comparison: watch the OS-reported process memory (container working set, /proc/<pid>/status on Linux, or Task Manager private bytes on Windows) against GC.GetGCMemoryInfo() or the GC Heap Size counter from inside the same process. A healthy managed process has these move together; a large and growing gap is the signature of native growth. From there, audit code for IDisposable types that wrap native handles, SafeHandle subclasses, native interop buffers, unmanaged image or compression libraries, and confirm every one of them is disposed on every code path, not just the happy path. Finalizers eventually reclaim undisposed SafeHandle-based resources, but under load finalization can fall behind the allocation rate badly enough that it is effectively still a leak from the application's point of view. Once the suspects are narrowed, true native-memory investigation moves outside the .NET diagnostics toolchain entirely, to tools such as a native heap profiler or allocation tracker appropriate to the platform, because SOS and dotnet-gcdump were never built to see memory the CLR itself did not allocate.

What interviewers look for: the working-set-versus-GC-heap comparison as the defining diagnostic signal, and the understanding that this category of bug requires a different toolset entirely, not just a different SOS command.

Common mistakes: running dumpheap -stat repeatedly on a native leak and concluding there is no leak because the managed heap looks clean; forgetting that finalizers are a backstop, not a substitute for explicit disposal.

Q8 You've found the leak, a static event subscription. How do you fix it without introducing a new leak or a regression elsewhere?#

Short answer: Make the subscriber explicitly unsubscribe through IDisposable, make sure every code path that creates one also disposes it, and add a targeted test that proves the object actually becomes collectible after disposal, rather than trusting that the fix works by inspection.

C#
public sealed class OrderAuditPanel : IDisposable
{
    private bool _disposed;

    public OrderAuditPanel() => OrderNotifier.OrderPlaced += OnOrderPlaced;

    private void OnOrderPlaced(object? sender, OrderEvent e) { /* ... */ }

    public void Dispose()
    {
        if (_disposed)
        {
            return;
        }
        OrderNotifier.OrderPlaced -= OnOrderPlaced;   // breaks the root; the panel can now be collected
        _disposed = true;
    }
}

The unsubscribe itself is the easy part; the discipline is making sure Dispose actually runs on every path, including exceptions, which usually means the object is created inside a using block or owned by a container that guarantees disposal. Resist the temptation to "fix" this by switching to a weak-reference-based event pattern everywhere: a weak event can mask a genuine lifecycle bug by letting the subscriber disappear earlier than the code actually intends, which trades a loud, diagnosable leak for a quiet, hard-to-reproduce missing-callback bug. Reserve weak events for cases where you genuinely cannot control the subscriber's lifetime, not as a default. Prove the fix with a test that does not rely on reading code by eye:

C#
var panel = new OrderAuditPanel();
var tracker = new WeakReference(panel);
panel.Dispose();
panel = null;

GC.Collect();
GC.WaitForPendingFinalizers();

Assert.False(tracker.IsAlive);   // proves the fix actually breaks the reference chain

What interviewers look for: going beyond "add the unsubscribe line" to the full discipline of guaranteed disposal and a test that verifies collectibility, plus the judgment to avoid over-correcting into weak events everywhere.

Common mistakes: adding Dispose but never calling it from every construction site; assuming the fix works without a WeakReference-based test to confirm the object is actually collectible.

Q9 How do you prevent a memory-leak regression like this from reaching production in the first place?#

Short answer: Put the pattern into code review as a named checklist item, prefer built-in eviction-aware types over hand-rolled static collections, and run a soak test under sustained load in CI or staging that fails the build if the GC heap trend is not flat.

Code review catches most of these before they ship if reviewers know specifically what to look for: every += on a static or singleton-owned event, every constructor-injected dependency into a singleton, and every hand-rolled Dictionary used as a cache without an eviction policy. Architecturally, defaulting to IMemoryCache or HybridCache with an explicit size limit, instead of a plain dictionary, removes an entire category of this bug by construction, because eviction is built into the type rather than relying on every engineer remembering to add it. For the regressions that slip past review, a soak test is the backstop: run representative, sustained load against a staging instance for long enough to see several full collections, record GC Heap Size throughout, and fail the pipeline if the trend does not flatten within a defined tolerance. This is slower than a unit test, so it typically runs nightly or before a release rather than on every commit, but it is the only check that actually exercises the failure mode a leak represents.

What interviewers look for: a layered answer, code review plus safer defaults plus an automated soak test, rather than relying on any single line of defense.

Follow-up questions:

  • What tolerance would you set for "heap size is flat enough" in an automated soak test, and why?
  • How would you keep a soak test from being flaky on shared CI infrastructure?

Q10 What monitoring and alerting would you put in place so a leak is caught long before it causes an outage?#

Short answer: Dashboard GC Heap Size (particularly the gen2 portion) alongside the process or container working set, alert on a sustained upward trend that does not correlate with request volume rather than a fixed threshold, and set the alert early enough, well below the container's memory limit, that on-call has time to roll back instead of racing an out-of-memory kill.

A single threshold alert is a poor fit for this problem because healthy services plateau at very different memory levels; what actually indicates a leak is the shape of the trend; a slope that keeps climbing over hours without flattening, uncorrelated with traffic. Charting the managed heap size next to the OS-level working set on the same dashboard catches both managed and native leaks with one view, since the two should move together in a healthy process. Tag the dashboard with deployment markers so a leak introduced by a specific release surfaces within hours of that release, not after a multi-day climb forces an emergency restart. For faster response once an alert fires, dotnet-monitor can be configured with automated collection rules that trigger a gcdump or trace the moment a metric crosses a threshold, which means the diagnostic artifact is often already captured and waiting by the time someone opens the incident, instead of the leak having to be reproduced live under pressure.

What interviewers look for: the "trend, not threshold" framing, and awareness that the goal of monitoring here is to buy the team time to fix the problem calmly, not just to confirm the outage after it happens.

Common mistakes: alerting only on a raw memory threshold close to the container limit, which leaves no time to respond; not correlating the memory dashboard with deploy events, which turns every investigation into starting from zero.

Quick-Fire Round#

QuestionAnswer
Which command captures only the managed heap graph, cheaply and repeatably?dotnet-gcdump collect.
Which SOS command ranks live types by count and total size?dumpheap -stat.
Which SOS command shows why an object is still reachable?gcroot <address>.
What's the classic managed-leak signature in gcroot?A chain ending at a static field or event.
What's the tell-tale sign of a native, not managed, leak?Working set climbs while GC Heap Size stays flat.
What DI mistake commonly causes a leak in ASP.NET Core?A singleton capturing a scoped dependency.
What should replace a hand-rolled dictionary used as a cache?IMemoryCache or HybridCache with a size limit.
What's a better alert condition than a fixed memory threshold?A sustained upward trend uncorrelated with load.

How to Prepare#

  • Practice the full sequence end to end on a deliberately leaky sample app: counters, two gcdump snapshots, a full dump, then dumpheap -stat and gcroot.
  • Be ready to explain, with a concrete criterion, how you tell a real leak apart from Server GC or DATAS sizing behavior.
  • Memorize the exact SOS commands; interviewers frequently ask you to narrate a dump analysis live.
  • Know at least three root-cause patterns well enough to sketch the buggy code and the fix from memory.
  • Have a working-set-versus-GC-heap answer ready for the unmanaged-leak follow-up; it is a common way interviewers test depth beyond SOS.
  • Read the garbage collection guide so generational behavior and DATAS are fresh context for the "is this really a leak" question.