Production Incident Triage for ASP.NET Core APIs: a Practical On-Call Guide

Learning outcome

By the end of this lesson, you should be able to triage an ASP.NET Core API incident without guessing: classify the symptom, separate observation from hypothesis, correlate logs/metrics/traces with request and correlation IDs, investigate saturation and dependency failure without assuming they are mutually exclusive, choose a safe mitigation, and communicate clearly while preserving evidence for the post-incident review.

Intuition

Production triage is not “find the bug fast”; it is “reduce uncertainty fast.” In an ASP.NET Core API, the same user-visible symptom can come from very different causes: a code regression, thread-pool starvation, a downstream database slowdown, a bad deploy, an exhausted connection pool, or an infrastructure fault. Good responders avoid the two classic mistakes: making every incident a code problem, and treating every error spike as a harmless blip.

The fastest path is to answer four questions in order:

1. What is broken for users? For example: 5xx rates, timeouts, slow p95 latency, partial failures, or data inconsistency.
2. What changed? A deployment, config update, feature flag change, dependency degradation, scaling event, or certificate rotation.
3. Where is the bottleneck? App CPU, memory, GC, request queueing, thread pool, connection pools, database latency, or upstream errors.
4. What is the safest mitigation? Roll back, disable a feature, shed load, throttle, or fail over—without destroying evidence.

A useful mental model is: symptom → scope → impact → bottleneck → mitigation. You do not need a final root cause to stabilize the service. You do need enough evidence to avoid making the situation worse.

Deep dive

Start by classifying the incident into one of a few buckets.

Symptom classification

A practical decision table helps separate the first-order signal:

Observation versus hypothesis

Treat every statement as one of two things:

• Observation: measurable or directly visible evidence.
• Hypothesis: an explanation that still needs confirmation.

For example:

• Observation: error rate rose from 0.2% to 8% after 09:12 UTC.
• Hypothesis: the new deployment introduced a regression.

This distinction matters because incident rooms often fill with confident but unverified narratives. Keep hypotheses testable.

Correlating logs, metrics, and traces

The most useful triage evidence is cross-signal correlation:

• Logs tell you what the app reported, especially exceptions, status codes, and validation failures.
• Metrics tell you whether the service is saturated, failing, or simply slow.
• Traces show where latency or failure accumulates across the request path.

For ASP.NET Core APIs, request IDs and correlation IDs are essential. Use the server’s request/trace identifiers as the controlled correlation path. A caller-supplied ID is untrusted input: validate its format and length, reject or sanitize invalid values, and store it separately if it is useful. Never trust it as identity or authorization, let it overwrite unrelated trace context, or use per-request IDs as metric labels. Otherwise one caller can forge a correlation value or create an unbounded metric label set. This lets you answer: “Are these failures all from the same request class?” and “Did the downstream service reject only the calls that had a certain shape?”

Saturation versus dependency failure

Use these as investigation branches, not an exhaustive or exclusive classification. A slow dependency can occupy local resources, grow queues, and trigger retries, so both branches may be active.

Saturation indicators

• CPU remains high under load.
• Requests queue up and latency climbs before failures.
• Thread pool or connection pool waits increase.
• GC pauses or memory pressure align with the slowdown.
• Error codes may be secondary symptoms, such as timeouts and cancellations.

Dependency failure indicators

• App CPU may be normal while failures rise.
• Traces show one downstream span dominates total latency.
• Errors cluster around a particular backend, tenant, or API path.
• Retrying increases pressure on the dependency and can worsen the incident.

Treat CPU, queue length, and slow spans as clues rather than verdicts. Compare the same time window, instance, route, and traffic mix. Low CPU does not rule out blocked local work; a slow dependency span alone does not prove that dependency is the initiating cause. Check sampling gaps and missing spans before interpreting an incomplete trace as a complete account.

Safe mitigation options

Mitigation should minimize blast radius and preserve evidence.

• Rollback a recent release when evidence supports the change and the rollback is compatible with current data/schema and dependencies. Use the approved runbook and an explicit success check.
• Disable a feature flag when impact is isolated to a feature path.
• Reduce traffic by throttling or shedding noncritical work if the service is saturating.
• Fail over or reroute if an upstream dependency or zone is unhealthy and the architecture supports it.
• Increase capacity carefully if the issue is confirmed saturation and the service can safely scale.

Avoid “blind restart” as a first move. A restart can erase diagnostic evidence, mask a bad deployment, and briefly hide the true failure mode. Preserve a bounded evidence sample when safe, but do not delay an urgent, authorized mitigation just to obtain perfect diagnostics. Define who may act, the smallest scope, the expected improvement, and a stop/revert condition.

Incident timeline and communications

Build a timeline as you go:

• Detection time
• First confirmed user impact
• First observation
• Major hypothesis changes
• Mitigation applied
• Recovery confirmed

Stakeholder communication should be short and factual:

• What is impacted?
• Who is impacted?
• What do we know?
• What are we doing now?
• When is the next update?

Do not promise a cause before it is known. Do provide confidence levels and next checkpoints.

Post-incident follow-up

Recovery is not closure. A good follow-up includes:

• a concise incident summary,
• impact window and user effect,
• contributing factors,
• what evidence was decisive,
• what mitigation worked,
• preventive actions with owners and due dates.

If you repeatedly see the same incident shape, the fix is often not “be faster next time” but to improve the system’s observability, rollback path, feature flag hygiene, or capacity guardrails.

Read-only triage checklist

Below is a conceptual triage checklist you can adapt into your runbook. It is intentionally non-executable and vendor-neutral.

1. Confirm impact
   - Which endpoints, tenants, regions, or client versions are affected?
   - Is the symptom 5xx, 4xx, latency, or data inconsistency?

2. Capture the time window
   - When did the first user-visible symptom begin?
   - What changed shortly before that time?

3. Check the three signals together
   - Logs: exception type, status code, correlation/request ID
   - Metrics: error rate, latency percentiles, CPU, memory, queue length
   - Traces: slow spans, failing dependency calls, retry amplification

4. Test the leading hypotheses
   - Compare CPU, queues, pool waits, and downstream spans in the same window.
   - Check whether dependency delay and local saturation coexist.
   - Record what would falsify each hypothesis.
5. Propose the safest authorized mitigation
   - Check rollback compatibility and the evidence for the suspected change.
   - Identify the smallest feature/traffic scope and recovery metric.
   - Execute only through the approved runbook and incident owner.

6. Communicate
   - State observations, hypotheses, mitigation, and next update time

7. Preserve evidence
   - Keep logs, traces, and deployment records for the review

Runbook walkthrough

This lesson does not include an executable sample because incident triage is primarily an operational skill. In practice, your “walkthrough” should read like a runbook:

• First, verify the symptom at the user boundary rather than inside a single process.
• Next, inspect whether the issue aligns with a deployment or config change.
• Then, correlate a small set of affected request IDs across logs and traces.
• Finally, test whether local saturation, dependency delay, or both explain the observed request path; keep alternatives open when evidence is missing.

A strong reviewer can explain why a safe rollback is often preferable to an immediate code hotfix: it shortens time-to-mitigation and reduces the chance of introducing a second fault during an active outage.

Failure modes

Common mistakes during production triage include:

• Anchoring on the newest change without evidence.
• Confusing correlation with causation when a deploy and an outage happen near each other.
• Treating all retries as helpful even when they amplify load.
• Skipping request IDs and then losing the path through logs and traces.
• Restarting too early and erasing diagnostic state.
• Over-communicating speculation instead of facts and confidence levels.
• Ignoring partial impact where only one tenant, route, or payload shape is failing.

Interview drill

Use these interview questions to test practical understanding:

1. How would you decide whether an ASP.NET Core API is saturated or blocked on a downstream dependency?
2. What evidence would convince you to roll back a deployment versus disable a feature flag?
3. How do request IDs and correlation IDs help when logs, metrics, and traces disagree?
4. What would you include in the first incident update to stakeholders?
5. How do you preserve evidence while still mitigating quickly?
6. Describe a timeline you would build for a real incident review.

Practical exercise:

• Scenario A: p95 latency doubles, CPU is high, error rate is flat, and queueing grows.
• Scenario B: CPU is normal, one downstream call is slow, and timeouts spike only on one endpoint.

For each scenario, write:

• the leading hypothesis,
• the next two observations you would seek,
• the safest mitigation,
• the message you would send to stakeholders.

Worked scenario answers

The following numbers are invented exercise evidence, not observations from a live system. Start with existing dashboards, logs, traces, and deployment records. Do not run load generators, mutate production data, change pool settings, or restart services merely to test a hypothesis. A trace or dump collection has its own overhead and data-handling risks; follow the approved diagnostic procedure before collecting one.

Scenario A: high CPU and queue growth. Between 09:12 and 09:17 UTC, one instance’s CPU rises from 45% to 94%, p95 increases from 300 ms to 900 ms, and queued requests rise from 20 to 600 while 5xx stays at 0.2%.

• Leading hypothesis: this instance cannot keep up with the offered work. This is not yet proof of a code regression or ThreadPool starvation.
• Next observations: compare request rate/route/payload mix with healthy instances; correlate the interval with deployment history and an already available CPU profile or hot-path evidence. A fleet average can hide the affected instance.
• Mitigation choice: if an optional report endpoint dominates the new work, propose the runbook’s narrow throttle or feature disable. Roll back only if the affected revision and compatibility evidence support it; adding instances can worsen a shared dependency bottleneck.
• Example update: “One API instance has elevated latency since 09:12 UTC; error rate is unchanged. Queue growth and CPU are confirmed. We’re comparing route mix and revisions before choosing a scoped mitigation. Next update at 09:22 UTC.”

Scenario B: one slow downstream call. The checkout route times out, application CPU is 35%, and affected traces contain a 2.8-second inventory span against a 3-second end-to-end budget.

• Leading hypothesis: inventory-path delay consumes most of checkout’s budget. Do not announce an inventory outage solely from the span name.
• Next observations: compare the same dependency call in unaffected routes/instances and inspect timeout/connect/pool-wait error categories; check attempt counts to see whether retries are amplifying traffic. A pool wait in the caller can resemble a slow dependency.
• Mitigation choice: use an approved scoped traffic reduction or already-designed degraded mode if the business rules allow it. Do not increase retries or timeout limits without analyzing their resource impact. Failover is conditional on the alternative’s health, capacity, and consistency behavior.
• Example update: “Checkout timeouts are elevated; other routes appear healthy. Traces place most observed delay on the inventory call path, but the initiating cause is unconfirmed. We’re checking connection waits and retry volume. Next update at 09:22 UTC.”

Recovery check: An instance restart or falling 5xx count alone is not recovery. Verify the affected user journey, request success and latency under representative traffic, draining queues, and any missed/duplicated business work over the runbook’s observation window. Distinguish “mitigation applied” from “recovery confirmed” in the timeline.

Revision checklist

• [ ] Classify the symptom before proposing a fix.
• [ ] Separate observations from hypotheses in the incident channel.
• [ ] Correlate logs, metrics, and traces with request/correlation IDs.
• [ ] Test local saturation and dependency hypotheses, including combinations.
• [ ] Choose an authorized, compatible, scoped mitigation with a recovery check.
• [ ] Record an incident timeline as you investigate.
• [ ] Communicate impact, scope, mitigation, and next update time.
• [ ] Preserve evidence for the post-incident review.
• [ ] Close the loop with action items, owners, and deadlines.

Social snippets

• Short post: Production incident triage is about reducing uncertainty fast: classify the symptom, correlate logs/metrics/traces, mitigate safely, then learn from the incident.
• Discussion prompt: In an ASP.NET Core API outage, when do you rollback, when do you disable a feature flag, and what evidence helps distinguish or connect local saturation and dependency failure?

Reference

• Microsoft .NET ThreadPool starvation diagnostics