Production Incident Triage for ASP.NET Core APIs: a Practical On-Call Guide

Learning outcome

By the end of this lesson, you should be able to triage an ASP.NET Core API incident without guessing: classify the symptom, separate observation from hypothesis, correlate logs/metrics/traces with request and correlation IDs, identify whether you are seeing saturation or dependency failure, choose a safe mitigation, and communicate clearly while preserving evidence for the post-incident review.

Intuition

Production triage is not “find the bug fast”; it is “reduce uncertainty fast.” In an ASP.NET Core API, the same user-visible symptom can come from very different causes: a code regression, thread-pool starvation, a downstream database slowdown, a bad deploy, an exhausted connection pool, or an infrastructure fault. Good responders avoid the two classic mistakes: making every incident a code problem, and treating every error spike as a harmless blip.

The fastest path is to answer four questions in order:

1. What is broken for users? For example: 5xx rates, timeouts, slow p95 latency, partial failures, or data inconsistency.
2. What changed? A deployment, config update, feature flag change, dependency degradation, scaling event, or certificate rotation.
3. Where is the bottleneck? App CPU, memory, GC, request queueing, thread pool, connection pools, database latency, or upstream errors.
4. What is the safest mitigation? Roll back, disable a feature, shed load, throttle, or fail over—without destroying evidence.

A useful mental model is: symptom → scope → impact → bottleneck → mitigation. You do not need a final root cause to stabilize the service. You do need enough evidence to avoid making the situation worse.

Deep dive

Start by classifying the incident into one of a few buckets.

Symptom classification

A practical decision table helps separate the first-order signal:

Observation versus hypothesis

Treat every statement as one of two things:

• Observation: measurable or directly visible evidence.
• Hypothesis: an explanation that still needs confirmation.

For example:

• Observation: error rate rose from 0.2% to 8% after 09:12 UTC.
• Hypothesis: the new deployment introduced a regression.

This distinction matters because incident rooms often fill with confident but unverified narratives. Keep hypotheses testable.

Correlating logs, metrics, and traces

The most useful triage evidence is cross-signal correlation:

• Logs tell you what the app reported, especially exceptions, status codes, and validation failures.
• Metrics tell you whether the service is saturated, failing, or simply slow.
• Traces show where latency or failure accumulates across the request path.

For ASP.NET Core APIs, request IDs and correlation IDs are essential. If a client provides an ID, preserve it; otherwise generate one and ensure it appears consistently in request logs, downstream calls, and trace context. This lets you answer: “Are these failures all from the same request class?” and “Did the downstream service reject only the calls that had a certain shape?”

Saturation versus dependency failure

These are the two most common triage branches.

Saturation indicators

• CPU remains high under load.
• Requests queue up and latency climbs before failures.
• Thread pool or connection pool waits increase.
• GC pauses or memory pressure align with the slowdown.
• Error codes may be secondary symptoms, such as timeouts and cancellations.

Dependency failure indicators

• App CPU may be normal while failures rise.
• Traces show one downstream span dominates total latency.
• Errors cluster around a particular backend, tenant, or API path.
• Retrying increases pressure on the dependency and can worsen the incident.

A simple rule: if the app is busy but the dependency is slow, the app may be the victim. If the app is busy because it is amplifying retries or expensive work, the app may be part of the problem.

Safe mitigation options

Mitigation should minimize blast radius and preserve evidence.

• Rollback a recent release when the timing and scope fit a regression.
• Disable a feature flag when impact is isolated to a feature path.
• Reduce traffic by throttling or shedding noncritical work if the service is saturating.
• Fail over or reroute if an upstream dependency or zone is unhealthy and the architecture supports it.
• Increase capacity carefully if the issue is confirmed saturation and the service can safely scale.

Avoid “blind restart” as a first move. A restart can erase diagnostic evidence, mask a bad deployment, and briefly hide the true failure mode.

Incident timeline and communications

Build a timeline as you go:

• Detection time
• First confirmed user impact
• First observation
• Major hypothesis changes
• Mitigation applied
• Recovery confirmed

Stakeholder communication should be short and factual:

• What is impacted?
• Who is impacted?
• What do we know?
• What are we doing now?
• When is the next update?

Do not promise a cause before it is known. Do provide confidence levels and next checkpoints.

Post-incident follow-up

Recovery is not closure. A good follow-up includes:

• a concise incident summary,
• impact window and user effect,
• contributing factors,
• what evidence was decisive,
• what mitigation worked,
• preventive actions with owners and due dates.

If you repeatedly see the same incident shape, the fix is often not “be faster next time” but to improve the system’s observability, rollback path, feature flag hygiene, or capacity guardrails.

Production code

Below is a conceptual triage checklist you can adapt into your runbook. It is intentionally non-executable and vendor-neutral.

1. Confirm impact
   - Which endpoints, tenants, regions, or client versions are affected?
   - Is the symptom 5xx, 4xx, latency, or data inconsistency?

2. Capture the time window
   - When did the first user-visible symptom begin?
   - What changed shortly before that time?

3. Check the three signals together
   - Logs: exception type, status code, correlation/request ID
   - Metrics: error rate, latency percentiles, CPU, memory, queue length
   - Traces: slow spans, failing dependency calls, retry amplification

4. Decide saturation vs dependency failure
   - If CPU/queue/thread pool pressure is high, suspect saturation.
   - If the app is idle but downstream spans are slow or failing, suspect dependency failure.

5. Choose the safest mitigation
   - Roll back the last suspected change
   - Disable the smallest feature scope that stops the impact
   - Reduce load or fail over if the architecture supports it

6. Communicate
   - State observations, hypotheses, mitigation, and next update time

7. Preserve evidence
   - Keep logs, traces, and deployment records for the review

Code walkthrough

This lesson does not include an executable sample because incident triage is primarily an operational skill. In practice, your “walkthrough” should read like a runbook:

• First, verify the symptom at the user boundary rather than inside a single process.
• Next, inspect whether the issue aligns with a deployment or config change.
• Then, correlate a small set of affected request IDs across logs and traces.
• Finally, determine whether the bottleneck is internal saturation or an external dependency.

A strong reviewer can explain why a safe rollback is often preferable to an immediate code hotfix: it shortens time-to-mitigation and reduces the chance of introducing a second fault during an active outage.

Failure modes

Common mistakes during production triage include:

• Anchoring on the newest change without evidence.
• Confusing correlation with causation when a deploy and an outage happen near each other.
• Treating all retries as helpful even when they amplify load.
• Skipping request IDs and then losing the path through logs and traces.
• Restarting too early and erasing diagnostic state.
• Over-communicating speculation instead of facts and confidence levels.
• Ignoring partial impact where only one tenant, route, or payload shape is failing.

Interview drill

Use these interview questions to test practical understanding:

1. How would you decide whether an ASP.NET Core API is saturated or blocked on a downstream dependency?
2. What evidence would convince you to roll back a deployment versus disable a feature flag?
3. How do request IDs and correlation IDs help when logs, metrics, and traces disagree?
4. What would you include in the first incident update to stakeholders?
5. How do you preserve evidence while still mitigating quickly?
6. Describe a timeline you would build for a real incident review.

Practical exercise:

• Scenario A: p95 latency doubles, CPU is high, error rate is flat, and queueing grows.
• Scenario B: CPU is normal, one downstream call is slow, and timeouts spike only on one endpoint.

For each scenario, write:

• the leading hypothesis,
• the next two observations you would seek,
• the safest mitigation,
• the message you would send to stakeholders.

Revision checklist

• [ ] Classify the symptom before proposing a fix.
• [ ] Separate observations from hypotheses in the incident channel.
• [ ] Correlate logs, metrics, and traces with request/correlation IDs.
• [ ] Decide whether the bottleneck is saturation or dependency failure.
• [ ] Prefer rollback or feature-flag mitigation over risky ad hoc changes.
• [ ] Record an incident timeline as you investigate.
• [ ] Communicate impact, scope, mitigation, and next update time.
• [ ] Preserve evidence for the post-incident review.
• [ ] Close the loop with action items, owners, and deadlines.

Social snippets

• Short post: Production incident triage is about reducing uncertainty fast: classify the symptom, correlate logs/metrics/traces, mitigate safely, then learn from the incident.
• Discussion prompt: In an ASP.NET Core API outage, when do you rollback, when do you disable a feature flag, and how do you prove saturation versus dependency failure?