Incident Management Interview Questions
5 questions for junior, mid-level and senior Incident Management interviews, with model answers, code examples and common mistakes. Answer each one out loud first, then reveal the answer.
On-Call Rotation and Escalation Basics
// interview question
You're about to go on-call for the first time. In your own words, what is an on-call rotation, and why do teams bother setting up a formal escalation policy instead of just pinging whoever happens to be online when something breaks?
Sample answer
An on-call rotation is a schedule that decides who is responsible for responding to production alerts at any given moment. Instead of "everyone is sort of responsible," one named person owns the pager for a defined window, then it rotates to the next person. An escalation policy is the set of rules for what happens when that person doesn't respond. A normal policy looks like: page the primary, wait a few minutes for an acknowledgement, and if nothing comes back, page the secondary. Still nothing? Page the on-call manager. Why formalize it instead of pinging whoever's around? A few reasons. First, it removes the bystander problem. If you message a busy channel at 3am, everyone assumes someone else has it and nobody picks it up. A rotation makes one person clearly accountable. Second, it removes a single point of failure. People sleep, lose signal, or are mid-flight. The escalation chain guarantees that if the first responder is unreachable, the alert still reaches a human instead of dying silently. Third, it protects people. A defined schedule means you know exactly when you're responsible and when you're truly off. Ad-hoc "just ping someone" means everyone is always a little bit on-call, which burns people out. The one distinction worth knowing on day one: acknowledging an alert means "I've got this, stop escalating." Resolving it means "the problem is actually fixed." New people forget to ack, the system thinks nobody responded, and it escalates to their manager at 3am. Don't be that person.
Why this matters
This is a warm-up question to confirm a junior candidate actually understands the basics before going on-call. You want to hear the two core concepts (rotation = who responds, escalation = what happens if they don't) and at least one reason the structure exists beyond "so we know who to call." The strongest junior answers mention the ack-vs-resolve distinction or the single-point-of-failure angle without being prompted. If they think on-call means they personally have to fix everything, that's a coaching flag, not a disqualifier.
Code examples
A basic escalation policy expressed as tiers and timeouts
# What "page the next person if nobody answers" actually means
escalation_policy:
name: payments-team
rules:
- level: 1
notify: primary_on_call
escalate_after: 5m # no ack in 5 min -> go to level 2
- level: 2
notify: secondary_on_call
escalate_after: 5m
- level: 3
notify: engineering_manager
escalate_after: 10m
# If level 3 still doesn't ack, loop back to level 1 once,
# then open a major incident. Never let an alert dead-end.Common mistakes to avoid
- Thinking on-call means you personally fix every problem. The job is to respond, triage, and pull in the right people. Escalating to someone who knows the system is a correct move, not a failure.
- Not acknowledging alerts. If you don't ack, the system assumes you're unreachable and escalates up the chain, often waking up your manager for something you were already handling.
- Confusing severity with escalation. Severity is how bad it is; escalation is who gets contacted and when. A low-severity alert shouldn't page the whole chain.
Likely follow-ups
- What's the difference between acknowledging an alert and resolving it, and why does it matter?
- It's the end of your shift and there's still an open incident. How do you hand off cleanly?
- What would you want in place before you ever get your first page?
Answer out loud first, then check yourself against the model answer.
Designing an On-Call Schedule
// interview question
You've got six engineers split across two time zones and you need 24/7 coverage. How would you actually design the rotation? Walk me through the trade-offs you'd weigh.
Sample answer
With two time zones, the goal I'd push for is follow-the-sun, so nobody gets paged at 3am on a regular basis. Each region covers their own daytime, and the pager hands off when their day ends. Concretely: split the six into two groups of three by region. Each region runs a weekly rotation during their working hours. When region A's day ends, coverage passes to region B who is just starting theirs. Overnight pages for region A land on region B, who is awake. That alone removes most of the pain of on-call. I'd also run a primary plus secondary, not primary only. The secondary is the safety net for when the primary misses a page. With three people per region you can do that without crushing anyone: roughly one week as primary and a different week as secondary out of every three. Rotation length: weekly is the usual sweet spot. Daily means constant context switching and bad handoffs. Monthly means one bad month ruins someone and they lose touch with the system between turns. A week is long enough to keep context and short enough to recover. The trade-offs I care about: - Fairness. Load should be even, and the schedule has to bend for PTO and holidays through overrides, not by silently dumping shifts on whoever's available. - Handoffs. Every shift change needs a written handoff: open incidents, risky changes shipping, anything being watched. A follow-the-sun handoff that loses context is worse than one tired person who remembers everything. - Coverage gaps. With only three per region, watch for the case where one person is on primary and there's nobody fresh for secondary. Never schedule the same person as both primary and secondary at once. - Compensation. On-call is work. Whether it's pay, time off, or reduced project load that week, it has to be acknowledged or people quietly resent it. The anti-pattern I'd avoid: a single 24/7 rotation in one region where someone eats overnight pages every shift. With two time zones you've been handed the fix for free, so use it.
Why this matters
This tests whether a mid-level engineer can reason about real scheduling constraints rather than reciting "we use PagerDuty." The detail that separates strong answers is recognizing that two time zones is an opportunity for follow-the-sun, not just a complication. Listen for primary plus secondary, sane rotation length with a justification, and the human factors: fairness, PTO overrides, handoffs, and compensation. Bonus points if they flag that a six-person team means rotations come around often and that's a retention risk.
Code examples
Follow-the-sun schedule with two regional layers (Terraform + PagerDuty)
# Two layers, each restricted to its region's working hours.
# PagerDuty fills overnight gaps in one layer from the other.
resource "pagerduty_schedule" "payments_24x7" {
name = "Payments 24x7"
time_zone = "UTC"
# Region A (e.g. Europe) covers 07:00-19:00 UTC
layer {
name = "EU daytime"
start = "2024-01-01T07:00:00Z"
rotation_virtual_start = "2024-01-01T07:00:00Z"
rotation_turn_length_seconds = 604800 # weekly
users = [var.eu_user_1, var.eu_user_2, var.eu_user_3]
restriction {
type = "daily_restriction"
start_time_of_day = "07:00:00"
duration_seconds = 43200 # 12h
}
}
# Region B (e.g. US) covers 19:00-07:00 UTC
layer {
name = "US daytime"
start = "2024-01-01T19:00:00Z"
rotation_virtual_start = "2024-01-01T19:00:00Z"
rotation_turn_length_seconds = 604800
users = [var.us_user_1, var.us_user_2, var.us_user_3]
restriction {
type = "daily_restriction"
start_time_of_day = "19:00:00"
duration_seconds = 43200
}
}
}Adding a PTO override so the schedule bends instead of breaking
# Engineer on the EU layer is out next week.
# Don't reshuffle the rotation, just override the window.
curl -s -X POST \
"https://api.pagerduty.com/schedules/PSCHED01/overrides" \
-H "Authorization: Token token=$PD_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"override": {
"start": "2024-06-10T07:00:00Z",
"end": "2024-06-17T07:00:00Z",
"user": { "id": "PUSER02", "type": "user_reference" }
}
}'Common mistakes to avoid
- Running a single 24/7 rotation where the same person absorbs overnight pages. With two time zones, follow-the-sun removes most night pages for free.
- Skipping the secondary. Primary-only means one missed page (phone on silent, dead zone) and the alert goes unanswered.
- Picking a rotation length without thinking. Daily rotations wreck handoffs and context; monthly rotations burn people out and let them lose touch between turns.
- Treating fairness and PTO as afterthoughts. No override mechanism means the schedule quietly becomes unfair and people stop trusting it.
Likely follow-ups
- One engineer goes on PTO during their scheduled week. How do you cover it without dumping it on one person?
- What would you change about this design if the team grew to 30 engineers?
- How do you actually compensate people for being on-call, and what happens if you don't?
Answer out loud first, then check yourself against the model answer.
Designing an Escalation Policy
// interview question
An alert fires at 3am and pages the primary on-call. Walk me through what your escalation policy should do from that moment, step by step, and tell me what failure modes you're designing around.
Sample answer
Step one: the primary gets paged through a channel that will actually wake them, push plus an escalating phone call, not just a Slack message. The policy starts an acknowledgement timer, say 5 to 8 minutes. Step two: if the primary acks, escalation stops and they own it. If the timer expires with no ack, the policy automatically pages the secondary. Same ack timer. Step three: if the secondary also doesn't ack, it escalates to the on-call manager or engineering manager. For a real major incident, this is also where you'd auto-create an incident channel and pull in an incident commander. That's the happy path. The failure modes I'm actually designing around are the interesting part: - The first responder is asleep or has no signal. That's why there's a secondary, and why the notification escalates from push to phone call rather than relying on one quiet channel. - The alert dead-ends. Every policy needs a final catch-all so an unacked alert eventually reaches a manager or opens an incident. It should never just stop. - The wrong team gets paged. Routing should send the alert to the team that owns the failing service, based on a service or team label, not to a single central pager. - Non-urgent things page humans. Severity-based routing matters here. A SEV1 pages immediately and notifies the manager and incident channel. A SEV3 should become a ticket or a Slack message, not a 3am phone call. If it doesn't need a human right now, it isn't a page. - Alert storms. When one outage trips fifty alerts, grouping and deduplication keep it to one notification instead of fifty phone calls. The edge case I'd call out: don't let escalation loop forever. Loop the chain once, maybe twice, then hand off to a guaranteed catch-all like a major-incident process. And every page should carry a link to its runbook, so the person you woke up at 3am has somewhere to start instead of a blank screen.
Why this matters
This is the core mid-level question on the topic. A weak candidate describes only the happy path (primary, then secondary, then manager). A strong one talks about what they're defending against: dead-ended alerts, single points of failure, wrong-team routing, and non-actionable pages. The phrase to listen for is severity-based routing, because that's where most teams' escalation pain actually comes from. Auto-creating an incident channel, attaching runbooks, and capping the escalation loop are senior-leaning signals.
Code examples
Escalation policy with ack timeouts and a manager catch-all (Terraform + PagerDuty)
resource "pagerduty_escalation_policy" "payments" {
name = "Payments escalation"
num_loops = 2 # loop the chain twice, then stop dead-ending
rule {
escalation_delay_in_minutes = 6 # ack window before next level
target {
type = "schedule_reference"
id = pagerduty_schedule.payments_primary.id
}
}
rule {
escalation_delay_in_minutes = 6
target {
type = "schedule_reference"
id = pagerduty_schedule.payments_secondary.id
}
}
rule {
escalation_delay_in_minutes = 10
target {
type = "user_reference"
id = var.engineering_manager_id # final catch-all
}
}
}Severity-based routing so only real urgency pages a human (Alertmanager)
route:
receiver: default-ticket
group_by: ['alertname', 'service']
group_wait: 30s
group_interval: 5m
routes:
# SEV1: page immediately, also notify the incident channel
- matchers: [severity="critical"]
receiver: pagerduty-page
continue: true
- matchers: [severity="critical"]
receiver: slack-incidents
# SEV3 and below: ticket, not a 3am phone call
- matchers: [severity=~"warning|info"]
receiver: default-ticket
# Don't page for every pod when the whole cluster is down
inhibit_rules:
- source_matchers: [alertname="ClusterDown"]
target_matchers: [severity="warning"]
equal: ['cluster']Common mistakes to avoid
- No acknowledgement timeout, or one so long the alert sits unanswered for 20 minutes before escalating.
- No final catch-all. If the last person in the chain misses the page, the alert silently dies and nobody finds out until customers complain.
- Paging on non-actionable or low-severity alerts. Every false page at 3am makes the next real one easier to ignore.
- Relying on a single notification channel like Slack. If their phone is on silent, a Slack ping does nothing.
Likely follow-ups
- How does your escalation policy change for a SEV1 versus a SEV3?
- What would you do differently when you've got 100 services instead of one?
- How do you stop a non-urgent alert from waking up an entire escalation chain?
Answer out loud first, then check yourself against the model answer.
Reducing On-Call Alert Fatigue
// interview question
Your on-call engineers are burning out. They're getting 40 to 50 pages a shift and they tell you most of it is noise they just ack and ignore. How do you fix this?
Sample answer
First I'd measure before touching anything. I'd pull pages grouped by alert name, by service, and by time of day, and split them into actionable versus auto-resolved. Almost always a handful of alerts generate most of the volume, and a big chunk auto-resolve before anyone does anything. That tells me where to aim. Then I work through it in order of impact: 1. Delete or downgrade non-actionable alerts. The test is simple: if a page doesn't require a human to do something right now, it is not a page. Make it a ticket or a dashboard panel. This alone usually cuts the worst noise. 2. Alert on symptoms, not causes. Page on what users feel: elevated error rate, latency past the SLO, a queue that's backing up. Don't page on CPU at 90 percent or memory pressure, because those are often fine and they fire constantly. If high CPU isn't hurting users, it isn't an incident. 3. Add duration guards. A `for:` clause stops flapping alerts that fire and clear every two minutes. If it can't stay broken for five minutes, it probably isn't worth a page. 4. Group and inhibit. When one outage trips twenty alerts, Alertmanager grouping collapses them into one notification, and inhibition rules suppress the downstream noise. If the cluster is down, don't also page for every pod on it. 5. Move to multi-window burn-rate alerts for SLOs. A static threshold either pages too early or too late. A fast-burn plus slow-burn pair catches a real budget burn quickly while staying quiet on blips. 6. Route by severity. Only genuine urgency pages. Everything else goes to Slack or a ticket queue that gets triaged during the day. 7. Set a budget and defend it. Pick a target like fewer than two pages per off-hours shift, and treat going over it as a bug with an owner, reviewed at every on-call handoff. Without a target, noise creeps back. The edge cases that bite people: don't just raise thresholds, because that hides real problems instead of removing noise. Watch the auto-resolving alerts, since a thing that flaps and clears all night is a real bug masquerading as harmless. And every remaining page must have a runbook. If a page has no runbook, either it's not worth paging on, or you owe it one.
Why this matters
This is the senior-level question that reveals whether someone has actually owned alerting in production. The tell is sequencing: a strong answer starts with measuring, then cuts and downgrades alerts, and treats symptom-based alerting and severity routing as the structural fixes. Weaker answers jump straight to "raise the thresholds" or "add a dashboard," both of which hide problems rather than fix them. Mentioning a pages-per-shift budget, multi-window burn-rate alerts, or making alert tuning part of the on-call retro are clear senior signals.
Code examples
A noisy cause-based alert rewritten as a quiet symptom-based one
# BAD: pages constantly, often when nothing is actually wrong
- alert: HighCPU
expr: instance:node_cpu_utilisation:rate5m > 0.9
labels: { severity: critical }
annotations:
summary: "CPU above 90% on {{ $labels.instance }}"
# GOOD: pages only when users are actually being hurt
- alert: CheckoutErrorBudgetFastBurn
# 2% of requests failing, sustained, on a user-facing path
expr: |
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m])) > 0.02
for: 5m # duration guard kills flapping
labels: { severity: critical }
annotations:
summary: "Checkout error rate above 2% for 5m"
runbook: "https://runbooks.internal/checkout-5xx"Multi-window burn-rate SLO alert: fast and slow, fewer false pages
# Page only when both a fast and a slow window agree the budget
# is burning. Catches real incidents, ignores momentary blips.
- alert: CheckoutSLOFastBurn
expr: |
(
slo:error_rate:ratio_rate5m{service="checkout"} > (14.4 * 0.001)
and
slo:error_rate:ratio_rate1h{service="checkout"} > (14.4 * 0.001)
)
for: 2m
labels: { severity: critical }
annotations:
summary: "Checkout burning 30-day budget ~14x too fast"Find your noisiest alerts so you fix the right ones first
# Top alert names by firing volume over the last 24h.
# Start cutting from the top of this list.
topk(10,
sum by (alertname) (
count_over_time(ALERTS{alertstate="firing"}[24h])
)
)Common mistakes to avoid
- Just raising the thresholds. That hides real degradation instead of removing noise, and the next real incident slips under the new bar.
- Silencing alerts without fixing the underlying cause. A muted alert that's still flapping is a bug you've agreed to stop looking at.
- Adding more dashboards instead of cutting alerts. Dashboards don't reduce pages; deleting non-actionable alerts does.
- Alerting on causes like CPU and memory instead of user-facing symptoms. Resource metrics belong on dashboards, not on the pager.
Likely follow-ups
- How do you decide whether a given alert should page someone versus go to a ticket?
- After you make these changes, how would you measure whether on-call fatigue actually went down?
- What's a multi-window burn-rate alert, and why is it better than a single static threshold?
Answer out loud first, then check yourself against the model answer.
Scaling an On-Call Program Across Many Teams
// interview question
You've been asked to design the on-call program for an org that grew from one team to fifteen in a year. Right now it's a free-for-all. What does a healthy on-call program look like at that scale, and how would you measure whether it's working?
Sample answer
The core shift is from central on-call to service ownership. The team that builds a service runs its on-call. You build it, you run it. A single central ops team can't own fifteen teams' services, because the people getting paged have no power to fix the root cause, so the noise never goes away. So I'd route alerts to the owning team based on a service or team label, and make sure every service has a named owner. No orphan services, no central pager catching everything. Then I'd standardize the things that should be consistent and let teams own the rest. Standardize: severity definitions, the incident process (who declares an incident, the incident commander role, comms), the paging tooling and a golden-path alerting setup, runbook expectations, and the compensation policy. Let each team own their own schedule and their own alerts within that frame. Consistency where it matters, autonomy where it doesn't. The part most people skip is governance through metrics instead of vibes. I'd track on-call health across teams: - Pages per shift, and specifically off-hours pages per shift. This is the burnout signal. - Percent of pages that were actionable versus auto-resolved. - Time to acknowledge, plus MTTA and MTTR. - Alerts with no runbook attached. - On-call satisfaction from a short recurring survey. Then set thresholds that trigger a review. For example, any team over a set number of off-hours pages per week gets a dedicated session to fix their alerting, treated as real work with time allocated, not a nag. On-call health becomes something with an owner and a dashboard, like a product. The edge cases at this scale: - Teams too small to staff 24/7 alone. Pool a few related teams into a shared rotation, or use follow-the-sun across regions. Don't pretend a four-person team can sustainably cover nights forever. - Shared infrastructure. The platform team owns the platform, but you need crisp boundaries so they don't become the dumping ground for every alert that's vaguely infra-shaped. Alerts route to the service owner first. - Culture. None of the metrics work without blameless postmortems. If reporting a noisy alert or a bad night gets you blamed, people hide it and the data goes dark. The failure mode I'd watch hardest: a central team quietly staying responsible for everyone's reliability. That breaks the feedback loop where the team that can fix the noise is the team that feels it.
Why this matters
This is a staff or senior-plus question about organizational design, not tooling. The single idea that has to be present is service ownership: you build it, you run it, with alerts routed to the owning team. From there, strong candidates separate what to standardize (severity, incident process, tooling, comp) from what to leave to teams (schedules, their own alerts), and they govern with on-call health metrics rather than just MTTR. Listen for off-hours page rate as the burnout signal, the small-team pooling problem, and the platform-team-as-dumping-ground trap. Tying it back to blameless culture shows they understand why the metrics survive contact with reality.
Code examples
Route alerts to the owning team, not a central pager (Alertmanager)
# Every alert carries a team label set by the owning service.
# Routing fans out to each team's own receiver. No central catch-all
# that silently absorbs everyone's noise.
route:
receiver: platform-fallback # only for truly unlabeled alerts
group_by: ['team', 'service']
routes:
- matchers: [team="payments"]
receiver: pd-payments
- matchers: [team="search"]
receiver: pd-search
- matchers: [team="checkout"]
receiver: pd-checkout
# Catch orphan services early: alert when something pages
# the fallback, because that means a service has no owner.
# - alert: UnownedServicePaging
# expr: count(ALERTS{team=""}) > 0On-call health SLIs you can actually put on a dashboard
# Off-hours pages per team this week (the burnout signal).
# Assumes a recording rule tags pages fired outside 08:00-20:00.
sum by (team) (
increase(oncall_pages_total{window="off_hours"}[7d])
)
# Share of pages that were actionable (not auto-resolved).
sum by (team) (increase(oncall_pages_total{actionable="true"}[30d]))
/ sum by (team) (increase(oncall_pages_total[30d]))
# Teams over the off-hours budget: these get a tuning session.
sum by (team) (increase(oncall_pages_total{window="off_hours"}[7d])) > 5Common mistakes to avoid
- Keeping a central ops team responsible for everyone's services. The people getting paged can't fix the root cause, so the noise is permanent.
- Measuring only MTTR and ignoring on-call load. A team can have great MTTR while quietly burning out from page volume.
- Mandating tooling without a migration path or a golden-path setup, so each team reinvents alerting badly.
- Treating on-call quality as each team's private business with no org-wide visibility. Without shared metrics, the worst rotations stay invisible until people leave.
Likely follow-ups
- How do you stop the central platform team from becoming the dumping ground for every alert?
- Which metrics would warn you that on-call is unhealthy before people start quitting?
- How do you handle a team that's too small to staff 24/7 coverage on its own?
Answer out loud first, then check yourself against the model answer.
More interview topics
Also worth your time on this topic
How to Build an Effective On-Call Rotation and Escalation Policy
Your phone buzzed at 3:14 AM for a disk warning that auto-resolved by 3:16. Nobody fixes the alert. The next person on rotation hates their life. Here is how to build on-call schedules, escalation policies, and alert rules that respect your engineers.
On-Call Rotation and Escalation Policy Checklist
A practical checklist for designing on-call schedules, defining escalation paths, and cutting alert fatigue so your team can sleep at night and still respond fast when things break.
60-120 minutes
Complete Monitoring Stack Setup
Deploy and configure Prometheus and Grafana for comprehensive application and infrastructure monitoring.
150 minutes