How to Triage a Production Incident with AI

At 21:47 on a Thursday, the on-call rotation at OpenCurtain, an online ticketing platform, paged Marta Oyelaran because the payment callback timeout rate had crossed its five percent threshold and stayed there for three straight minutes. By 21:52 the number was 18 percent, roughly 3,400 checkout attempts were stuck waiting on a webhook that never arrived, and the support queue was filling with customers asking whether their concert tickets had actually been bought.

This workflow is what Marta runs in the first ninety minutes and then again the next morning. ChatGPT is not the pager and it is not the fix, but it is a fast scribe and a stubborn second reader. It turns a noisy dashboard into ranked hypotheses, drafts the status update while Marta is still in the terminal, and later produces a blameless postmortem she can circulate without starting a blame war.

What a Complete Incident Record Looks Like

An incident is not finished when the graph goes flat. It is finished when the record contains five things: a severity and a scope statement, a timeline of what happened and when, a root-cause hypothesis with the contributing factors listed separately, the customer-facing and internal communications, and a set of action items with owners.

For OpenCurtain that record is a page in the #inc-4821 channel plus a postmortem doc. The rule is simple. Every fact in the record is either something you observed or something you decided, and the two are labeled differently so the next reader can tell them apart.

Minute Zero: Frame Severity Before You Touch a Dashboard

The first prompt does not ask the model to solve anything. It asks for a triage frame so a half-awake on-call engineer stops staring and starts deciding.

You are an incident commander assistant for an online ticketing platform.
Alert: payment callback timeout rate above 5% for 3 minutes.
Money is involved, customers may have paid without receiving tickets, and a show
starts in four hours.
Give me:
1. A one-line severity recommendation with a reason (assuming a Sev-1 to Sev-3 scale).
2. A scope statement of what is likely affected and what is definitely not.
3. The first three questions I should answer before touching any code.
4. Who I should wake up right now, in priority order.
Answer in under 200 words. Do not speculate about root cause yet.

What you get: a calm severity call, a scope boundary, and a wake-up list. The instruction to hold off on root cause is deliberate. At 21:47 you need a decision and a second pair of hands, not a theory.

First Fifteen Minutes: Turn Noisy Signals into Ranked Hypotheses

Paste the raw signals, not a summary of them. Let the model generate competing explanations and then rank them by how cheaply each one can be checked.

Here are the live signals. Rank the plausible causes.
Signals:
- callback-worker p99 latency climbed from 400 ms to 18 s at 21:44.
- PSP webhook delivery volume doubled at 21:43, then tripled.
- checkout-api error rate 18%, all on the "confirm payment" step.
- A schema migration shipped at 21:30.
- No deploy to checkout-api in six hours.
Rank the top five causes by likelihood AND by how fast each can be confirmed or ruled out.
For each, give the single command or dashboard to check and what result would confirm it.
Flag any cause that could be making the others look worse.

What you get: a ranked, testable list instead of a guess. In this run the top item was the schema migration removing an index that the callback lookup relied on, with a PSP retry storm as the amplifier that turned a slow query into a timeout cascade.

Keep the model honest by asking it to state, for each hypothesis, what evidence would rule it out. A ranking you cannot falsify is just a hunch with a number next to it. Marta used the rule-out column to cross off the checkout-api deploy and the database failover in the first four minutes, which left the migration and the retry storm as the only live threads when the second engineer joined.

When a hypothesis points at a specific function, switching into debugging mode on that code is faster than staying in incident mode. The loop in our debug code with ChatGPT workflow is built for exactly that handoff.

While the Fix Is Rolling: Draft the Status Update

Communication is the part on-call engineers drop when they are busy, and it is the part customers notice when it is missing. Draft it in parallel with the mitigation, not after.

Draft a customer-facing status update for the OpenCurtain status page.
Facts to include: some payments are processing slowly, tickets may take a few minutes
to appear, no payment has been double-charged, a fix is being rolled out.
Rules:
- 60 words maximum.
- No technical jargon. No "root cause". No timeline promises yet.
- Reassure customers who already paid.
Then draft a 120-word internal update for the support team with what to tell users
right now and what NOT to promise.
Label both drafts clearly.

What you get: a short public note and a support script that keeps agents from inventing answers. The “what not to promise” section is the useful one, because it stops the team from committing to a refund policy nobody has approved.

After Mitigation: Write the Root Cause and the Contributing Factors

A single root cause is usually a convenient fiction. Force the model to separate the trigger from the conditions that let it become an outage.

Here is the raw timeline from the whole incident. Write a root-cause analysis.
Structure:
- Trigger: the change or event that started the cascade.
- Contributing factors: at least three conditions that turned a slow query into an outage.
- Detection gap: how long until we noticed, and why that long.
- Customer impact: recoveries, delayed tickets, and support volume, as ranges if exact
  numbers are not final.
- What went right: two things that limited the blast radius.
Keep it factual. Do not assign blame to any individual.

What you get: an analysis that names the missing-index migration as the trigger but also flags the absent index check in CI, the retry storm, and the timeout default of 20 seconds as contributing factors. That last one, a config default nobody had reviewed in a year, is exactly the kind of factor a single-cause writeup loses.

Next Morning: Draft the Blameless Postmortem

The postmortem is the deliverable. It should be readable by someone who was asleep through all of it.

Turn the analysis into a blameless postmortem document with these sections:
1. Summary for people who were not on call (5 sentences).
2. Impact in numbers.
3. Timeline as a table of time, event, and who observed it.
4. Root cause and contributing factors.
5. Action items: for each, an owner role, a due date window, and a type
   (prevent, detect, mitigate, or document). Write at least six action items.
Tone: neutral, systems-focused, no individual named as at fault.
End with three questions the team should discuss at the review meeting.

What you get: a postmortem with a timeline table and six or more owned action items. The “document” action type matters, because the cheapest fix here was writing down the 20-second timeout default so the next person does not rediscover it during an outage.

Read the draft once for anything that reads as a verdict on a person rather than on the system. Swap “the engineer missed the alert” for “the alert routed to a channel that is not monitored at night”, and the same fact stops being a blame sentence and starts being a fixable gap. That single habit is what makes the review meeting short enough to actually happen.

For the weekly pattern that catches repeat incidents, the habit in our AI weekly review workflow turns these postmortems into a running reliability backlog instead of a folder of forgotten docs.

Deliverables Checklist Before You Close the Incident

  • Severity and scope statement agreed by the incident commander.
  • Timestamped timeline of observations from page to recovery.
  • Ranked hypothesis list with a disposition for each one.
  • Customer status updates and an internal support script.
  • Root cause plus at least three contributing factors, kept separate.
  • Blameless postmortem with six or more owned action items.

Common Traps in AI-Assisted Incident Response

Postmortem prematurely. Asking the model for a root cause before you have the timeline invites a confident story built on the alert text alone. Gather signals first, theorize second.

Trusting a single root cause. Insist on contributing factors. Most outages need three things to go wrong, and fixing only the trigger leaves the next one armed.

Letting the model write customer messages unattended. Draft fast, but a human approves anything on the status page. The model does not know your refund policy.

Blame language creeping back in. Phrases like “the engineer forgot” undo a blameless review. Ask for a systems-focused tone and check the final text yourself.

If you also want the daily handoff covered, the shape in our daily standup summary workflow complements this one nicely.

Sources & Notes

The triage-to-postmortem sequence is our own framing of standard incident response practice, reviewed on 2026-10-09. OpenCurtain, Marta Oyelaran, and incident #inc-4821 are illustrative examples, and the latency and error figures are representative rather than measured. Severity scales and postmortem templates vary by organization, so map the action-item types onto whatever your team already uses. Language models change between versions, so re-check any prompt that leans on your specific tooling or on-call policy.

声明:1.本站所有文章,如无特殊说明或标注,均为本站原创发布。本网站中【文本生成/文生图】使用了DeeSseek人工智能生成内容技术, 算法来源说明:本网站未自行开发、训练或部署深度合成算法,所使用的人工智能算法来自:DeepSeek、Chatgpt、Google Gemini。 2.为了网站加载速度优化,本站预览图片进行压缩处理,所以有些不是太清晰,并不影响AI提示词生图、AI设计等内容。3.任何个人或组织,在未征得本站同意时,禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益,可联系我们进行处理。--GUUNN.COM