Declare Early, Declare Loudly. Nobody Agrees What Counts As Major.

Somebody types "anyone else seeing 502s on the API?" into a channel at 11:40 at night, then sits there with the cursor blinking and does not type the word incident. Maybe it's the cache again. Maybe they can fix it before anyone important notices, and nobody has to be woken over a blip.

That pause is one of the most expensive things in operations, and no org chart has a line for it.

Twenty-five years in, I had the decision inside it framed wrong. I thought declaring was a judgment about size. It isn't, and there's a measurement from a different industry entirely that shows why.

The comfortable theory

Declare when it's big enough. Hence the severity matrix every org has — customer-facing or internal, revenue-affecting or not, P1 through P4, pick a row. It assumes "big enough" sits on the surface of the event, waiting to be read off by a competent person. Which puts the weight on individual judgment, and means that when someone hesitates for twenty minutes, the official explanation is that they judged poorly.

Three pictures and 178 professionals

Pre-hospital medicine has been arguing about this longer than we have, and somebody actually measured it.

In How and Why Do You Declare a Major Incident? (Prehospital and Disaster Medicine, 2008), Smart and Maconochie showed three pictorial scenarios to 178 UK emergency-response professionals and asked one question: would you declare a major incident here?

• Road traffic incident: 101 said yes — 57%
• Coach rollover: 82 — 46%
• Rail crash: 156 — 87%

Sit with the middle one. Shown an identical photograph of an overturned coach, trained professionals split roughly down the middle.

Experience did not resolve it. Among the 46 respondents with prior major-incident experience — 44 of whom had formal training too — the figures were 54%, 54% and 96%.

The limits are real: the paper is paywalled, I'm working from its abstract and published figures, and a photograph is not a shift. But the authors name the mechanism themselves. The core problem, they write, is "the interpretation and subjective nature of the word 'major'."

If trained responders looking at the same photograph split 54/46 on whether to declare, then "declare when it's major" is not a decision rule. It is a request for a judgment the situation cannot supply.

Swap the coach for "the portal is slow for some people in one region" and you have our version of the same photograph.

What declaring actually does

Here's the part I had backwards.

The Incident Command System is not a severity scale. It's an organizational chart already in effect whether or not anybody filled it in. FEMA's ICS review document states the rule directly: "Responsibility for functions that subordinates perform defaults to the next higher supervisory position until the supervisor delegates those responsibilities."

Google's SRE workbook gives the software version just as plainly: "the person who declares the incident typically steps into the IC role," and "By default, the IC assumes all roles that have not been delegated yet." There are three to go around — incident commander, operations lead, communications lead.

Read those together against 11:40 p.m. Before anyone has declared anything, somebody is already holding all three. Diagnosing is operations. Deciding how bad it is and who else is needed is command. Answering the director's DM is communications. Nobody appointed them; the roles default to whoever is standing there.

Declaring doesn't create that structure. It names one that already exists — and naming it is the only thing that lets you give two-thirds of it away.

Which is why the span-of-control figure matters more than it looks. FEMA puts the optimum at "one supervisor to five subordinates," as a guideline rather than a law. One engineer running command, operations and comms at once is past a sane ratio before the second responder joins the call.

PagerDuty's response documentation makes the split a rule: "Delegate all repair actions, the Incident Commander is NOT a resolver." Not because commanders are too senior to type — because command and diagnosis draw on the same attention, and whichever one you're doing, the other is being neglected. Treat that as a strong default rather than a law; Google's workbook explicitly lets an IC hand off command and take operations instead.

What the undeclared version costs

Google published its own worked example of skipping the step. A bug in Assistant version 1.88 "caused speaker recognition files to be fetched 50 times more often than expected." On Monday, May 22, 2017, the on-call developer halted the release at 25% of users. It restarted, passed 50% by May 31, and hit 100% on Saturday, June 3 — at which point users lost half their requests across the weekend. The workbook's own line: "The team did not declare an incident, but continued to troubleshoot the issue via 'normal' methods."

Thirteen days, on a defect known from day one with an owner watching it the whole time.

The same chapter carries the contrast. PagerDuty's NTP clock-drift incident, October 6, 2017: alert at 7:53 p.m., declared at 9:49 p.m., then ten-plus hours of managed response with the IC rotating every four hours — and very minimal customer impact. Declaring cost them a staffing rotation instead of a weekend.

Why it ended up this way

Because the system was built for fires and we inherited it sideways. Southern California, 1970: thirteen days of fire, sixteen deaths, over 500,000 acres burned. The finding afterward wasn't about equipment or tactics — by the program's own history, agencies arrived using different terminology and different structures and set up separate command posts for the same incident. The design that came out of it was renamed the Incident Command System in June 1974.

So it was never a way to rate how bad something is. It was a fix for too many capable people responding to one event without an agreed structure — a fair description of a bad night in production.

Where I would be careful

Declaring isn't free. Laura Maguire's Managing the Hidden Costs of Coordination argues that coordination carries real cognitive cost — an IC doing too much assessment starves coordination, and too much coordination starves assessment — and that ICS can become a bottleneck in a fast escalation. That's a practitioner analysis rather than a controlled study, and I hold it as a well-argued observation.

Early declaration is also necessary without being sufficient. In Atlassian's post-incident review of its April 2022 outage, a deletion script removed 883 sites belonging to 775 customers. Internal monitoring never fired, because "the sites were deleted via a standard workflow." The first customer ticket landed at 07:46 UTC, the major incident process triggered at 08:17, and it was escalated to Critical at 08:24 — half an hour from first external signal, which most organizations would take happily. A subset of those customers were still down fourteen days later, and Atlassian's own conclusion is that the program was "not designed to handle the depth, expansiveness, and duration of this type of incident." Fast declaration bought them a response. It did not buy them a restore.

What I would do

Replace the severity question with a structural one. "Is this major?" has no reliable answer — that's what the 178 professionals demonstrate. "Am I holding more than one role right now?" has one, instantly. Diagnosing and writing stakeholder updates in the same ten minutes means declare. The SRE book's criteria are a decent floor under that: a second team is needed, the outage is customer-visible, or an hour of concentrated analysis hasn't cracked it.

Make declaring cheap enough to be wrong. From the same chapter: "It is better to declare an incident early and then find a simple fix and close out the incident." If closing one out needs a form, a meeting and an explanation of why you overreacted, you've priced the hesitation in — and you'll get it at 11:40 p.m.

Count the ones you closed in ten minutes and treat that number rising as a health signal, not noise. It's the cheapest evidence you'll get that the pause is shrinking.

Separate declaring from commanding. Whoever declares holds command by default, so their first act is handing off one of the three roles — even if it's to one other person who is now writing things down.

I still pause. The difference is that I now know what I'm doing during it: three jobs at once, all of them badly, while deciding whether to admit that out loud.