Incident management process

A B2B payments platform in New York (260 engineers in 28 teams, 2,100 customers, $4B processed a month, a 99.95% SLA with service credits) runs about 180 services on Kubernetes across two AWS regions, with a shared PostgreSQL cluster for the ledger. In the last 12 months: 31 customer-impacting incidents, median time to detect 22 minutes (customers detected 40% of them first), median time to mitigate 3 h 10 min, $1.3M paid in SLA credits, two incidents in which nobody was sure who was in charge for over an hour, and a CEO email about "outages we hear about from clients". On-call exists in 12 of the 28 teams, unpaid, fed by alerts from six different tools — 3,400 alerts a month, 85% of them noise; there is no severity scale, status-page updates are written by whoever is around, and postmortems happen for some incidents, in various formats, with action items that are rarely tracked (11 of 64 closed). A SOC 2 Type II audit in eight months will test incident response. Engineers push back against "carrying a pager for other teams' code".

Define the incident management process: severity levels and what each one triggers; roles (incident commander, communications, scribe, subject-matter responders) and how they are staffed 24x7 across 28 teams; the on-call structure, rotations, compensation and the rules for alert quality; detection and escalation paths; internal and customer communications (status page, account managers, regulators when required) with their timings; postmortems (when mandatory, format, blameless review, ownership and tracking of actions); the metrics and reviews that show whether it works; and how it is introduced across the teams without waiting for the audit.