Incident Management: From Detection to Post-Mortem Excellence

A clear severity classification system ensures proportional response to every incident. TaptiPM uses four severity levels: SEV-1 (complete service outage or data breach affecting all users — response: all-hands, 15-minute update cadence, executive notification), SEV-2 (major feature degradation affecting >25% of users — response: on-call team, 30-minute update cadence, stakeholder notification), SEV-3 (minor feature issue affecting <25% of users — response: on-call engineer, hourly updates, internal tracking), SEV-4 (cosmetic or non-impacting issues — response: normal sprint backlog, no incident process).

Severity is determined by impact (number of affected users) and urgency (rate of degradation). A slow memory leak that will cause an outage in 4 hours is SEV-2 now, not SEV-4 because the system is still running. Classification should happen within 5 minutes of detection — it is better to over-classify and de-escalate than to under-classify and scramble when the situation worsens.

Effective on-call rotations balance responsiveness with sustainability. TaptiPM's HRMS module manages on-call schedules with three principles: primary and secondary on-call for every service (secondary takes over if primary does not acknowledge within 10 minutes), weekly rotations to prevent burnout (no engineer should be on-call for more than 7 consecutive days), and compensatory time off after high-severity incidents (engineers who work a SEV-1 on a weekend get a day off the following week).

Escalation paths are defined per service and per severity. A SEV-2 on the authentication service escalates to the platform team lead after 30 minutes without resolution, then to the VP of Engineering after 60 minutes. These escalation paths are documented in TaptiPM's runbook for each service and trigger automatically — no human needs to remember to escalate during the stress of an active incident.

Incident communication follows the Incident Commander model. One person (not necessarily the most senior engineer) is designated as Incident Commander and is responsible for: coordinating responders, maintaining the incident timeline, communicating status to stakeholders, and making decisions about severity changes and escalations. The IC does not debug — they manage the response process, freeing engineers to focus on resolution.

External communication (to clients and end users) follows templated formats: initial acknowledgment ("We are aware of an issue affecting [service]. Our team is investigating."), progress updates every N minutes based on severity ("We have identified the root cause and are deploying a fix. Estimated resolution: 30 minutes."), and resolution notice ("The issue has been resolved. All services are operating normally. We will publish a detailed post-mortem within 48 hours."). TaptiPM's incident module generates these communications from templates and posts them to the status page automatically.

Every SEV-1 and SEV-2 incident requires a post-mortem within 48 hours. The post-mortem is blameless: it focuses on system failures (missing monitoring, inadequate testing, unclear runbooks) rather than individual failures ("John pushed a bad config"). Blameless does not mean accountable — the post-mortem identifies what went wrong and assigns owners to corrective actions. But it avoids the blame that causes engineers to hide problems, avoid risky improvements, or add defensive process that slows everyone down.

TaptiPM's post-mortem template includes five sections: Timeline (minute-by-minute account of the incident), Root Cause Analysis (using the Five Whys technique to reach systemic causes), Impact Assessment (affected users, duration, financial impact), Corrective Actions (specific, assignable actions with deadlines), and Lessons Learned (insights that apply beyond this specific incident). Corrective actions automatically become sprint backlog items with a "post-mortem" tag, ensuring they are tracked to completion alongside feature work.