Skip to content

Incident Response

Last audited: 2026-05-30.

Severity

  • SEV1: production unavailable, payment/subscription corruption, broad data exposure, or destructive data loss.
  • SEV2: major feature broken for many users, webhook/payment processing delayed, repeated 5xx, or production deploy regression with workaround.
  • SEV3: limited user impact, noisy degradation, non-critical job failure, or integration-only incident.

Ownership

The first responder owns the incident until they explicitly hand off. For SEV1 and SEV2, assign:

  • Incident owner: coordinates decisions and timeline.
  • Technical lead: investigates and fixes.
  • Communications owner: updates users or stakeholders.
  • Scribe: records timestamps, evidence, and actions.

Acknowledgement Targets

  • SEV1: acknowledge within 10 minutes.
  • SEV2: acknowledge within 30 minutes.
  • SEV3: acknowledge during the same working day.

Initial Triage

  1. Confirm environment and start time.
  2. Check newest deploy, GitLab pipeline, New Relic incidents and Discord notification delivery, and CloudWatch backend logs. Confirm the alert reached the intended channel before treating an alarm as acknowledged.
  3. Capture requestId, correlationId, spawnId, Stripe event IDs, user/account references needed for support, and release SHA. Do not copy raw secrets, tokens, payment payloads, message bodies, or unnecessary personal data into the incident record.
  4. Decide whether to roll back, disable a workflow, replay a webhook, or continue diagnosis.

Rollback Triggers

Rollback or redeploy last known good when:

  • 5xx or health alarms start within 30 minutes of deploy.
  • New Relic shows new or regressed production errors tied to the release SHA.
  • Payment, subscription, login, or upload flows fail broadly.
  • The team cannot identify a safer mitigation within the acknowledgement target.

Customer Communications

Communicate for SEV1, user-visible SEV2, payment/subscription incidents, or data integrity risk. Include what is affected, current workaround, next update time, and final resolution.

Post-Incident

Within two working days, record:

  • Timeline and detection source
  • User/account impact
  • Root cause and contributing factors
  • What worked and what slowed diagnosis
  • Follow-up owners and due dates