Incident Triage¶
Current monitoring stack: New Relic and CloudWatch. Use Warsaw time for incident windows and record the corresponding UTC boundaries.
| Symptom | First place | Corroborating evidence |
|---|---|---|
| Backend ERROR or unhandled exception | New Relic Backend ERROR logs, APM errors and traces |
Structured CloudWatch logs and stack trace; release and request correlation |
| API unavailable or slow | New Relic backend health, ALB/ECS or integration host metrics | Public health endpoint, CloudWatch logs, deployment status |
| Mobile crash | New Relic mobile occurrences and crash condition | Exact build/release, readable frames, native diagnostics |
| Admin JavaScript error | New Relic browser errors and source maps | Exact served build and private source-map verification |
| Failed payment or webhook | New Relic critical workflow and Stripe backlog/dead-event metrics | Stripe event status and backend CloudWatch logs |
| Infrastructure lifecycle failure | New Relic MamNajemAwsEvent and AWS polling metrics |
ECS/EventBridge/SNS/Lambda CloudWatch evidence |
| Failed delivery or ingestion | New Relic notification records and telemetry freshness | Forwarder Lambda logs, workflow/destination status and IAM scope |
| Failed deployment or scheduled maintenance | Exact GitLab pipeline and failed job | Release marker and post-deploy New Relic/CloudWatch window |
- Take a current read-only snapshot using the local
mamnajem-alert-triageskill. Query recovered incidents as well as currently open issues; follow pagination. - Check release/deployment timestamps and correlate each activation with logs. A closed issue demonstrates recovery, not a permanent fix.
- Identify expected tests separately. Routine checks should use an isolated policy or scoped maintenance muting; preserve detection of genuine errors.
- Classify the finding, reproduce its source and recommend an exact fix. Implement only when remediation is authorized.
- After an authorized fix, verify New Relic receipt, condition behavior, Discord provider delivery records and the post-deploy CloudWatch window. Follow guarded delivery.
Backend log coverage¶
Both environments have a critical Backend ERROR logs condition:
SELECT count(*) FROM Log
WHERE `entity.name` = 'mamnajem-integration-backend' AND level = 'ERROR'
Use the production application name for production. Count above zero qualifies, evaluated at least once in a 300-second duration with 60-second aggregation. The generic exception handler logs at ERROR with its throwable. A WARN stack trace or a trace printed outside the logger is not covered by this severity filter; compare CloudWatch with New Relic before declaring full coverage. The separate APM condition excludes expected HTTP 4xx responses and retains server/status-less failures.
See observability and alerts for live coverage and remaining verification gates. Physical iPhone production crash verification remains deferred by the owner.