Observability Maintenance¶
Last audited: 2026-05-30.
Use this runbook to keep backend, mobile, infrastructure, and operator docs in sync with the current observability behavior.
Current Request Tracing Contract¶
The system currently supports these privacy-safe tracing headers and telemetry fields:
| Header | Log/New Relic/telemetry field | Purpose |
|---|---|---|
x-request-id |
requestId |
Current backend request identifier. |
x-correlation-id |
correlationId |
Logical workflow identifier shared across related calls. |
x-spawn-id |
spawnId |
Spawned operation identifier for separating related calls inside a workflow. |
Expected behavior:
- Mobile API requests send all three headers.
- Backend responses expose all three headers.
- Backend logs include all three fields in MDC where request context exists.
- Backend handled 5xx capture includes the tracing fields when available.
- Backend outbound calls made through injected
WebClient.Builderinstances propagate the tracing headers. - Mobile API success and failure telemetry includes the tracing fields.
- New Relic privacy filters allow these tracing keys and headers while continuing to scrub raw user, tenant, property, lease, payment, token, message, and secret data.
Routine Review¶
Run this review after observability, New Relic, telemetry, logging, API client, webhook, or deployment changes.
- Confirm backend request filtering still normalizes safe tracing IDs and replaces unsafe or missing values.
- Confirm CORS allows and exposes
x-request-id,x-correlation-id, andx-spawn-id. - Confirm backend logs and handled 5xx New Relic context carry the tracing fields.
- Confirm outbound backend clients propagate the tracing headers where the request context is available.
- Confirm mobile API middleware still sends the tracing headers and records the returned values in API telemetry.
- Confirm New Relic privacy allowlists include only low-risk tracing identifiers and do not allow raw domain identifiers or payload data.
- Confirm the operations docs still describe the current tracing behavior, alert flow, dashboards, and first-response checks.
Suggested Verification¶
Backend targeted checks:
mvn -q -Djacoco.skip=true \
-Dtest=CorrelationIdFilterTest,RequestLoggingFilterTest,SecurityConfigCoverageTest,NewRelicHandledFailureReporterTest,ObservabilityWebClientCustomizerTest,GlobalExceptionHandlerTest \
test
Backend broad check:
mvn -q test
Mobile targeted checks:
pnpm exec prettier --check api/apiClient.ts api/apiClient.test.ts utils/telemetry/telemetry.ts utils/telemetry/telemetry.test.ts utils/telemetry/telemetryEvents.ts utils/telemetry/newrelicPrivacy.ts utils/telemetry/newrelicPrivacy.test.ts
pnpm test api/apiClient.test.ts utils/telemetry/telemetry.test.ts utils/telemetry/newrelicPrivacy.test.ts utils/telemetry/telemetryEvents.test.ts --runInBand
Architecture checks when alerting, dashboards, or Terraform change:
terraform fmt -recursive -check
./scripts/tf.sh all-validate
Docs check:
./scripts/build-docs-site.sh
Manual Review Checklist¶
Before pushing observability changes:
- Confirm no real secrets were added.
- Confirm no raw user IDs, emails, addresses, Stripe IDs, tokens, property IDs, lease IDs, payment payloads, or free-text message content were added to metrics, analytics, logs, or New Relic tags.
- Confirm backend metric tags are low-cardinality.
- Confirm New Relic context is privacy-safe.
- Confirm Firebase event names and properties are typed and documented.
- Confirm Terraform changes were validated and not applied unless explicitly approved for the target environment.
- Confirm backend and architecture repositories are on
integrationbefore pushing.
Incident Use¶
When diagnosing a live issue:
- Start from the symptom-specific operations runbook.
- Capture
requestId,correlationId, andspawnIdfrom the user-facing response, mobile telemetry, New Relic event, or backend logs when present. - Use
correlationIdto follow related calls andspawnIdto separate spawned operations inside the same workflow. - Use timestamps, release SHA, route/workflow tags, Stripe event IDs, and CloudWatch metrics to fill gaps when a tracing ID is missing.