Skip to content

Observability Maintenance

Last audited: 2026-05-30.

Use this runbook to keep backend, mobile, infrastructure, and operator docs in sync with the current observability behavior.

Current Request Tracing Contract

The system currently supports these privacy-safe tracing headers and telemetry fields:

Header Log/New Relic/telemetry field Purpose
x-request-id requestId Current backend request identifier.
x-correlation-id correlationId Logical workflow identifier shared across related calls.
x-spawn-id spawnId Spawned operation identifier for separating related calls inside a workflow.

Expected behavior:

  • Mobile API requests send all three headers.
  • Backend responses expose all three headers.
  • Backend logs include all three fields in MDC where request context exists.
  • Backend handled 5xx capture includes the tracing fields when available.
  • Backend outbound calls made through injected WebClient.Builder instances propagate the tracing headers.
  • Mobile API success and failure telemetry includes the tracing fields.
  • New Relic privacy filters allow these tracing keys and headers while continuing to scrub raw user, tenant, property, lease, payment, token, message, and secret data.

Routine Review

Run this review after observability, New Relic, telemetry, logging, API client, webhook, or deployment changes.

  1. Confirm backend request filtering still normalizes safe tracing IDs and replaces unsafe or missing values.
  2. Confirm CORS allows and exposes x-request-id, x-correlation-id, and x-spawn-id.
  3. Confirm backend logs and handled 5xx New Relic context carry the tracing fields.
  4. Confirm outbound backend clients propagate the tracing headers where the request context is available.
  5. Confirm mobile API middleware still sends the tracing headers and records the returned values in API telemetry.
  6. Confirm New Relic privacy allowlists include only low-risk tracing identifiers and do not allow raw domain identifiers or payload data.
  7. Confirm the operations docs still describe the current tracing behavior, alert flow, dashboards, and first-response checks.

Suggested Verification

Backend targeted checks:

mvn -q -Djacoco.skip=true \
  -Dtest=CorrelationIdFilterTest,RequestLoggingFilterTest,SecurityConfigCoverageTest,NewRelicHandledFailureReporterTest,ObservabilityWebClientCustomizerTest,GlobalExceptionHandlerTest \
  test

Backend broad check:

mvn -q test

Mobile targeted checks:

pnpm exec prettier --check api/apiClient.ts api/apiClient.test.ts utils/telemetry/telemetry.ts utils/telemetry/telemetry.test.ts utils/telemetry/telemetryEvents.ts utils/telemetry/newrelicPrivacy.ts utils/telemetry/newrelicPrivacy.test.ts
pnpm test api/apiClient.test.ts utils/telemetry/telemetry.test.ts utils/telemetry/newrelicPrivacy.test.ts utils/telemetry/telemetryEvents.test.ts --runInBand

Architecture checks when alerting, dashboards, or Terraform change:

terraform fmt -recursive -check
./scripts/tf.sh all-validate

Docs check:

./scripts/build-docs-site.sh

Manual Review Checklist

Before pushing observability changes:

  • Confirm no real secrets were added.
  • Confirm no raw user IDs, emails, addresses, Stripe IDs, tokens, property IDs, lease IDs, payment payloads, or free-text message content were added to metrics, analytics, logs, or New Relic tags.
  • Confirm backend metric tags are low-cardinality.
  • Confirm New Relic context is privacy-safe.
  • Confirm Firebase event names and properties are typed and documented.
  • Confirm Terraform changes were validated and not applied unless explicitly approved for the target environment.
  • Confirm backend and architecture repositories are on integration before pushing.

Incident Use

When diagnosing a live issue:

  1. Start from the symptom-specific operations runbook.
  2. Capture requestId, correlationId, and spawnId from the user-facing response, mobile telemetry, New Relic event, or backend logs when present.
  3. Use correlationId to follow related calls and spawnId to separate spawned operations inside the same workflow.
  4. Use timestamps, release SHA, route/workflow tags, Stripe event IDs, and CloudWatch metrics to fill gaps when a tracing ID is missing.