Observability And Alerts¶
Monitoring status: 2026-10-05.
New Relic account 8573192 (EU) is the selected destination for operational
metrics, logs, distributed traces, backend errors, admin JavaScript errors and
mobile crashes. Firebase remains the optional product analytics provider.
Product Outcomes reporting¶
Product reports use shared backend aggregate definitions and a separate private Product Outcomes dashboard. They require a dedicated runtime reporting query identity; the workstation administration key must never be injected into the backend. Publication stays disabled until runtime identity, effective retention, and billing are confirmed. The current change set prepares collection/history, admin reports, aggregate publication, and isolated non-paging warnings; it does not extend the deployed operational cutover evidence below into reporting acceptance. See the implementation brief and reporting runbook.
New Relic setup and cutover¶
The integration backend is deployed with Spring Boot 4.0.8 and source revision
34c777248f73bfcb51746f1327e0344aafa4bb7b. Pipeline 2910938493 passed its
security, build, API compatibility and deployment gates. Local Maven verification
passed 6,660 tests with no failures or errors. Public health, liveness, readiness
and subscription plans returned HTTP 200 after deployment.
New Relic receipt is verified for backend transactions, distributed spans,
application logs, business metrics and health (3=UP), host metrics, system,
Caddy and PostgreSQL logs, and backup/deployment success (1). A real PostgreSQL
backup reached S3 and its success metric was read back. New Relic recorded
successful Discord notifications for a real failed deployment and its recovery.
The agent requires Fluent Bit as an explicit dependency because the bootstrap
uses --no-install-recommends. Caddy forwarding covers both the bootstrap and
TLS log paths; Java agent logs are queried by entity.name because they do not
carry the infrastructure collector's environment attribute.
All 49 MamNajem CloudWatch metric alarms were removed after their New Relic replacements and Discord delivery were verified: 17 integration and 32 production. The New Relic policies contain 30 integration and 39 production conditions. AWS log groups, metric filters, SNS topics and ECS EventBridge rules remain; deleting the alarms removes their recurring alarm charges.
The existing event-forwarder Lambda now sends safe infrastructure events to
New Relic, which evaluates the alerts and delivers Discord notifications.
A labelled MAMNAJEM_MIGRATION_TEST published to each environment's SNS topic
was received as MamNajemAwsEvent, and each workflow recorded a successful
activated notification. This check uses the real SNS subscription and Lambda
without failing or restarting a production task.
Coverage includes integration and production Cognito/Lambda errors and
throttles, SNS delivery failures, ECS deployment failures and unexpected task
exits, backend restart bursts, and account notification errors. A restart burst
means more than one backend startup within a rolling 15-minute window; a
single expected deployment startup is allowed. Stale export processing already
increments data.export.archive.failed, so the export-failure condition covers
it without relying on an unpublished stale gauge.
Backend, admin and mobile use New Relic in integration and production. Deployment and runtime verification have passed for the authorized cutover; the owner deferred the physical iPhone production crash check.
| Surface | Current result | Remaining gate |
|---|---|---|
| Integration backend and host | Live New Relic telemetry, backup/deployment metrics and Discord delivery verified | No remaining deployment gate |
| Admin | Revision f591a69d1d9eefcee26620b3138fbeeec28aa7a5 deployed to both environments; live login form, marker, served asset, security headers and authenticated activation reset verified; production page view and scrubbed JavaScript error received in New Relic |
No remaining deployment gate |
| React Native | Revision d836145f passed all four NVIDIA CI shards; signed integration build 991 and production build 997 delivered; production Android crash and ten readable framework frames received, with successful Discord delivery |
Owner confirmed readable Android JavaScript file/function names; physical iPhone production crash verification deferred by the owner |
| Production | Backend pipeline 2910968044 and mobile root 2911478424 passed; live backend and admin monitoring, cloud polling and Discord verified; Android build 997 uploaded to its internal track, iOS build 997 observed as VALID in its separate production TestFlight app |
No remaining deployment gate; physical iPhone production crash verification deferred by the owner |
Admin integration pipeline 2907572052 passed build, security, source-map,
smoke and deployment checks. The live verifier independently confirmed the
sealed source revision, runtime-contract checksum and security headers. A real
protected GitLab job authenticated with the reset controller before AWS writes;
its cleanup disabled integration activation and cleared the scoped activation
SHA. Production activation remained disabled. The temporary inactive schedule
was removed after the one-shot deployment; its pipeline evidence is retained.
The controller runs on the existing integration host using architecture revision
9f6cc425. Its GitLab key is restricted to the
admin project and the read/update operations needed to verify jobs and disarm
activation. The service reads its protected systemd credential from the
read-only credential mount. It rejects unsafe credential paths or permissions.
A production startup check found that the admin client still rejected
production environment settings before authentication and diagnostics could
initialize. Revision f591a69d binds each environment to its exact admin and
API URLs and retains rejection of cross-environment settings. The new production
acceptance regression failed before the fix. After the fix, 134 application
tests, 49 delivery helper tests, 34 Python checks, a production build and a local
browser login check passed. Integration pipeline 2911418673 and production
pipeline 2911448639 passed. Both live login forms, build markers, served assets,
security headers and authenticated activation resets were verified. An anonymous
production browser session produced a PageView and a scrubbed
MamNajemMigrationProbe JavaScript error, both tagged with production and
release mamnajem-admin@f591a69d1d9eefcee26620b3138fbeeec28aa7a5.
A fresh anonymous browser session produced PageView and JavaScriptError
events for integration browser application 538909644. The controlled
MamNajemMigrationProbe was received with the deployed release, environment
integration and message Diagnostic error, confirming scrubbing. This probe
verifies browser ingestion; it is not evidence of a mobile crash or production
receipt. Source maps are uploaded privately and read back against the exact
asset URL, release and build commit. A duplicate upload is accepted only after
that read-back; the source-map listing API does not expose a content checksum.
React Native verification¶
The owner confirmed readable JavaScript file/function names in a New Relic
occurrence from integration Android build 1.1.16+2000000908. This closes the
UI-only JavaScript stack check. Physical iPhone production crash verification
was deferred by the owner on October 4, 2026; it has not been demonstrated.
Integration uses the New Relic React Native SDK for native crashes, handled
exceptions and JavaScript errors. SDK 1.9.1 reports JavaScript errors through
MobileJSError. Crash and handled-error alert queries include that event and
separate fatal from nonfatal errors. Architecture revision 7c85f0d3 passed
pipeline 2907765835; its New Relic query configuration was applied to both
environments. This does not establish a production application rollout.
The obsolete mobile EXPO_PUBLIC_NEW_RELIC_LICENSE_KEY CI variable was removed from the
global, integration-build and integration-e2e-device scopes, with absence
confirmed by read-back. Historical New Relic events remain available. Backend, admin and mobile
project-level New Relic DSN, organization and project variables were subsequently
removed and absence verified. Shared credentials and historical New Relic projects
were preserved.
Signed Android build 1.1.16+2000000904, source 88dd18ba, produced a controlled
native crash after diagnostic consent was enabled. New Relic received the crash
for application 538909505; occurrence read-back returned ten readable Android
framework frames. Notification history confirmed successful delivery of its
incident to mamnajem-integration-discord. The shell-induced framework crash
proves native ingestion and readable framework frames; it does not prove
application ProGuard deobfuscation. This release is unminified, and its artifact
contract records that no ProGuard map is required.
Signed Android build 1.1.16+2000000908, source 68d39fdc, produced controlled
JavaScript network errors with bundle identity
pl.mamnajem.integration@1.1.16+2000000908. New Relic received MobileJSError
events and successfully delivered the handled-error incident to Discord. Its
exact JavaScript source map was uploaded privately and read back. The mobile
JavaScript errors view shows the ApiFailure and Error groups, but readable
JavaScript frames were confirmed readable by the owner in the UI occurrence.
The classic exception API returned no frames and did not establish that result. Connectivity was
restored, diagnostic consent withdrawn, fixtures cleaned and the owned
emulator stopped after the runtime probe.
Symbol publication is bound to each signed artifact's native version and bundle identity. Android requires a ProGuard map only for obfuscated bundles. iOS requires its dSYM archive and JavaScript map. Build, upload and diagnostic-job success alone do not prove runtime symbolication. The physical iPhone production crash and readable-stack check is deferred by the owner.
Production Android build 1.1.16+2000000997, source d836145f, was installed
from the exact signed CI APK after its checksum and release identity were
verified. On an owned disposable NVIDIA emulator, its anonymous login screen
rendered and diagnostic consent was seeded only in local fixture storage.
A controlled Android platform crash reached New Relic application 538909741
with the production environment and exact JavaScript bundle identity.
Occurrence read-back returned ten readable Android framework frames, and a
production mobile session was received. New Relic notification history also
confirmed successful delivery of its real crash incident to
mamnajem-production-discord. This verifies platform Java crash
ingestion, not NDK or application ProGuard symbolication. The APK is unminified.
Consent was disabled and verified before the owned emulator was removed; no
production user or backend data was created or changed.
Android delivery gate¶
Mobile revision 536a545107b5083204dd126d8c4bd69c389b57f8 passed the complete
integration delivery in
pipeline 2910646711.
The root, workload, Android release and iOS release pipelines all succeeded.
JUnit verification confirmed all 220 required selectors in their assigned
shards: 210 passed, ten expected skips and zero failures. All 5,388 application
tests passed across 746 suites; quality, security and compatibility checks passed.
Android 1.1.16+2000000965 was observed on the integration Play alpha track,
and iOS 1.1.16 (965) completed TestFlight processing.
Before retrying that existing commit, a complete local run used the exact application and instrumentation APKs from its earlier CI attempt, four fresh NVIDIA GPU emulators and the tested runner storage settings. It passed the same 220-selector inventory with the same ten expected skips. The current application tests, runner checks and quality checks also passed locally. This retry required no additional mobile code push.
The local MamNajem runner 54582069 has a limit of four and two Docker tmpfs
mounts: /opt/android-avd uses rw,size=10g, and /tmp/android-unknown uses
rw,size=10g,mode=0700. The second mount holds QEMU's writable disk overlays;
putting only the AVD directory in RAM leaves those writes on the host filesystem.
A local storage reproduction produced a provider-guard cleanup timeout. The
complete local and CI runs passed with both mounts in RAM. All four live CI
containers were checked for these mounts, NVIDIA GLES rendering and host GPU
processes; software-rendering fallback was disabled. The runner setting is in
its private workstation configuration and must be preserved during runner
reconfiguration.
Each shard uses a distinct emulator port and nine distinct owned fixture identities. All 36 identities were disjoint in the successful CI run. Bounded fixture cleanup passed in all four shards, and all owned emulators stopped. Future mobile changes require the complete local gate before an integration push, then terminal CI verification and the applicable signed release checks.
After the owner installed NVIDIA driver 595.91.07, the exact Android CI image
booted its emulator locally with usable KVM and the NVIDIA RTX 4080 SUPER
OpenGL renderer. The desktop now uses display :0; the protected project
variable DISPLAY=:0, scoped to integration-e2e-device, overrides the older
job default. The private Xauthority mirror was refreshed, and its existing
desktop-session watcher remains enabled. MamNajem GPU runner 54582069 is
online and unpaused, with both RAM mounts and hardware-only rendering
preserved. The existing integration pipeline was retried at the same tested
mobile revision, without another source push. Workload pipeline 2911399249
passed all four device shards and its release quality gate: 223 selectors,
213 passes, ten expected skips, zero failures and zero errors. Each shard ran
on local runner 54582069, and each live emulator reported the NVIDIA RTX
4080 SUPER OpenGL renderer. The root, workload, Android release and iOS release
pipelines all succeeded. Android 1.1.16+2000000991 was observed on the
integration Play alpha track, and iOS 1.1.16 (991) was observed as VALID in
the separate integration TestFlight app.
The owner confirmed readable Android JavaScript frames. A green build does not
establish physical iOS crash reporting; the owner deferred that production check.
Both iOS environments now use the protected project setting
IOS_CLEAN_DERIVED_DATA=1. Integration initially restored cached Xcode output
and failed the exact JavaScript source-map selection check after a successful
archive. Its clean build at the same source revision passed the New Relic
source-map upload. Production already used this clean-build setting. Keep the
exact-map guard enabled; archive success alone is insufficient evidence.
The dependency gate verifies the pinned node-forge fix for
GHSA-86w9-cpqp-85rv and the
pinned upstream braces depth-validation patch for
GHSA-vfj7-8cjw-p6xm. These
are validated patches, not advisory suppressions. Patch, lockfile, installed
runtime-file hashes and focused regressions are bound into the audit receipt;
altered evidence or another unresolved high/critical finding fails the gate.
Architecture revision 9f6cc425 installed the separate protected docs reset
controller on the existing integration host. Docs revision 4cd5f8ea passed
pipeline 2908731679 and publication verification. Its authenticated deployment
receipt proves that activation was disabled and its source SHA cleared after
deployment; the controller seal remains bound to the deployed source. Production
deployment is not established by these receipts.
The production admin runtime contract binds browser application 538909745 to
EU account 8573192, with protected file and checksum variables verified by
read-back. Its controller environment, endpoint and audience are configured.
Backend, architecture and admin revisions are now promoted and verified in
production. Mobile revision d836145f was promoted by fast-forward merge request
!6 after both its integration and merge-request checks passed. Production
root 2911478424, Android release 2911484352 and iOS release 2911483906
all succeeded. Signed Android 1.1.16+2000000997 is uploaded to the
production internal track; signed iOS 1.1.16 (997) was observed as VALID in
the production TestFlight app.
Empty panels are not evidence of application health or crash-free operation.
| Environment | Application prefix | Dashboard | Alert policy |
|---|---|---|---|
| Integration | mamnajem-integration- |
mamnajem-integration-observability |
mamnajem-integration |
| Production | mamnajem-production- |
mamnajem-production-observability |
mamnajem-production |
Application suffixes are backend, admin, mobile-android and mobile-ios.
The integration host is mamnajem-integration-api. Open the private
integration overview
for the verified backend and host surfaces. Production rollout remains manual.
New Relic alert delivery goes directly through an environment-filtered workflow and secure webhook destination to the existing Discord channel. Notifications cover open, acknowledged and resolved issues, respect fully muted issues and disable Discord mentions. The integration notification test was delivered.
From architecture, inspect then apply the environment configuration:
node scripts/configure-newrelic-discord.cjs --environment integration
node scripts/configure-newrelic-discord.cjs --environment integration --apply
node scripts/configure-newrelic-observability.cjs --environment integration
node scripts/configure-newrelic-observability.cjs --environment integration --apply
node scripts/configure-newrelic-aws.cjs
node scripts/configure-newrelic-aws.cjs --apply
The commands reconcile named resources and refuse ambiguous duplicates. The
Discord helper reads the existing AWS webhook secret without printing it. Add
--test only when one explicitly labelled Discord notification is intended.
Preparing New Relic production resources does not deploy production applications.
Private credentials live in ~/.config/mamnajem/newrelic.env, owned by the
operator with mode 0600. NEW_RELIC_LICENSE_KEY is the ingest key for agents;
NEW_RELIC_API_KEY is the User API key for NerdGraph and symbol uploads. CI names
the latter NEW_RELIC_USER_API_KEY. Mobile app tokens are platform-specific,
protected environment inputs. Never put private keys in source or terminal output.
Useful integration queries:
FROM SystemSample SELECT count(*) WHERE entityName = 'mamnajem-integration-api' SINCE 20 minutes ago
FROM Log SELECT count(*) WHERE environment = 'integration' OR `entity.name` = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM Log SELECT count(*) WHERE environment = 'integration' AND service = 'postgres-backup' SINCE 30 hours ago
FROM Metric SELECT latest(mamnajem.backup.success) WHERE environment = 'integration' SINCE 30 hours ago
FROM Transaction SELECT count(*) WHERE appName = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM Span SELECT count(*) WHERE appName = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM MamNajemMetric SELECT latest(value) WHERE environment = 'integration' AND metricName = 'health' SINCE 20 minutes ago
Health values are 3=UP, 2=OUT_OF_SERVICE, 1=DOWN, 0=UNKNOWN. Host and
backend health alerts detect loss of a previously received signal after five
minutes; the daily backup allows 30 hours. Loss-of-signal detection needs an
initial sample, so validate first receipt as part of deployment.
The dashboard covers requests, latency, errors, mobile crashes, admin errors,
logs, spans, host CPU/memory/disk, backup/deployment and account ingest. The
integration paging policy uses critical thresholds only. The integration host
memory condition is disabled; host memory remains visible on the dashboard.
The account-wide ingest budget alert
uses NrMTDConsumption and pages above 80% of the reported monthly free
allowance: above 80 GB while the allowance is 100 GB. It divides the latest
monthly GigabytesIngested by the reported freeLimit; hourly release/E2E bursts
alone do not page. Consumption is reported hourly and can lag by approximately
three hours; the 10-minute event timer can further delay notifications. This
leaves 20% allowance headroom and does not impose a billing hard cap. See
New Relic usage queries.
The owner confirmed the Free plan with no payment
method in the account UI; the API cannot independently verify that billing
state. Continue checking calendar-month usage and billing settings separately;
consumption reporting can lag incoming data.
Mobile diagnostics require consent. Session replay, automatic HTTP-body capture, web-view instrumentation, mobile log forwarding, offline storage and background reporting are disabled. Diagnostic messages and stack URLs are sanitized; opt-out stops the native agent and clears identity. Re-enable after shutdown requires an app restart. CI binds JavaScript maps, Android mappings and iOS dSYMs to the exact release artifact before store gates. An opted-in device crash with readable symbols remains required runtime evidence.
The admin agent disables cookies, replay, automatic Ajax collection, session traces and log forwarding. It sanitizes errors, uploads source maps to the EU endpoint, reads them back by release and URL, and excludes maps from public ZIPs. The Java agent obfuscates SQL, strips exception messages, excludes sensitive attributes and limits logs/spans. Host samples run every 60 seconds and process samples are disabled. Avoid forwarding Java logs again through a second collector.
Cutover verification covers deployed backend transactions, spans, logs and custom metrics, browser errors, native Android crashes and Discord delivery. Production ECS, ALB, RDS and Lambda coverage comes from the linked New Relic AWS integration; the integration Lightsail agent covers its own host. The production ingest secret and execution-role grant are provisioned and the backend rollout is verified. Historical CloudWatch logs and New Relic events are retained. Platform Lambda and RDS logs remain in AWS; cloud polling supplies their metrics and inventory rather than forwarding all platform logs. Physical iPhone production crash verification is deferred by the owner.
RDS creates /aws/rds/instance/mamnajem-production-postgres/postgresql itself
when log export starts, with no expiry. Terraform does not manage it because the
guarded apply rejects imported resources, so its 30-day retention was set
directly with aws logs put-retention-policy on 2026-11-08. A recreated
database or log group needs the same command again.
Manual production preparation¶
architecture/terraform/aws-production now declares the retained
mamnajem-production-new-relic-license-key secret container. Its value is never
managed by Terraform. A dedicated new_relic_ingest inline policy grants the
existing execution role access to this secret only, leaving its existing runtime
secret grant unchanged during monitoring preparation.
new_relic_enabled defaults to true after authorized provisioning of its
AWSCURRENT value outside Terraform. Use the
new_relic_license_secret_arn output for protected backend
NEW_RELIC_LICENSE_SECRET_ARN.
new_relic_aws_polling_enabled now defaults to true, preserving the applied
role. The authorized scoped apply created a metrics and inventory role with no credential or application
data access. Its trust is restricted to New Relic AWS account 754728514883 and
external ID 8573192. Link new_relic_polling_role_arn in New Relic EU, enable
ECS, RDS and ALB polling at 300 seconds or longer, then verify received service
samples before adding their alerts. Polling still uses AWS CloudWatch metrics
and may incur AWS API charges; New Relic Free does not remove those charges.
The owner authorized production deployments on 2026-10-04. The reviewed
preparation was applied as four creates: the ingest-secret container, its exact
execution-role grant, the New Relic polling role and its metrics policy. The
license was provisioned privately into AWSCURRENT; protected backend
NEW_RELIC_LICENSE_SECRET_ARN points to that container. No compute capacity,
updates, replacements or deletions were included.
A separate guarded one-create apply adds read-only Lambda inventory permissions
for the retained Cognito functions. The IAM-role mirror normalization accepts
only its inline-policy projection and null-to-empty tags; trust, role name,
session limits and all other attributes remain exact. Lambda polling is scoped
to functions tagged Application=mamnajem in eu-west-1, covering both
environments. Linked New Relic cloud account 71700 polls ECS, RDS, ALB, Lambda
and SNS metrics every 900 seconds and inventory every 3,600 seconds. Each poll is
a billed CloudWatch GetMetricData read: at 300 seconds it cost about 14 dollars a
month, so on 2026-11-08 polling moved to 900 seconds (about a third of that). The
alert windows for these metrics are 900 seconds to match, so AWS-metric alerts
react up to 15 minutes later; ALB unhealthy targets needs three consecutive polls,
while a full outage still pages on the first poll. SNS extended
inventory is disabled. Its metadata permissions are restricted to the two
MamNajem alert topics. Production ECS/RDS/ALB metrics and Cognito function
inventory have been received; dormant functions need no invented zero metrics.
Production Discord workflow delivery was tested successfully. The production
admin build, asset, headers and authenticated activation reset are verified.
Backend revision c953f0cc passed integration after a production packaging
failure: CI now transfers the New Relic agent directory with the JAR. A local
Docker build using the transferred-artifact layout passed, and all 139 CI helper
tests passed. Container vulnerability scanning then blocked 13 high and one
critical finding in Jackson 2/3, Bouncy Castle and SCRAM. Revision 34c77724
patches those dependencies; its full local Maven gate passed 6,660 tests with
zero failures, API compatibility passed and the patched container scan returned
zero high or critical findings. Production pipeline 2910968044 subsequently
passed deployment and runtime verification. New Relic receives transactions,
spans, logs, health (3=UP) and business metrics from the production backend.
Instrumented startup took approximately 164 seconds on the existing 0.5 vCPU/1 GB
tier. ECS health-check grace is now 300 seconds; desired count remains one and
the deployment circuit breaker still rolls back failures. Architecture revision
ad7ac8a4 passed both integration and production CI.
Signed production Android build 1.1.16+2000000973 and its New Relic diagnostic
artifacts passed verification and the build was uploaded to the production
internal track. Owner Google access allowed the missing production publisher
quota and Remote Config permissions to be repaired through a scoped four-create
IAM apply. Both environments' IAM grants were read back; no compute resources
were created. The production iOS build 1.1.16(973) passed signing and diagnostic
verification after its scoped Firebase Analytics setting was corrected and
derived-data cache was cleared. Its source map and dSYM upload passed, and
TestFlight upload succeeded, and Apple processing returned VALID.
Integration uses bundle pl.mamnajem.integration, App Store Connect app
6801274157; production uses pl.mamnajem, app 6801273572. Both branches
deliver to their own TestFlight app. Release observation checks the exact
processed TestFlight build. Public required-update publication is a separate
opt-in action requiring the exact publicly distributed build before changing
Firebase Remote Config.
Mobile revision d836145facc87f169787f75b6c3d7dfee451bc44 adds that production
publication distinction. Its local gate passed lint, types, formatting, 5,397
application tests, 1,189 general CI helper tests (three expected skips), and 443
Android helper tests (one expected skip). Its integration root 2911196168,
workload 2911399249, Android release 2911407696 and iOS release 2911407698
all passed, including the current 223-selector device inventory. The exact
revision then passed merge-request checks and was promoted by fast-forward MR
!6. Production root 2911478424, Android release 2911484352 and iOS release
2911483906 all passed. Both signed production platforms delivered build 997:
Android to pl.mamnajem internal testing and iOS to the separate production
TestFlight app, with Apple processing VALID. Their New Relic source-map
checks passed; iOS also passed its exact dSYM check. Public store launch and
mandatory-update publication remain separate guarded actions. The production
Android runtime crash and Discord delivery are verified above. Physical iPhone
production crash reporting and readable-stack verification are deferred by the
owner.
The alert retirement adds no compute capacity or secret containers. It reuses the account ingest secret and the existing event-forwarder Lambda. The earlier ingest-secret container retains its AWS storage charge, and cloud polling, CloudWatch metrics/logs, SNS and Lambda use can still incur AWS charges. All 49 billed CloudWatch alarm resources are removed, rather than merely disabling their actions. Their historical log data and source metrics remain; deleted alarm resources no longer expose their alarm history.
The forwarder can decrypt the ingest secret through Secrets Manager only; its KMS grant is restricted to that secret encryption context and the existing key.
New Relic subscription cost remains $0 within the Free plan's 100 GB monthly allowance. On October 5, observed collection suggested 30–50 GB per month, with about 40 GB as the planning estimate; sustained release/test bursts would increase this. Check actual calendar-month consumption. Free-plan ingestion can stop at the allowance, so usage alerts provide warning rather than a billing cap or a guarantee of uninterrupted monitoring.
References: New Relic pricing, infrastructure log forwarding, NRQL alerts.
Retained AWS collection and historical data¶
Applications and alerts use New Relic. Historical New Relic projects and events remain available, while their application configuration is removed. CloudWatch service metrics remain the AWS polling source, and log groups retain their configured history. All MamNajem CloudWatch metric alarm resources are removed. The sections below describe the current investigation paths.
Primary Alert And Error Surfaces¶
| Service/tool | What it catches | What to look for |
|---|---|---|
| New Relic errors | Backend exceptions, admin JavaScript errors, mobile crashes, handled mobile errors and release regressions. | Issues by environment, release, route, workflow, exception type, requestId, correlationId, and spawnId where present. |
| AWS CloudWatch Logs | Backend logs, Caddy logs, PostgreSQL logs, system logs, Lambda logs, ECS logs. | Request failures, startup errors, webhook failures, DB errors, proxy errors, and tracing IDs in backend application logs. |
| AWS CloudWatch Metrics | Health, route metrics, Stripe webhook metrics, deploy metrics, ECS/ALB/RDS metrics. | 5xx spikes, latency, unhealthy targets, dead webhooks, retry age, deploy failures. |
| New Relic alerts | Alert rules over collected application, host and AWS signals. | Incidents, condition queries, thresholds, loss of signal and notification delivery. |
| AWS event forwarder / New Relic workflow | Existing Lambda forwards safe SNS infrastructure events into New Relic; the New Relic workflow sends Discord notifications. | MamNajemAwsEvent, Lambda failures, SNS failures and New Relic notification status. |
| AWS SNS | Infrastructure event delivery path. Terraform and backend config also contain Android SNS mobile-push resources, but current backend notification publishing uses ExpoPushSender. |
Failed infrastructure event delivery or event-forwarder subscription issues. Do not assume app push uses SNS without checking current code. |
| GitLab CI/CD | Build, test, deploy, release, and security failures. | Failed jobs in Maven, tests, dependency check, Gitleaks, Semgrep, deploy, Android release, iOS release, Terraform plan/apply. |
| Spring Actuator | Backend health endpoint. | /actuator/health, readiness after deploy, missing health response. |
| Docker/ECS health checks | Container/backend process health. | Unhealthy containers, ECS task restarts, ALB target health failures. |
| Docker Compose on the integration laptop | Integration backend, PostgreSQL and Caddy containers; deployer state per component. | Container restarts, docker compose logs, failed JAR startup or readiness, failed or rolled-back deployer runs. |
| Stripe Dashboard | Webhook delivery, payment errors, disputes, checkout/payment state. | Failed webhook attempts, payment intent failures, disputes, Connect issues. |
| Firebase Analytics / DebugView | Mobile and web product telemetry. | Missing events, failed workflow events, required-update events, privacy-safe event properties. |
| Firebase Remote Config | Required app update config behavior. | Bad update rules, unexpected app blocking, stale minimum version config. |
| Google Play / App Store Connect | Mobile release, in-app purchase, TestFlight/internal release problems. | Rejected builds, failed uploads, subscription/product configuration issues. |
| PostgreSQL logs / DB metrics | Query, runtime, and database health failures. | Connection exhaustion, disk/storage pressure, slow or failed DB operations. |
| Caddy logs | Integration reverse proxy behavior. | TLS/domain/proxy failures, edge 4xx/5xx responses. |
Alert Flow¶
Infrastructure event alert:
ECS deployment failure or unexpected stopped task
-> EventBridge rule
-> existing SNS alert topic
-> existing event-forwarder Lambda
-> New Relic MamNajemAwsEvent
-> New Relic alert condition and environment workflow
-> Discord notification
Collected metrics and logs alert directly through New Relic conditions.
NrIntegrationError with category=NotificationError identifies destination
failures; check NrAiNotification for notification status. If Discord is
unavailable, its failure remains visible in New Relic.
Application error alert:
Application exception or crash
-> New Relic error or mobile crash event
-> New Relic issue/alert condition
-> investigate by environment, release, route, workflow, breadcrumbs, and tracing IDs
GitLab pipeline alert:
Push, schedule, release, or manual job
-> GitLab pipeline
-> failed stage/job
-> inspect whether failure is test, contract, security, deploy, release, or infrastructure
Stripe operational issue:
Stripe event/payment/dispute
-> Stripe webhook delivery
-> backend unified webhook flow
-> backend logs, New Relic metrics and errors, DB state
Request Tracing IDs¶
The backend and mobile app use three privacy-safe HTTP tracing identifiers:
| Header / field | Meaning | Where to look |
|---|---|---|
x-request-id / requestId |
The normalized ID attached to the current backend request. The backend currently aligns it to the accepted correlation ID. | Backend response headers, backend logs, New Relic context, mobile API telemetry. |
x-correlation-id / correlationId |
The logical workflow trace. Mobile sends it on API requests; backend accepts a safe incoming value, falls back to x-request-id, or generates one. |
Mobile API telemetry, backend logs, New Relic, outbound backend WebClient calls. |
x-spawn-id / spawnId |
A per-client or per-backend spawned operation ID for separating related calls inside the same correlated workflow. | Mobile API telemetry, backend logs, New Relic, outbound backend WebClient calls. |
Backend handling:
- Incoming IDs must match the safe ID format before being reused.
- Missing or unsafe IDs are replaced with generated backend IDs.
- The accepted IDs are returned as response headers.
- The IDs are added to MDC/log context and Reactor context.
- Injected backend
WebClient.Builderinstances propagate the IDs on outbound calls. - Diagnostic privacy filtering allows safe tracing IDs, but still scrubs raw user, payment, lease, property, tenant, message, token, and secret data.
Mobile handling:
- API requests include
x-request-id,x-correlation-id, andx-spawn-id. - API success and failure telemetry includes
requestId,correlationId, andspawnId. - Diagnostic privacy filtering treats safe tracing identifiers as diagnostic metadata.
What Each Tool Is Best For¶
| Tool | Best question |
|---|---|
| New Relic errors | What exception happened, in what release, on what route, workflow, or tracing ID? |
| CloudWatch Logs | What did the backend, proxy, database, Lambda, or host actually log for the tracing ID or timestamp? |
| New Relic metrics/alerts | Is the system unhealthy, degraded, or trending badly? |
| GitLab CI/CD | Did tests, deploy, security, or release fail before users saw it? |
| Stripe Dashboard | Did Stripe send the event, and what is the payment/subscription/dispute state? |
| Firebase DebugView | Is mobile telemetry being emitted correctly by an integration build? |
| Cognito | Is login, signup, token, redirect, or identity-provider configuration broken? |
| SNS/event forwarder and New Relic notifications | Did infrastructure events reach New Relic and did alerts reach Discord? |
| Expo Push sender | Did backend push publication to Expo fail, and did the stored token format match what the sender expects? |
Important Names¶
| Item | Value |
|---|---|
| Integration API | https://api.mamnajem.win/v1 |
| Integration web | https://www.mamnajem.win |
| Integration admin | https://admin.mamnajem.win |
| Integration docs | https://docs.mamnajem.win |
Integration web app (web-frontend) |
https://app.mamnajem.win |
| Production API | https://api.mamnajem.pl |
| Production web | https://mamnajem.pl |
| Backend health | /actuator/health |
| Stripe webhook | /v1/webhooks/stripe |
| Integration CloudWatch namespace | MamNajem/integration |
| Production CloudWatch namespace | MamNajem/production |
| Integration backend runtime | backend service of the mamnajem-integration Docker Compose project on the integration laptop |
First Checks By Area¶
Backend/API issue:
- Check GitLab pipeline for a recent failed deploy or test.
- Check
/actuator/health. - Check New Relic alerts and backend logs.
- Check New Relic backend issues by environment, release, route, and tracing IDs.
Payment issue:
- Check Stripe Dashboard event/payment state.
- Check Stripe webhook delivery status.
- Check backend webhook logs.
- Check New Relic Stripe webhook metrics and dead-webhook conditions.
- Check database payment attempt/request state.
Mobile issue:
- Check New Relic mobile errors or crashes by release and environment.
- Check Firebase DebugView for integration builds.
- Check API request telemetry and backend logs by
requestId,correlationId, orspawnId. - Check app environment variables and telemetry-disabled flags.
- For integration URL problems, verify whether the failing build was configured from the mobile
.pldefaults or the architecture.proweb/API values.
Auth issue:
- Check Cognito user pool/client configuration.
- Check redirect/logout URLs.
- Check Amplify configuration.
- Check backend auth errors and New Relic.
Deploy issue:
- Check the GitLab deploy job.
- Check CloudWatch
DeploySuccessmetric where applicable. - Check
/actuator/health. - For integration, check the laptop deployer state for the commit and the
backendcontainer's readiness and logs. - For production, check ECS service deployment and ALB target health.
Backend ERROR alert coverage¶
Both environments have enabled critical Backend ERROR logs conditions, scoped to entity.name = mamnajem-<environment>-backend and level = ERROR. One matching forwarded log qualifies; the 60-second aggregation is evaluated with AT_LEAST_ONCE in a 300-second duration. Five minutes without a matching signal closes the issue without opening a new silence issue. Stack traces logged at ERROR are covered through the log record; bare stdout/stderr or WARN traces require separate receipt/severity investigation. The APM Backend errors condition excludes expected HTTP 4xx responses while keeping server/status-less exceptions.
Use only New Relic dashboards, metrics, incidents and notification records plus CloudWatch logs for operational alert audits. Current private access instructions are in the Codex workflow; source and installed skill copies are maintained by the workspace bootstrap.