Observability runbook

Musubi exposes Prometheus metrics on a private listener and writes structured JSON logs to stdout/stderr. The implementation lives in apps/api/src/metrics.ts; dashboards and alerts live under ops/.

Metrics endpoint

Property	Value
Listener	Separate from the public API
Port	METRICS_PORT, default 9464; 0 disables it
Path	GET /metrics
Bind address	0.0.0.0
Prefix	musubi_
Constant label	service="api"

The bundled Compose files do not publish this port to the host. Scrape it from a trusted Docker network:

scrape_configs:
  - job_name: musubi-api
    static_configs:
      - targets: ["musubi-api:9464"]

Keep the job name musubi-api; the shipped dashboards and alert rules expect it.

Metric catalog

HTTP

Metric	Type	Labels
musubi_http_requests_total	counter	method, route, status
musubi_http_request_duration_seconds	histogram	method, route, status
musubi_http_requests_in_flight	gauge	method

Routes use registered Express patterns, never concrete URLs. UUIDs and invite tokens therefore do not become high-cardinality labels.

Provider sync

musubi_external_sync_failures_total{stage,provider} counts bounded failure classes. stage is one of account, discovery, push, or scheduler; provider is a known provider name, all, or unknown.

musubi_scheduled_task_skips_total{task} increments when cleanup or external_sync is still running at its next tick. One increment means overlap was prevented, not that two runs executed. Investigate duration, provider latency, and interval sizing; the shipped MusubiScheduledTaskOverlap alert fires on any increase.

Usage snapshot

The API computes current-state gauges from PostgreSQL and caches the snapshot for 60 seconds:

Metric	Meaning
musubi_users_total	Non-federated local users
musubi_events_total	Live events
musubi_calendars_total	Calendars
musubi_active_users	Users with a valid session
musubi_active_sessions	Non-expired sessions
musubi_sync_accounts{provider,status}	Connected credential/provider accounts

CalDAV credentials live outside Better Auth and have no OAuth account status, so they appear with status="active". Use the failure counter and logs for their health.

Process and live connections

prom-client Node/process defaults are exported under the same prefix. SSE gauges are per process:

• musubi_sse_connections;
• musubi_sse_users; and
• musubi_process_* / musubi_nodejs_*.

The supported deployment has one API replica, so these gauges describe the whole service. A future coordinated multi-replica design would need to aggregate them with sum(...).

Structured logs

Every completed request emits http.request.completed with:

• method, registered route, status, and durationMs;
• requestId, also returned as the x-request-id response header; and
• userId when authenticated.

Status determines level: 5xx is error, 4xx is warn, everything else is info. Provider-sync debug logs add per-account/per-calendar counts and timing when LOG_LEVEL=debug.

Caution

Do not add request bodies, passwords, OAuth codes, refresh tokens, CalDAV credentials, invite tokens, or member tokens to logs. A request ID is the safe join key between client reports and server activity.

Shipped operations assets

Path	Purpose
ops/prometheus/musubi-alerts.yml	Target-down, 5xx, and provider-sync alerts
ops/grafana/musubi-usage-dashboard.json	Users, calendars, events, sessions, accounts, sync and SSE
ops/grafana/musubi-api-dashboard.json	Throughput, failures, latency and process health
ops/grafana/musubi-logs-metrics-dashboard.json	Combined Prometheus and Loki investigation

Import dashboard JSON through Grafana’s Dashboards → New → Import. Select the Prometheus/Loki data sources requested by the template variables.

These assets are checked against the code

Observability fails quietly by nature. A panel whose metric was renamed draws an empty graph, which looks exactly like a healthy system; an alert whose metric was renamed simply never fires, and a rule that never matches is indistinguishable from a rule that never had cause to. Both are discovered during the incident they existed to catch.

So the files above are read as a contract in CI, under the Observability job:

• apps/api/src/metrics.test.ts fails if any series a dashboard panel or an alert rule reads is missing from the registry, and if an alert has no severity label or summary annotation — one nothing can route, the other a page that says only the rule’s name. It also pins the route label to registered Express patterns, which is what keeps label cardinality finite and invite tokens out of the metric store.
• packages/config/src/logger.test.ts covers the properties that make the logs usable: one JSON object per line, warnings and errors on stderr, and the redaction that makes logger.error("failed", { error }) safe to write without first inspecting what the object carries.
• promtool check rules loads musubi-alerts.yml the way Prometheus would.

Renaming a metric is allowed. Renaming it without updating ops/ is not.

Useful queries:

# State growth is derived from stored gauge history
delta(musubi_events_total[7d])
delta(musubi_users_total[24h])

# Connected external providers
sum by (provider) (
  musubi_sync_accounts{provider=~"google|microsoft|caldav"}
)

# API 5xx ratio
sum(rate(musubi_http_requests_total{status=~"5.."}[5m]))
/
sum(rate(musubi_http_requests_total[5m]))

Incident triage

1. Define the blast radius.

   Check API reachability, 5xx ratio, latency, database availability, sync failures, and whether only one provider/account is affected.

2. Capture a request ID.

   Ask for the x-request-id from the failing response, or reproduce once. Find the same ID across request, auth, handler, sync, and error log entries.

3. Separate local persistence from provider delivery.

   Calendar-level provider mutations are attempted before the local mirror changes. Event provider push is currently best-effort after local persistence; a successful API response can therefore still require reconciliation. See Sync failure semantics.

4. Inspect the account and cursor.

   Determine provider, exact connection, syncStatus, calendar cursor/window, last successful sync, and whether a legitimate reset is in progress.

5. Enable debug logs briefly.

   Set LOG_LEVEL=debug, reproduce one bounded operation, then restore the production level. Never collect secrets while increasing detail.

6. Verify recovery.

   Confirm a subsequent pull converges database, provider, and client cache. Record the alert, request ID, provider/account scope, cause, and prevention.

Current scaling boundary

The scheduler, cleanup timers, SSE registry, rate limiter, and metrics registry are process-local. Musubi therefore enforces one active API process per database:

• apps/api/src/singleton.ts holds a dedicated, session-scoped PostgreSQL advisory lock for the process lifetime;
• a second API pointed at the same database exits before listening;
• both shipped Compose files declare deploy.replicas: 1; and
• losing the lock connection terminates the process instead of silently serving with an invalid coordination assumption.

Cleanup and external sync additionally use an in-process single-flight guard. If one run exceeds its interval, the next tick is skipped and observable through logs plus musubi_scheduled_task_skips_total.

True horizontal scaling still requires:

• coordinate or singleton-run scheduled jobs;
• use shared pub/sub for cross-replica SSE delivery;
• replace the in-memory rate limiter for a global policy; and
• aggregate per-instance metrics.

Do not remove or bypass the singleton lock until those pieces exist. A database alone does not coordinate these in-memory behaviors; its advisory lock currently prevents the unsupported topology rather than making it safe.