Sync Engine

Two-way external calendar sync is Musubi’s most distinctive subsystem. It lives in apps/api/src/sync/ and is provider-agnostic: the engine does not talk to a provider directly — it routes through a small adapter contract. Google, Microsoft Graph, and CalDAV are the current implementations.

Mental model

flowchart LR
    subgraph P[External providers]
      G[Google]
      M[Microsoft]
      C[CalDAV]
    end
    subgraph EN[engine.ts]
      SP["syncProvider(adapter, user, account)<br/>1 · listCalendars → reconcile mirrors<br/>2 · fetchChanges(cursor) → upsert<br/>3 · push local edits (pushCreate/Update/Delete)"]
    end
    subgraph DB[Local DB]
      T["external_* · events · calendar_*"]
    end
    P -->|"pull · fetchChanges"| EN
    EN -->|"push · pushCreate/Update/Delete"| P
    EN --> DB

Each connected provider account produces mirror calendars — ordinary Musubi calendars with an externalCalendars mapping. Their events flow into the unified events table like any other event (that’s the point of “events beyond calendars”). Sync is bidirectional:

• Pull (provider → local): fetchChanges returns normalized events; the engine upserts them.
• Push (local → provider): editing an event in a mirror calendar calls the adapter’s pushCreate/Update/Delete.

The adapter contract

sync/adapter.ts defines three data shapes and the interface every provider implements.

type NormalizedEvent = {
  externalId: string;
  status: "active" | "cancelled";   // cancelled ⇒ delete locally
  title: string;
  start: Date; end: Date;           // UTC; all-day end is exclusive
  isAllDay: boolean;
  description: string | null; location: string | null;
  organizer: string | null;
  recurrence: string | null;        // iCal RRULE text (+ EXDATE lines)
  url: string | null;
  etag?: string | null;             // CalDAV conflict detection
};

type ExternalCalendarInfo = { externalId; name; color; readOnly?: boolean };
type FetchChangesResult   = { changes: NormalizedEvent[]; nextCursor: string | null; reset?: boolean };

interface CalendarAdapter {
  listAccounts(userID): Promise<{ id; label }[]>;
  listCalendars(userID, accountId): Promise<ExternalCalendarInfo[]>;
  fetchChanges(userID, accountId, externalCalendarId, cursor): Promise<FetchChangesResult>;
  pushCreate(userID, accountId, externalCalendarId, event): Promise<{ externalEventId }>;
  pushUpdate(userID, accountId, externalCalendarId, externalEventId, event): Promise<void>;
  pushDelete(userID, accountId, externalCalendarId, externalEventId): Promise<void>;

  // calendar-level writes — create / rename+recolor / delete the calendar itself
  createCalendar(userID, accountId, { name, color }): Promise<{ externalId }>;
  updateCalendar(userID, accountId, externalCalendarId, { name, color }): Promise<void>;
  deleteCalendar(userID, accountId, externalCalendarId): Promise<void>;
}

NormalizedEvent is the currency of the whole system — every adapter translates its provider’s format to and from it.

Calendar-level writes

Mirrors are editable and deletable from Musubi, and POST /calendars can create a calendar into a connected account. The calendar handlers apply these provider-first: the adapter call runs before the local write, and a provider rejection (Google refusing to delete a primary calendar, an expired token…) prevents the local mutation. A later local database failure can still leave a provider-only change, so this is ordering, not a distributed transaction.

Only the user whose account backs the mirror may perform these operations (external_calendars.userID check); read-only mirrors map to viewer, so assertCan blocks writes. Provider quirks stay in adapters: Google color lives on the per-user calendarList entry, Microsoft maps hex to its nearest preset, and CalDAV uses raw MKCALENDAR / PROPPATCH / DELETE.

Write path and failure semantics

Event writes use a different contract. Every dependent local write is transactional: create commits the event, creator-attendee, and links together; update commits the event, link diff, and removed mappings together; delete commits unlinking and the possible final tombstone together.

Provider calls remain outside PostgreSQL. Event handlers preflight the full known intent, then commit local event/link changes and event_outbox rows together. Remote delete addresses are captured before unlink removes ordinary mappings. The request claims each persisted operation before its first conditional delivery; there is no second legacy inline sender. Postcommit failures return truthful localCommitted, revision and sanitized partial/unconfirmed receipts.

K07 preserves pending/attempting intent across process exit, but does not yet retry it. Unresolved predecessors block later attempts. K08 owns restart recovery, provider-specific idempotence/reconciliation, pull coordination and disconnect cleanup. This is not a distributed transaction or a guarantee of eventual delivery.

Mutation	Ordering	Current failure result
Calendar create/update/delete	Provider, then local DB	Provider rejection aborts local change; later DB failure can diverge
Event create/update/delete/link/unlink/fork	Preflight, local CAS + outbox transaction, claimed provider delivery	Precommit failure rolls back; postcommit conflict/unconfirmed response retains local revision and durable intent
Scheduled pull	Provider fetch, then local reconciliation	Account/provider failures are isolated and retried next cycle; pending-write coordination remains K08

Disconnecting one provider account uses removeExternalAccountData() to remove all of its local mirrors, events, opt-out tombstones, and (for CalDAV) stored credentials in one transaction. Google revocation and Better Auth OAuth unlink still sit outside PostgreSQL and therefore keep the same cross-system boundary.

See the observability runbook.

The engine

sync/engine.ts orchestrates one account at a time in syncProvider():

1. Reconcile calendars. listCalendars() → delete local mirrors gone on the provider (removeCalendar cascades orphans), import new ones (importExternalCalendar), and refresh the read-only flag. readOnly: true maps to calendar_members.role = "viewer".

2. Pull events. For each mirror, fetchChanges(cursor). If reset: true (e.g. Google 410 cursor-expired), clearCalendarEvents() soft-deletes all first, then upserts fresh. Per event: status: "cancelled" → deleteExternalEvent; else upsertExternalEvent (transactional — creates or revives the event row + calendar_events + external_events mapping). Persist nextCursor.

syncUser(userID) discovers accounts per provider, then wraps every account sync independently. One expired Google account therefore neither aborts a second Google account nor CalDAV. Account failures include the concrete accountId in structured logs; provider-level discovery failures use sync.provider.failed.

The CalDAV connect handler uses the same path in an account-scoped strict mode (syncUser(userID, { provider: "caldav", accountId, throwOnError: true })). This makes the first calendar/event import part of a successful connection and returns an import failure to the client. Scheduled syncs keep the default best-effort behaviour.

Near-realtime: the scheduled sync

Provider changes reach the app without a manual pull-to-refresh through a three-piece loop:

1. Scheduler (apps/api/src/index.ts): every EXTERNAL_SYNC_INTERVAL_MIN minutes (default 5, 0 disables), the server runs syncUser for every user who has at least one provider mirror (getExternalSyncUserIDs).
2. Change detection (the part that makes this sane): upsertExternalEvent returns whether it actually wrote — with an etag (CalDAV) an unchanged, alive event is a verified no-op: no write, no updatedAt bump, so neither the broadcast nor the client delta fire for nothing. Deletions are detected by a sweep: on a full-set fetch (reset: true — CalDAV always, Google after a 410), events whose external id wasn’t in the fetched set get tombstoned. The sweep replaced the old tombstone-everything-then-revive reset, which churned every event on every sync.
3. Broadcast → silent refresh: when a calendar really changed, syncUser sends an SSE external_sync frame to its members; the client (useEventsStream) responds with a silent delta refresh — refresh({ providerSync: false }), skipping the provider-sync trigger so it can’t loop.

Note

Why polling and not push? True Google push (watch webhooks) needs a publicly reachable HTTPS endpoint plus channel management/renewal — hostile to self-hosting — and CalDAV has no push protocol at all. A uniform poll with real change detection gives both providers near-realtime for free. If webhook push ever lands, it slots in as an additional trigger for the same syncUser path.

Caution

The API supports one process per database and enforces that boundary with a session-scoped PostgreSQL advisory lock. Within that process, the scheduler is single-flight: if a run outlasts its interval, the next tick is skipped, logged as sync.scheduler.skipped, and counted by musubi_scheduled_task_skips_total{task="external_sync"}. Do not bypass the singleton lock to scale horizontally; shared pub/sub and rate limiting are still required. See Current scaling boundary.

Two correctness details worth knowing: provider deletions tombstone (deletedAt) rather than hard-delete, so offline clients learn about them through the delta’s deletedIds (a hard delete just vanished — stale caches kept the event forever); and a client-originated push echoes back on the next poll as a write (values identical), which is harmless.

Note

organizer is NOT NULL in the schema but adapters may hand back null — the engine falls back to "". color on synced events comes from the mirror calendar, not the provider event.

Account disconnect semantics

The current client disconnects exactly one { provider, accountId } through POST /api/v1/users/connections/disconnect. The handler:

1. best-effort revokes that Google account’s refresh token (Microsoft has no equivalent step here);
2. removes only mirror calendars and disabled-calendar markers for that account;
3. asks Better Auth to unlink that exact account; and
4. if Better Auth refuses—most commonly because it is the user’s final login account—clears calendar tokens/scope only on the matching (user, provider, accountId) row.

This preserves sibling Google/Microsoft accounts. The legacy POST /api/v1/users/connections/google/revoke route has no accountId and is intentionally provider-wide for old clients; do not use or describe it as an exact-account operation.

Credential crypto

sync/crypto.ts encrypts CalDAV passwords with AES-256-GCM before they hit caldav_accounts.encryptedPassword:

encryptSecret(plaintext) → "base64(iv):base64(tag):base64(ciphertext)"   (fresh 12-byte IV each time)
decryptSecret(blob)      → verifies the auth tag, throws on tampering

The key is CALDAV_ENC_KEY (64 hex chars = 32 bytes; generate with openssl rand -hex 32). Plaintext is decrypted only when building a CalDAV client, never stored or logged. Without the key set, CalDAV connection fails at use time (not boot).

Provider specifics (the gotcha zones)

Most sync bugs are recurrence and all-day edge cases. The existing adapters already handle these — study them before writing a new one.

Google (adapters/google.ts):

• The adapter refreshes expired access tokens against Google’s token endpoint so it can retain the safe OAuth error/error_subtype codes that Better Auth’s generic getAccessToken error otherwise discards. Tokens and error descriptions are never logged.
• invalid_grant is permanent until the user authorizes again. The account row is marked sync_status = reconnect_required, its dead tokens are cleared, and listAccounts() excludes it from later scheduled runs. Existing mirrors remain visible with a reconnect action. A successful Better Auth OAuth relink resets the status to active through the account update hook.
• Network failures, rate limits, 5xx responses, and server OAuth configuration errors remain retryable and do not disable an individual user’s account.
• sanitizeRecurrence() fixes three iCal quirks: EXDATE;TZID=… (the rrule lib ignores TZID → convert to UTC), all-day EXDATE must be VALUE=DATE, and FREQ=YEARLY;BYMONTHDAY without BYMONTH (RFC expands monthly → anchor the month).
• Incremental sync via syncToken; 410 Gone → return reset: true.
• All-day end is exclusive (±1 day on pull/push).

Microsoft (adapters/microsoft.ts):

• OAuth mirrors the Google lifecycle exactly — both adapters share sync/oauth.ts (token refresh, invalid_grant → reconnect_required) and the generic queries/oauth.ts account queries. One Microsoft quirk handled there: the refresh token rotates on every refresh and must be re-persisted.
• Incremental sync via Graph calendarView delta. v1.0 has no unbounded event delta, so the delta token is bound to a fixed date window (WINDOW_* constants, currently −180d…+730d). The cursor stores {link, windowEnd}; when the future edge gets within 90 days, the adapter forces reset: true with a fresh window. Events outside the window don’t exist in the mirror — a rolling view, not full history.
• Recurring series arrive pre-expanded: calendarView returns occurrences and exceptions as individual events, so recurrence is always null on pull — no RRULE conversion, and Outlook-side exceptions come for free. The flip side: pushing a Musubi-created recurring event into an Outlook mirror is rejected with a clear error (Graph models recurrence as structured patterns, not RRULE/EXDATE; converting is the upgrade path).
• Prefer: outlook.timezone="UTC", outlook.body-content-type="text" on every read — times come back UTC and bodies as plain text.
• Graph only accepts preset color names (hexColor is read-only), so create and update map the chosen Musubi hex to the nearest Graph preset.

CalDAV (adapters/caldav.ts + caldav_client.ts):

• Wraps tsdav; Basic auth; principal/home discovery is automatic (incl. iCloud hosts).
• Events parsed via ical.js; externalId is the resource URL, not the UID.
• Always returns reset: true — full fetch every sync (a deliberate simplification; upgrade path is WebDAV sync-collection). This cleanly handles deletions.
• Passwords are trimmed (mobile clipboards add whitespace) then decrypted per request.

Caution

Recurrence must round-trip on update. When pushing an edit to a recurring event, always emit the full RRULE + all EXDATE lines — not a delta. Omitting them strips the recurrence off the provider’s copy on every save. Both adapters do this carefully; preserve it.

How to add a provider

1. Create adapters/<provider>.ts implementing CalendarAdapter. Translate the provider’s API to/from NormalizedEvent.

2. Credentials. OAuth: add the provider to Better Auth (packages/auth) and implement listAccounts() against the account table. Preserve the provider’s machine-readable refresh error code so permanent revocation can be distinguished from a transient outage. Password-based: model it on caldav_accounts and encrypt with encryptSecret().

3. fetchChanges. Paginate internally, accumulate all pages into one changes array, return the provider’s opaque nextCursor. Map cursor-expiry errors to reset: true.

4. push* idempotency. Deletes must treat 404/410 as success.

5. Register in the engine — add <provider>: <adapter> to the registry map in engine.ts. The externalCalendars.provider / externalEvents.provider columns are free-form strings, so no schema change is needed.

6. Wire connect/disconnect handlers (model on handlers/caldav.ts / handlers/google.ts) and trigger an initial syncUser().

Testing provider boundaries

pnpm test includes two provider-neutral safety nets:

• sync/orchestrator.test.ts proves discovery/account failures are isolated, retryable accounts run again on the next cycle, strict connect flows throw, and provider/account filters do not leak work; and
• sync/adapters/provider_http.test.ts runs local fake Google/Graph servers to exercise multi-page deltas, 410 cursor reset, partial-result discard, Microsoft series-master cache reset/reuse and safe caller retry.

For stored credential behavior, run pnpm test:provider-db against a disposable migrated PostgreSQL database. Its fake token endpoint verifies encrypted access and refresh-token rotation, account-scoped invalid_grant, sibling preservation and transient failure retry. See Commands for the required test-only environment.

These checks intentionally do not impersonate a live provider. Before a release that changes an adapter, still perform one sandbox round trip with real credentials: connect, initial pull, incremental pull, create/update/delete, token refresh and disconnect.

Adapter checklist

• Cursor expiry → reset: true so the engine wipes and refetches
• Read-only calendars (holidays, subscriptions) → readOnly: true
• All-day end is exclusive — adjust ±1 day on pull/push
• Recurrence: normalise provider format to iCal RRULE; EXDATE value-type must match DTSTART
• EXDATE with a TZID → convert to UTC (rrule lib ignores TZID)
• Push includes full RRULE + EXDATE on every update
• organizer may be null → engine defaults to ""
• Multiple accounts per provider supported via listAccounts()
• Deletes idempotent (404/410 = success)
• Calendar-level ops (createCalendar/updateCalendar/deleteCalendar) throw on provider rejection — the handlers rely on that to abort the local write

Reference

For setup and provider-specific debugging, use the dedicated guides: Google, Microsoft / Outlook, and Apple / CalDAV.

File	Contents
sync/engine.ts	syncProvider, syncUser, prepareEventWrites, registry
sync/orchestrator.ts	Provider/account failure boundary, strict/scoped selection
sync/adapter.ts	NormalizedEvent, ExternalCalendarInfo, FetchChangesResult, CalendarAdapter
sync/adapters/google.ts	Google Calendar adapter
sync/adapters/microsoft.ts	Microsoft Outlook adapter (Graph calendarView delta)
sync/adapters/caldav.ts + caldav_client.ts	CalDAV adapter (Apple/iCloud + generic)
sync/oauth.ts	shared OAuth access-token refresh (google + microsoft)
sync/crypto.ts	AES-256-GCM encryptSecret/decryptSecret
packages/db/src/queries/external.ts	upsertExternalEvent, clearCalendarEvents, removeExternalAccountData, cursor & link helpers