Every request-path service — core-api, mcp-server, spa, portal, widget — runs as paired -a / -b replicas defined in docker-compose.prod.yml. Caddy fronts each pair with a reverse_proxy upstream pool, lb_policy round_robin, a /health upstream probe, and passive failover via fail_duration. The sibling carries traffic while its twin recreates.

scripts/deploycrm typechecks the SPA on the workstation first, SSHes to the box, and rolls one replica at a time with docker compose up --build --force-recreate --wait --wait-timeout 180, gating on the Docker healthcheck before the next replica flips. Code, env, or credential changes deploy with no user-visible 502 window.

The same topology carries to ECS / Fargate, EKS, or Fly.io by swapping the orchestrator. Service code, Dockerfiles, and the Caddy upstream contract are unchanged.

Eight phases, one audit row

Every deploy — dev or prod — runs the same pipeline. Halt-on-any-failure. The trap-on-EXIT writes the terminal outcome to deploy_history regardless of which phase failed, so the audit trail is complete even when the deploy aborts mid-flight.

  1. Build — one image tagged with the new commit SHA, reused by every later phase.
  2. Migrate pre-roll — apply-pending --phase additive,expand,switch,data. Old code keeps running against the new schema.
  3. Record-start — writes a deploy_history row in outcome=in_progress.
  4. Roll A-side — recreate core-api-a, mcp-server-a, spa-a, portal-a, widget-a with --wait. Then a 5-minute soak. Then slo verify --window 5m --max-errors 10 reads recent server_errors and exits non-zero on breach.
  5. Roll B-side — same shape. Second 5-minute soak. Second slo verify gate. Caddy reloads last, so the routing config sees healthy upstreams from moment zero.
  6. Migrate contract — apply-pending --phase contract. Destructive SQL runs now that the old code is gone.
  7. Smoke — an HMAC bearer derived at runtime from PLATFORM_MASTER_KEY via HKDF (subsystem label deploy-smoke) posts to /api/v1/health/deploy-smoke. The endpoint runs three canaries server-side — users.reachable, metadata.reachable, agent.write_path — proxies for auth, entity read, and the agent write path.
  8. Housekeeping and record-finish — dangling images pruned, BuildKit cache capped at 5 GB, terminal outcome recorded.

Zero new env vars for the smoke surface. The master key is the credential; the bearer derives at runtime; a 60-second clock-skew window makes it effectively single-use.

Expand, switch, contract

Every migration file declares its phase in a header — -- phase: additive | expand | switch | contract | data — and the discipline is enforced in two independent places.

.github/workflows/migration_policy.yml rejects any PR that pairs destructive SQL (DROP COLUMN, RENAME COLUMN, ALTER COLUMN ... TYPE, DROP TABLE, NOT NULL without DEFAULT) with any phase other than contract. The runtime SchemaMigrationService refuses to apply a file whose header is missing or unrecognised, so the discipline holds even if a developer bypasses the lint.

The two buckets the pipeline runs — pre-roll for additive, expand, switch, data; post-roll for contract only — guarantee the previous-version application can serve traffic against the new schema for one full release cycle. That is what makes rollback safe.

Backward-compatible changes ship in one deploycrm. Destructive changes ship in two, with a gap: the first stops using the old shape; the second drops it. A column rename becomes three migrations across two deploys — expand, switch, contract. The naïve one-shot RENAME COLUMN fails the lint before it merges.

One-shot migrate container, sha256 drift detection

The migrate container is a one-shot process the pipeline invokes with docker compose run --rm -T migrate .... Two flags are non-negotiable: -T disables the pseudo-TTY, < /dev/null detaches container stdin. Without both, an SSH heredoc silently consumes the next command as stdin and the deploy aborts with outcome=aborted and migration_count=0. The invariant is documented in the runbook.

The same SchemaMigrationService powers both this CLI and the in-app /platform/operations/db-admin runner. Sha256 recording, drift detection, and pg_advisory_lock serialisation are identical across surfaces. A drifted file surfaces as a diff between the recorded hash and the on-disk file. One code path, one audit trail.

Every paid feature has a live probe

/platform/operations is the single page that tells an operator whether the platform is functional end-to-end. Container health is one row; the other rows exercise every paid feature. Six tiers: per-replica HA health across every paired service, host resources (root filesystem pressure, degraded at 80% used, down at 90%), both S3 buckets probed via HEAD with region rendered, agent runtime (Anthropic, OpenAI, embeddings, Whisper, workflow worker pulse, billing worker pulse, meeting-recorder retention sweeper heartbeat), Postgres backup recency, and the OTEL collector.

LLM probes are 30-second server-cached so the Refresh button doesn’t hammer providers. A degraded probe raises a platform admin banner on every /platform/* page, so an operator sees pressure the moment they open any admin page. The pitch line is literal: every paid feature on the platform has a live reachability probe so silent vendor outages surface to operators before customers do.

Fail-loudly substrate

Every uncaught failure on either side of the wire becomes a forensic row in the same Postgres the application writes to. One global FastAPI @app.exception_handler(Exception) writes to server_errors with full traceback, request_id, exception_type, method, path, status, tenant, and actor. Browser-tier failures land in frontend_errors from window.onerror, window.onunhandledrejection, and the React root error boundary. Both surfaces poll their triage UIs every 30 seconds.

Routers hold no endpoint-level catch surface. A 2026-05-22 sweep removed 178 silent catches across 17 routers; the pattern except Exception: return api_error("server_error", ...) is structurally impossible now. The error-boundary copy “Root admin has been notified” is literally true — by the time the recovery UI paints, the forensic row is in the database.

OTEL across every tier

core-api, mcp-server, and the SPA all emit through the OpenTelemetry Web SDK and OTLP/HTTP. FetchInstrumentation auto-injects W3C traceparent on every /api/* call, so one trace spans browser → core_api → mcp_server. Caddy’s /otel/* route proxies to the collector for same-origin export. OTLP is a public specification with multiple compatible backends — an open-standards property, vendor-neutral by construction.

The takeaway

Deploys are boring on purpose. Two invocations of deploycrm cover the 1% of destructive changes; one covers the other 99%. The audit trail is a table you can query. Rollback is one command against the last outcome=success SHA, drilled quarterly under SOC 2 CC7.4. For a technical buyer, the story sits at the layer that matters: durable execution on Postgres, open standards at every boundary, and an operations posture that reads like infrastructure.