Every multi-tenant SaaS runs on the same implicit prayer: that every engineer, on every query, on every code path, remembers to add WHERE tenant_id = %s. Miss it once, in one repository method, on one debugging session, on one raw INSERT ... SELECT migration — and one tenant reads another tenant’s rows.

Actionary puts that guarantee in Postgres. Row-level security enforces tenant isolation at the database. The application layer keeps its WHERE tenant_id = %s clauses as defence in depth; the substrate is the guarantor. If the tenant clause is missing, the substrate returns zero rows.

The stance is documented at docs/09_security/tenant_isolation_rls_spec.md and gets its own section in the substrate story. This page is the shape of it.

The substrate enforces, the app assumes

Every table with a tenant_id column carries a Postgres row-level security policy keyed on the request-scoped GUC app.current_tenant_id. The request path acquires the crm_app role at connection acquisition, sets the tenant from the resolved request context, and every query the caller issues is filtered by the substrate.

crm_app is NOSUPERUSER NOBYPASSRLS by definition. It is the only role a request path ever holds. A missed WHERE tenant_id = %s, an ad-hoc psql session run as crm_app, a raw INSERT ... SELECT inside a migration, and a new physical table created via Schema Admin all fail closed. The database applies the filter, and application code has no opt-out.

Two roles cover legitimate cross-tenant reach

There are exactly two BYPASSRLS roles in the system, and both are enumerated.

  • crm_platform_admin — staff-platform routes and cross-tenant worker scans. Reached only by a session_kind === "staff_platform" session or an operator worker. Never issued to a tenant path.
  • crm_migrate — the one-shot migration runner. Reached only by python -m services.core_api.cli migrate in the migrate container.

Every other role — application, background workflow, agent — holds crm_app. Every bypass surface is documented; adding a third BYPASSRLS role would fail the review gate at docs/09_security/rls_ship_review_checklist.md.

Three CI gates keep the substrate from rotting

Substrate protection is only as strong as the discipline that shipped it, so three independent AST tests fail the build on any drift.

  1. tests/test_router_rls_posture.py walks every mounted router under services/core_api/api/v1/routers/ and asserts each declares a tenant posture — a get_tenant / require_root_admin / require_auth dep, an internal set_current_tenant, or an explicit bypass-role acquisition. A new host-agnostic router that lands without a tenant posture cannot ship.
  2. tests/test_migration_rls_invariant.py walks every migration numbered above 338 and asserts every CREATE TABLE ... tenant_id ... declares an explicit RLS posture. A new per-tenant table that lands without RLS cannot ship.
  3. tests/test_back_door_router_tenant_scope.py exercises each host-agnostic router’s tenant-resolution helper and asserts set_current_tenant (or a bypass-role acquisition) fires at the correct point. A router with the surface declaration but the wrong firing order cannot ship.

Zero network, zero database, pure AST. The spec is in the code, the test, and the doc. A regression cannot ship.

Runtime observability — drift visible in the first cursor

The connection layer at shared/db/connection.py counts every unscoped crm_app cursor acquisition and emits one WARNING per unique caller-site per process:

db.unscoped_tenant_read
  role=crm_app
  site=<file:lineno>
  state=no_tenant_guc_set
  action=rls_returns_zero_rows
  next_step=set_current_tenant_or_acquire_bypass

The cumulative counter unscoped_read_counter() is public for metric scraping. Zero is the healthy value. Any non-zero delta names a bug — with the file and line to fix it at.

Two additional signals cover the substrate’s own state. db.role_bootstrap_fallback fires once per role per process when migration 338 has not yet applied. db.role_missing_but_policies_exist fires on every cursor when a role is missing but RLS policies exist — a substrate-degraded incident the log collector alerts on until the role is restored.

Staged rollout, drilled rollback

The deploy pipeline (scripts/deploycrm) splits its replica rollout into A-side and B-side stages. Between them: a five-minute soak, then python -m services.core_api.cli slo verify --window 5m --max-errors 10 reads recent server_errors and exits non-zero if the count breaches. Deploy halts before touching B-side on any A-side breach. Both stage outcomes persist to deploy_history.soak_summary.

Every substrate migration ships with a companion rollback at db/rollback/<N>_rollback.sql and an operator playbook. For migration 338, the rollback is idempotent and lives outside the standard migrations tree so the pipeline runner leaves it in place. Before every substrate deploy reaches production, the rollback drill runs on a fresh dev environment and the outcome is captured on the independent-reviewer checklist. Incident response is documented at docs/08_operations/rls_incident_runbook.md.

Cross-tenant probes look identical to a missing tenant

An attacker who guesses tenant slugs and looks for differential responses gets nothing. A probe against a tenant the caller does not belong to returns HTTP 404, with a body identical to “tenant does not exist”. The response shape matches a slug that was never provisioned — same status, same envelope, same timing profile. The substrate’s zero-row answer and the router’s 404 shape compose into a single indistinguishable response — enumeration through timing or shape returns no signal.

The takeaway

Every enterprise security review asks the tenant-isolation question. The confident answer is a database property with a name, a role, a GUC, three CI gates, three runtime signals, a staged rollout, and a drilled rollback. Actionary ships it as the default posture on every tenant. If you want the deeper stance behind it, read the design principles and the security posture; if you want the CISO-level walkthrough, start there.

“A query that forgets the tenant clause returns zero rows.” That is the sentence to remember.