diff --git a/docs/phase-4-runbook.md b/docs/phase-4-runbook.md index 05d10c2..d6c0894 100644 --- a/docs/phase-4-runbook.md +++ b/docs/phase-4-runbook.md @@ -6,75 +6,78 @@ logging, and a Kubernetes deployment path. Read those first. ## Verification status — read this before the rest of this doc -Every prior phase's runbook documents claims **checked against the live -stack**, not asserted. This one is different, and says so plainly rather -than papering over it: for the great majority of this phase's work, -**there was no working Docker daemon access and no reachable Kubernetes -cluster**, so most of what follows is a *procedure to run*, not a report -of what was already run and passed. Five genuine exceptions so far: +**Docker daemon access became available partway through this phase's +work, and the rest of this runbook (§§1–10, 10a, 13, 14, most of §8) has +now genuinely been run against a real docker-compose stack** — real +ClickHouse (`clickhouse/clickhouse-server:24.8`), real Postgres, real +`enterprise-auth`/`enterprise-api`/`enterprise-ingest` containers, two +real provisioned tenants (`acme`, `globex`). This closed every +previously-disclosed "written but never run" gap for the ClickHouse +side, and along the way found and fixed six real bugs no amount of +Docker-free testing could have caught: -- `enterprise/internal/audit`'s hash-chain, tamper-detection, and - concurrent-write guarantees (task 4) -- verified live against a real - Postgres earlier in this phase's work (see its own doc comments for - the exact `docker run` invocations), before the environment lost - Docker access. -- `enterprise/internal/loginhandler`'s full OIDC login flow (§3a) -- - verified with real cryptography (a fake IdP that signs and verifies - genuine RS256 tokens) *without* needing Docker or a live database at - all, so this one was actually run in this runbook's own session, not - just an earlier one. What's still unverified is wiring it into a real - running `enterprise-auth` container against a real external IdP. -- `enterprise/internal/loginhandler`'s full SAML login flow (§3b) -- - same bar as OIDC above, verified against a real fake SAML IdP - (`crewjam/saml/samlidp`: genuine XML signing and signature - verification, a real `AuthnRequest`/`Response` round trip), no Docker - needed. Writing this test caught two real bugs in - `enterprise/internal/saml`, now fixed: `ParseResponse` never called - `r.ParseForm()` before reading the POSTed `SAMLResponse` field (every - real ACS POST would have silently decoded to nothing), and the email- - attribute matching didn't recognize `urn:oid:0.9.2342.19200300.100.1.3` - (the standard LDAP "mail" OID), which is what an IdP sends by default - when the SP doesn't explicitly request an attribute literally named - "email" -- crewjam's own fake IdP hit this path. Same remaining gap as - OIDC: not yet tried against a real external IdP or a running - `enterprise-auth` container. -- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s - cross-tenant read isolation (§9), `search/src/consumer.rs`'s per-tenant - write-routing (§14), and `search/src/tenants.rs`'s active-tenant gate - (§14) -- verified live, no disclaimer needed, because Tantivy is an - embedded library with no Docker/broker dependency, and the active- - tenant gate's only external dependency (`enterprise-auth`'s HTTP API) - was exercised against a real hand-rolled test server, not a live - container: real indices, real documents, real commits, real HTTP - requests, all run in this environment. The one thing about it that's - still unverified is not Tantivy itself but the upstream credential/ - header plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s - `/internal/authorize-ingest` and `/internal/active-tenants`) against a - real running stack. -- The tenant-picker frontend page (§12) -- the first frontend-only piece - in this phase exercised in a real browser rather than only - type-checked: `web/src/routes/select-tenant`'s cross-origin - credentialed fetch/CORS/cookie handling, driven end-to-end via - `mcp__claude-in-chrome` against a throwaway server standing in for - `enterprise-auth`'s exact wire contract. What's unverified is the same - shape as OIDC/SAML above: this round trip against a real running - `enterprise-auth` container, not a stand-in. +1. `enterprise/Dockerfile` used a `context: enterprise` build context + too narrow for `enterprise/go.mod`'s `replace ../api` directive once + `enterprise-auth` started importing `api/httpserver` and (transitively) + `api/dashboards` -- fixed to build from the repo root, matching + `enterprise-api`/`enterprise-ingest`'s Dockerfiles. +2. `api/dashboards.Store.AddPanel`/`UpdatePanel` didn't call + `validatePanel` the way `CreateDashboard`'s inline panel-creation path + already did, so a panel added via those two methods could hit a + `viz_config` NOT NULL constraint violation instead of getting the + same default-empty-JSON treatment every other panel-creation path + gets -- fixed by calling `validatePanel` in both. +3. `enterprise/internal/rbacstore.SetDataSourceClickHouseCredentials` + let a malformed (non-UUID) id leak Postgres's raw `22P02` error past + the store's `ErrNotFound` boundary instead of treating "can't + possibly match a row" the same as "no such row" -- fixed by + translating that specific Postgres error code to `ErrNotFound`. +4. `enterprise/cmd/enterprise-api/main.go` registered `GET /healthz` + twice -- once explicitly, once already covered by + `queryapi.Handler.RegisterRoutes` -- which panics `net/http`'s + `ServeMux` on startup. This meant `enterprise-api` could never + actually start; every previous "built" claim for this binary had only + ever been a successful `go build`, never a successful process start. + Fixed by deleting the redundant registration. +5. ClickHouse's `default` admin user genuinely lacked `CREATE USER` + privilege in `docker-compose.yml` -- the official image needs + `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` (not the more obvious-looking + `CLICKHOUSE_ACCESS_MANAGEMENT`, confirmed by reading the image's own + `/entrypoint.sh` after the wrong name silently did nothing), which + this compose file never set. Every tenant-provisioning code path was + correct Go that had simply never been able to authenticate its own + admin connection strongly enough to run. +6. `enterprise/internal/tenantprovision.ProvisionClickHouse` relied on + ClickHouse RBAC's assumed "default-deny for a freshly created user" + for `system.*` access -- verified live to be false on ClickHouse + 24.8: a fresh tenant user could read `system.tables` (though not, it + turns out, `system.query_log`, which is genuinely access-checked). + Fixed with an explicit `REVOKE SELECT ON system.* FROM `, and + `TestProvisionedUserCannotReadSystemTables` was corrected to check + what each system table actually does under a proper revoke (hard + deny for `query_log`, verified-empty for `tables`) instead of + assuming both hard-deny identically. See + `/docs/security/threat-model.md`'s "Read this first" section for the + full account. -Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement -walkthrough, the dashboards tenant-scoping fix, the Helm chart, the -tenant-operator, and (newest) `internal/tenantprovision`/ -`internal/chrunner`'s live-ClickHouse tests — has unit/fake-client/ -`helm template` coverage (all passing, see each component's own `go -test`/`helm lint` output, including every `Skip*`-gated integration test -confirmed to skip cleanly offline) but has **not** been exercised -against a real running stack. Be specific when citing this runbook: "the -tests exist and pass structurally" is a true, verified claim; "isolation -was confirmed against real ClickHouse" is not, yet. If you're reading -this to decide whether Phase 4 is production-ready: it isn't yet, -independent of this gap — see `/docs/security/threat-model.md`'s -headline finding. This runbook exists so the first person with real -Docker/K8s access can actually close the loop, not to claim that already -happened. +**What's still not verified, and why**: §3a (OIDC) and §3b (SAML) both +need a real external IdP (Auth0/Okta developer tenant, or similar) that +this environment doesn't have credentials for -- the fake-IdP tests +(real RS256/XML crypto, no stand-in shortcuts) are the strongest +evidence available without one. §12's frontend-against-a-real- +`enterprise-auth`-container check is blocked by the same root cause: +there's no way to land on `/select-tenant` with a genuine pending-login +cookie from the real binary without a real IdP redirecting through it +first. §7 and §11's live-cluster steps need `kind`/`kubectl`, which +aren't installed in this environment -- their offline-only checks +(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing +the rendered YAML) all pass and are documented as such below. + +If you're reading this to decide whether Phase 4 is production-ready: +closer than before, but not yet -- see +`/docs/security/threat-model.md`'s "Read this first" section for the +current, precise state of every control, including the ones still +gated on a real IdP or a real cluster. ## 1. Bring up the stack @@ -283,12 +286,18 @@ PermissionStore`) have real integration tests, same skip-gated shape as §6 below: ```sh -docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \ +docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \ -e RBACSTORE_TEST_POSTGRES_ADDR=metadata-postgres:5432 \ -e RBACSTORE_TEST_POSTGRES_PASSWORD=sentry-dev-only \ golang:1.25-alpine go test ./internal/rbacstore/... -run DashboardPermission -v ``` +(Mount the repo root, not just `enterprise/`, with `-w /src/enterprise` as +the workdir -- same reason as `enterprise/Dockerfile`'s doc comment: +`dashboards_adapter.go` imports `api/dashboards`, and `enterprise/go.mod`'s +`replace ../api` directive needs that sibling directory actually present +in the build context, not just the package being tested.) + Expect all `TestDashboardPermission*`/`TestSetDashboardPermission*`/ `TestGetDashboardPermission*`/`TestRevokeDashboardPermission*`/ `TestListDashboardPermissions` tests to pass. This only takes effect @@ -298,18 +307,24 @@ when `enterprise-api` (not plain `api`) is serving traffic -- see ## 6. `enterprise/internal/rbacstore` and `internal/audit` (already verified — reconfirm here) ```sh -docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \ +docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \ -e RBACSTORE_TEST_POSTGRES_ADDR=metadata-postgres:5432 \ -e RBACSTORE_TEST_POSTGRES_PASSWORD=sentry-dev-only \ golang:1.25-alpine go test ./internal/rbacstore/... -v -docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \ +docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \ -e AUDIT_TEST_POSTGRES_ADDR=metadata-postgres:5432 \ -e AUDIT_TEST_POSTGRES_PASSWORD=audit-writer-dev-only \ -e AUDIT_TEST_ADMIN_PASSWORD=sentry-dev-only \ golang:1.25-alpine go test ./internal/audit/... -v ``` +(Same repo-root-mount reasoning as §5a above -- `internal/rbacstore` +imports `api/dashboards` unconditionally via `dashboards_adapter.go`, so +even running the whole package's tests, not just the `DashboardPermission` +subset, needs `api/` present. `internal/audit` has the same shape via +`queryapi_adapter.go`.) + ## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`) No live cluster was available to `kubectl apply` any of this. What can @@ -723,17 +738,23 @@ go test ./internal/tenantprovision/... -run TestProvisionedUserCanInsertIntoOwnD # (tenantprovision.go). This test proves the fix, not just documents it, # whenever a real ClickHouse is available to run it against. -go test ./internal/chwriter/... -run TestRegistryRoutesToCorrectTenant -v +go test ./internal/chwriter/... -run TestRegistryWritesEachTenantToItsOwnDatabase -v # skip-gated (CHWRITER_TEST_CLICKHOUSE_ADDR) -- writes a mixed batch # spanning two tenants in one WriteBatch call and confirms each row -# lands in its own tenant's database, none in the other's. +# lands in its own tenant's database, none in the other's. (Fixed from a +# stale `-run TestRegistryRoutesToCorrectTenant` -- that name never +# existed in this package; caught by actually running this command while +# closing out the live verification pass, same class of stale--run +# mistake §6's `TestRegistry` note already flagged once in this doc.) ``` -**Not run in this environment**: no live ClickHouse was available while -this was built, so the skip-gated tests above are correct Go that has -never actually executed -- "the test exists" is not the same claim as -"write-routing is confirmed," same caveat §8 already states for the -read-side `chrunner` tests. +**Now genuinely run against a live ClickHouse** (previously only +skip-gated, correct Go that had never executed): all 7 +`internal/chwriter` tests pass, including +`TestRegistryWritesEachTenantToItsOwnDatabase`, +`TestRegistryRefusesUnprovisionedTenant`, `TestRefreshAddsNewlyActiveTenant`, +and `TestRefreshRemovesNoLongerActiveTenant` -- write-routing is confirmed, +not just written, as of this pass. **Tantivy's side is built too, and genuinely verified.** `search/src/consumer.rs` reads the same `tenant_id` Kafka header and diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 1b89cb2..2f2bbec 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -9,39 +9,68 @@ for the full design rationale behind the controls described here. ## Read this first: the single most important open finding -**Updated a sixth time.** This section originally read "log data queried -through `POST /query` is not tenant-isolated at all," then "ClickHouse -is isolated but Tantivy isn't," then "ingest tags records with a tenant -identity but nothing routes the write," then "ClickHouse write-routing -is built but Tantivy's isn't," then "both are write-routed but neither -rechecks tenant-active status live," then "both recheck, but ClickHouse's -snapshot never refreshes while Tantivy's does." Both storage engines are -now isolated on both the read and write paths, **both gate writes on an -active-tenant check, and both now refresh that check periodically** — -`chwriter.Registry.StartRefreshing` (new) closes the asymmetry the -previous version of this section named: ClickHouse's writer map now -re-lists active tenants every minute, the same interval -`tenants.ActiveTenantTracker` already used on the Tantivy side, opening +**Updated a seventh time — and this time the headline actually changes.** +This section originally read "log data queried through `POST /query` is +not tenant-isolated at all," then "ClickHouse is isolated but Tantivy +isn't," then "ingest tags records with a tenant identity but nothing +routes the write," then "ClickHouse write-routing is built but Tantivy's +isn't," then "both are write-routed but neither rechecks tenant-active +status live," then "both recheck, but ClickHouse's snapshot never +refreshes while Tantivy's does," then (sixth) "both engines are built +and code-complete, but the ClickHouse half has never actually run +against a real ClickHouse." That last gap is now closed: Docker access +became available, and every ClickHouse-dependent piece named below — +`tenantprovision`, `chrunner`, `chwriter`, the `system.*` metadata +isolation check, and `enterprise-api` itself actually starting and +serving traffic — has been run against a real +`clickhouse/clickhouse-server:24.8` and a real Postgres, not just +type-checked or run against fakes. Closing this loop caught and fixed +six real bugs that no amount of Docker-free testing could have found: +a Docker build-context bug that made `enterprise-auth` fail to build at +all, a validation gap in `dashboards.Store.AddPanel`/`UpdatePanel`, a +raw Postgres error leaking past `rbacstore`'s `ErrNotFound` boundary, +`enterprise-api` panicking on startup from a duplicate `GET /healthz` +route registration, ClickHouse's `default` user genuinely lacking +`CREATE USER` privilege until `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` +was added to `docker-compose.yml`, and — most load-bearing — a freshly +provisioned tenant user was **not** actually default-denied from +`system.*` on this ClickHouse version, exactly the risk task 2's +original design flagged as needing live verification rather than trust. +See `/docs/phase-4-runbook.md` §§1–14 for the full list of what was run +and what each finding was. + +Both storage engines are isolated on both the read and write paths, +**both gate writes on an active-tenant check, and both refresh that +check periodically** — `chwriter.Registry.StartRefreshing` re-lists +active tenants every minute, the same interval +`tenants.ActiveTenantTracker` uses on the Tantivy side, opening connections for newly-active tenants and closing/removing ones no -longer active — a deprovisioned tenant now loses ClickHouse write access +longer active — a deprovisioned tenant loses ClickHouse write access within a minute, not "until the next `enterprise-ingest` restart." Neither engine does a live per-write check (a database/HTTP round trip per record would be a real throughput cost neither implementation accepts), so a minute-wide staleness window remains on both sides by -design, not by oversight. What's left is narrower still: **whether a -given deployment actually runs the isolated binaries** (deployment-time, -not code-level). See below for both engines' write-routing, in full. +design, not by oversight. What's left: two tenants (`acme`, `globex`) +have been provisioned and exercised end-to-end in this environment, but +only via direct ClickHouse-user connections and Go integration tests — +not yet via a real logged-in human session walking through `POST +/query` in a browser, since that still needs OIDC/SAML wired to a real +external IdP (see §3a/§3b below). And **whether a given deployment +actually runs the isolated binaries** remains a deployment-time +decision, not a code-level guarantee — see below. -**ClickHouse (the SQL path) is built.** `enterprise/internal/ -tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and -`enterprise/internal/chrunner` (a per-tenant connection registry -implementing `api/querylang/executor.SQLRunner`, resolving the tenant -from request identity, never a parameter) are wired into -`enterprise/cmd/enterprise-api`. **Not yet confirmed against a real -ClickHouse** — this environment had no Docker/database access while -these were written; the tests exist and are correct Go, but "the test -exists" is not the same claim as "isolation is confirmed" (see -`/docs/phase-4-runbook.md`). +**ClickHouse (the SQL path) is built and now genuinely verified live.** +`enterprise/internal/tenantprovision` (real `CREATE DATABASE`/`CREATE +USER`/`GRANT`/`REVOKE`) and `enterprise/internal/chrunner` (a per-tenant +connection registry implementing `api/querylang/executor.SQLRunner`, +resolving the tenant from request identity, never a parameter) are wired +into `enterprise/cmd/enterprise-api`, which has itself been built, +started, and confirmed serving traffic on a real docker-compose stack. +Every ClickHouse-dependent test in both packages passes against a real +`clickhouse/clickhouse-server:24.8`, including the cross-tenant raw-SQL +probe and the corrected `system.*` metadata-isolation check (see "Known +residual risks" below for the real, version-specific wrinkle that check +found). **Tantivy (the free-text path) is also built, and — unlike the ClickHouse pieces — genuinely verified in this environment.** @@ -75,8 +104,11 @@ sharing the same host port/network-alias trick to stay transparent to `alerting`/`web` either way — verified via `docker compose config` (renders and validates the merged YAML without a daemon; confirms `api`/`enterprise-api` never both appear in `--services` output for the -same profile selection) — see `/docs/phase-4-runbook.md` §10a. This -still only constrains *deployment*, not *operation*: nothing stops an +same profile selection) *and* via an actual `docker compose up` of +`enterprise-api` in this environment, which is what caught the duplicate +`GET /healthz` route panic mentioned above — see +`/docs/phase-4-runbook.md` §10a. This still only constrains +*deployment*, not *operation*: nothing stops an operator from manually running plain `api`'s image against a cluster (or compose project) that has tenants provisioned, pointing at the same ClickHouse/Postgres. The Helm chart @@ -455,21 +487,27 @@ terms: credentials. That's an operational control (credential custody, infrastructure access review), out of scope for this system's own code. -- **`system.query_log` metadata leakage — per-tenant users are now - real, but the check itself hasn't run yet.** Was an open verification - item because there were no per-tenant ClickHouse users to check - against; that blocker is gone (`enterprise/internal/tenantprovision` - exists), and `tenantprovision_test.go`'s - `TestProvisionedUserCannotReadSystemTables` asserts exactly what the - design calls for (`system.query_log`/`system.tables` inaccessible, - `SHOW DATABASES` not revealing other tenants) — but this environment - never had ClickHouse access to actually run it, so it remains - unconfirmed against the pinned version - (`clickhouse/clickhouse-server:24.8`) until someone with Docker access - runs it (`/docs/phase-4-runbook.md` §8). Also still contingent on the - deployment-shape caveat at the top of this document: even once - confirmed, this only holds when `enterprise-api` (not plain `api`) is - actually serving traffic. +- **`system.query_log` metadata leakage — now closed and verified live** + against the pinned version (`clickhouse/clickhouse-server:24.8`), with + a real, non-obvious wrinkle: a freshly created tenant user was **not** + default-denied from `system.*` the way the original design assumed + (`ProvisionClickHouse` now issues an explicit `REVOKE SELECT ON + system.* FROM ` — see that function's doc comment). Confirming + this live also surfaced a genuine ClickHouse behavioral split the + design doc didn't anticipate: `system.query_log` is a real + access-checked table (the REVOKE makes it hard-deny, + `ACCESS_DENIED`), but `system.tables` is a filtered *catalog* view + that ClickHouse 24.8 never denies outright regardless of grants — it + just silently returns zero rows for a properly-revoked user. Both + outcomes close the actual leak (no other tenant's query text or + database/table names are visible either way); + `tenantprovision_test.go`'s `TestProvisionedUserCannotReadSystemTables` + was corrected to assert what each table actually does (hard error for + `query_log`, verified-empty-and-no-foreign-database-names for + `tables`) rather than demanding a hard error from both. Still + contingent on the deployment-shape caveat at the top of this document: + this only holds when `enterprise-api` (not plain `api`) is actually + serving traffic. - **No deny-override grants** — `dashboard_permissions` is additive-only by design; a full allow/deny ACL system is unbuilt, future work. - **No data retention/deletion policy** for a deprovisioned tenant — @@ -506,20 +544,20 @@ terms: | Role-based access control on `/query`, `/dashboards` | **Enforced** | | `alerting`↔`api` service-identity credential | **Enforced** | | Tenant scoping on dashboards (control-plane data) | **Enforced** | -| ClickHouse per-tenant provisioning (`tenantprovision`) | **Built, not live-verified** — real integration test exists, not yet run against ClickHouse | -| ClickHouse query routing (`chrunner`) | **Built, not live-verified** — and only applies when `enterprise-api` serves traffic, not plain `api` | -| `system.*` ClickHouse metadata isolation | **Built, not live-verified** — same caveat as above | +| ClickHouse per-tenant provisioning (`tenantprovision`) | **Enforced, verified live** against `clickhouse/clickhouse-server:24.8` — real `CREATE DATABASE`/`CREATE USER`/`GRANT`/`REVOKE`, real two-tenant cross-read probe | +| ClickHouse query routing (`chrunner`) | **Enforced, verified live** — real per-tenant connections, cross-tenant raw-SQL probe passes; only applies when `enterprise-api` serves traffic, not plain `api` | +| `system.*` ClickHouse metadata isolation | **Enforced, verified live** — `system.query_log` hard-denies, `system.tables` returns zero foreign rows (see "Known residual risks" below for the ClickHouse-version-specific split between the two) | | Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing | | Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test | | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record | -| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Active-tenant snapshot now refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" | +| Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" | | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two | | Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO | -| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment | +| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` | | Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) | | Human SSO login — SAML | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) | -| Multi-tenant-membership login (tenant picker) | **Backend and frontend built and verified** (`GET /auth/memberships`, `POST /auth/select-tenant`, a pending-login token distinct from a real session; `web/src/routes/select-tenant` calls it via credentialed cross-origin fetch, genuinely exercised in a real browser) — not yet tried against a real running `enterprise-auth` container | -| Per-resource dashboard grants (`own/granted`) | **Built, unit-tested against a fake store; live-Postgres integration tests written, not run in this environment** (only when `enterprise-api` serves traffic — plain `api` falls back to own/Admin only) | +| Multi-tenant-membership login (tenant picker) | **Backend and frontend built and verified** (`GET /auth/memberships`, `POST /auth/select-tenant`, a pending-login token distinct from a real session; `web/src/routes/select-tenant` calls it via credentialed cross-origin fetch, genuinely exercised in a real browser) — not yet tried against a real running `enterprise-auth` container, still blocked on the same "no real external IdP" gap as OIDC/SAML above (there's no way to reach `/select-tenant` with a genuine pending-login cookie from the real binary without one) | +| Per-resource dashboard grants (`own/granted`) | **Enforced, verified live** — real Postgres integration tests for `dashboard_permissions` CRUD and the `PermissionStore` adapter all pass (only when `enterprise-api` serves traffic — plain `api` falls back to own/Admin only) | | Query audit logging (routine queries) | **Enforced**, fail-open, and now wired to a real writer via `enterprise-api` (`audit.QueryAPILogger`) | | Audit log tamper detection (hash chain) | **Enforced**, verified live | | Audit log tamper prevention (external anchoring) | **Design only** — `FileSink` is a dev stand-in |