Update Phase 4 docs: the ClickHouse-side live verification actually ran

Docker access became available and §§1-10, 10a, 13, 14, and most of §8
of the runbook have now genuinely been run against a real docker-
compose stack, not just documented as a procedure to run. Rewrites the
runbook's "Verification status" section and the threat model's "Read
this first" finding to describe what was actually confirmed (including
the six real bugs this pass found and fixed) versus what's still gated
on a real external IdP (§3a/§3b/§12) or a real Kubernetes cluster
(§7/§11's live-cluster halves). Also fixes two stale runbook commands:
§5a/§6's docker run mounts needed the repo root, not just enterprise/,
for the same replace-directive reason the Dockerfile fix does; §14 cited
a chwriter test name that never existed.

docs/phase-4-runbook.md

@@ -6,75 +6,78 @@ logging, and a Kubernetes deployment path. Read those first.
 
 ## Verification status — read this before the rest of this doc
 
-Every prior phase's runbook documents claims **checked against the live
+**Docker daemon access became available partway through this phase's
-stack**, not asserted. This one is different, and says so plainly rather
+work, and the rest of this runbook (§§1–10, 10a, 13, 14, most of §8) has
-than papering over it: for the great majority of this phase's work,
+now genuinely been run against a real docker-compose stack** — real
-**there was no working Docker daemon access and no reachable Kubernetes
+ClickHouse (`clickhouse/clickhouse-server:24.8`), real Postgres, real
-cluster**, so most of what follows is a *procedure to run*, not a report
+`enterprise-auth`/`enterprise-api`/`enterprise-ingest` containers, two
-of what was already run and passed. Five genuine exceptions so far:
+real provisioned tenants (`acme`, `globex`). This closed every
+previously-disclosed "written but never run" gap for the ClickHouse
+side, and along the way found and fixed six real bugs no amount of
+Docker-free testing could have caught:
 
-- `enterprise/internal/audit`'s hash-chain, tamper-detection, and
+1. `enterprise/Dockerfile` used a `context: enterprise` build context
-  concurrent-write guarantees (task 4) -- verified live against a real
+   too narrow for `enterprise/go.mod`'s `replace ../api` directive once
-  Postgres earlier in this phase's work (see its own doc comments for
+   `enterprise-auth` started importing `api/httpserver` and (transitively)
-  the exact `docker run` invocations), before the environment lost
+   `api/dashboards` -- fixed to build from the repo root, matching
-  Docker access.
+   `enterprise-api`/`enterprise-ingest`'s Dockerfiles.
-- `enterprise/internal/loginhandler`'s full OIDC login flow (§3a) --
+2. `api/dashboards.Store.AddPanel`/`UpdatePanel` didn't call
-  verified with real cryptography (a fake IdP that signs and verifies
+   `validatePanel` the way `CreateDashboard`'s inline panel-creation path
-  genuine RS256 tokens) *without* needing Docker or a live database at
+   already did, so a panel added via those two methods could hit a
-  all, so this one was actually run in this runbook's own session, not
+   `viz_config` NOT NULL constraint violation instead of getting the
-  just an earlier one. What's still unverified is wiring it into a real
+   same default-empty-JSON treatment every other panel-creation path
-  running `enterprise-auth` container against a real external IdP.
+   gets -- fixed by calling `validatePanel` in both.
-- `enterprise/internal/loginhandler`'s full SAML login flow (§3b) --
+3. `enterprise/internal/rbacstore.SetDataSourceClickHouseCredentials`
-  same bar as OIDC above, verified against a real fake SAML IdP
+   let a malformed (non-UUID) id leak Postgres's raw `22P02` error past
-  (`crewjam/saml/samlidp`: genuine XML signing and signature
+   the store's `ErrNotFound` boundary instead of treating "can't
-  verification, a real `AuthnRequest`/`Response` round trip), no Docker
+   possibly match a row" the same as "no such row" -- fixed by
-  needed. Writing this test caught two real bugs in
+   translating that specific Postgres error code to `ErrNotFound`.
-  `enterprise/internal/saml`, now fixed: `ParseResponse` never called
+4. `enterprise/cmd/enterprise-api/main.go` registered `GET /healthz`
-  `r.ParseForm()` before reading the POSTed `SAMLResponse` field (every
+   twice -- once explicitly, once already covered by
-  real ACS POST would have silently decoded to nothing), and the email-
+   `queryapi.Handler.RegisterRoutes` -- which panics `net/http`'s
-  attribute matching didn't recognize `urn:oid:0.9.2342.19200300.100.1.3`
+   `ServeMux` on startup. This meant `enterprise-api` could never
-  (the standard LDAP "mail" OID), which is what an IdP sends by default
+   actually start; every previous "built" claim for this binary had only
-  when the SP doesn't explicitly request an attribute literally named
+   ever been a successful `go build`, never a successful process start.
-  "email" -- crewjam's own fake IdP hit this path. Same remaining gap as
+   Fixed by deleting the redundant registration.
-  OIDC: not yet tried against a real external IdP or a running
+5. ClickHouse's `default` admin user genuinely lacked `CREATE USER`
-  `enterprise-auth` container.
+   privilege in `docker-compose.yml` -- the official image needs
-- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s
+   `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` (not the more obvious-looking
-  cross-tenant read isolation (§9), `search/src/consumer.rs`'s per-tenant
+   `CLICKHOUSE_ACCESS_MANAGEMENT`, confirmed by reading the image's own
-  write-routing (§14), and `search/src/tenants.rs`'s active-tenant gate
+   `/entrypoint.sh` after the wrong name silently did nothing), which
-  (§14) -- verified live, no disclaimer needed, because Tantivy is an
+   this compose file never set. Every tenant-provisioning code path was
-  embedded library with no Docker/broker dependency, and the active-
+   correct Go that had simply never been able to authenticate its own
-  tenant gate's only external dependency (`enterprise-auth`'s HTTP API)
+   admin connection strongly enough to run.
-  was exercised against a real hand-rolled test server, not a live
+6. `enterprise/internal/tenantprovision.ProvisionClickHouse` relied on
-  container: real indices, real documents, real commits, real HTTP
+   ClickHouse RBAC's assumed "default-deny for a freshly created user"
-  requests, all run in this environment. The one thing about it that's
+   for `system.*` access -- verified live to be false on ClickHouse
-  still unverified is not Tantivy itself but the upstream credential/
+   24.8: a fresh tenant user could read `system.tables` (though not, it
-  header plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
+   turns out, `system.query_log`, which is genuinely access-checked).
-  `/internal/authorize-ingest` and `/internal/active-tenants`) against a
+   Fixed with an explicit `REVOKE SELECT ON system.* FROM <user>`, and
-  real running stack.
+   `TestProvisionedUserCannotReadSystemTables` was corrected to check
-- The tenant-picker frontend page (§12) -- the first frontend-only piece
+   what each system table actually does under a proper revoke (hard
-  in this phase exercised in a real browser rather than only
+   deny for `query_log`, verified-empty for `tables`) instead of
-  type-checked: `web/src/routes/select-tenant`'s cross-origin
+   assuming both hard-deny identically. See
-  credentialed fetch/CORS/cookie handling, driven end-to-end via
+   `/docs/security/threat-model.md`'s "Read this first" section for the
-  `mcp__claude-in-chrome` against a throwaway server standing in for
+   full account.
-  `enterprise-auth`'s exact wire contract. What's unverified is the same
-  shape as OIDC/SAML above: this round trip against a real running
-  `enterprise-auth` container, not a stand-in.
 
-Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement
+**What's still not verified, and why**: §3a (OIDC) and §3b (SAML) both
-walkthrough, the dashboards tenant-scoping fix, the Helm chart, the
+need a real external IdP (Auth0/Okta developer tenant, or similar) that
-tenant-operator, and (newest) `internal/tenantprovision`/
+this environment doesn't have credentials for -- the fake-IdP tests
-`internal/chrunner`'s live-ClickHouse tests — has unit/fake-client/
+(real RS256/XML crypto, no stand-in shortcuts) are the strongest
-`helm template` coverage (all passing, see each component's own `go
+evidence available without one. §12's frontend-against-a-real-
-test`/`helm lint` output, including every `Skip*`-gated integration test
+`enterprise-auth`-container check is blocked by the same root cause:
-confirmed to skip cleanly offline) but has **not** been exercised
+there's no way to land on `/select-tenant` with a genuine pending-login
-against a real running stack. Be specific when citing this runbook: "the
+cookie from the real binary without a real IdP redirecting through it
-tests exist and pass structurally" is a true, verified claim; "isolation
+first. §7 and §11's live-cluster steps need `kind`/`kubectl`, which
-was confirmed against real ClickHouse" is not, yet. If you're reading
+aren't installed in this environment -- their offline-only checks
-this to decide whether Phase 4 is production-ready: it isn't yet,
+(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing
-independent of this gap — see `/docs/security/threat-model.md`'s
+the rendered YAML) all pass and are documented as such below.
-headline finding. This runbook exists so the first person with real
+
-Docker/K8s access can actually close the loop, not to claim that already
+If you're reading this to decide whether Phase 4 is production-ready:
-happened.
+closer than before, but not yet -- see
+`/docs/security/threat-model.md`'s "Read this first" section for the
+current, precise state of every control, including the ones still
+gated on a real IdP or a real cluster.
 
 ## 1. Bring up the stack
 
@@ -283,12 +286,18 @@ PermissionStore`) have real integration tests, same skip-gated shape as
 §6 below:
 
 ```sh
-docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
+docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \
   -e RBACSTORE_TEST_POSTGRES_ADDR=metadata-postgres:5432 \
   -e RBACSTORE_TEST_POSTGRES_PASSWORD=sentry-dev-only \
   golang:1.25-alpine go test ./internal/rbacstore/... -run DashboardPermission -v
 ```
 
+(Mount the repo root, not just `enterprise/`, with `-w /src/enterprise` as
+the workdir -- same reason as `enterprise/Dockerfile`'s doc comment:
+`dashboards_adapter.go` imports `api/dashboards`, and `enterprise/go.mod`'s
+`replace ../api` directive needs that sibling directory actually present
+in the build context, not just the package being tested.)
+
 Expect all `TestDashboardPermission*`/`TestSetDashboardPermission*`/
 `TestGetDashboardPermission*`/`TestRevokeDashboardPermission*`/
 `TestListDashboardPermissions` tests to pass. This only takes effect
@@ -298,18 +307,24 @@ when `enterprise-api` (not plain `api`) is serving traffic -- see
 ## 6. `enterprise/internal/rbacstore` and `internal/audit` (already verified — reconfirm here)
 
 ```sh
-docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
+docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \
   -e RBACSTORE_TEST_POSTGRES_ADDR=metadata-postgres:5432 \
   -e RBACSTORE_TEST_POSTGRES_PASSWORD=sentry-dev-only \
   golang:1.25-alpine go test ./internal/rbacstore/... -v
 
-docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
+docker run --rm --network sentry_default -v $(pwd):/src -w /src/enterprise \
   -e AUDIT_TEST_POSTGRES_ADDR=metadata-postgres:5432 \
   -e AUDIT_TEST_POSTGRES_PASSWORD=audit-writer-dev-only \
   -e AUDIT_TEST_ADMIN_PASSWORD=sentry-dev-only \
   golang:1.25-alpine go test ./internal/audit/... -v
 ```
 
+(Same repo-root-mount reasoning as §5a above -- `internal/rbacstore`
+imports `api/dashboards` unconditionally via `dashboards_adapter.go`, so
+even running the whole package's tests, not just the `DashboardPermission`
+subset, needs `api/` present. `internal/audit` has the same shape via
+`queryapi_adapter.go`.)
+
 ## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`)
 
 No live cluster was available to `kubectl apply` any of this. What can
@@ -723,17 +738,23 @@ go test ./internal/tenantprovision/... -run TestProvisionedUserCanInsertIntoOwnD
 # (tenantprovision.go). This test proves the fix, not just documents it,
 # whenever a real ClickHouse is available to run it against.
 
-go test ./internal/chwriter/... -run TestRegistryRoutesToCorrectTenant -v
+go test ./internal/chwriter/... -run TestRegistryWritesEachTenantToItsOwnDatabase -v
 # skip-gated (CHWRITER_TEST_CLICKHOUSE_ADDR) -- writes a mixed batch
 # spanning two tenants in one WriteBatch call and confirms each row
-# lands in its own tenant's database, none in the other's.
+# lands in its own tenant's database, none in the other's. (Fixed from a
+# stale `-run TestRegistryRoutesToCorrectTenant` -- that name never
+# existed in this package; caught by actually running this command while
+# closing out the live verification pass, same class of stale--run
+# mistake §6's `TestRegistry` note already flagged once in this doc.)
 ```
 
-**Not run in this environment**: no live ClickHouse was available while
+**Now genuinely run against a live ClickHouse** (previously only
-this was built, so the skip-gated tests above are correct Go that has
+skip-gated, correct Go that had never executed): all 7
-never actually executed -- "the test exists" is not the same claim as
+`internal/chwriter` tests pass, including
-"write-routing is confirmed," same caveat §8 already states for the
+`TestRegistryWritesEachTenantToItsOwnDatabase`,
-read-side `chrunner` tests.
+`TestRegistryRefusesUnprovisionedTenant`, `TestRefreshAddsNewlyActiveTenant`,
+and `TestRefreshRemovesNoLongerActiveTenant` -- write-routing is confirmed,
+not just written, as of this pass.
 
 **Tantivy's side is built too, and genuinely verified.**
 `search/src/consumer.rs` reads the same `tenant_id` Kafka header and

docs/security/threat-model.md

@@ -9,39 +9,68 @@ for the full design rationale behind the controls described here.
 
 ## Read this first: the single most important open finding
 
-**Updated a sixth time.** This section originally read "log data queried
+**Updated a seventh time — and this time the headline actually changes.**
-through `POST /query` is not tenant-isolated at all," then "ClickHouse
+This section originally read "log data queried through `POST /query` is
-is isolated but Tantivy isn't," then "ingest tags records with a tenant
+not tenant-isolated at all," then "ClickHouse is isolated but Tantivy
-identity but nothing routes the write," then "ClickHouse write-routing
+isn't," then "ingest tags records with a tenant identity but nothing
-is built but Tantivy's isn't," then "both are write-routed but neither
+routes the write," then "ClickHouse write-routing is built but Tantivy's
-rechecks tenant-active status live," then "both recheck, but ClickHouse's
+isn't," then "both are write-routed but neither rechecks tenant-active
-snapshot never refreshes while Tantivy's does." Both storage engines are
+status live," then "both recheck, but ClickHouse's snapshot never
-now isolated on both the read and write paths, **both gate writes on an
+refreshes while Tantivy's does," then (sixth) "both engines are built
-active-tenant check, and both now refresh that check periodically** —
+and code-complete, but the ClickHouse half has never actually run
-`chwriter.Registry.StartRefreshing` (new) closes the asymmetry the
+against a real ClickHouse." That last gap is now closed: Docker access
-previous version of this section named: ClickHouse's writer map now
+became available, and every ClickHouse-dependent piece named below —
-re-lists active tenants every minute, the same interval
+`tenantprovision`, `chrunner`, `chwriter`, the `system.*` metadata
-`tenants.ActiveTenantTracker` already used on the Tantivy side, opening
+isolation check, and `enterprise-api` itself actually starting and
+serving traffic — has been run against a real
+`clickhouse/clickhouse-server:24.8` and a real Postgres, not just
+type-checked or run against fakes. Closing this loop caught and fixed
+six real bugs that no amount of Docker-free testing could have found:
+a Docker build-context bug that made `enterprise-auth` fail to build at
+all, a validation gap in `dashboards.Store.AddPanel`/`UpdatePanel`, a
+raw Postgres error leaking past `rbacstore`'s `ErrNotFound` boundary,
+`enterprise-api` panicking on startup from a duplicate `GET /healthz`
+route registration, ClickHouse's `default` user genuinely lacking
+`CREATE USER` privilege until `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1`
+was added to `docker-compose.yml`, and — most load-bearing — a freshly
+provisioned tenant user was **not** actually default-denied from
+`system.*` on this ClickHouse version, exactly the risk task 2's
+original design flagged as needing live verification rather than trust.
+See `/docs/phase-4-runbook.md` §§1–14 for the full list of what was run
+and what each finding was.
+
+Both storage engines are isolated on both the read and write paths,
+**both gate writes on an active-tenant check, and both refresh that
+check periodically** — `chwriter.Registry.StartRefreshing` re-lists
+active tenants every minute, the same interval
+`tenants.ActiveTenantTracker` uses on the Tantivy side, opening
 connections for newly-active tenants and closing/removing ones no
-longer active — a deprovisioned tenant now loses ClickHouse write access
+longer active — a deprovisioned tenant loses ClickHouse write access
 within a minute, not "until the next `enterprise-ingest` restart."
 Neither engine does a live per-write check (a database/HTTP round trip
 per record would be a real throughput cost neither implementation
 accepts), so a minute-wide staleness window remains on both sides by
-design, not by oversight. What's left is narrower still: **whether a
+design, not by oversight. What's left: two tenants (`acme`, `globex`)
-given deployment actually runs the isolated binaries** (deployment-time,
+have been provisioned and exercised end-to-end in this environment, but
-not code-level). See below for both engines' write-routing, in full.
+only via direct ClickHouse-user connections and Go integration tests —
+not yet via a real logged-in human session walking through `POST
+/query` in a browser, since that still needs OIDC/SAML wired to a real
+external IdP (see §3a/§3b below). And **whether a given deployment
+actually runs the isolated binaries** remains a deployment-time
+decision, not a code-level guarantee — see below.
 
-**ClickHouse (the SQL path) is built.** `enterprise/internal/
+**ClickHouse (the SQL path) is built and now genuinely verified live.**
-tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
+`enterprise/internal/tenantprovision` (real `CREATE DATABASE`/`CREATE
-`enterprise/internal/chrunner` (a per-tenant connection registry
+USER`/`GRANT`/`REVOKE`) and `enterprise/internal/chrunner` (a per-tenant
-implementing `api/querylang/executor.SQLRunner`, resolving the tenant
+connection registry implementing `api/querylang/executor.SQLRunner`,
-from request identity, never a parameter) are wired into
+resolving the tenant from request identity, never a parameter) are wired
-`enterprise/cmd/enterprise-api`. **Not yet confirmed against a real
+into `enterprise/cmd/enterprise-api`, which has itself been built,
-ClickHouse** — this environment had no Docker/database access while
+started, and confirmed serving traffic on a real docker-compose stack.
-these were written; the tests exist and are correct Go, but "the test
+Every ClickHouse-dependent test in both packages passes against a real
-exists" is not the same claim as "isolation is confirmed" (see
+`clickhouse/clickhouse-server:24.8`, including the cross-tenant raw-SQL
-`/docs/phase-4-runbook.md`).
+probe and the corrected `system.*` metadata-isolation check (see "Known
+residual risks" below for the real, version-specific wrinkle that check
+found).
 
 **Tantivy (the free-text path) is also built, and — unlike the
 ClickHouse pieces — genuinely verified in this environment.**
@@ -75,8 +104,11 @@ sharing the same host port/network-alias trick to stay transparent to
 `alerting`/`web` either way — verified via `docker compose config`
 (renders and validates the merged YAML without a daemon; confirms
 `api`/`enterprise-api` never both appear in `--services` output for the
-same profile selection) — see `/docs/phase-4-runbook.md` §10a. This
+same profile selection) *and* via an actual `docker compose up` of
-still only constrains *deployment*, not *operation*: nothing stops an
+`enterprise-api` in this environment, which is what caught the duplicate
+`GET /healthz` route panic mentioned above — see
+`/docs/phase-4-runbook.md` §10a. This still only constrains
+*deployment*, not *operation*: nothing stops an
 operator from manually running plain `api`'s image against a cluster
 (or compose project) that has tenants
 provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
@@ -455,21 +487,27 @@ terms:
   credentials. That's an operational control (credential custody,
   infrastructure access review), out of scope for this system's own
   code.
-- **`system.query_log` metadata leakage — per-tenant users are now
+- **`system.query_log` metadata leakage — now closed and verified live**
-  real, but the check itself hasn't run yet.** Was an open verification
+  against the pinned version (`clickhouse/clickhouse-server:24.8`), with
-  item because there were no per-tenant ClickHouse users to check
+  a real, non-obvious wrinkle: a freshly created tenant user was **not**
-  against; that blocker is gone (`enterprise/internal/tenantprovision`
+  default-denied from `system.*` the way the original design assumed
-  exists), and `tenantprovision_test.go`'s
+  (`ProvisionClickHouse` now issues an explicit `REVOKE SELECT ON
-  `TestProvisionedUserCannotReadSystemTables` asserts exactly what the
+  system.* FROM <user>` — see that function's doc comment). Confirming
-  design calls for (`system.query_log`/`system.tables` inaccessible,
+  this live also surfaced a genuine ClickHouse behavioral split the
-  `SHOW DATABASES` not revealing other tenants) — but this environment
+  design doc didn't anticipate: `system.query_log` is a real
-  never had ClickHouse access to actually run it, so it remains
+  access-checked table (the REVOKE makes it hard-deny,
-  unconfirmed against the pinned version
+  `ACCESS_DENIED`), but `system.tables` is a filtered *catalog* view
-  (`clickhouse/clickhouse-server:24.8`) until someone with Docker access
+  that ClickHouse 24.8 never denies outright regardless of grants — it
-  runs it (`/docs/phase-4-runbook.md` §8). Also still contingent on the
+  just silently returns zero rows for a properly-revoked user. Both
-  deployment-shape caveat at the top of this document: even once
+  outcomes close the actual leak (no other tenant's query text or
-  confirmed, this only holds when `enterprise-api` (not plain `api`) is
+  database/table names are visible either way);
-  actually serving traffic.
+  `tenantprovision_test.go`'s `TestProvisionedUserCannotReadSystemTables`
+  was corrected to assert what each table actually does (hard error for
+  `query_log`, verified-empty-and-no-foreign-database-names for
+  `tables`) rather than demanding a hard error from both. Still
+  contingent on the deployment-shape caveat at the top of this document:
+  this only holds when `enterprise-api` (not plain `api`) is actually
+  serving traffic.
 - **No deny-override grants** — `dashboard_permissions` is additive-only
   by design; a full allow/deny ACL system is unbuilt, future work.
 - **No data retention/deletion policy** for a deprovisioned tenant —
@@ -506,20 +544,20 @@ terms:
 | Role-based access control on `/query`, `/dashboards` | **Enforced** |
 | `alerting`↔`api` service-identity credential | **Enforced** |
 | Tenant scoping on dashboards (control-plane data) | **Enforced** |
-| ClickHouse per-tenant provisioning (`tenantprovision`) | **Built, not live-verified** — real integration test exists, not yet run against ClickHouse |
+| ClickHouse per-tenant provisioning (`tenantprovision`) | **Enforced, verified live** against `clickhouse/clickhouse-server:24.8` — real `CREATE DATABASE`/`CREATE USER`/`GRANT`/`REVOKE`, real two-tenant cross-read probe |
-| ClickHouse query routing (`chrunner`) | **Built, not live-verified** — and only applies when `enterprise-api` serves traffic, not plain `api` |
+| ClickHouse query routing (`chrunner`) | **Enforced, verified live** — real per-tenant connections, cross-tenant raw-SQL probe passes; only applies when `enterprise-api` serves traffic, not plain `api` |
-| `system.*` ClickHouse metadata isolation | **Built, not live-verified** — same caveat as above |
+| `system.*` ClickHouse metadata isolation | **Enforced, verified live** — `system.query_log` hard-denies, `system.tables` returns zero foreign rows (see "Known residual risks" below for the ClickHouse-version-specific split between the two) |
 | Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
 | Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
 | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
-| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Active-tenant snapshot now refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
+| Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
 | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
 | Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
-| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
+| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` |
 | Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
 | Human SSO login — SAML | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
-| Multi-tenant-membership login (tenant picker) | **Backend and frontend built and verified** (`GET /auth/memberships`, `POST /auth/select-tenant`, a pending-login token distinct from a real session; `web/src/routes/select-tenant` calls it via credentialed cross-origin fetch, genuinely exercised in a real browser) — not yet tried against a real running `enterprise-auth` container |
+| Multi-tenant-membership login (tenant picker) | **Backend and frontend built and verified** (`GET /auth/memberships`, `POST /auth/select-tenant`, a pending-login token distinct from a real session; `web/src/routes/select-tenant` calls it via credentialed cross-origin fetch, genuinely exercised in a real browser) — not yet tried against a real running `enterprise-auth` container, still blocked on the same "no real external IdP" gap as OIDC/SAML above (there's no way to reach `/select-tenant` with a genuine pending-login cookie from the real binary without one) |
-| Per-resource dashboard grants (`own/granted`) | **Built, unit-tested against a fake store; live-Postgres integration tests written, not run in this environment** (only when `enterprise-api` serves traffic — plain `api` falls back to own/Admin only) |
+| Per-resource dashboard grants (`own/granted`) | **Enforced, verified live** — real Postgres integration tests for `dashboard_permissions` CRUD and the `PermissionStore` adapter all pass (only when `enterprise-api` serves traffic — plain `api` falls back to own/Admin only) |
 | Query audit logging (routine queries) | **Enforced**, fail-open, and now wired to a real writer via `enterprise-api` (`audit.QueryAPILogger`) |
 | Audit log tamper detection (hash chain) | **Enforced**, verified live |
 | Audit log tamper prevention (external anchoring) | **Design only** — `FileSink` is a dev stand-in |