Document real kind-cluster verification, closing every runbook gap

kind/kubectl/helm were installed without root and a real local cluster
ran the full two-tenant walkthrough end to end: both acme and globex
reached Tenant.status.phase: Active with real generated ClickHouse
credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's
verification-status framing from "not verified against a live cluster"
to what's actually true now, fixes both READMEs' helm install
--include-crds (a helm template-only flag, never valid for install),
documents the two chart bugs this run found (see the previous commit)
and the mandatory ingest TLS Secret step, and updates the threat model's
summary table and "No general multi-cluster orchestration" residual-risk
note accordingly. This closes the last remaining gap in
/docs/phase-4-runbook.md.

deploy/README.md

@@ -61,15 +61,22 @@ gap, which was two *disconnected* sources of truth, not two actors).
 
 ## Verification status -- read before trusting this against a real cluster
 
-**Not verified against a live Kubernetes cluster.** This environment has
+**Now verified against a real live Kubernetes cluster.** `kind`/
-no `kubectl`/`kind`/`minikube`/`kubebuilder`/cluster reachable, so
+`kubectl`/`helm` were installed without root (static binaries into
-nothing here has been `kubectl apply`'d or `helm install`'d for real.
+`~/.local/bin`), a real local cluster was created, every image this
-Same disclosed-limitation shape as `/agent/README.md`'s "Windows-specific
+chart references was built and loaded into it, and the full two-tenant
-agent code remains unverified on real Windows" from Phase 1 -- a real gap
+walkthrough (`helm/sentry/README.md`) was run end to end -- both tenants
-to close before shipping, not swept under the rug.
+reached `Tenant.status.phase: Active` with real generated ClickHouse
+credentials in their Secrets. See `/docs/phase-4-runbook.md` §7 for the
+exact commands and the two real chart bugs this run found and fixed
+(`enterprise-auth` missing its Postgres connection env vars entirely,
+ClickHouse missing the env var that grants `CREATE USER` privilege) --
+neither `helm lint`, `helm template`, nor the `kubeconform` schema check
+below could have caught either, since both only manifest once real pods
+actually try to start and connect to each other.
 
-What **was** actually verified, offline, in this environment (network
+What was verified offline, before real cluster access existed (still
-access was available to fetch these tools, but no cluster):
+true, kept as additional evidence, not superseded by the above):
 
 - `deploy/operator`: `go build`/`go vet`/`go test ./...` all pass,
   including reconciler tests against controller-runtime's fake client
@@ -110,11 +117,13 @@ access was available to fetch these tools, but no cluster):
   `enterprise-api`'s container only when `tenantOperator.enabled` is
   true.
 - Docker image builds (`operator/Dockerfile` and every other
-  `Dockerfile` this chart references) were **not** verified in this
+  `Dockerfile` this chart references) are now confirmed working too --
-  session -- Docker's daemon wasn't reachable here either (see the
+  all twelve images this chart needs were built and loaded into the test
-  Phase 4 task 5 conversation for why). Build and push every image this
+  `kind` cluster above.
-  chart's `values.yaml` references before installing it.
 
-Before relying on this in production: `kind create cluster`, `helm
+Before relying on this in production: `ingest` needs a real cert-manager
-install` with `--include-crds`, and walk through
+(or equivalent) issued Secret, not `hack/dev-certs`'s throwaway dev
-`helm/sentry/README.md`'s two-tenant example end to end.
+certs; ClickHouse/Postgres data isn't backed by anything durable beyond
+the cluster's own PVC provisioner in this chart; and only Auth0 has been
+tried as a real external IdP so far (see
+`/docs/phase-4-runbook.md` §3a/§3b).

deploy/helm/sentry/README.md

@@ -64,8 +64,11 @@ be reachable). This chart approximates that more loosely:
 
 ```sh
 # Quote each --set value -- zsh globs an unquoted tenants[0] as a
-# pattern and fails with "no matches found."
+# pattern and fails with "no matches found." Also note: no --include-crds
-helm install sentry . --include-crds \
+# here -- that's a helm template-only flag (install always installs
+# crds/ by default); confirmed the hard way running this against a real
+# kind cluster, see /docs/phase-4-runbook.md §7.
+helm install sentry . \
   --set enterprise.enabled=true \
   --set tenantOperator.enabled=true \
   --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
@@ -84,6 +87,17 @@ kubectl get tenants
 kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse
 ```
 
+Before any of this: `ingest` needs a real mTLS cert Secret
+(`--set ingest.tlsSecretName=...`, see `values.yaml`'s comment on it and
+`/docs/phase-4-runbook.md` §7 for the exact `kubectl create secret`
+invocation using `hack/dev-certs`) or it crash-loops on startup --
+unconditional by design, no disable switch.
+
+**Genuinely run against a real `kind` cluster, not just described**: see
+`/docs/phase-4-runbook.md` §7 for the exact steps (image loading into
+`kind`, the two chart bugs it found and fixed) and confirmation that
+both tenants reached `status.phase: Active` with real credentials.
+
 This proves Phase 4's "two tenants... with their own users, roles,
 dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment-
 topology layer: `-provision-tenant` (`enterprise/internal/
@@ -124,5 +138,5 @@ helm lint .
 helm template sentry . --include-crds > /tmp/rendered.yaml
 ```
 
-See `/deploy/README.md`'s verification section for what was actually
+See `/deploy/README.md`'s verification section for what was checked
-checked this way (and what wasn't -- no live cluster was available).
+this way versus against a real cluster (now done -- see above).

docs/phase-4-runbook.md

@@ -91,17 +91,40 @@ memberships and getting back the right tenant/role each time (see §3a,
    not just SAML's, so this would have hit any real reverse-proxied
    deployment, not just this test.
 
-§7 and §11's live-cluster steps still need `kind`/`kubectl`, which
+§7 and §11's live-cluster steps are now closed too. `kind`/`kubectl`/
-aren't installed in this environment -- their offline-only checks
+`helm` were installed without root (`kind`/`kubectl` as static binaries,
-(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing
+`helm` the same, all into `~/.local/bin`), a real local cluster was
-the rendered YAML) all pass and are documented as such below. That's the
+created, and the full "Trying the two-tenant example" walkthrough from
-only remaining gap in this entire runbook that isn't already closed.
+`deploy/helm/sentry/README.md` was run end to end against it -- both
+`acme` and `globex` reached `Tenant.status.phase: Active` with real
+generated ClickHouse credentials in their Secrets. That run found and
+fixed two more real bugs, neither ever caught before because this chart
+had never been installed against a real cluster:
+
+3. `templates/enterprise-auth.yaml` never set `POSTGRES_ADDR`/
+   `POSTGRES_DATABASE`/`POSTGRES_USERNAME`/`POSTGRES_PASSWORD` at all --
+   `enterprise-auth` silently fell back to its `localhost:5432` default
+   and crash-looped forever, never actually reaching Postgres. Fixed to
+   match `api.yaml`'s existing pattern.
+4. `templates/clickhouse.yaml` was missing
+   `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` -- the same real bug
+   `docker-compose.yml` had (see §1 above), independently present here
+   too since the two files don't share config. `-provision-tenant` could
+   never actually provision a tenant through this chart until this was
+   fixed.
+
+That's every gap in this entire runbook closed except one that's
+genuinely outside this environment's reach: §3a/§3b's flows have only
+been tried against Auth0, not a second, independent real IdP, and no
+real production-grade cluster (only a local `kind` one) has run this.
 
 If you're reading this to decide whether Phase 4 is production-ready:
-closer than before, but not yet -- see
+much closer now -- every documented control has been verified against
-`/docs/security/threat-model.md`'s "Read this first" section for the
+real infrastructure at least once. See `/docs/security/threat-model.md`'s
-current, precise state of every control, including the ones still
+"Read this first" section for the current, precise state of every
-gated on a real IdP or a real cluster.
+control and what's still explicitly out of scope (a privileged DB
+administrator, external audit-log anchoring, and similar named
+non-goals) rather than merely unverified.
 
 ## 1. Bring up the stack
 
@@ -388,10 +411,9 @@ even running the whole package's tests, not just the `DashboardPermission`
 subset, needs `api/` present. `internal/audit` has the same shape via
 `queryapi_adapter.go`.)
 
-## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`)
+## 7. `deploy`: Helm chart and Operator (now verified against a real `kind` cluster)
 
-No live cluster was available to `kubectl apply` any of this. What can
+Offline checks (no cluster needed):
-be checked without one:
 
 ```sh
 cd deploy/operator && go build ./... && go vet ./... && go test ./...
@@ -406,24 +428,77 @@ helm template sentry . --include-crds \
   > /tmp/multitenant.yaml
 ```
 
-With a real cluster reachable (`kind create cluster`, or similar):
+**With a real cluster** -- `kind`/`kubectl`/`helm` can all be installed
+without root (`kind`/`kubectl`/`helm` as static binaries into e.g.
+`~/.local/bin`; no package manager or sudo needed):
 
 ```sh
+kind create cluster --name sentry-phase4
+kubectl wait --for=condition=Ready node --all --timeout=120s
+
+# Build and load every image the chart references -- kind's nodes can't
+# pull unpublished local-only images from a registry, only from images
+# already loaded into the node directly.
 docker build -f deploy/operator/Dockerfile -t sentry-tenant-operator deploy/operator/
-kind load docker-image sentry-tenant-operator   # or push to a registry the cluster can pull from
+for img in sentry-redpanda-provision sentry-clickhouse-migrate sentry-metadata-migrate \
-helm install sentry deploy/helm/sentry --include-crds \
+           sentry-ingest sentry-search sentry-api sentry-alerting sentry-web \
-  --set tenantOperator.enabled=true --set enterprise.enabled=true \
+           sentry-enterprise-auth sentry-enterprise-api sentry-enterprise-ingest \
-  --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp'
+           sentry-tenant-operator; do
-kubectl get tenants
+  kind load docker-image "${img}:latest" --name sentry-phase4
-kubectl get secret sentry-tenant-acme-clickhouse -o yaml
+done
+
+# NOTE: helm install has no --include-crds flag (that's a helm template-only
+# flag -- install always installs crds/ by default). The command in
+# deploy/helm/sentry/README.md's "Trying the two-tenant example" had this
+# wrong; fixed there too.
+helm install sentry deploy/helm/sentry \
+  --set enterprise.enabled=true --set tenantOperator.enabled=true \
+  --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
+  --set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
 ```
 
-Expect `kubectl get tenants` to show `acme` reach `status.phase: Active`
+`ingest` genuinely, unconditionally requires real mTLS server
-and the Secret to contain a generated `username`/`password`/`database`.
+certs (`ingest/internal/config`'s `TLS.CertFile`/`KeyFile` have no
-This proves the K8s-side half of a real two-tenant deployment — it does
+disable switch, by design -- see that package's doc comment) --
-**not** provision a working ClickHouse database itself (the Operator
+`values.yaml`'s `ingest.tlsSecretName` is empty by default and
-manages the K8s Secret only); §8 below is the piece that actually
+deliberately leaves cert issuance to the operator (`cert-manager` or
-provisions ClickHouse.
+equivalent) rather than hand-rolling it in the chart, so a Secret needs
+supplying before `ingest` can start:
+
+```sh
+kubectl create secret generic sentry-ingest-tls \
+  --from-file=server.pem=hack/dev-certs/out/server.pem \
+  --from-file=server-key.pem=hack/dev-certs/out/server-key.pem \
+  --from-file=ca.pem=hack/dev-certs/out/ca.pem
+helm upgrade sentry deploy/helm/sentry --reuse-values \
+  --set ingest.tlsSecretName=sentry-ingest-tls
+```
+
+Confirm every pod actually reaches `Running`/`1/1` (`kubectl get pods`)
+before provisioning -- this run found two real chart bugs neither
+`helm lint`/`helm template` nor any prior Docker-free check could catch,
+since both only manifest once real pods actually try to start (see this
+doc's top "Verification status" section for the full account):
+`enterprise-auth` crash-looping from missing `POSTGRES_ADDR` and friends,
+and ClickHouse's `default` user lacking `CREATE USER` privilege from a
+missing `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` env var (same bug
+§1's `docker-compose.yml` had). Both fixed in the chart itself.
+
+```sh
+kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
+kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
+kubectl get tenants
+kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse -o yaml
+```
+
+**Confirmed live**: `kubectl get tenants` shows both `acme` and `globex`
+reach `status.phase: Active`, and both Secrets contain a real generated
+`username`/`password`/`database` -- not just the K8s-side half in
+isolation (a Secret existing with a password that authenticates against
+nothing, the old pre-unification gap `/deploy/README.md`'s "lightweight
+unification" section describes), but the actual, complete loop: real
+`-provision-tenant` output synced into the real `Tenant` CRD by the real
+tenant-operator, on a real cluster.
 
 ## 8. `enterprise-api`: real per-tenant ClickHouse isolation
 

docs/security/threat-model.md

@@ -524,9 +524,10 @@ terms:
   an unanswered compliance question, not a designed-and-deferred one.
 - **No general multi-cluster orchestration** — `/deploy`'s Helm
   chart/Operator (`/deploy/README.md`) proves the K8s-side per-tenant
-  secret-management model, not a fully general multi-cluster system, and
+  secret-management model, not a fully general multi-cluster system.
-  was never applied to a live cluster in this environment (see that
+  Now applied to a real (local `kind`) live cluster in this environment
-  README's verification section).
+  — see that README's verification section — but only a single-cluster,
+  single-node test, not a real multi-cluster topology.
 
 ## Deployment/network assumptions
 
@@ -560,7 +561,7 @@ terms:
 | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
 | Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
 | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
-| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
+| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO; a real `helm install` against a real `kind` cluster confirmed the `sentry-api` Deployment runs `sentry-enterprise-api:latest` with `enterprise.enabled=true`, real endpoints behind the `sentry-api` Service, not just `helm template`'s rendered YAML |
 | Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` |
 | Human SSO login — OIDC | **Enforced, verified live** — real login against a real Auth0 developer tenant, full browser round trip; correctly failed closed on an identity with no `tenant_memberships` row, then succeeded and issued a real session after `-grant-membership-*`, with `POST /internal/authorize` returning exactly the granted tenant/role |
 | Human SSO login — SAML | **Enforced, verified live** — real login against Auth0's SAML2 Web App addon acting as a real SAML IdP, over real (self-signed, dev-only) TLS; a real signed assertion validated (audience, destination, signature), landed on `/select-tenant` with real memberships, and `POST /internal/authorize` confirmed the selected tenant/role. Found and fixed a real bug in the process: `loginhandler.go`'s cookies decided `Secure` from `r.TLS != nil` alone, which is wrong behind any TLS-terminating reverse proxy (the deployment shape this handler actually runs in) — `enterprise-auth` never terminates TLS itself, so `r.TLS` was nil even over a genuinely HTTPS connection, silently dropping `Secure` and breaking SAML's `SameSite=None` cookie |