Document real kind-cluster verification, closing every runbook gap
kind/kubectl/helm were installed without root and a real local cluster ran the full two-tenant walkthrough end to end: both acme and globex reached Tenant.status.phase: Active with real generated ClickHouse credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's verification-status framing from "not verified against a live cluster" to what's actually true now, fixes both READMEs' helm install --include-crds (a helm template-only flag, never valid for install), documents the two chart bugs this run found (see the previous commit) and the mandatory ingest TLS Secret step, and updates the threat model's summary table and "No general multi-cluster orchestration" residual-risk note accordingly. This closes the last remaining gap in /docs/phase-4-runbook.md.
This commit is contained in:
+100
-25
@@ -91,17 +91,40 @@ memberships and getting back the right tenant/role each time (see §3a,
|
||||
not just SAML's, so this would have hit any real reverse-proxied
|
||||
deployment, not just this test.
|
||||
|
||||
§7 and §11's live-cluster steps still need `kind`/`kubectl`, which
|
||||
aren't installed in this environment -- their offline-only checks
|
||||
(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing
|
||||
the rendered YAML) all pass and are documented as such below. That's the
|
||||
only remaining gap in this entire runbook that isn't already closed.
|
||||
§7 and §11's live-cluster steps are now closed too. `kind`/`kubectl`/
|
||||
`helm` were installed without root (`kind`/`kubectl` as static binaries,
|
||||
`helm` the same, all into `~/.local/bin`), a real local cluster was
|
||||
created, and the full "Trying the two-tenant example" walkthrough from
|
||||
`deploy/helm/sentry/README.md` was run end to end against it -- both
|
||||
`acme` and `globex` reached `Tenant.status.phase: Active` with real
|
||||
generated ClickHouse credentials in their Secrets. That run found and
|
||||
fixed two more real bugs, neither ever caught before because this chart
|
||||
had never been installed against a real cluster:
|
||||
|
||||
3. `templates/enterprise-auth.yaml` never set `POSTGRES_ADDR`/
|
||||
`POSTGRES_DATABASE`/`POSTGRES_USERNAME`/`POSTGRES_PASSWORD` at all --
|
||||
`enterprise-auth` silently fell back to its `localhost:5432` default
|
||||
and crash-looped forever, never actually reaching Postgres. Fixed to
|
||||
match `api.yaml`'s existing pattern.
|
||||
4. `templates/clickhouse.yaml` was missing
|
||||
`CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` -- the same real bug
|
||||
`docker-compose.yml` had (see §1 above), independently present here
|
||||
too since the two files don't share config. `-provision-tenant` could
|
||||
never actually provision a tenant through this chart until this was
|
||||
fixed.
|
||||
|
||||
That's every gap in this entire runbook closed except one that's
|
||||
genuinely outside this environment's reach: §3a/§3b's flows have only
|
||||
been tried against Auth0, not a second, independent real IdP, and no
|
||||
real production-grade cluster (only a local `kind` one) has run this.
|
||||
|
||||
If you're reading this to decide whether Phase 4 is production-ready:
|
||||
closer than before, but not yet -- see
|
||||
`/docs/security/threat-model.md`'s "Read this first" section for the
|
||||
current, precise state of every control, including the ones still
|
||||
gated on a real IdP or a real cluster.
|
||||
much closer now -- every documented control has been verified against
|
||||
real infrastructure at least once. See `/docs/security/threat-model.md`'s
|
||||
"Read this first" section for the current, precise state of every
|
||||
control and what's still explicitly out of scope (a privileged DB
|
||||
administrator, external audit-log anchoring, and similar named
|
||||
non-goals) rather than merely unverified.
|
||||
|
||||
## 1. Bring up the stack
|
||||
|
||||
@@ -388,10 +411,9 @@ even running the whole package's tests, not just the `DashboardPermission`
|
||||
subset, needs `api/` present. `internal/audit` has the same shape via
|
||||
`queryapi_adapter.go`.)
|
||||
|
||||
## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`)
|
||||
## 7. `deploy`: Helm chart and Operator (now verified against a real `kind` cluster)
|
||||
|
||||
No live cluster was available to `kubectl apply` any of this. What can
|
||||
be checked without one:
|
||||
Offline checks (no cluster needed):
|
||||
|
||||
```sh
|
||||
cd deploy/operator && go build ./... && go vet ./... && go test ./...
|
||||
@@ -406,24 +428,77 @@ helm template sentry . --include-crds \
|
||||
> /tmp/multitenant.yaml
|
||||
```
|
||||
|
||||
With a real cluster reachable (`kind create cluster`, or similar):
|
||||
**With a real cluster** -- `kind`/`kubectl`/`helm` can all be installed
|
||||
without root (`kind`/`kubectl`/`helm` as static binaries into e.g.
|
||||
`~/.local/bin`; no package manager or sudo needed):
|
||||
|
||||
```sh
|
||||
kind create cluster --name sentry-phase4
|
||||
kubectl wait --for=condition=Ready node --all --timeout=120s
|
||||
|
||||
# Build and load every image the chart references -- kind's nodes can't
|
||||
# pull unpublished local-only images from a registry, only from images
|
||||
# already loaded into the node directly.
|
||||
docker build -f deploy/operator/Dockerfile -t sentry-tenant-operator deploy/operator/
|
||||
kind load docker-image sentry-tenant-operator # or push to a registry the cluster can pull from
|
||||
helm install sentry deploy/helm/sentry --include-crds \
|
||||
--set tenantOperator.enabled=true --set enterprise.enabled=true \
|
||||
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp'
|
||||
kubectl get tenants
|
||||
kubectl get secret sentry-tenant-acme-clickhouse -o yaml
|
||||
for img in sentry-redpanda-provision sentry-clickhouse-migrate sentry-metadata-migrate \
|
||||
sentry-ingest sentry-search sentry-api sentry-alerting sentry-web \
|
||||
sentry-enterprise-auth sentry-enterprise-api sentry-enterprise-ingest \
|
||||
sentry-tenant-operator; do
|
||||
kind load docker-image "${img}:latest" --name sentry-phase4
|
||||
done
|
||||
|
||||
# NOTE: helm install has no --include-crds flag (that's a helm template-only
|
||||
# flag -- install always installs crds/ by default). The command in
|
||||
# deploy/helm/sentry/README.md's "Trying the two-tenant example" had this
|
||||
# wrong; fixed there too.
|
||||
helm install sentry deploy/helm/sentry \
|
||||
--set enterprise.enabled=true --set tenantOperator.enabled=true \
|
||||
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
|
||||
--set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
|
||||
```
|
||||
|
||||
Expect `kubectl get tenants` to show `acme` reach `status.phase: Active`
|
||||
and the Secret to contain a generated `username`/`password`/`database`.
|
||||
This proves the K8s-side half of a real two-tenant deployment — it does
|
||||
**not** provision a working ClickHouse database itself (the Operator
|
||||
manages the K8s Secret only); §8 below is the piece that actually
|
||||
provisions ClickHouse.
|
||||
`ingest` genuinely, unconditionally requires real mTLS server
|
||||
certs (`ingest/internal/config`'s `TLS.CertFile`/`KeyFile` have no
|
||||
disable switch, by design -- see that package's doc comment) --
|
||||
`values.yaml`'s `ingest.tlsSecretName` is empty by default and
|
||||
deliberately leaves cert issuance to the operator (`cert-manager` or
|
||||
equivalent) rather than hand-rolling it in the chart, so a Secret needs
|
||||
supplying before `ingest` can start:
|
||||
|
||||
```sh
|
||||
kubectl create secret generic sentry-ingest-tls \
|
||||
--from-file=server.pem=hack/dev-certs/out/server.pem \
|
||||
--from-file=server-key.pem=hack/dev-certs/out/server-key.pem \
|
||||
--from-file=ca.pem=hack/dev-certs/out/ca.pem
|
||||
helm upgrade sentry deploy/helm/sentry --reuse-values \
|
||||
--set ingest.tlsSecretName=sentry-ingest-tls
|
||||
```
|
||||
|
||||
Confirm every pod actually reaches `Running`/`1/1` (`kubectl get pods`)
|
||||
before provisioning -- this run found two real chart bugs neither
|
||||
`helm lint`/`helm template` nor any prior Docker-free check could catch,
|
||||
since both only manifest once real pods actually try to start (see this
|
||||
doc's top "Verification status" section for the full account):
|
||||
`enterprise-auth` crash-looping from missing `POSTGRES_ADDR` and friends,
|
||||
and ClickHouse's `default` user lacking `CREATE USER` privilege from a
|
||||
missing `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` env var (same bug
|
||||
§1's `docker-compose.yml` had). Both fixed in the chart itself.
|
||||
|
||||
```sh
|
||||
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
|
||||
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
|
||||
kubectl get tenants
|
||||
kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse -o yaml
|
||||
```
|
||||
|
||||
**Confirmed live**: `kubectl get tenants` shows both `acme` and `globex`
|
||||
reach `status.phase: Active`, and both Secrets contain a real generated
|
||||
`username`/`password`/`database` -- not just the K8s-side half in
|
||||
isolation (a Secret existing with a password that authenticates against
|
||||
nothing, the old pre-unification gap `/deploy/README.md`'s "lightweight
|
||||
unification" section describes), but the actual, complete loop: real
|
||||
`-provision-tenant` output synced into the real `Tenant` CRD by the real
|
||||
tenant-operator, on a real cluster.
|
||||
|
||||
## 8. `enterprise-api`: real per-tenant ClickHouse isolation
|
||||
|
||||
|
||||
@@ -524,9 +524,10 @@ terms:
|
||||
an unanswered compliance question, not a designed-and-deferred one.
|
||||
- **No general multi-cluster orchestration** — `/deploy`'s Helm
|
||||
chart/Operator (`/deploy/README.md`) proves the K8s-side per-tenant
|
||||
secret-management model, not a fully general multi-cluster system, and
|
||||
was never applied to a live cluster in this environment (see that
|
||||
README's verification section).
|
||||
secret-management model, not a fully general multi-cluster system.
|
||||
Now applied to a real (local `kind`) live cluster in this environment
|
||||
— see that README's verification section — but only a single-cluster,
|
||||
single-node test, not a real multi-cluster topology.
|
||||
|
||||
## Deployment/network assumptions
|
||||
|
||||
@@ -560,7 +561,7 @@ terms:
|
||||
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
|
||||
| Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
|
||||
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO; a real `helm install` against a real `kind` cluster confirmed the `sentry-api` Deployment runs `sentry-enterprise-api:latest` with `enterprise.enabled=true`, real endpoints behind the `sentry-api` Service, not just `helm template`'s rendered YAML |
|
||||
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` |
|
||||
| Human SSO login — OIDC | **Enforced, verified live** — real login against a real Auth0 developer tenant, full browser round trip; correctly failed closed on an identity with no `tenant_memberships` row, then succeeded and issued a real session after `-grant-membership-*`, with `POST /internal/authorize` returning exactly the granted tenant/role |
|
||||
| Human SSO login — SAML | **Enforced, verified live** — real login against Auth0's SAML2 Web App addon acting as a real SAML IdP, over real (self-signed, dev-only) TLS; a real signed assertion validated (audience, destination, signature), landed on `/select-tenant` with real memberships, and `POST /internal/authorize` confirmed the selected tenant/role. Found and fixed a real bug in the process: `loginhandler.go`'s cookies decided `Secure` from `r.TLS != nil` alone, which is wrong behind any TLS-terminating reverse proxy (the deployment shape this handler actually runs in) — `enterprise-auth` never terminates TLS itself, so `r.TLS` was nil even over a genuinely HTTPS connection, silently dropping `Secure` and breaking SAML's `SameSite=None` cookie |
|
||||
|
||||
Reference in New Issue
Block a user