diff --git a/docs/clickhouse-credential-rotation-runbook.md b/docs/clickhouse-credential-rotation-runbook.md new file mode 100644 index 0000000..d15a5da --- /dev/null +++ b/docs/clickhouse-credential-rotation-runbook.md @@ -0,0 +1,120 @@ +# Runbook: rotating the ClickHouse `default` password + +Applies to the docker-compose deployments (`proto.cairnobs.org`, +`demo.cairnobs.org`). For Helm deployments the value is a K8s `Secret` +(`/deploy/helm/cairnobs/templates/secrets.yaml`) and the procedure is a +secret update plus a rollout restart — only steps 1 and 5 below apply. + +## What you are rotating + +Not a login gate. `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` (set in +`docker-compose.yml` for Phase 4's per-tenant provisioning) makes the +`default` user a ClickHouse superuser: + +``` +$ docker exec cairnobs-clickhouse clickhouse-client --query "SHOW GRANTS FOR default" +GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, ... ACCESS MANAGEMENT, +CLUSTER ON *.* TO default WITH GRANT OPTION +``` + +Read/write over every tenant's logs, plus the ability to create further +users. See `/docs/security/threat-model.md` for how this sits in the +wider model. + +## Before you start + +- **Confirm the blast radius is what you think it is.** ClickHouse + publishes only to loopback on these boxes (`127.0.0.1:8123`, + `127.0.0.1:9000`); the only `0.0.0.0` listener is `cairnobs-ingest:4317`, + the mTLS agent endpoint. Verify with `docker ps --format "{{.Names}}\t{{.Ports}}"` + before assuming an exposure is or isn't reachable. +- **The boxes do not share a password.** Rotate them independently, and + do not assume a value recovered from one applies to the other. +- **Rotate demo before proto**, matching the deploy risk order. + +## The trap + +`CLICKHOUSE_PASSWORD` is read by the official image's entrypoint **only +when it initialises an empty data volume**. On a box that has already +been running, editing the env var and restarting does *not* change the +stored password — ClickHouse keeps the old one, the clients present the +new one, and every query starts returning 403. The password must be +changed *inside* ClickHouse first. + +## Procedure + +Four compose services declare the credential: `clickhouse`, +`clickhouse-migrate`, `api`, `ingest`. Only three of them are +long-running — `clickhouse-migrate` is a one-shot that has already +exited on a running box, which is why `docker ps` shows three. **All +four must be updated in the override file**, or the next `docker compose +up` will run the migrate step with a stale password and fail. + +`web`, `alerting`, `search`, `enterprise-auth`, `metadata-postgres`, and +`redpanda` do not carry it and do not need restarting. + +**1. Generate a new value** and keep it somewhere you can paste from. + +```sh +openssl rand -base64 24 +``` + +**2. Change it inside ClickHouse first.** + +```sh +docker exec cairnobs-clickhouse clickhouse-client \ + --query "ALTER USER default IDENTIFIED BY ''" +``` + +Existing connections keep working; this affects authentication of new +ones. Expect the next `api`/`ingest` query to fail until step 4. + +**3. Update the override file** — `docker-compose.override.yml` in the +deployment directory (`/opt/sentry` on proto, `/home/john/cairnobs-demo` +on demo). This file is gitignored precisely because it carries real +credentials; it is not in the repo and rsync will not overwrite it +unless you tell it to. + +Update every occurrence, not just the `clickhouse` service's — `api` and +`ingest` each set their own copy. + +**4. Restart the three services that carry it.** + +```sh +docker compose up -d clickhouse api ingest +``` + +**5. Verify.** All three must pass: + +```sh +# a) the credential works +docker exec cairnobs-clickhouse clickhouse-client \ + --user default --password '' --query "SELECT 1" + +# b) the app can still read logs (non-zero, and climbing on a live box) +docker exec cairnobs-clickhouse clickhouse-client \ + --query "SELECT count() FROM cairnobs.logs" + +# c) no auth failures since the restart +docker compose logs --since 5m api ingest | grep -iE 'auth|403|denied' +``` + +If (a) passes but (b) or (c) fail, the override file still holds the old +value somewhere — recheck every service block in step 3. + +## Rollback + +Re-run step 2 with the previous password and restart the three services. +There is no schema or data change here, so rollback is symmetric and +carries no data-loss risk. + +## Known gap + +The compose path keeps the password in the container environment, where +`docker inspect` exposes it to anyone with Docker access — which on +these boxes is root-equivalent anyway. The Helm path already avoids this +via `secretKeyRef`. Closing it for compose would need a +`CLICKHOUSE_PASSWORD_FILE` variant in the config loaders +(`api/internal/config`, `ingest/internal/config`); only `TLS_*_FILE` +paths support file-sourced values today. Worth doing only if these hosts +ever gain more than single-operator access.