cairnobs/docs/clickhouse-credential-rotation-runbook.md

Runbook: rotating the ClickHouse default password

Applies to the docker-compose deployments (proto.cairnobs.org, demo.cairnobs.org). For Helm deployments the value is a K8s Secret (/deploy/helm/cairnobs/templates/secrets.yaml) and the procedure is a secret update plus a rollout restart. The users_xml mechanics in "The trap" below apply there too — the chart's ClickHouse pod builds the same default-user.xml from the same env var — so the shape is identical: change the value, recreate the pod, verify. Only the sed-the-override steps differ.

What you are rotating

Not a login gate. CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 (set in docker-compose.yml for Phase 4's per-tenant provisioning) makes the default user a ClickHouse superuser:

$ docker exec cairnobs-clickhouse clickhouse-client --query "SHOW GRANTS FOR default"
GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, ... ACCESS MANAGEMENT,
CLUSTER ON *.* TO default WITH GRANT OPTION

Read/write over every tenant's logs, plus the ability to create further users. See /docs/security/threat-model.md for how this sits in the wider model.

Before you start

• Confirm the blast radius is what you think it is. ClickHouse publishes only to loopback on these boxes (127.0.0.1:8123, 127.0.0.1:9000); the only 0.0.0.0 listener is cairnobs-ingest:4317, the mTLS agent endpoint. Verify with docker ps --format "{{.Names}}\t{{.Ports}}" before assuming an exposure is or isn't reachable.
• The boxes do not share a password. Rotate them independently, and do not assume a value recovered from one applies to the other.
• Rotate demo before proto, matching the deploy risk order.

The trap: do NOT reach for ALTER USER

The obvious move — change the password inside ClickHouse, then update the env to match — does not work here, and fails loudly:

$ docker exec cairnobs-clickhouse clickhouse-client \
    --query "SELECT name, storage FROM system.users"
default	users_xml

storage = users_xml means the default user is defined by a config file, not by SQL. ClickHouse treats that storage as read-only and rejects ALTER USER against it. There is no in-database password to change.

The password lives in /etc/clickhouse-server/users.d/default-user.xml, which the official image's entrypoint regenerates from CLICKHOUSE_PASSWORD on every container start — not only on first init. That directory is part of the container filesystem; only /var/lib/clickhouse (the data) is a volume, so the file is rebuilt each time and the data survives untouched.

So the env var is the source of truth, and recreating the container is what applies it. Verify by comparing the file's mtime against docker inspect <container> --format '{{.State.StartedAt}}' — they match to the second, before and after a rotation.

Procedure

Four compose services declare the credential: clickhouse, clickhouse-migrate, api, ingest. Only three of them are long-running — clickhouse-migrate is a one-shot that has already exited on a running box, which is why docker ps shows three. All four must be updated in the override file, or the next docker compose up will run the migrate step with a stale password and fail.

web, alerting, search, enterprise-auth, metadata-postgres, and redpanda do not carry it and do not need restarting.

1. Back up the override file. It is the only copy of these credentials, and it is the rollback path.

cd /opt/sentry   # or /home/john/cairnobs-demo on demo
cp -p docker-compose.override.yml \
      docker-compose.override.yml.bak.$(date +%Y%m%d-%H%M%S)

2. Generate the value on the box, and never let it leave. Use hex, not base64: this string is interpolated into the generated default-user.xml, and base64's /, +, and = are needless risk in XML. Generating it remotely also keeps it out of your terminal scrollback and out of any AI-assistant transcript.

NEW="$(openssl rand -hex 24)"

3. Update all four occurrences in the override file. Match only indented KEY: value lines, so the header comment (which mentions CLICKHOUSE_PASSWORD in prose) is left alone:

sed -i -E "s|^( +)CLICKHOUSE_PASSWORD: .*|\1CLICKHOUSE_PASSWORD: \"${NEW}\"|" \
  docker-compose.override.yml
grep -cE '^ +CLICKHOUSE_PASSWORD: ' docker-compose.override.yml   # expect 4

The file is gitignored precisely because it carries real credentials; it is not in the repo, and an rsync deploy will not overwrite it unless you explicitly tell it to.

4. Validate before touching anything live.

docker compose config -q && echo OK

If this fails, restore the backup and stop.

5. Recreate the three long-running services. This is what applies the new password — the entrypoint rewrites default-user.xml as clickhouse comes up.

docker compose up -d clickhouse api ingest

clickhouse-migrate and the other one-shots will run and exit cleanly as dependencies; that is expected, and is why step 3 had to update their copy too.

6. Verify. All four must pass:

# a) the OLD password is now rejected -- read it from the backup rather
#    than retyping it anywhere
OLD=$(grep -m1 -E '^ +CLICKHOUSE_PASSWORD: ' docker-compose.override.yml.bak.* \
      | sed -E 's/.*: *"?([^"]*)"?$/\1/')
docker exec cairnobs-clickhouse clickhouse-client --user default \
  --password "$OLD" --query "SELECT 1" && echo "FAILED: old still works"

# b) the new password is accepted
docker exec cairnobs-clickhouse clickhouse-client --user default \
  --password "$NEW" --query "SELECT 1"

# c) the app can still read logs, and the count climbs on a live box
docker exec cairnobs-clickhouse clickhouse-client \
  --query "SELECT count() FROM cairnobs.logs"

# d) no auth failures since the restart
docker compose logs --since 5m api ingest | grep -icE '403|auth.*fail|denied'

If (b) passes but (c) or (d) fail, the override still holds the old value somewhere — recheck every service block from step 3.

Rollback

Restore the backup from step 1 and recreate the same three services:

cp -p docker-compose.override.yml.bak.<timestamp> docker-compose.override.yml
docker compose config -q && docker compose up -d clickhouse api ingest

Nothing else has to be undone. The rotation changes no schema and no data — /var/lib/clickhouse is a volume and is never rewritten by this procedure — so rollback is symmetric and carries no data-loss risk.

Verification status

This procedure was executed end-to-end against proto.cairnobs.org on 2026-08-23: old password confirmed rejected, new one accepted, default-user.xml mtime matching the new StartedAt, zero auth failures, and the ingest row count still climbing afterwards. No downtime beyond the recreate of clickhouse, api, and ingest.

An earlier draft of this runbook had it backwards — it opened with ALTER USER and claimed the entrypoint only reads CLICKHOUSE_PASSWORD at volume-init. Both were wrong, and /opt/sentry's own docker-compose.override.yml header had already recorded the correct mechanism during the 2026-08-19 rotation. Check that file before trusting anything here.

Known gap

The compose path keeps the password in the container environment, where docker inspect exposes it to anyone with Docker access — which on these boxes is root-equivalent anyway. The Helm path already avoids this via secretKeyRef. Closing it for compose would need a CLICKHOUSE_PASSWORD_FILE variant in the config loaders (api/internal/config, ingest/internal/config); only TLS_*_FILE paths support file-sourced values today. Worth doing only if these hosts ever gain more than single-operator access.