Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary
Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.
Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.
Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."
Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.
This commit is contained in:
+37
-21
@@ -79,7 +79,7 @@ This split is not to be changed without discussion — see CLAUDE.md.
|
||||
| `search` (Rust, Phase 1) | Consumes the same Redpanda topic `ingest` does (own offset tracking), builds a Tantivy full-text index over `message`, serves matches over gRPC. One shared index for every tenant today — see "Tenant isolation" below. |
|
||||
| `api` (Go) | gRPC + REST gateway. `POST /query` compiles pipe-syntax or raw SQL to one IR, executed across ClickHouse/Tantivy (`/docs/query-language-design.md`). `internal/dashboards` is CRUD only — panel query execution happens client-side, reusing `/query`. `internal/authz` (Phase 4) enforces RBAC via a network call to `enterprise-auth`, never an import. |
|
||||
| `alerting` (Go, Phase 3) | Evaluates alert rules on an interval, calls `api`'s `POST /query` (via a `RoleService` credential once Phase 4 auth is configured — see `/docs/phase-4-isolation-design.md`'s alerting↔api gap), delivers firing/resolved notifications (webhook/Slack/PagerDuty). |
|
||||
| `enterprise` (Go, commercial license, Phase 4) | SSO (OIDC/SAML protocol mechanics), RBAC storage (`internal/rbacstore`), session/service-token issuance (`internal/session`), the append-only audit log (`internal/audit`), and `enterprise-auth`'s HTTP surface (`/internal/authorize`, `/auth/features`). Never imported by core — see "Licensing boundary" below. Does **not** yet include per-tenant ClickHouse/Tantivy connection routing or the OIDC/SAML login HTTP handlers — see `/docs/security/threat-model.md`. |
|
||||
| `enterprise` (Go, commercial license, Phase 4) | SSO (OIDC/SAML protocol mechanics), RBAC storage (`internal/rbacstore`), session/service-token issuance (`internal/session`), the append-only audit log (`internal/audit`), `enterprise-auth`'s HTTP surface (`/internal/authorize`, `/auth/features`), per-tenant ClickHouse provisioning (`internal/tenantprovision`) and query routing (`internal/chrunner`), and `cmd/enterprise-api` — a second binary combining core's `api/queryapi`/`api/dashboards` handlers with these tenant-aware implementations. Never imported by core — see "Licensing boundary" below. Does **not** yet include per-tenant Tantivy routing or the OIDC/SAML login HTTP handlers — see `/docs/security/threat-model.md`. |
|
||||
| `web` (SvelteKit, static build) | Query bar, dashboards, alerts, and (Phase 4) a settings page that renders SSO status via a runtime capability check (`GET /auth/features`) rather than bundling enterprise-licensed components. |
|
||||
| `cli` (`sentryctl`) | `ping`, `query`, `dashboards` (list/get/apply), `alerts` (list/get/apply). `$SENTRYCTL_TOKEN`, if set, is forwarded as a Bearer credential (Phase 4). |
|
||||
| `deploy` | A Helm chart covering every `docker-compose.yml` service, plus (Phase 4) a small Go Operator managing one CRD (`Tenant`) that provisions a per-tenant ClickHouse credential Secret. Never applied to a live cluster in the environment this was built in — see `/deploy/README.md`'s verification section before trusting it. |
|
||||
@@ -101,32 +101,48 @@ through a tenant-scoped connection the database's own access control
|
||||
enforces — not at the query-compiler layer, since Phase 2's raw-SQL
|
||||
escape hatch is opaque to any compiler-injected filter.
|
||||
|
||||
**As built, through Phase 4 task 8:**
|
||||
**As built, currently:**
|
||||
|
||||
- Role-based access control (`api/internal/authz`) is live on `/query`
|
||||
- Role-based access control (`api/authz`) is live on `/query`
|
||||
and `/dashboards`, resolved via `enterprise-auth` over HTTP.
|
||||
- Control-plane tenant scoping is live for dashboards
|
||||
(`api/internal/dashboards`'s store filters every query by the
|
||||
(`api/dashboards`'s store filters every query by the
|
||||
authenticated identity's tenant, never a client-supplied field).
|
||||
- The `alerting`↔`api` service-identity gap (task 2's finding) is
|
||||
closed: a `RoleService` credential, distinct from every human role.
|
||||
- **The connection-layer isolation itself — the actual design above —
|
||||
is not built.** `api/internal/querylang/executor.SQLRunner`/
|
||||
`SearchClient` and `search`'s gRPC service carry no tenant field
|
||||
anywhere. There is one shared ClickHouse connection and one shared
|
||||
Tantivy index for every tenant. RBAC controls *who* can run a query;
|
||||
nothing yet controls *what data* that query can see.
|
||||
- `deploy/operator`'s `Tenant` CRD manages only the K8s-side artifact (a
|
||||
credential Secret) — it doesn't call ClickHouse or provision anything
|
||||
ClickHouse-side. `enterprise/internal/tenantprovision` (the piece that
|
||||
would) is unbuilt.
|
||||
- **ClickHouse connection-layer isolation is built**, but lives in a
|
||||
second binary: `enterprise/internal/tenantprovision` (real `CREATE
|
||||
DATABASE`/`CREATE USER`/`GRANT` against ClickHouse) and
|
||||
`enterprise/internal/chrunner` (a per-tenant connection registry
|
||||
implementing `api/querylang/executor.SQLRunner`, resolving the
|
||||
right tenant's connection from the authenticated identity in request
|
||||
context) are wired into `enterprise/cmd/enterprise-api` — a binary
|
||||
that imports both `api`'s handler packages and enterprise's
|
||||
tenant-aware implementations (the allowed `enterprise → api` import
|
||||
direction; core still never imports `enterprise/`). Real integration
|
||||
tests assert a tenant cannot read another tenant's database by
|
||||
fully-qualified name, and that `system.query_log`/`system.tables`/
|
||||
`SHOW DATABASES` don't leak across tenants either — written but not
|
||||
yet run against a live ClickHouse in this environment, see
|
||||
`/docs/security/threat-model.md` and `/docs/phase-4-runbook.md`'s
|
||||
verification-status sections. Plain `api/cmd/api` still exists,
|
||||
unchanged, with its single shared connection — nothing forces a
|
||||
deployment to run `enterprise-api` instead, and nothing flags it if it
|
||||
doesn't.
|
||||
- **Tantivy connection-layer isolation is not built.** `search`'s gRPC
|
||||
service and `proto/sentry/search/v1/search.proto`'s `SearchRequest`
|
||||
still carry no tenant field. Every tenant's free-text queries hit the
|
||||
same shared Tantivy index regardless of which binary serves the
|
||||
request.
|
||||
- `deploy/operator`'s `Tenant` CRD still manages only the K8s-side
|
||||
artifact (a credential Secret); the Helm chart has no service
|
||||
definition for `enterprise-api` yet.
|
||||
|
||||
Building `enterprise/internal/chrunner` + `internal/searchclient` (the
|
||||
tenant-scoped implementations of the two interfaces above) and wiring
|
||||
them into `api/internal/queryapi.Handler` in place of the single shared
|
||||
connection `api/cmd/api/main.go` opens today is the single largest
|
||||
remaining gap between this system and the isolation model it was
|
||||
designed to have.
|
||||
Building `enterprise/internal/searchclient` (the Tantivy-side sibling of
|
||||
`chrunner`) and giving the deployment topology (Helm chart, or at least
|
||||
clear documentation) an actual way to route traffic to `enterprise-api`
|
||||
instead of `api` are the two largest remaining gaps between this system
|
||||
and the isolation model it was designed to have.
|
||||
|
||||
## Licensing boundary
|
||||
|
||||
@@ -137,7 +153,7 @@ CI by `hack/check-tenant-boundary.sh`, which greps every build for the
|
||||
import edge. Where core needs a decision only `enterprise/` can make
|
||||
(is this request authorized, what SSO is configured), it calls
|
||||
`enterprise-auth` over plain HTTP instead
|
||||
(`api/internal/authz.HTTPAuthorizer`, `web`'s `GET /auth/features`) —
|
||||
(`api/authz.HTTPAuthorizer`, `web`'s `GET /auth/features`) —
|
||||
the same "network boundary, not import boundary" shape `/alerting`↔`api`
|
||||
already used before `enterprise/` existed.
|
||||
|
||||
|
||||
+84
-33
@@ -8,28 +8,33 @@ logging, and a Kubernetes deployment path. Read those first.
|
||||
|
||||
Every prior phase's runbook documents claims **checked against the live
|
||||
stack**, not asserted. This one is different, and says so plainly rather
|
||||
than papering over it: **this session had no working Docker daemon
|
||||
access and no reachable Kubernetes cluster**, so most of what follows is
|
||||
a *procedure to run*, not a report of what was already run and passed.
|
||||
Two exceptions, genuinely verified live against a real Postgres during
|
||||
earlier Phase 4 tasks (see their own doc comments for the exact `docker
|
||||
run` invocations):
|
||||
than papering over it: for the great majority of this phase's work,
|
||||
**there was no working Docker daemon access and no reachable Kubernetes
|
||||
cluster**, so most of what follows is a *procedure to run*, not a report
|
||||
of what was already run and passed. One genuine exception, verified live
|
||||
against a real Postgres earlier in this phase's work (see its own doc
|
||||
comments for the exact `docker run` invocations, and note this was
|
||||
before the environment lost Docker access, not a claim about this
|
||||
runbook's own session):
|
||||
|
||||
- `enterprise/internal/audit`'s hash-chain, tamper-detection, and
|
||||
concurrent-write guarantees (task 4).
|
||||
- `enterprise/internal/rbacstore`'s CRUD, run against a live Postgres
|
||||
the same way.
|
||||
|
||||
Everything else below — the auth-enforcement walkthrough, the dashboards
|
||||
tenant-scoping fix, the Helm chart, the tenant-operator — has unit/fake-
|
||||
client/`helm template` coverage (all passing, see each component's own
|
||||
`go test`/`helm lint` output) but has **not** been exercised against a
|
||||
real running stack in this session. If you're reading this to decide
|
||||
whether Phase 4 is production-ready: it isn't yet, independent of this
|
||||
gap — see `/docs/security/threat-model.md`'s headline finding (log-data
|
||||
query isolation isn't built). This runbook exists so the first person
|
||||
with real Docker/K8s access can actually close the loop, not to claim
|
||||
that already happened.
|
||||
Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement
|
||||
walkthrough, the dashboards tenant-scoping fix, the Helm chart, the
|
||||
tenant-operator, and (newest) `internal/tenantprovision`/
|
||||
`internal/chrunner`'s live-ClickHouse tests — has unit/fake-client/
|
||||
`helm template` coverage (all passing, see each component's own `go
|
||||
test`/`helm lint` output, including every `Skip*`-gated integration test
|
||||
confirmed to skip cleanly offline) but has **not** been exercised
|
||||
against a real running stack. Be specific when citing this runbook: "the
|
||||
tests exist and pass structurally" is a true, verified claim; "isolation
|
||||
was confirmed against real ClickHouse" is not, yet. If you're reading
|
||||
this to decide whether Phase 4 is production-ready: it isn't yet,
|
||||
independent of this gap — see `/docs/security/threat-model.md`'s
|
||||
headline finding. This runbook exists so the first person with real
|
||||
Docker/K8s access can actually close the loop, not to claim that already
|
||||
happened.
|
||||
|
||||
## 1. Bring up the stack
|
||||
|
||||
@@ -172,28 +177,74 @@ kubectl get secret sentry-tenant-acme-clickhouse -o yaml
|
||||
Expect `kubectl get tenants` to show `acme` reach `status.phase: Active`
|
||||
and the Secret to contain a generated `username`/`password`/`database`.
|
||||
This proves the K8s-side half of a real two-tenant deployment — it does
|
||||
**not** prove either tenant has a working ClickHouse database, since
|
||||
`enterprise/internal/tenantprovision` (the piece that would create one)
|
||||
isn't built. See `/deploy/README.md` and
|
||||
`/docs/security/threat-model.md`.
|
||||
**not** provision a working ClickHouse database itself (the Operator
|
||||
manages the K8s Secret only); §8 below is the piece that actually
|
||||
provisions ClickHouse.
|
||||
|
||||
## 8. `enterprise-api`: real per-tenant ClickHouse isolation
|
||||
|
||||
This is new since this runbook was first written — `enterprise/internal/
|
||||
tenantprovision` and `enterprise/internal/chrunner` now exist, closing
|
||||
the headline gap §"Known gaps" below used to describe as completely
|
||||
unbuilt. It's still a second binary you have to choose to run, though —
|
||||
see `/docs/security/threat-model.md`'s "Read this first" section.
|
||||
|
||||
```sh
|
||||
docker compose build enterprise-api
|
||||
docker compose run --rm enterprise-api -provision-tenant=acme -display-name="Acme Corp"
|
||||
docker compose run --rm enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
|
||||
docker compose up -d enterprise-api
|
||||
curl -s http://localhost:8083/healthz
|
||||
```
|
||||
|
||||
There's still no OIDC/SAML login handler and no CLI for minting a human
|
||||
session token (see `/docs/security/threat-model.md`) -- so a real
|
||||
`curl -X POST http://localhost:8083/query` walkthrough as tenant acme
|
||||
isn't possible yet. Confirm isolation end to end against the live stack (this is the same
|
||||
assertion `enterprise/internal/chrunner/chrunner_test.go`'s
|
||||
`TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` makes, run here
|
||||
as an integration test instead of a curl walkthrough since there's no
|
||||
login flow to drive it through curl yet):
|
||||
|
||||
```sh
|
||||
docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
|
||||
-e CHRUNNER_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
|
||||
-e CHRUNNER_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
|
||||
golang:1.25-alpine go test ./internal/chrunner/... -v
|
||||
|
||||
docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
|
||||
-e TENANTPROVISION_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
|
||||
-e TENANTPROVISION_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
|
||||
golang:1.25-alpine go test ./internal/tenantprovision/... -v
|
||||
```
|
||||
|
||||
Expect all tests to pass, including
|
||||
`TestProvisionedUserCannotReadSystemTables` (item 2 of
|
||||
`/docs/phase-4-isolation-design.md`'s verification plan, closed this
|
||||
pass) and `TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` (item 1,
|
||||
closed through the actual production code path, not just
|
||||
tenantprovision's raw grants).
|
||||
|
||||
## Known gaps (do not treat this phase as done without reading these)
|
||||
|
||||
Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
|
||||
- **No tenant isolation on log data.** `POST /query` executes against
|
||||
one shared ClickHouse connection and one shared Tantivy index for
|
||||
every tenant, regardless of RBAC. This is Phase 4's originally-stated
|
||||
highest-risk item and it is not resolved.
|
||||
- **ClickHouse isolation exists but is opt-in.** `enterprise-api`
|
||||
(§8) gives real per-tenant ClickHouse isolation, but plain `api`
|
||||
(still the default in `docker-compose.yml`/`web`'s base URL) has none,
|
||||
and nothing flags which one a given deployment is actually running.
|
||||
- **No Tantivy/free-text isolation at all**, regardless of which binary
|
||||
serves the request -- `enterprise/internal/searchclient` (chrunner's
|
||||
Tantivy-side sibling) doesn't exist.
|
||||
- **No human SSO login.** OIDC/SAML protocol wiring exists;
|
||||
the HTTP login/callback handlers that would use it don't.
|
||||
- **No per-resource dashboard grants** (`dashboard_permissions` has a
|
||||
schema, no handler reads it).
|
||||
- Four adversarial ClickHouse/Tantivy probes named in
|
||||
`/docs/phase-4-isolation-design.md`'s verification plan are stubbed as
|
||||
explicitly-skipped tests in `api/internal/queryapi/
|
||||
tenant_isolation_gap_test.go`, blocked on the tenant-scoped connection
|
||||
work above.
|
||||
- Two of the four adversarial ClickHouse/Tantivy probes named in
|
||||
`/docs/phase-4-isolation-design.md`'s verification plan are closed
|
||||
(§8); the other two (Tantivy cross-tenant search, mid-provisioning-race
|
||||
handling) are still stubbed as explicitly-skipped tests in
|
||||
`api/queryapi/tenant_isolation_gap_test.go`.
|
||||
|
||||
## Tearing down
|
||||
|
||||
@@ -214,13 +265,13 @@ be too.
|
||||
set.**
|
||||
Check `api/cmd/api/main.go` actually left `authorizer` nil when
|
||||
`cfg.EnterpriseAuthURL == ""` — a nil `Authorizer` must be a no-op
|
||||
(`api/internal/authz.RequireRole`'s doc comment). If this regresses, it
|
||||
(`api/authz.RequireRole`'s doc comment). If this regresses, it
|
||||
breaks every existing Phase 0-3 deployment silently.
|
||||
|
||||
**A dashboard created by one tenant is visible to another.**
|
||||
This is the exact bug found and fixed in task 7 — see
|
||||
`/docs/security/threat-model.md`'s "application-layer tenant scoping"
|
||||
section and `api/internal/dashboards/handler_test.go`'s
|
||||
section and `api/dashboards/handler_test.go`'s
|
||||
`TestCrossTenant*` tests. If this regresses, `Handler.tenantID` or
|
||||
`store.go`'s `WHERE tenant_id = ...` filters have been bypassed
|
||||
somewhere — check every store method still takes and uses a `tenantID`
|
||||
|
||||
@@ -9,28 +9,58 @@ for the full design rationale behind the controls described here.
|
||||
|
||||
## Read this first: the single most important open finding
|
||||
|
||||
**Log data queried through `POST /query` is not tenant-isolated today.**
|
||||
Every authenticated tenant's ad hoc queries and dashboard panel queries
|
||||
execute against the same shared ClickHouse connection and the same
|
||||
shared Tantivy index — there is no per-tenant database, user, or index
|
||||
routing anywhere in the query execution path
|
||||
(`api/internal/querylang/executor.SQLRunner`/`SearchClient`, `search`'s
|
||||
gRPC service, `proto/sentry/search/v1/search.proto`). Confirmed by
|
||||
reading the actual code, not assumed: neither interface, nor the
|
||||
`search` proto, carries a tenant field anywhere.
|
||||
**Updated**: this section originally read "log data queried through
|
||||
`POST /query` is not tenant-isolated at all." That's now only half
|
||||
true, and the half that's no longer true matters — read carefully,
|
||||
because the remaining gap (Tantivy/free-text) is easy to miss if you
|
||||
stop at "ClickHouse is isolated now."
|
||||
|
||||
This is exactly the mechanism `/docs/phase-4-isolation-design.md`
|
||||
specifies as the core deliverable of tenant isolation (one dedicated
|
||||
ClickHouse database/user and one dedicated Tantivy index directory per
|
||||
tenant) — it is **designed but not built**. What *is* built and live:
|
||||
role-based access control (below) and tenant-scoped control-plane data
|
||||
(dashboards, below). Until `enterprise/internal/chrunner` and
|
||||
`enterprise/internal/searchclient` exist and are wired into
|
||||
`api/internal/queryapi.Handler` in place of the single shared connection
|
||||
`api/cmd/api/main.go` opens today, **treat any deployment of this system
|
||||
as single-tenant only**, regardless of how many `Tenant` CRs or
|
||||
`tenant_memberships` rows exist. RBAC controls who can run a query; they
|
||||
do not control what data that query can see.
|
||||
**ClickHouse (the SQL path) is now built, but only if you run the right
|
||||
binary — and it has not yet been confirmed against a real ClickHouse.**
|
||||
`enterprise/internal/tenantprovision` (real `CREATE DATABASE`/`CREATE
|
||||
USER`/`GRANT` against ClickHouse) and `enterprise/internal/chrunner` (a
|
||||
per-tenant `driver.Conn` registry implementing api's
|
||||
`querylang/executor.SQLRunner`, resolving which tenant's connection to
|
||||
use from the authenticated identity in request context — never from a
|
||||
client-suppliable field) now exist, and a new binary,
|
||||
`enterprise/cmd/enterprise-api`, wires them into the same
|
||||
`api/queryapi.Handler`/`api/dashboards.Handler` core already ships. Real
|
||||
integration tests exist and would prove the core adversarial claim —
|
||||
`enterprise/internal/tenantprovision/tenantprovision_test.go`'s
|
||||
`TestProvisionedUserCannotReadOtherTenantDatabase` and
|
||||
`TestProvisionedUserCannotReadSystemTables`,
|
||||
`enterprise/internal/chrunner/chrunner_test.go`'s
|
||||
`TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` — but this
|
||||
environment had no Docker/ClickHouse access while these were written, so
|
||||
they've only been confirmed to skip cleanly offline, not to pass for
|
||||
real. See `/docs/phase-4-runbook.md`'s verification-status section
|
||||
before treating "the test exists" as "isolation is confirmed."
|
||||
|
||||
**But plain `api/cmd/api` still runs with one shared connection**, and
|
||||
nothing in this repo automatically routes traffic to `enterprise-api`
|
||||
instead — `docker-compose.yml` includes it "available, not defaulted
|
||||
into the traffic path" (same shape as `enterprise-auth`'s own addition),
|
||||
and the Helm chart has no service for it at all yet. **A deployment is
|
||||
only as isolated as which binary is actually serving traffic** — this
|
||||
is an operational decision nothing currently enforces or even surfaces
|
||||
as a warning.
|
||||
|
||||
**Tantivy (the free-text path) is still fully unisolated.** There is no
|
||||
`enterprise/internal/searchclient` (the Tantivy-side equivalent of
|
||||
chrunner) — `search`'s gRPC service and
|
||||
`proto/sentry/search/v1/search.proto`'s `SearchRequest` still carry no
|
||||
tenant field anywhere, confirmed by reading the code. Every tenant's
|
||||
free-text queries hit the same shared Tantivy index regardless of which
|
||||
binary (`api` or `enterprise-api`) serves the HTTP request. A query that
|
||||
resolves to a pure pipe-syntax free-text search (e.g. `message:"error"`)
|
||||
is not protected by chrunner at all.
|
||||
|
||||
**What this means concretely**: treat a deployment as tenant-isolated
|
||||
for structured/SQL queries *only if* it runs `enterprise-api` fronting
|
||||
provisioned tenants, and treat it as **not isolated at all** for
|
||||
free-text search regardless of which binary runs. RBAC (below) and
|
||||
dashboard tenant-scoping (below) hold regardless of which binary is
|
||||
running; the ClickHouse/Tantivy split above is what changed.
|
||||
|
||||
## System overview
|
||||
|
||||
@@ -38,12 +68,19 @@ do not control what data that query can see.
|
||||
Browser ──▶ web (SvelteKit, static)
|
||||
│
|
||||
▼
|
||||
Browser ──▶ api ──▶ ClickHouse (log data, SQL path)
|
||||
│ └─▶ search (gRPC) ──▶ Tantivy (log data, full-text path)
|
||||
Browser ──▶ api OR enterprise-api ──▶ ClickHouse (log data, SQL path)
|
||||
│ └─▶ search (gRPC) ──▶ Tantivy (log data, full-text path)
|
||||
└─▶ Postgres (control plane: dashboards, alert_rules,
|
||||
tenants, users, tenant_memberships, audit_log)
|
||||
|
||||
alerting ──▶ api (POST /query, RoleService credential)
|
||||
# api: one shared ClickHouse connection, nil AuditLogger -- Phase 0-3 behavior.
|
||||
# enterprise-api: enterprise/internal/chrunner (per-tenant ClickHouse
|
||||
# connections) + enterprise/internal/audit.QueryAPILogger (real audit
|
||||
# writes) wired into the SAME api/queryapi.Handler/api/dashboards.Handler
|
||||
# core -- see this document's "Read this first" section. Either binary
|
||||
# can be running; nothing forces the isolated one.
|
||||
|
||||
alerting ──▶ api or enterprise-api (POST /query, RoleService credential)
|
||||
alerting ──▶ Postgres (rulestore, notifystore)
|
||||
|
||||
api/alerting ──▶ enterprise-auth (POST /internal/authorize, HTTP only —
|
||||
@@ -67,9 +104,9 @@ doesn't yet cover, not just an implementation gap.
|
||||
session issuance) is never imported by AGPL core (`/api`, `/alerting`,
|
||||
`/web`, `/cli`) — enforced in CI by `hack/check-tenant-boundary.sh`,
|
||||
which greps for the import edge on every build. Core calls
|
||||
`enterprise-auth` over plain HTTP (`api/internal/authz.HTTPAuthorizer`),
|
||||
`enterprise-auth` over plain HTTP (`api/authz.HTTPAuthorizer`),
|
||||
forwarding only the `Cookie`/`Authorization` headers, never the full
|
||||
request (`api/internal/authz/httpauthz_test.go` asserts this — an
|
||||
request (`api/authz/httpauthz_test.go` asserts this — an
|
||||
unrelated header like `X-Forwarded-For` is never forwarded). This means
|
||||
core's authorization decision is only as trustworthy as the network path
|
||||
to `enterprise-auth` — see "Deployment/network assumptions" below.
|
||||
@@ -95,9 +132,9 @@ a network-reachable endpoint) and configured via `API_SERVICE_TOKEN`.
|
||||
`enterprise/internal/session.Manager` issues and validates this token;
|
||||
`enterprise/internal/authhandler`'s `POST /internal/authorize` resolves
|
||||
it. `RoleService` is a distinct, non-comparable lane on the `Role` type
|
||||
(`api/internal/authz.Role.Satisfies`) — a service credential can never
|
||||
(`api/authz.Role.Satisfies`) — a service credential can never
|
||||
satisfy a human-role check and vice versa, verified by exhaustive
|
||||
table-driven tests (`api/internal/authz/authz_test.go`).
|
||||
table-driven tests (`api/authz/authz_test.go`).
|
||||
|
||||
**Session/token integrity.** Tokens are HS256-signed JWTs with a single
|
||||
shared signing key (`ENTERPRISE_SESSION_SIGNING_KEY`, ≥32 bytes,
|
||||
@@ -115,10 +152,10 @@ mode — bad signature, malformed token, expired — into one
|
||||
|
||||
**Live and enforced.** `POST /query` and every `/dashboards` endpoint in
|
||||
`api` require a minimum role, resolved per-request via
|
||||
`api/internal/authz.RequireRole`/`RequireRoleOrService` calling
|
||||
`api/authz.RequireRole`/`RequireRoleOrService` calling
|
||||
`enterprise-auth`. Roles: Viewer < Editor < Admin < Owner, plus the
|
||||
separate `RoleService` lane above. `GET /dashboards` is Viewer+;
|
||||
create/update/delete require Editor+ (`api/internal/dashboards/
|
||||
create/update/delete require Editor+ (`api/dashboards/
|
||||
handler.go`). A nil `Authorizer` (no `ENTERPRISE_AUTH_URL` configured)
|
||||
is a deliberate no-op, matching Phase 0-3's no-auth behavior — this is
|
||||
correct default-open-for-single-tenant behavior, not an oversight, but
|
||||
@@ -135,12 +172,12 @@ dashboard in that tenant, not just their own/granted ones.
|
||||
|
||||
**Application-layer tenant scoping (dashboards only).** Every
|
||||
`dashboards` store query filters `WHERE tenant_id = $identity.TenantID`
|
||||
(`api/internal/dashboards/store.go`), and the handler resolves that
|
||||
(`api/dashboards/store.go`), and the handler resolves that
|
||||
tenant ID from the RBAC-authenticated identity's context
|
||||
(`authz.IdentityFromContext`), **never** from a client-supplied request
|
||||
field. This closes a real gap found during this document's own review:
|
||||
`Dashboard.TenantID` is a JSON-tagged, client-settable field
|
||||
(`api/internal/dashboards/types.go`), and the original handler/store
|
||||
(`api/dashboards/types.go`), and the original handler/store
|
||||
implementation trusted it directly on create/update and applied no
|
||||
`tenant_id` filter at all on list/get/update/delete — meaning any
|
||||
authenticated user could read, modify, or delete any other tenant's
|
||||
@@ -149,7 +186,7 @@ dashboards simply by supplying (or guessing) their UUID, or spoof
|
||||
to. Fixed as part of this task, with regression tests proving
|
||||
cross-tenant access now returns 404 (not 403, which would itself leak
|
||||
that the ID exists under a different tenant) —
|
||||
`api/internal/dashboards/handler_test.go`'s
|
||||
`api/dashboards/handler_test.go`'s
|
||||
`TestCrossTenant*`/`TestCreateDashboardIgnoresClientSuppliedTenantID`/
|
||||
`TestImportIgnoresExportedTenantID`. **This same class of bug should be
|
||||
assumed present anywhere else client-supplied identifiers cross a tenant
|
||||
@@ -200,7 +237,7 @@ altered after the fact" claim actually holds against a privileged
|
||||
insider.
|
||||
|
||||
**Fail-open by design for routine queries.** `queryapi.Handler.logAudit`
|
||||
(`api/internal/queryapi/handler.go`) logs a write failure and otherwise
|
||||
(`api/queryapi/handler.go`) logs a write failure and otherwise
|
||||
ignores it — an audit-log outage does not take down the query path. This
|
||||
is a deliberate availability-over-completeness tradeoff: it means a
|
||||
brief audit outage produces an under-logged (not over-blocked) window.
|
||||
@@ -234,15 +271,21 @@ terms:
|
||||
credentials. That's an operational control (credential custody,
|
||||
infrastructure access review), out of scope for this system's own
|
||||
code.
|
||||
- **`system.query_log` metadata leakage** (task 2's finding): once
|
||||
per-tenant ClickHouse users exist, `system.query_log` and related
|
||||
`system.*` tables can expose other tenants' query *text* (predicate
|
||||
values, field names) even if row-level isolation between databases
|
||||
works perfectly. The design calls for revoking `system.*` access from
|
||||
every tenant user explicitly, not relying on ClickHouse's default
|
||||
template — this can only be verified once per-tenant users actually
|
||||
exist (they don't yet; see the top of this document), so it remains
|
||||
an open verification item, not a closed one.
|
||||
- **`system.query_log` metadata leakage — per-tenant users are now
|
||||
real, but the check itself hasn't run yet.** Was an open verification
|
||||
item because there were no per-tenant ClickHouse users to check
|
||||
against; that blocker is gone (`enterprise/internal/tenantprovision`
|
||||
exists), and `tenantprovision_test.go`'s
|
||||
`TestProvisionedUserCannotReadSystemTables` asserts exactly what the
|
||||
design calls for (`system.query_log`/`system.tables` inaccessible,
|
||||
`SHOW DATABASES` not revealing other tenants) — but this environment
|
||||
never had ClickHouse access to actually run it, so it remains
|
||||
unconfirmed against the pinned version
|
||||
(`clickhouse/clickhouse-server:24.8`) until someone with Docker access
|
||||
runs it (`/docs/phase-4-runbook.md` §8). Also still contingent on the
|
||||
deployment-shape caveat at the top of this document: even once
|
||||
confirmed, this only holds when `enterprise-api` (not plain `api`) is
|
||||
actually serving traffic.
|
||||
- **No deny-override grants** — `dashboard_permissions` is additive-only
|
||||
by design; a full allow/deny ACL system is unbuilt, future work.
|
||||
- **No data retention/deletion policy** for a deprovisioned tenant —
|
||||
@@ -278,12 +321,16 @@ terms:
|
||||
|---|---|
|
||||
| Role-based access control on `/query`, `/dashboards` | **Enforced** |
|
||||
| `alerting`↔`api` service-identity credential | **Enforced** |
|
||||
| Tenant scoping on dashboards (control-plane data) | **Enforced** (fixed this task) |
|
||||
| Tenant isolation on log data (`/query` → ClickHouse/Tantivy) | **Not implemented** |
|
||||
| Tenant scoping on dashboards (control-plane data) | **Enforced** |
|
||||
| ClickHouse per-tenant provisioning (`tenantprovision`) | **Built, not live-verified** — real integration test exists, not yet run against ClickHouse |
|
||||
| ClickHouse query routing (`chrunner`) | **Built, not live-verified** — and only applies when `enterprise-api` serves traffic, not plain `api` |
|
||||
| `system.*` ClickHouse metadata isolation | **Built, not live-verified** — same caveat as above |
|
||||
| Tantivy/free-text tenant isolation | **Not implemented** — no per-tenant index routing at all |
|
||||
| Deployment actually routing traffic to `enterprise-api` | **Not implemented** — no Helm service, no default wiring |
|
||||
| Human SSO login (OIDC/SAML) | **Not implemented** |
|
||||
| Per-resource dashboard grants (`own/granted`) | **Not implemented** |
|
||||
| Query audit logging (routine queries) | **Enforced**, fail-open |
|
||||
| Query audit logging (routine queries) | **Enforced**, fail-open, and now wired to a real writer via `enterprise-api` (`audit.QueryAPILogger`) |
|
||||
| Audit log tamper detection (hash chain) | **Enforced**, verified live |
|
||||
| Audit log tamper prevention (external anchoring) | **Design only** — `FileSink` is a dev stand-in |
|
||||
| `system.*` ClickHouse metadata isolation | **Unverified** — depends on unbuilt per-tenant users |
|
||||
| Mid-provisioning-race handling (evaluator ticks against a not-yet-active tenant) | **Unverified** — see `api/queryapi/tenant_isolation_gap_test.go` |
|
||||
| Protection against a privileged DB administrator | **Explicit non-goal** |
|
||||
|
||||
Reference in New Issue
Block a user