Build per-tenant ClickHouse write-routing for ingest (Tantivy still deferred)
ingest tags every record with a tenant_id Kafka header (built previously), but nothing consumed it to actually route the write. This closes that for ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) reuses ingest/consumer's own flush loop unchanged, with enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer registry -- swapped in as the writer. A batch pulled from the single shared Redpanda topic can mix records from many tenants, so WriteBatch groups by TenantID and dispatches each group to its own tenant's connection, fail- closed on an empty or unrecognized tenant_id. ingest/consumer and ingest/clickhousewriter move out of internal/ (same reason api/internal/* moved earlier this phase: enterprise/ can't import anything under another module's internal/). Their New() constructors now take small local Config structs instead of ingest/internal/config types, so enterprise/ doesn't need that import either. Building this surfaced a real bug: tenantprovision.ProvisionClickHouse only granted SELECT on a tenant's ClickHouse user, correct for chrunner's read-only use but not enough for chwriter reusing the same credential to write -- every real per-tenant write would have failed closed with a permission error. Fixed by widening the grant to SELECT, INSERT; no cross-tenant boundary is crossed by also allowing INSERT within a tenant's own database. Helm gates enterprise-ingest's Deployment on the same ingest.requireTenantCredential flag that already gates tag validation -- write-routing is meaningless without tagging already being required, so they're one decision, not two. docker-compose.yml's version is a disclosed, weaker approximation: it can't achieve Helm's genuine -mode=server/-mode=consumer split, so with the enterprise profile active both ingest and enterprise-ingest independently consume every message via different consumer groups -- harmless duplication for local verification only. Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs) still doesn't read the tenant_id header at all -- every record still lands in the one shared index regardless of tenant. Not run: the live-ClickHouse- gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT regression test) -- no Docker/database access in this environment; they're correct Go that has never executed, disclosed as such in docs/security/ threat-model.md and docs/phase-4-runbook.md §14.
This commit is contained in:
+42
-9
@@ -14,17 +14,46 @@ binary, selected with `--mode`:
|
||||
and the ClickHouse writer both read the same Redpanda messages and need
|
||||
to agree on the same ID for the same record to join search hits back to
|
||||
rows — two consumers generating their own IDs would produce mismatched
|
||||
ones for what's supposed to be the same record.
|
||||
ones for what's supposed to be the same record. Also (Phase 4):
|
||||
resolves an optional per-tenant `Authorization: Bearer <token>`
|
||||
credential via `internal/grpcserver.TenantResolver` (nil by default --
|
||||
single-tenant behavior unchanged) and attaches the resolved tenant ID
|
||||
to every produced Kafka message as a `tenant_id` header
|
||||
(`consumer.TenantIDHeaderKey`) -- see "Multi-tenant write-routing"
|
||||
below.
|
||||
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
||||
shape (`internal/normalize`), and batch-writes via the native protocol
|
||||
driver. Commits Redpanda offsets only after a successful ClickHouse
|
||||
write, so a ClickHouse outage causes redelivery on restart rather than
|
||||
data loss.
|
||||
data loss. Reads each message's `tenant_id` header (if any) but this
|
||||
package's own writer (`clickhousewriter.Writer`, used by
|
||||
`cmd/ingest`'s single-tenant mode) ignores it -- every record still
|
||||
lands in the one shared ClickHouse database regardless of tag. See
|
||||
"Multi-tenant write-routing" below for where the tag actually gets
|
||||
used.
|
||||
- **all** (default) — both, in one process. This is what docker-compose
|
||||
runs. Splitting into two deployments later (e.g. to scale them
|
||||
independently in k8s) is a manifest change, not a code change — see
|
||||
`--mode`.
|
||||
|
||||
## Multi-tenant write-routing
|
||||
|
||||
This package (AGPL core) only ever writes to one shared ClickHouse
|
||||
database, regardless of any `tenant_id` tag a message carries -- routing
|
||||
a tagged record into its own tenant's dedicated database is
|
||||
`enterprise/internal/chwriter` and `enterprise/cmd/enterprise-ingest`'s
|
||||
job (commercial-licensed, per `/CLAUDE.md`'s licensing boundary), not
|
||||
this package's. `consumer` and `clickhousewriter` live outside
|
||||
`internal/` (moved there once `enterprise/internal/chwriter` needed to
|
||||
import them directly -- Go's compiler-enforced `internal/` visibility
|
||||
rule blocks a separate module from importing anything under
|
||||
`ingest/internal/...`, the same reason several `api/internal/...`
|
||||
packages moved out earlier in Phase 4) specifically so `enterprise/` can
|
||||
reuse this package's own flush loop and ClickHouse batch-insert logic
|
||||
unchanged, rather than reimplementing either. See
|
||||
`/enterprise/README.md`'s "Ingest tenant identity"/write-routing
|
||||
sections for the full story, including what's still not built.
|
||||
|
||||
## Why Redpanda stays in the path
|
||||
|
||||
Confirmed with the project owner during Phase 0 planning: the gRPC front
|
||||
@@ -65,6 +94,7 @@ full list and defaults) — no config file format for Phase 0:
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
|
||||
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
|
||||
| `ENTERPRISE_AUTH_URL` | (empty) | Enables `internal/grpcserver.TenantResolver` -- empty means PushBatch never requires a bearer credential and no `tenant_id` header is ever attached, same as every Phase 0-3 deployment |
|
||||
|
||||
## Building & testing
|
||||
|
||||
@@ -87,11 +117,14 @@ docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
|
||||
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
|
||||
`consumer` and `internal/grpcserver` depend on Redpanda and ClickHouse
|
||||
only through small interfaces (`reader`/`chWriter` in consumer,
|
||||
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
|
||||
unit-tested against fakes — no embedded broker or database needed. What's
|
||||
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
|
||||
wiring and the ClickHouse native-protocol driver itself. Those are only
|
||||
exercised by the docker-compose end-to-end flow described in
|
||||
`/docs/phase-0-runbook.md`.
|
||||
unit-tested against fakes — no embedded broker or database needed. This
|
||||
includes the tenant_id tagging/extraction round trip end to end (a fake
|
||||
`TenantResolver` in grpcserver's tests, a fake Kafka header in
|
||||
consumer's) — real logic, fake transport, no live enterprise-auth or
|
||||
Redpanda needed. What's *not* covered by these tests: the real
|
||||
`kafka.Reader`/`kafka.Writer` wiring and the ClickHouse native-protocol
|
||||
driver itself. Those are only exercised by the docker-compose end-to-end
|
||||
flow described in `/docs/phase-0-runbook.md`.
|
||||
|
||||
Reference in New Issue
Block a user