Build per-tenant Tantivy write-routing, closing the last ingest write gap
search/src/consumer.rs now resolves each record's tenant_id Kafka header through the same IndexRegistry the read side (search/src/registry.rs + enterprise/internal/searchclient) already used, and writes into that tenant's own index instead of always the default one. The periodic Tantivy commit now commits every tenant index that's actually seen a write (IndexRegistry::commit_all), not just the default index. Unlike ClickHouse, this needed no "second binary": Tantivy has no grant system to gate a commercially-licensed credential behind, so IndexRegistry already lived directly in this AGPL-core search binary -- there was never an import-boundary reason to split the write side into an enterprise/ binary the way chwriter/enterprise-ingest was for ClickHouse. Read and write simply share one registry. Because Tantivy is an embedded library, this is genuinely verified in this environment, not just written: registry.rs's commit_all_commits_default_and_every_opened_tenant_index writes into the default index plus two tenant indices, confirms nothing is searchable pre-commit, then confirms all three are post-commit. consumer.rs's tenant_id_from_headers is factored out as a small pure helper (mirroring ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a guard test against the "tenant_id" header-key literal drifting from the Go side's -- the same guard-test pattern ingest/cmd/ingest already used for its own two Go copies of the constant, now mirrored a third time across the language boundary. One gap is disclosed, not fixed, by this change: unlike chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup, so an unrecognized tenant_id is refused outright) and unlike the read side (gated by searchclient.TenantChecker), this consumer's registry.resolve() call has no active-tenant check at all -- search has no Postgres access to check tenant status against. A still-valid-but-should-be-revoked ingest credential can cause an index directory to be created for a tenant that's no longer active. Narrow blast radius (an orphan, isolated, empty index, not cross-tenant leakage, and only reachable with a real signed credential), but real -- see registry.rs's doc comment on resolve(). Closing it fully would mean giving search some way to learn which tenants are active without an enterprise/ import, which isn't designed yet. This closes the last of Phase 4's ingest write-routing gaps (ClickHouse was closed last commit). The one remaining gap in the whole phase is now the tenant-picker frontend page, deliberately deferred earlier in this phase as out of scope for this environment.
This commit is contained in:
+14
-10
@@ -182,14 +182,13 @@ silently left out:
|
||||
a cross-origin `fetch` with credentials from `web`'s origin needs
|
||||
it), neither of which is verifiable in this environment without a
|
||||
live backend and a browser session to exercise.
|
||||
- **Ingest write-routing, for Tantivy** -- `search/src/consumer.rs` (a
|
||||
completely independent Redpanda consumer, not called through `ingest`
|
||||
or `enterprise-ingest` at all) still writes every record into the one
|
||||
shared (default) Tantivy index, regardless of tenant. The ClickHouse
|
||||
half is now built (see "Ingest write-routing" below); Tantivy's is
|
||||
real, disclosed, separate follow-up work -- a different codebase
|
||||
(Rust) and a different consumer process, not just "the same fix
|
||||
applied twice."
|
||||
|
||||
Ingest write-routing (both ClickHouse and Tantivy) is no longer on this
|
||||
list -- see "Ingest write-routing (ClickHouse)" below and
|
||||
`/search/README.md`'s "Per-tenant indices" section for Tantivy, which
|
||||
needed no code in this module at all: `search`'s `IndexRegistry` already
|
||||
lived in AGPL core, so its write side didn't need an `enterprise/`
|
||||
counterpart the way ClickHouse's did.
|
||||
|
||||
Deployment-topology routing (does traffic actually reach `enterprise-api`
|
||||
instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and
|
||||
@@ -307,8 +306,13 @@ ingest must never import enterprise/) is the client side of `internal/
|
||||
authhandler`'s new `POST /internal/authorize-ingest` -- see "Ingest
|
||||
tenant identity" above.
|
||||
|
||||
Future additions: per-tenant write-routing for ingest (ClickHouse and
|
||||
Tantivy both) -- see "Ingest tenant identity" above.
|
||||
Per-tenant write-routing for ingest is built for both ClickHouse (this
|
||||
module's `cmd/enterprise-ingest` + `internal/chwriter`, see "Ingest
|
||||
write-routing (ClickHouse)" above) and Tantivy (`search/src/consumer.rs`
|
||||
+ `search/src/registry.rs`, entirely in AGPL core -- see
|
||||
`/search/README.md`'s "Per-tenant indices" section, since Tantivy's lack
|
||||
of a grant system meant there was never an import-boundary reason to put
|
||||
any of it here).
|
||||
|
||||
## Why OIDC and SAML aren't hand-rolled
|
||||
|
||||
|
||||
Reference in New Issue
Block a user