Build per-tenant Tantivy write-routing, closing the last ingest write gap

search/src/consumer.rs now resolves each record's tenant_id Kafka header
through the same IndexRegistry the read side (search/src/registry.rs +
enterprise/internal/searchclient) already used, and writes into that
tenant's own index instead of always the default one. The periodic
Tantivy commit now commits every tenant index that's actually seen a
write (IndexRegistry::commit_all), not just the default index.

Unlike ClickHouse, this needed no "second binary": Tantivy has no grant
system to gate a commercially-licensed credential behind, so
IndexRegistry already lived directly in this AGPL-core search binary --
there was never an import-boundary reason to split the write side into
an enterprise/ binary the way chwriter/enterprise-ingest was for
ClickHouse. Read and write simply share one registry.

Because Tantivy is an embedded library, this is genuinely verified in
this environment, not just written: registry.rs's
commit_all_commits_default_and_every_opened_tenant_index writes into the
default index plus two tenant indices, confirms nothing is searchable
pre-commit, then confirms all three are post-commit. consumer.rs's
tenant_id_from_headers is factored out as a small pure helper (mirroring
ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a
guard test against the "tenant_id" header-key literal drifting from the
Go side's -- the same guard-test pattern ingest/cmd/ingest already used
for its own two Go copies of the constant, now mirrored a third time
across the language boundary.

One gap is disclosed, not fixed, by this change: unlike chwriter.Registry
(an active-tenants-only snapshot built at enterprise-ingest startup, so
an unrecognized tenant_id is refused outright) and unlike the read side
(gated by searchclient.TenantChecker), this consumer's registry.resolve()
call has no active-tenant check at all -- search has no Postgres access
to check tenant status against. A still-valid-but-should-be-revoked
ingest credential can cause an index directory to be created for a
tenant that's no longer active. Narrow blast radius (an orphan, isolated,
empty index, not cross-tenant leakage, and only reachable with a real
signed credential), but real -- see registry.rs's doc comment on
resolve(). Closing it fully would mean giving search some way to learn
which tenants are active without an enterprise/ import, which isn't
designed yet.

This closes the last of Phase 4's ingest write-routing gaps (ClickHouse
was closed last commit). The one remaining gap in the whole phase is now
the tenant-picker frontend page, deliberately deferred earlier in this
phase as out of scope for this environment.
This commit is contained in:
2026-08-14 20:16:09 -07:00
parent 1de77b969f
commit bdd42e06f6
9 changed files with 458 additions and 151 deletions
+14 -10
View File
@@ -182,14 +182,13 @@ silently left out:
a cross-origin `fetch` with credentials from `web`'s origin needs
it), neither of which is verifiable in this environment without a
live backend and a browser session to exercise.
- **Ingest write-routing, for Tantivy** -- `search/src/consumer.rs` (a
completely independent Redpanda consumer, not called through `ingest`
or `enterprise-ingest` at all) still writes every record into the one
shared (default) Tantivy index, regardless of tenant. The ClickHouse
half is now built (see "Ingest write-routing" below); Tantivy's is
real, disclosed, separate follow-up work -- a different codebase
(Rust) and a different consumer process, not just "the same fix
applied twice."
Ingest write-routing (both ClickHouse and Tantivy) is no longer on this
list -- see "Ingest write-routing (ClickHouse)" below and
`/search/README.md`'s "Per-tenant indices" section for Tantivy, which
needed no code in this module at all: `search`'s `IndexRegistry` already
lived in AGPL core, so its write side didn't need an `enterprise/`
counterpart the way ClickHouse's did.
Deployment-topology routing (does traffic actually reach `enterprise-api`
instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and
@@ -307,8 +306,13 @@ ingest must never import enterprise/) is the client side of `internal/
authhandler`'s new `POST /internal/authorize-ingest` -- see "Ingest
tenant identity" above.
Future additions: per-tenant write-routing for ingest (ClickHouse and
Tantivy both) -- see "Ingest tenant identity" above.
Per-tenant write-routing for ingest is built for both ClickHouse (this
module's `cmd/enterprise-ingest` + `internal/chwriter`, see "Ingest
write-routing (ClickHouse)" above) and Tantivy (`search/src/consumer.rs`
+ `search/src/registry.rs`, entirely in AGPL core -- see
`/search/README.md`'s "Per-tenant indices" section, since Tantivy's lack
of a grant system meant there was never an import-boundary reason to put
any of it here).
## Why OIDC and SAML aren't hand-rolled