Close search's active-tenant write-routing gap with a polled allowlist
search/src/consumer.rs's write-routing (built last pass) had no active- tenant check at all: IndexRegistry.resolve() would open-or-create an index directory for any syntactically-valid tenant_id, active or not -- unlike ClickHouse's chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup) or the read side (gated by searchclient.TenantChecker, a direct rbacstore query). search is AGPL core with no Postgres access and no enterprise/ import allowed, so it needed a network boundary instead -- the same shape ingest's TenantResolver already uses against enterprise-auth, just Rust calling Go instead of Go calling Go. New GET /internal/active-tenants endpoint on enterprise-auth (rbacstore.ListActiveTenantIDs + authhandler.handleActiveTenants), gated on a RoleService Bearer credential -- server-to-server auth, the same shape alerting presents to api, minted via the already-generic enterprise-auth -mint-service-token search. search/src/tenants.rs's ActiveTenantTracker polls it every 60s, blocking startup on the first fetch succeeding (fail-closed cold start -- a control-plane outage at boot must not silently accept every tenant_id) and keeping the last- known-good set on any later refresh failure (a transient blip shouldn't stop every tenant's indexing, only prevent the allowlist from growing/ shrinking until connectivity resumes). consumer.rs refuses any tagged record whose tenant isn't in the polled set, before ever calling resolve() -- IndexRegistry itself stays policy-free, matching the same mechanism/policy split clickhousewriter.Writer vs. chwriter.Registry already draws on the ClickHouse side. Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are both set (search/src/config.rs rejects exactly one being set) -- every existing deployment is unaffected. Verified with real HTTP round trips in this environment: tenants.rs's tests exercise real reqwest requests (actual Authorization: Bearer header, actual JSON parsing) against a hand-rolled dependency-free TCP test server, including both fail-closed paths (rejected first fetch, unreachable server). authhandler's new tests cover the credential-kind distinction this endpoint exists to enforce -- a real human session, even for a genuine Owner, must not satisfy a check meant for a service identity. One asymmetry remains, disclosed rather than fixed: chwriter.Registry's snapshot still never refreshes (stale until enterprise-ingest restarts), while ActiveTenantTracker's 60s poll gives Tantivy a materially tighter staleness window. Neither is a live per-write check -- that would mean a database/HTTP round trip per record, a throughput cost neither implementation accepts -- so both have some staleness window by design; the gap between the two windows is what's disclosed, not a claim either is fully live.
This commit is contained in:
+15
-11
@@ -171,17 +171,21 @@ escape hatch is opaque to any compiler-injected filter.
|
||||
index. No "second binary" needed here, unlike ClickHouse — Tantivy has
|
||||
no grant system to gate a commercially-licensed credential behind, so
|
||||
`IndexRegistry` already lived directly in this AGPL-core binary, and
|
||||
read/write just share it. One gap is disclosed rather than closed by
|
||||
this change: unlike `chwriter.Registry` (an active-tenants-only
|
||||
snapshot built at startup) and unlike the read side (gated by
|
||||
`searchclient.TenantChecker`), this consumer's `resolve()` call has no
|
||||
active-tenant check — the process has no Postgres access to check
|
||||
against — so a still-valid-but-should-be-revoked ingest credential can
|
||||
cause an index directory to be created for a tenant that's no longer
|
||||
active. Narrow blast radius (an orphan, isolated, empty index, not
|
||||
cross-tenant leakage, and only reachable with a real signed
|
||||
credential), but real; see `search/src/registry.rs`'s doc comment on
|
||||
`resolve`.
|
||||
read/write just share it. The active-tenant gap this design left open
|
||||
is now closed too: `search/src/tenants.rs`'s `ActiveTenantTracker`
|
||||
polls a new `GET /internal/active-tenants` endpoint on
|
||||
`enterprise-auth` — `search` has no Postgres access, so unlike
|
||||
`chwriter.Registry`'s direct `rbacstore` query or the read side's
|
||||
`searchclient.TenantChecker`, this needed a network call instead (the
|
||||
same "network boundary, not import boundary" shape `ingest`'s
|
||||
`TenantResolver` already uses against the same service, authenticated
|
||||
with a RoleService credential like `alerting`↔`api`) — and
|
||||
`consumer.rs` refuses any tagged record whose tenant isn't in the
|
||||
polled allowlist. Off unless configured, fail-closed on the first
|
||||
fetch, last-known-good on later refresh failures; see
|
||||
`search/src/tenants.rs`'s doc comment for the full design and
|
||||
`search/src/registry.rs`'s for how mechanism (index lifecycle) and
|
||||
policy (the gate) responsibilities split.
|
||||
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified, deliberately lightweight: `-provision-tenant` stays
|
||||
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
|
||||
|
||||
+68
-29
@@ -39,14 +39,18 @@ of what was already run and passed. Two genuine exceptions:
|
||||
OIDC: not yet tried against a real external IdP or a running
|
||||
`enterprise-auth` container.
|
||||
- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s
|
||||
cross-tenant read isolation (§9) and, since this pass,
|
||||
`search/src/consumer.rs`'s per-tenant write-routing (§14) -- verified
|
||||
live, no disclaimer needed, because Tantivy is an embedded library with
|
||||
no Docker/broker dependency: real indices, real documents, real
|
||||
commits, run in this environment. The one thing about it that's still
|
||||
unverified is not Tantivy itself but the upstream credential/header
|
||||
plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
|
||||
`/internal/authorize-ingest`) against a real running stack.
|
||||
cross-tenant read isolation (§9), `search/src/consumer.rs`'s per-tenant
|
||||
write-routing (§14), and `search/src/tenants.rs`'s active-tenant gate
|
||||
(§14) -- verified live, no disclaimer needed, because Tantivy is an
|
||||
embedded library with no Docker/broker dependency, and the active-
|
||||
tenant gate's only external dependency (`enterprise-auth`'s HTTP API)
|
||||
was exercised against a real hand-rolled test server, not a live
|
||||
container: real indices, real documents, real commits, real HTTP
|
||||
requests, all run in this environment. The one thing about it that's
|
||||
still unverified is not Tantivy itself but the upstream credential/
|
||||
header plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
|
||||
`/internal/authorize-ingest` and `/internal/active-tenants`) against a
|
||||
real running stack.
|
||||
- The tenant-picker frontend page (§12) -- the first frontend-only piece
|
||||
in this phase exercised in a real browser rather than only
|
||||
type-checked: `web/src/routes/select-tenant`'s cross-origin
|
||||
@@ -746,21 +750,54 @@ cargo test --quiet
|
||||
# ingest/cmd/ingest's own guard test does on the Go side.
|
||||
```
|
||||
|
||||
**What's still not built, for either engine**: a live active-tenant
|
||||
recheck at write time. `chwriter.Registry`'s per-tenant writer map is a
|
||||
snapshot built once at `enterprise-ingest` startup from
|
||||
`rbacstore.ListProvisionedDataSources` (active tenants only) -- an
|
||||
unrecognized `tenant_id` is refused outright, but a tenant deprovisioned
|
||||
*after* startup keeps writing successfully until the next restart.
|
||||
`IndexRegistry.resolve()` has no allowlist at all on either the read or
|
||||
write side -- `search` has no Postgres access to check tenant status
|
||||
against -- so a still-valid-but-should-be-revoked ingest credential can
|
||||
cause an orphan Tantivy index directory to be created for a tenant
|
||||
that's no longer active. Narrow blast radius either way (isolated, not
|
||||
cross-tenant leakage, reachable only with a real signed credential), but
|
||||
real and disclosed, not silently accepted -- see
|
||||
`search/src/registry.rs`'s doc comment on `resolve` and
|
||||
`/docs/security/threat-model.md`'s "Read this first".
|
||||
**Tantivy's write path is now active-tenant-gated too.**
|
||||
`search/src/tenants.rs`'s `ActiveTenantTracker` polls a new
|
||||
`GET /internal/active-tenants` endpoint on `enterprise-auth` (Go side:
|
||||
`rbacstore.ListActiveTenantIDs` + `authhandler.handleActiveTenants`,
|
||||
RoleService-credentialed -- mint one with
|
||||
`enterprise-auth -mint-service-token search`, the same generic flag
|
||||
`alerting` already uses, just a different subject name) and
|
||||
`consumer.rs` refuses any tagged record whose tenant isn't in the polled
|
||||
set. Off unless `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are
|
||||
both set (`search/src/config.rs` rejects exactly one being set); startup
|
||||
blocks on the first fetch succeeding (fail-closed cold start), and a
|
||||
later refresh failure keeps serving the last-known-good set rather than
|
||||
clearing it. Genuinely verified in this environment, no live
|
||||
enterprise-auth needed:
|
||||
|
||||
```sh
|
||||
cd search
|
||||
cargo test --quiet tenants::
|
||||
# start_fetches_and_serves_the_initial_list / start_sends_the_bearer_token
|
||||
# run against a real (hand-rolled, dependency-free) TCP server -- actual
|
||||
# reqwest request construction (the Authorization: Bearer header, the
|
||||
# /internal/active-tenants path) and actual JSON response parsing, not a
|
||||
# fake HTTP client. start_fails_closed_when_the_first_fetch_fails and
|
||||
# start_fails_closed_when_the_server_is_unreachable prove the cold-start
|
||||
# refusal: ActiveTenantTracker::start returns Err rather than falling
|
||||
# back to an empty (accept-nothing, silently-safe-looking-but-wrong-for-
|
||||
# operators) or permissive (accept-anything, the exact bug being closed)
|
||||
# default.
|
||||
go test ./internal/authhandler/... -run TestActiveTenants -v
|
||||
# GET /internal/active-tenants requires a RoleService credential -- a
|
||||
# valid human session (even for a real Owner) is rejected the same way
|
||||
# an invalid/missing token is, the regression test for this endpoint's
|
||||
# whole reason to distinguish token kinds.
|
||||
```
|
||||
|
||||
**One asymmetry remains, disclosed rather than fixed**: `chwriter.
|
||||
Registry`'s per-tenant writer map is a snapshot built once at
|
||||
`enterprise-ingest` startup, never refreshed -- a tenant deprovisioned
|
||||
after startup keeps writing successfully to ClickHouse until the next
|
||||
restart, a real staleness window `ActiveTenantTracker`'s 60-second
|
||||
refresh doesn't have on the Tantivy side. Neither is a live per-write
|
||||
check (a database/HTTP round trip on every record would be a real
|
||||
throughput cost neither implementation accepts), so both have *some*
|
||||
staleness window by design -- the gap between the two windows is what's
|
||||
disclosed as inconsistent here, not a claim that either is fully live.
|
||||
Closing it would mean adding a periodic refresh to `chwriter.Registry`
|
||||
too; not done in this pass. See `/docs/security/threat-model.md`'s "Read
|
||||
this first".
|
||||
|
||||
**Also not built**: Helm/`docker-compose.yml` do gate *whether*
|
||||
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
|
||||
@@ -811,12 +848,14 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
see §14). `search/src/consumer.rs` reads the same header and routes
|
||||
each record into its own tenant's Tantivy index -- genuinely verified
|
||||
in this environment, unlike the ClickHouse side, since Tantivy needs
|
||||
no Docker to exercise real logic. Both engines share one remaining,
|
||||
disclosed gap: neither write path rechecks tenant-active status live
|
||||
(ClickHouse: a startup-time snapshot, stale until restart; Tantivy: no
|
||||
allowlist at all, since `search` has no Postgres access) -- narrow
|
||||
blast radius, not cross-tenant leakage, but real; see §14 and
|
||||
`/docs/security/threat-model.md`'s "Read this first". A
|
||||
no Docker to exercise real logic. Both engines now also gate writes on
|
||||
an active-tenant check: ClickHouse's is a startup-time snapshot with
|
||||
no refresh (stale until the next `enterprise-ingest` restart), while
|
||||
Tantivy's `ActiveTenantTracker` polls `enterprise-auth` every 60
|
||||
seconds, a tighter staleness bound the ClickHouse side wasn't updated
|
||||
to match -- a real, disclosed asymmetry between the two, not a gap in
|
||||
either alone; see §14 and `/docs/security/threat-model.md`'s "Read
|
||||
this first". A
|
||||
newly-provisioned tenant's ClickHouse database and Tantivy index are
|
||||
both now real, isolated, and actually populated by write-routed
|
||||
traffic (the ClickHouse claim pending live confirmation, the Tantivy
|
||||
|
||||
@@ -9,21 +9,27 @@ for the full design rationale behind the controls described here.
|
||||
|
||||
## Read this first: the single most important open finding
|
||||
|
||||
**Updated a fourth time.** This section originally read "log data
|
||||
queried through `POST /query` is not tenant-isolated at all," then
|
||||
"ClickHouse is isolated but Tantivy isn't," then "ingest tags records
|
||||
with a tenant identity but nothing routes the write," then "ClickHouse
|
||||
write-routing is built but Tantivy's isn't." Both storage engines are
|
||||
now isolated on both the read and write paths. What's left is narrower:
|
||||
**whether a given deployment actually runs the isolated binaries**
|
||||
(deployment-time, not code-level), and two disclosed write-side gaps,
|
||||
different in kind: ClickHouse's write registry is a startup-time
|
||||
snapshot of active tenants with no live recheck (a *deprovisioned*
|
||||
tenant can keep writing successfully until the next `enterprise-ingest`
|
||||
restart), while Tantivy's write path has no active-tenant allowlist at
|
||||
all — it opens an index for *any* syntactically-valid `tenant_id` a
|
||||
record carries, active, deprovisioned, or never real. See below for
|
||||
both, in full.
|
||||
**Updated a fifth time.** This section originally read "log data queried
|
||||
through `POST /query` is not tenant-isolated at all," then "ClickHouse
|
||||
is isolated but Tantivy isn't," then "ingest tags records with a tenant
|
||||
identity but nothing routes the write," then "ClickHouse write-routing
|
||||
is built but Tantivy's isn't," then "both are write-routed but neither
|
||||
rechecks tenant-active status live." Both storage engines are now
|
||||
isolated on both the read and write paths, **and both now gate writes
|
||||
on an active-tenant check** — ClickHouse's (`chwriter.Registry`) and
|
||||
Tantivy's (`search/src/tenants.rs`'s `ActiveTenantTracker`, new) differ
|
||||
in staleness bound, not in whether the gate exists at all: ClickHouse's
|
||||
is a startup-time snapshot with no refresh (a *deprovisioned* tenant can
|
||||
keep writing successfully until the next `enterprise-ingest` restart);
|
||||
Tantivy's polls `enterprise-auth` every 60 seconds and keeps serving the
|
||||
last-known-good set through a transient refresh failure, a materially
|
||||
tighter staleness window with no code changed on the ClickHouse side to
|
||||
match it — a real, disclosed asymmetry between the two, not a claim
|
||||
they're now identical. What's left is narrower still: **whether a given
|
||||
deployment actually runs the isolated binaries** (deployment-time, not
|
||||
code-level), and closing ClickHouse's staleness gap to match Tantivy's
|
||||
if that inconsistency matters for a given deployment. See below for
|
||||
both engines' write-routing, in full.
|
||||
|
||||
**ClickHouse (the SQL path) is built.** `enterprise/internal/
|
||||
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
|
||||
@@ -117,25 +123,35 @@ library with no Docker dependency. No "second binary" was needed here,
|
||||
unlike ClickHouse: Tantivy has no grant system to gate a
|
||||
commercially-licensed credential behind, so `IndexRegistry` already
|
||||
lived directly in this AGPL-core `search` binary, and read/write simply
|
||||
share it.
|
||||
share it. This write path is now also active-tenant-gated:
|
||||
`search/src/tenants.rs`'s `ActiveTenantTracker` polls a new
|
||||
`GET /internal/active-tenants` endpoint on `enterprise-auth` every 60
|
||||
seconds (RoleService-credentialed, the same auth shape `alerting` uses
|
||||
against `api`) and `consumer.rs` refuses any tagged record whose tenant
|
||||
isn't in the polled allowlist — fail-closed on the first fetch (startup
|
||||
blocks until it succeeds), last-known-good on any later refresh failure.
|
||||
Off unless `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are
|
||||
both set. Genuinely verified here too: real HTTP round trips (the Bearer
|
||||
header actually sent, JSON parsing, both fail-closed paths) against a
|
||||
hand-rolled TCP test server, no live enterprise-auth needed since
|
||||
`reqwest` doesn't care that the other end is real.
|
||||
|
||||
**Both engines share one open question** — deployment topology, covered
|
||||
above — and Tantivy specifically has one gap ClickHouse's design doesn't:
|
||||
`chwriter.Registry`'s map is built once at startup from an
|
||||
active-tenants-only query, so an unrecognized `tenant_id` is refused
|
||||
outright; `IndexRegistry.resolve()` (used for both read and write) has
|
||||
no equivalent allowlist at all, because `search` has no Postgres access
|
||||
to check tenant status against — a syntactically-valid `tenant_id` on a
|
||||
still-valid-but-should-be-revoked ingest credential can cause an orphan
|
||||
index directory to be created for a tenant that's no longer active.
|
||||
Narrow blast radius (isolated, empty except for that traffic, not
|
||||
cross-tenant leakage, and reachable only with a real signed credential),
|
||||
but real, and not closed by this change; see
|
||||
`search/src/registry.rs`'s doc comment on `resolve`. A newly-provisioned
|
||||
tenant's ClickHouse database and Tantivy index are both now real,
|
||||
isolated, and actually populated by write-routed agent traffic (the
|
||||
ClickHouse claim pending live confirmation, the Tantivy claim already
|
||||
verified).
|
||||
above — and now differ only in staleness bound, not in whether an
|
||||
active-tenant gate exists at all: `chwriter.Registry`'s map is built
|
||||
once at `enterprise-ingest` startup from an active-tenants-only query
|
||||
and never refreshed, so a tenant deprovisioned after startup keeps
|
||||
writing successfully until the next restart; `ActiveTenantTracker`
|
||||
refreshes every 60 seconds, a materially tighter window, with no
|
||||
corresponding change made to the ClickHouse side to match it. Neither
|
||||
is a live per-write check (that would mean a database round trip on
|
||||
every record, a real throughput cost neither implementation accepts),
|
||||
so both have *some* staleness window by design — the asymmetry between
|
||||
the two windows is the one thing disclosed as inconsistent, not fixed,
|
||||
here. A newly-provisioned tenant's ClickHouse database and Tantivy index
|
||||
are both now real, isolated, and actually populated by write-routed
|
||||
agent traffic (the ClickHouse claim pending live confirmation, the
|
||||
Tantivy claim already verified).
|
||||
|
||||
## System overview
|
||||
|
||||
@@ -494,7 +510,7 @@ terms:
|
||||
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
|
||||
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
|
||||
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart |
|
||||
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. No active-tenant allowlist at all on write (`search` has no Postgres access) — a syntactically-valid `tenant_id` on a still-valid credential can create an orphan index for a no-longer-active tenant; narrow, disclosed, not cross-tenant leakage |
|
||||
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — tighter staleness bound than ClickHouse's startup-only snapshot, an asymmetry disclosed above, not a gap on Tantivy's side specifically |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
|
||||
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
|
||||
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
|
||||
|
||||
Reference in New Issue
Block a user