Close search's active-tenant write-routing gap with a polled allowlist

search/src/consumer.rs's write-routing (built last pass) had no active-
tenant check at all: IndexRegistry.resolve() would open-or-create an
index directory for any syntactically-valid tenant_id, active or not --
unlike ClickHouse's chwriter.Registry (an active-tenants-only snapshot
built at enterprise-ingest startup) or the read side (gated by
searchclient.TenantChecker, a direct rbacstore query). search is AGPL
core with no Postgres access and no enterprise/ import allowed, so it
needed a network boundary instead -- the same shape ingest's
TenantResolver already uses against enterprise-auth, just Rust calling
Go instead of Go calling Go.

New GET /internal/active-tenants endpoint on enterprise-auth
(rbacstore.ListActiveTenantIDs + authhandler.handleActiveTenants),
gated on a RoleService Bearer credential -- server-to-server auth, the
same shape alerting presents to api, minted via the already-generic
enterprise-auth -mint-service-token search. search/src/tenants.rs's
ActiveTenantTracker polls it every 60s, blocking startup on the first
fetch succeeding (fail-closed cold start -- a control-plane outage at
boot must not silently accept every tenant_id) and keeping the last-
known-good set on any later refresh failure (a transient blip shouldn't
stop every tenant's indexing, only prevent the allowlist from growing/
shrinking until connectivity resumes). consumer.rs refuses any tagged
record whose tenant isn't in the polled set, before ever calling
resolve() -- IndexRegistry itself stays policy-free, matching the same
mechanism/policy split clickhousewriter.Writer vs. chwriter.Registry
already draws on the ClickHouse side.

Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are both
set (search/src/config.rs rejects exactly one being set) -- every
existing deployment is unaffected.

Verified with real HTTP round trips in this environment: tenants.rs's
tests exercise real reqwest requests (actual Authorization: Bearer
header, actual JSON parsing) against a hand-rolled dependency-free TCP
test server, including both fail-closed paths (rejected first fetch,
unreachable server). authhandler's new tests cover the credential-kind
distinction this endpoint exists to enforce -- a real human session,
even for a genuine Owner, must not satisfy a check meant for a service
identity.

One asymmetry remains, disclosed rather than fixed: chwriter.Registry's
snapshot still never refreshes (stale until enterprise-ingest restarts),
while ActiveTenantTracker's 60s poll gives Tantivy a materially tighter
staleness window. Neither is a live per-write check -- that would mean
a database/HTTP round trip per record, a throughput cost neither
implementation accepts -- so both have some staleness window by design;
the gap between the two windows is what's disclosed, not a claim either
is fully live.
This commit is contained in:
2026-08-14 23:47:25 -07:00
parent 5a845f06ee
commit 088677643f
19 changed files with 1139 additions and 141 deletions
+18 -11
View File
@@ -267,17 +267,24 @@ gate a commercially-licensed credential behind, so there was never an
import-boundary reason to split it out), so read and write share one import-boundary reason to split it out), so read and write share one
registry directly. The periodic Tantivy commit now commits every tenant registry directly. The periodic Tantivy commit now commits every tenant
index that's seen a write, not just the default one index that's seen a write, not just the default one
(`IndexRegistry::commit_all`). **One gap disclosed, not fixed, in this (`IndexRegistry::commit_all`). **The active-tenant gap this same change
same change**: unlike `chwriter.Registry` (built from an originally disclosed is now closed too**: `search/src/tenants.rs`'s
active-tenants-only snapshot at startup) and unlike the read side (gated `ActiveTenantTracker` polls a new `GET /internal/active-tenants`
by `searchclient.TenantChecker`), this consumer's `resolve()` call has endpoint on `enterprise-auth` (`search` has no Postgres access, unlike
no active-tenant check — this process has no Postgres access to check `chwriter.Registry`'s direct `rbacstore` query or the read side's
against, so a still-valid-but-should-be-revoked ingest credential can `searchclient.TenantChecker`, so this needed a network call — the same
cause an index directory to be created for a tenant that's no longer "network boundary, not import boundary" shape `ingest`'s
active. Narrow blast radius (an orphan, isolated, empty index — not `TenantResolver` already uses against the same service, authenticated
cross-tenant leakage — and only reachable with a real signed with a RoleService credential the same way `alerting` authenticates to
credential), but real; see `search/src/registry.rs`'s doc comment on `api`) and `consumer.rs` refuses any tagged record whose tenant isn't in
`resolve`. **The tenant-picker page is now built too**: the polled allowlist. Off unless both `ENTERPRISE_AUTH_URL` and
`ENTERPRISE_AUTH_SERVICE_TOKEN` are set (same "off unless configured"
default as everything else optional in this codebase); when they are,
startup blocks on the first fetch succeeding and later refresh failures
keep serving the last-known-good set rather than clearing it. Verified
with real HTTP round trips against a hand-rolled TCP test server in this
environment, no live enterprise-auth needed. **The tenant-picker page is
now built too**:
`web/src/routes/select-tenant` calls `GET /auth/memberships`/ `web/src/routes/select-tenant` calls `GET /auth/memberships`/
`POST /auth/select-tenant` via `fetch(..., {credentials: 'include'})` `POST /auth/select-tenant` via `fetch(..., {credentials: 'include'})`
(new `$lib/api.ts` functions), which needed a second CORS posture (new `$lib/api.ts` functions), which needed a second CORS posture
+10
View File
@@ -187,6 +187,16 @@ services:
# -- found by actually checking `docker compose logs search` and # -- found by actually checking `docker compose logs search` and
# seeing nothing, same silent-logging gap the agent had in Phase 0. # seeing nothing, same silent-logging gap the agent had in Phase 0.
RUST_LOG: "info" RUST_LOG: "info"
# ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are deliberately
# NOT set here -- same reason enterprise-auth's own service below
# doesn't wire alerting's API_SERVICE_TOKEN in by default: the
# token can't be known ahead of time (mint it with
# `enterprise-auth -mint-service-token search` after enterprise-auth
# is up), so this is a manual step, not a default. Unset means
# search/src/tenants.rs's ActiveTenantTracker never starts and
# write-routing has no active-tenant gate, same as every deployment
# before this tracker existed -- see search/README.md's "Per-tenant
# indices" section for how to turn it on for manual testing.
volumes: volumes:
- search-index-data:/var/lib/sentry-search - search-index-data:/var/lib/sentry-search
+15 -11
View File
@@ -171,17 +171,21 @@ escape hatch is opaque to any compiler-injected filter.
index. No "second binary" needed here, unlike ClickHouse — Tantivy has index. No "second binary" needed here, unlike ClickHouse — Tantivy has
no grant system to gate a commercially-licensed credential behind, so no grant system to gate a commercially-licensed credential behind, so
`IndexRegistry` already lived directly in this AGPL-core binary, and `IndexRegistry` already lived directly in this AGPL-core binary, and
read/write just share it. One gap is disclosed rather than closed by read/write just share it. The active-tenant gap this design left open
this change: unlike `chwriter.Registry` (an active-tenants-only is now closed too: `search/src/tenants.rs`'s `ActiveTenantTracker`
snapshot built at startup) and unlike the read side (gated by polls a new `GET /internal/active-tenants` endpoint on
`searchclient.TenantChecker`), this consumer's `resolve()` call has no `enterprise-auth``search` has no Postgres access, so unlike
active-tenant check — the process has no Postgres access to check `chwriter.Registry`'s direct `rbacstore` query or the read side's
against — so a still-valid-but-should-be-revoked ingest credential can `searchclient.TenantChecker`, this needed a network call instead (the
cause an index directory to be created for a tenant that's no longer same "network boundary, not import boundary" shape `ingest`'s
active. Narrow blast radius (an orphan, isolated, empty index, not `TenantResolver` already uses against the same service, authenticated
cross-tenant leakage, and only reachable with a real signed with a RoleService credential like `alerting``api`) — and
credential), but real; see `search/src/registry.rs`'s doc comment on `consumer.rs` refuses any tagged record whose tenant isn't in the
`resolve`. polled allowlist. Off unless configured, fail-closed on the first
fetch, last-known-good on later refresh failures; see
`search/src/tenants.rs`'s doc comment for the full design and
`search/src/registry.rs`'s for how mechanism (index lifecycle) and
policy (the gate) responsibilities split.
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant` - `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
are now unified, deliberately lightweight: `-provision-tenant` stays are now unified, deliberately lightweight: `-provision-tenant` stays
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
+68 -29
View File
@@ -39,14 +39,18 @@ of what was already run and passed. Two genuine exceptions:
OIDC: not yet tried against a real external IdP or a running OIDC: not yet tried against a real external IdP or a running
`enterprise-auth` container. `enterprise-auth` container.
- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s - Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s
cross-tenant read isolation (§9) and, since this pass, cross-tenant read isolation (§9), `search/src/consumer.rs`'s per-tenant
`search/src/consumer.rs`'s per-tenant write-routing (§14) -- verified write-routing (§14), and `search/src/tenants.rs`'s active-tenant gate
live, no disclaimer needed, because Tantivy is an embedded library with (§14) -- verified live, no disclaimer needed, because Tantivy is an
no Docker/broker dependency: real indices, real documents, real embedded library with no Docker/broker dependency, and the active-
commits, run in this environment. The one thing about it that's still tenant gate's only external dependency (`enterprise-auth`'s HTTP API)
unverified is not Tantivy itself but the upstream credential/header was exercised against a real hand-rolled test server, not a live
plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s container: real indices, real documents, real commits, real HTTP
`/internal/authorize-ingest`) against a real running stack. requests, all run in this environment. The one thing about it that's
still unverified is not Tantivy itself but the upstream credential/
header plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
`/internal/authorize-ingest` and `/internal/active-tenants`) against a
real running stack.
- The tenant-picker frontend page (§12) -- the first frontend-only piece - The tenant-picker frontend page (§12) -- the first frontend-only piece
in this phase exercised in a real browser rather than only in this phase exercised in a real browser rather than only
type-checked: `web/src/routes/select-tenant`'s cross-origin type-checked: `web/src/routes/select-tenant`'s cross-origin
@@ -746,21 +750,54 @@ cargo test --quiet
# ingest/cmd/ingest's own guard test does on the Go side. # ingest/cmd/ingest's own guard test does on the Go side.
``` ```
**What's still not built, for either engine**: a live active-tenant **Tantivy's write path is now active-tenant-gated too.**
recheck at write time. `chwriter.Registry`'s per-tenant writer map is a `search/src/tenants.rs`'s `ActiveTenantTracker` polls a new
snapshot built once at `enterprise-ingest` startup from `GET /internal/active-tenants` endpoint on `enterprise-auth` (Go side:
`rbacstore.ListProvisionedDataSources` (active tenants only) -- an `rbacstore.ListActiveTenantIDs` + `authhandler.handleActiveTenants`,
unrecognized `tenant_id` is refused outright, but a tenant deprovisioned RoleService-credentialed -- mint one with
*after* startup keeps writing successfully until the next restart. `enterprise-auth -mint-service-token search`, the same generic flag
`IndexRegistry.resolve()` has no allowlist at all on either the read or `alerting` already uses, just a different subject name) and
write side -- `search` has no Postgres access to check tenant status `consumer.rs` refuses any tagged record whose tenant isn't in the polled
against -- so a still-valid-but-should-be-revoked ingest credential can set. Off unless `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are
cause an orphan Tantivy index directory to be created for a tenant both set (`search/src/config.rs` rejects exactly one being set); startup
that's no longer active. Narrow blast radius either way (isolated, not blocks on the first fetch succeeding (fail-closed cold start), and a
cross-tenant leakage, reachable only with a real signed credential), but later refresh failure keeps serving the last-known-good set rather than
real and disclosed, not silently accepted -- see clearing it. Genuinely verified in this environment, no live
`search/src/registry.rs`'s doc comment on `resolve` and enterprise-auth needed:
`/docs/security/threat-model.md`'s "Read this first".
```sh
cd search
cargo test --quiet tenants::
# start_fetches_and_serves_the_initial_list / start_sends_the_bearer_token
# run against a real (hand-rolled, dependency-free) TCP server -- actual
# reqwest request construction (the Authorization: Bearer header, the
# /internal/active-tenants path) and actual JSON response parsing, not a
# fake HTTP client. start_fails_closed_when_the_first_fetch_fails and
# start_fails_closed_when_the_server_is_unreachable prove the cold-start
# refusal: ActiveTenantTracker::start returns Err rather than falling
# back to an empty (accept-nothing, silently-safe-looking-but-wrong-for-
# operators) or permissive (accept-anything, the exact bug being closed)
# default.
go test ./internal/authhandler/... -run TestActiveTenants -v
# GET /internal/active-tenants requires a RoleService credential -- a
# valid human session (even for a real Owner) is rejected the same way
# an invalid/missing token is, the regression test for this endpoint's
# whole reason to distinguish token kinds.
```
**One asymmetry remains, disclosed rather than fixed**: `chwriter.
Registry`'s per-tenant writer map is a snapshot built once at
`enterprise-ingest` startup, never refreshed -- a tenant deprovisioned
after startup keeps writing successfully to ClickHouse until the next
restart, a real staleness window `ActiveTenantTracker`'s 60-second
refresh doesn't have on the Tantivy side. Neither is a live per-write
check (a database/HTTP round trip on every record would be a real
throughput cost neither implementation accepts), so both have *some*
staleness window by design -- the gap between the two windows is what's
disclosed as inconsistent here, not a claim that either is fully live.
Closing it would mean adding a periodic refresh to `chwriter.Registry`
too; not done in this pass. See `/docs/security/threat-model.md`'s "Read
this first".
**Also not built**: Helm/`docker-compose.yml` do gate *whether* **Also not built**: Helm/`docker-compose.yml` do gate *whether*
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same `enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
@@ -811,12 +848,14 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
see §14). `search/src/consumer.rs` reads the same header and routes see §14). `search/src/consumer.rs` reads the same header and routes
each record into its own tenant's Tantivy index -- genuinely verified each record into its own tenant's Tantivy index -- genuinely verified
in this environment, unlike the ClickHouse side, since Tantivy needs in this environment, unlike the ClickHouse side, since Tantivy needs
no Docker to exercise real logic. Both engines share one remaining, no Docker to exercise real logic. Both engines now also gate writes on
disclosed gap: neither write path rechecks tenant-active status live an active-tenant check: ClickHouse's is a startup-time snapshot with
(ClickHouse: a startup-time snapshot, stale until restart; Tantivy: no no refresh (stale until the next `enterprise-ingest` restart), while
allowlist at all, since `search` has no Postgres access) -- narrow Tantivy's `ActiveTenantTracker` polls `enterprise-auth` every 60
blast radius, not cross-tenant leakage, but real; see §14 and seconds, a tighter staleness bound the ClickHouse side wasn't updated
`/docs/security/threat-model.md`'s "Read this first". A to match -- a real, disclosed asymmetry between the two, not a gap in
either alone; see §14 and `/docs/security/threat-model.md`'s "Read
this first". A
newly-provisioned tenant's ClickHouse database and Tantivy index are newly-provisioned tenant's ClickHouse database and Tantivy index are
both now real, isolated, and actually populated by write-routed both now real, isolated, and actually populated by write-routed
traffic (the ClickHouse claim pending live confirmation, the Tantivy traffic (the ClickHouse claim pending live confirmation, the Tantivy
+49 -33
View File
@@ -9,21 +9,27 @@ for the full design rationale behind the controls described here.
## Read this first: the single most important open finding ## Read this first: the single most important open finding
**Updated a fourth time.** This section originally read "log data **Updated a fifth time.** This section originally read "log data queried
queried through `POST /query` is not tenant-isolated at all," then through `POST /query` is not tenant-isolated at all," then "ClickHouse
"ClickHouse is isolated but Tantivy isn't," then "ingest tags records is isolated but Tantivy isn't," then "ingest tags records with a tenant
with a tenant identity but nothing routes the write," then "ClickHouse identity but nothing routes the write," then "ClickHouse write-routing
write-routing is built but Tantivy's isn't." Both storage engines are is built but Tantivy's isn't," then "both are write-routed but neither
now isolated on both the read and write paths. What's left is narrower: rechecks tenant-active status live." Both storage engines are now
**whether a given deployment actually runs the isolated binaries** isolated on both the read and write paths, **and both now gate writes
(deployment-time, not code-level), and two disclosed write-side gaps, on an active-tenant check** — ClickHouse's (`chwriter.Registry`) and
different in kind: ClickHouse's write registry is a startup-time Tantivy's (`search/src/tenants.rs`'s `ActiveTenantTracker`, new) differ
snapshot of active tenants with no live recheck (a *deprovisioned* in staleness bound, not in whether the gate exists at all: ClickHouse's
tenant can keep writing successfully until the next `enterprise-ingest` is a startup-time snapshot with no refresh (a *deprovisioned* tenant can
restart), while Tantivy's write path has no active-tenant allowlist at keep writing successfully until the next `enterprise-ingest` restart);
all — it opens an index for *any* syntactically-valid `tenant_id` a Tantivy's polls `enterprise-auth` every 60 seconds and keeps serving the
record carries, active, deprovisioned, or never real. See below for last-known-good set through a transient refresh failure, a materially
both, in full. tighter staleness window with no code changed on the ClickHouse side to
match it — a real, disclosed asymmetry between the two, not a claim
they're now identical. What's left is narrower still: **whether a given
deployment actually runs the isolated binaries** (deployment-time, not
code-level), and closing ClickHouse's staleness gap to match Tantivy's
if that inconsistency matters for a given deployment. See below for
both engines' write-routing, in full.
**ClickHouse (the SQL path) is built.** `enterprise/internal/ **ClickHouse (the SQL path) is built.** `enterprise/internal/
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
@@ -117,25 +123,35 @@ library with no Docker dependency. No "second binary" was needed here,
unlike ClickHouse: Tantivy has no grant system to gate a unlike ClickHouse: Tantivy has no grant system to gate a
commercially-licensed credential behind, so `IndexRegistry` already commercially-licensed credential behind, so `IndexRegistry` already
lived directly in this AGPL-core `search` binary, and read/write simply lived directly in this AGPL-core `search` binary, and read/write simply
share it. share it. This write path is now also active-tenant-gated:
`search/src/tenants.rs`'s `ActiveTenantTracker` polls a new
`GET /internal/active-tenants` endpoint on `enterprise-auth` every 60
seconds (RoleService-credentialed, the same auth shape `alerting` uses
against `api`) and `consumer.rs` refuses any tagged record whose tenant
isn't in the polled allowlist — fail-closed on the first fetch (startup
blocks until it succeeds), last-known-good on any later refresh failure.
Off unless `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are
both set. Genuinely verified here too: real HTTP round trips (the Bearer
header actually sent, JSON parsing, both fail-closed paths) against a
hand-rolled TCP test server, no live enterprise-auth needed since
`reqwest` doesn't care that the other end is real.
**Both engines share one open question** — deployment topology, covered **Both engines share one open question** — deployment topology, covered
above — and Tantivy specifically has one gap ClickHouse's design doesn't: above — and now differ only in staleness bound, not in whether an
`chwriter.Registry`'s map is built once at startup from an active-tenant gate exists at all: `chwriter.Registry`'s map is built
active-tenants-only query, so an unrecognized `tenant_id` is refused once at `enterprise-ingest` startup from an active-tenants-only query
outright; `IndexRegistry.resolve()` (used for both read and write) has and never refreshed, so a tenant deprovisioned after startup keeps
no equivalent allowlist at all, because `search` has no Postgres access writing successfully until the next restart; `ActiveTenantTracker`
to check tenant status against — a syntactically-valid `tenant_id` on a refreshes every 60 seconds, a materially tighter window, with no
still-valid-but-should-be-revoked ingest credential can cause an orphan corresponding change made to the ClickHouse side to match it. Neither
index directory to be created for a tenant that's no longer active. is a live per-write check (that would mean a database round trip on
Narrow blast radius (isolated, empty except for that traffic, not every record, a real throughput cost neither implementation accepts),
cross-tenant leakage, and reachable only with a real signed credential), so both have *some* staleness window by design — the asymmetry between
but real, and not closed by this change; see the two windows is the one thing disclosed as inconsistent, not fixed,
`search/src/registry.rs`'s doc comment on `resolve`. A newly-provisioned here. A newly-provisioned tenant's ClickHouse database and Tantivy index
tenant's ClickHouse database and Tantivy index are both now real, are both now real, isolated, and actually populated by write-routed
isolated, and actually populated by write-routed agent traffic (the agent traffic (the ClickHouse claim pending live confirmation, the
ClickHouse claim pending live confirmation, the Tantivy claim already Tantivy claim already verified).
verified).
## System overview ## System overview
@@ -494,7 +510,7 @@ terms:
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test | | Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record | | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse**`enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart | | Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse**`enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart |
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified**`search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. No active-tenant allowlist at all on write (`search` has no Postgres access) — a syntactically-valid `tenant_id` on a still-valid credential can create an orphan index for a no-longer-active tenant; narrow, disclosed, not cross-tenant leakage | | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified**`search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — tighter staleness bound than ClickHouse's startup-only snapshot, an asymmetry disclosed above, not a gap on Tantivy's side specifically |
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced**`api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO | | Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced**`api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced**`api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment | | Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced**`api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) | | Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
+16 -1
View File
@@ -284,7 +284,7 @@ this environment.
## Package layout ## Package layout
``` ```
cmd/enterprise-auth/ config loading, OIDC discovery at startup, health/authorize/features/authorize-ingest endpoints, -mint-service-token, -create-tenant, -grant-membership-*, -revoke-membership-*, -list-memberships-tenant, -transfer-owner-*, -create-ingest-credential-tenant, -list-ingest-credentials-tenant, -revoke-ingest-credential cmd/enterprise-auth/ config loading, OIDC discovery at startup, health/authorize/features/authorize-ingest/active-tenants endpoints, -mint-service-token, -create-tenant, -grant-membership-*, -revoke-membership-*, -list-memberships-tenant, -transfer-owner-*, -create-ingest-credential-tenant, -list-ingest-credentials-tenant, -revoke-ingest-credential
cmd/enterprise-api/ multi-tenant-aware alternative to api/cmd/api -- see its own doc comment cmd/enterprise-api/ multi-tenant-aware alternative to api/cmd/api -- see its own doc comment
cmd/enterprise-ingest/ multi-tenant-aware alternative to ingest -mode=consumer -- see its own doc comment cmd/enterprise-ingest/ multi-tenant-aware alternative to ingest -mode=consumer -- see its own doc comment
internal/tenant/ the ID type -- see its package doc comment before touching it internal/tenant/ the ID type -- see its package doc comment before touching it
@@ -390,6 +390,21 @@ TOKEN=$(docker compose run --rm enterprise-auth -mint-service-token=alerting)
# alerting: set API_SERVICE_TOKEN=$TOKEN and restart # alerting: set API_SERVICE_TOKEN=$TOKEN and restart
``` ```
Same shape for `search`'s write-side active-tenant gate
(`search/src/tenants.rs` -- see `/search/README.md`'s "Per-tenant
indices" section):
```sh
docker compose up -d enterprise-auth
SEARCH_TOKEN=$(docker compose run --rm enterprise-auth -mint-service-token=search)
# search: set ENTERPRISE_AUTH_URL=http://enterprise-auth:8082 and
# ENTERPRISE_AUTH_SERVICE_TOKEN=$SEARCH_TOKEN, then restart -- unlike
# alerting/api above, this doesn't turn on request-level auth
# enforcement anywhere; it only gates which tenant_ids search will
# write-route into their own index, refusing anything not in
# GET /internal/active-tenants' response.
```
```sh ```sh
docker build -f Dockerfile -t sentry-enterprise-auth . # context is enterprise/, not the repo root docker build -f Dockerfile -t sentry-enterprise-auth . # context is enterprise/, not the repo root
``` ```
+1 -1
View File
@@ -205,7 +205,7 @@ func main() {
OIDCEnabled: cfg.OIDC.IssuerURL != "", OIDCEnabled: cfg.OIDC.IssuerURL != "",
SAMLEnabled: cfg.SAML.IDPMetadataURL != "", SAMLEnabled: cfg.SAML.IDPMetadataURL != "",
} }
authhandler.New(logger, sessionManager, features, rbac).RegisterRoutes(mux) authhandler.New(logger, sessionManager, features, rbac, rbac).RegisterRoutes(mux)
loginhandler.New(logger, oidcProvider, samlProvider, sessionManager, rbac, cfg.PostLoginRedirectURL, cfg.SelectTenantRedirectURL).RegisterRoutes(mux) loginhandler.New(logger, oidcProvider, samlProvider, sessionManager, rbac, cfg.PostLoginRedirectURL, cfg.SelectTenantRedirectURL).RegisterRoutes(mux)
// Credentialed, not plain, CORS: GET /auth/memberships and POST // Credentialed, not plain, CORS: GET /auth/memberships and POST
+58 -2
View File
@@ -13,6 +13,9 @@
// api/authz) and a different credential type (an ingest bearer token // api/authz) and a different credential type (an ingest bearer token
// checked against rbacstore's ingest_credentials table, not a // checked against rbacstore's ingest_credentials table, not a
// session.Manager JWT) -- see ingestCredentialValidator's doc comment. // session.Manager JWT) -- see ingestCredentialValidator's doc comment.
//
// GET /internal/active-tenants is a third sibling, for `search`
// (core/AGPL, Rust) -- see tenantLister's doc comment.
package authhandler package authhandler
import ( import (
@@ -50,21 +53,36 @@ type ingestCredentialValidator interface {
ValidateIngestCredential(ctx context.Context, token string) (tenantID string, err error) ValidateIngestCredential(ctx context.Context, token string) (tenantID string, err error)
} }
// tenantLister is the narrow interface GET /internal/active-tenants
// needs -- *rbacstore.Store is the production implementation (the same
// concrete type ingestCredentials above already wires in, just a
// second narrow interface it happens to also satisfy). Backs
// search/src/tenants.rs's ActiveTenantTracker: search is AGPL core with
// no Postgres access and no enterprise/ import allowed, so its write-
// routing needed a network boundary to learn which tenants are active,
// the same shape ingest/internal/grpcserver.TenantResolver already uses
// against this exact service (see that package's doc comment).
type tenantLister interface {
ListActiveTenantIDs(ctx context.Context) ([]string, error)
}
type Handler struct { type Handler struct {
logger *slog.Logger logger *slog.Logger
manager *session.Manager manager *session.Manager
features Features features Features
ingestCredentials ingestCredentialValidator ingestCredentials ingestCredentialValidator
tenants tenantLister
} }
func New(logger *slog.Logger, manager *session.Manager, features Features, ingestCredentials ingestCredentialValidator) *Handler { func New(logger *slog.Logger, manager *session.Manager, features Features, ingestCredentials ingestCredentialValidator, tenants tenantLister) *Handler {
return &Handler{logger: logger, manager: manager, features: features, ingestCredentials: ingestCredentials} return &Handler{logger: logger, manager: manager, features: features, ingestCredentials: ingestCredentials, tenants: tenants}
} }
func (h *Handler) RegisterRoutes(mux *http.ServeMux) { func (h *Handler) RegisterRoutes(mux *http.ServeMux) {
mux.HandleFunc("POST /internal/authorize", h.handleAuthorize) mux.HandleFunc("POST /internal/authorize", h.handleAuthorize)
mux.HandleFunc("GET /auth/features", h.handleFeatures) mux.HandleFunc("GET /auth/features", h.handleFeatures)
mux.HandleFunc("POST /internal/authorize-ingest", h.handleAuthorizeIngest) mux.HandleFunc("POST /internal/authorize-ingest", h.handleAuthorizeIngest)
mux.HandleFunc("GET /internal/active-tenants", h.handleActiveTenants)
} }
type featuresResponse struct { type featuresResponse struct {
@@ -146,6 +164,44 @@ func (h *Handler) handleAuthorizeIngest(w http.ResponseWriter, r *http.Request)
_ = json.NewEncoder(w).Encode(authorizeIngestResponse{TenantID: tenantID}) _ = json.NewEncoder(w).Encode(authorizeIngestResponse{TenantID: tenantID})
} }
type activeTenantsResponse struct {
TenantIDs []string `json:"tenant_ids"`
}
// handleActiveTenants is search/src/tenants.rs's ActiveTenantTracker's
// server side -- polled periodically, not per-write, to build a local
// allowlist for its write-routing gate (see that module's doc comment).
// Requires a RoleService Bearer credential (session.Manager-issued,
// Role == "service"), not a human session -- server-to-server, the same
// authentication shape /alerting presents to /api, minted via
// `enterprise-auth -mint-service-token search` (the flag is already
// generic over caller name; no change needed there for a new caller).
// Deliberately does NOT accept the tenant-scoped credential a human
// session or an ingest credential would carry: this endpoint answers
// "which tenants exist," a question no single tenant's identity should
// be able to ask on its own.
func (h *Handler) handleActiveTenants(w http.ResponseWriter, r *http.Request) {
token := bearerToken(r.Header.Get("Authorization"))
if token == "" {
http.Error(w, "no credentials presented", http.StatusUnauthorized)
return
}
claims, err := h.manager.Validate(token)
if err != nil || claims.Role != "service" {
http.Error(w, "invalid or expired credentials", http.StatusUnauthorized)
return
}
ids, err := h.tenants.ListActiveTenantIDs(r.Context())
if err != nil {
h.logger.Error("listing active tenant ids", "error", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(activeTenantsResponse{TenantIDs: ids})
}
func bearerToken(header string) string { func bearerToken(header string) string {
const prefix = "Bearer " const prefix = "Bearer "
if !strings.HasPrefix(header, prefix) { if !strings.HasPrefix(header, prefix) {
@@ -36,13 +36,27 @@ type fakeNotFoundError struct{}
func (*fakeNotFoundError) Error() string { return "not found" } func (*fakeNotFoundError) Error() string { return "not found" }
// fakeTenantLister is an in-memory stand-in for *rbacstore.Store's
// ListActiveTenantIDs.
type fakeTenantLister struct {
ids []string
err error
}
func (f *fakeTenantLister) ListActiveTenantIDs(_ context.Context) ([]string, error) {
if f.err != nil {
return nil, f.err
}
return f.ids, nil
}
func testHandler(t *testing.T) (*Handler, *session.Manager) { func testHandler(t *testing.T) (*Handler, *session.Manager) {
t.Helper() t.Helper()
m, err := session.NewManager([]byte("this-is-a-32-byte-test-signing-key!")) m, err := session.NewManager([]byte("this-is-a-32-byte-test-signing-key!"))
if err != nil { if err != nil {
t.Fatalf("session.NewManager: %v", err) t.Fatalf("session.NewManager: %v", err)
} }
return New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, newFakeIngestCredentialValidator()), m return New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, newFakeIngestCredentialValidator(), &fakeTenantLister{}), m
} }
func doAuthorize(t *testing.T, h *Handler, mutate func(*http.Request)) *httptest.ResponseRecorder { func doAuthorize(t *testing.T, h *Handler, mutate func(*http.Request)) *httptest.ResponseRecorder {
@@ -146,7 +160,7 @@ func TestFeaturesReflectsConfiguredMechanisms(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("session.NewManager: %v", err) t.Fatalf("session.NewManager: %v", err)
} }
h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{OIDCEnabled: true, SAMLEnabled: false}, newFakeIngestCredentialValidator()) h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{OIDCEnabled: true, SAMLEnabled: false}, newFakeIngestCredentialValidator(), &fakeTenantLister{})
mux := http.NewServeMux() mux := http.NewServeMux()
h.RegisterRoutes(mux) h.RegisterRoutes(mux)
@@ -219,7 +233,7 @@ func TestAuthorizeIngestResolvesTenant(t *testing.T) {
} }
validator := newFakeIngestCredentialValidator() validator := newFakeIngestCredentialValidator()
validator.tenantByToken["real-token"] = "acme" validator.tenantByToken["real-token"] = "acme"
h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, validator) h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, validator, &fakeTenantLister{})
rec := doAuthorizeIngest(t, h, func(r *http.Request) { rec := doAuthorizeIngest(t, h, func(r *http.Request) {
r.Header.Set("Authorization", "Bearer real-token") r.Header.Set("Authorization", "Bearer real-token")
@@ -273,3 +287,96 @@ func TestAuthorizeIngestRejectsSessionToken(t *testing.T) {
t.Fatalf("status = %d, want 401 (a session token must not validate as an ingest credential)", rec.Code) t.Fatalf("status = %d, want 401 (a session token must not validate as an ingest credential)", rec.Code)
} }
} }
func doActiveTenants(t *testing.T, h *Handler, mutate func(*http.Request)) *httptest.ResponseRecorder {
t.Helper()
mux := http.NewServeMux()
h.RegisterRoutes(mux)
req := httptest.NewRequest(http.MethodGet, "/internal/active-tenants", nil)
if mutate != nil {
mutate(req)
}
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec
}
func TestActiveTenantsViaServiceToken(t *testing.T) {
m, err := session.NewManager([]byte("this-is-a-32-byte-test-signing-key!"))
if err != nil {
t.Fatalf("session.NewManager: %v", err)
}
h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, newFakeIngestCredentialValidator(), &fakeTenantLister{ids: []string{"acme", "globex"}})
token, err := m.IssueServiceToken("search")
if err != nil {
t.Fatalf("IssueServiceToken: %v", err)
}
rec := doActiveTenants(t, h, func(r *http.Request) {
r.Header.Set("Authorization", "Bearer "+token)
})
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
}
var body activeTenantsResponse
if err := json.NewDecoder(rec.Body).Decode(&body); err != nil {
t.Fatalf("decoding response: %v", err)
}
if len(body.TenantIDs) != 2 || body.TenantIDs[0] != "acme" || body.TenantIDs[1] != "globex" {
t.Fatalf("unexpected response: %+v", body)
}
}
func TestActiveTenantsNoCredentialsIsUnauthorized(t *testing.T) {
h, _ := testHandler(t)
rec := doActiveTenants(t, h, nil)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", rec.Code)
}
}
// TestActiveTenantsRejectsHumanSession is the regression test for this
// endpoint's whole reason to distinguish token kinds: a human session
// (even a real, validly-signed one) must not be able to list every
// active tenant in the deployment -- only a RoleService credential can.
func TestActiveTenantsRejectsHumanSession(t *testing.T) {
h, m := testHandler(t)
sessionToken, err := m.IssueUserSession("acme", "u1", "owner")
if err != nil {
t.Fatalf("IssueUserSession: %v", err)
}
rec := doActiveTenants(t, h, func(r *http.Request) {
r.Header.Set("Authorization", "Bearer "+sessionToken)
})
if rec.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401 (a human session must not satisfy the service-only active-tenants endpoint)", rec.Code)
}
}
func TestActiveTenantsInvalidTokenIsUnauthorized(t *testing.T) {
h, _ := testHandler(t)
rec := doActiveTenants(t, h, func(r *http.Request) {
r.Header.Set("Authorization", "Bearer not-a-real-token")
})
if rec.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", rec.Code)
}
}
func TestActiveTenantsStoreErrorIsInternalError(t *testing.T) {
m, err := session.NewManager([]byte("this-is-a-32-byte-test-signing-key!"))
if err != nil {
t.Fatalf("session.NewManager: %v", err)
}
h := New(slog.New(slog.NewTextHandler(io.Discard, nil)), m, Features{}, newFakeIngestCredentialValidator(), &fakeTenantLister{err: errNotFound})
token, err := m.IssueServiceToken("search")
if err != nil {
t.Fatalf("IssueServiceToken: %v", err)
}
rec := doActiveTenants(t, h, func(r *http.Request) {
r.Header.Set("Authorization", "Bearer "+token)
})
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want 500", rec.Code)
}
}
@@ -198,6 +198,34 @@ func (s *Store) TenantIsActive(ctx context.Context, tenantID string) (bool, erro
return t.Status == "active", nil return t.Status == "active", nil
} }
// ListActiveTenantIDs backs enterprise-auth's GET /internal/active-tenants
// -- search (AGPL core) polls this to learn which tenant_ids it may
// safely write-route into their own Tantivy index, since it has no
// Postgres access of its own (see search/src/tenants.rs's doc comment
// for the full design). Deliberately narrower than
// ListProvisionedDataSources (which also requires ClickHouse
// credentials to be present, since chwriter.Registry needs those to
// actually connect): search's write path needs nothing but the id.
func (s *Store) ListActiveTenantIDs(ctx context.Context) ([]string, error) {
rows, err := s.pool.Query(ctx, `SELECT id FROM tenants WHERE status = 'active'`)
if err != nil {
return nil, fmt.Errorf("rbacstore: listing active tenant ids: %w", err)
}
defer rows.Close()
var ids []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("rbacstore: scanning active tenant id: %w", err)
}
ids = append(ids, id)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("rbacstore: iterating active tenant ids: %w", err)
}
return ids, nil
}
// SetTenantStatus is the only way a tenant's status column changes -- // SetTenantStatus is the only way a tenant's status column changes --
// every tenant-resolution path elsewhere must re-check this via // every tenant-resolution path elsewhere must re-check this via
// GetTenant, never cache/assume 'active', per // GetTenant, never cache/assume 'active', per
@@ -470,6 +470,41 @@ func createTestDashboard(t *testing.T, s *Store, tenantID, createdBy string) str
return id return id
} }
func TestListActiveTenantIDsExcludesNonActive(t *testing.T) {
s := testStore(t)
ctx := context.Background()
active := "test-tenant-" + uniqueSuffix()
provisioning := "test-tenant-" + uniqueSuffix()
for _, id := range []string{active, provisioning} {
if _, err := s.CreateTenant(ctx, id, id); err != nil {
t.Fatalf("CreateTenant %s: %v", id, err)
}
}
if err := s.SetTenantStatus(ctx, active, "active"); err != nil {
t.Fatalf("SetTenantStatus active: %v", err)
}
// provisioning stays in 'provisioning' (CreateTenant's default).
ids, err := s.ListActiveTenantIDs(ctx)
if err != nil {
t.Fatalf("ListActiveTenantIDs: %v", err)
}
foundActive := false
for _, id := range ids {
if id == provisioning {
t.Fatalf("non-active tenant %q leaked into ListActiveTenantIDs", provisioning)
}
if id == active {
foundActive = true
}
}
if !foundActive {
t.Fatal("expected the active tenant to be in the list")
}
}
func TestSetDashboardPermissionThenGet(t *testing.T) { func TestSetDashboardPermissionThenGet(t *testing.T) {
s := testStore(t) s := testStore(t)
ctx := context.Background() ctx := context.Background()
+353
View File
@@ -308,6 +308,17 @@ dependencies = [
"subtle", "subtle",
] ]
[[package]]
name = "displaydoc"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.3",
]
[[package]] [[package]]
name = "downcast-rs" name = "downcast-rs"
version = "1.2.1" version = "1.2.1"
@@ -382,6 +393,15 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "form_urlencoded"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf"
dependencies = [
"percent-encoding",
]
[[package]] [[package]]
name = "fs4" name = "fs4"
version = "0.8.4" version = "0.8.4"
@@ -667,13 +687,16 @@ version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [ dependencies = [
"base64",
"bytes", "bytes",
"futures-channel", "futures-channel",
"futures-util", "futures-util",
"http", "http",
"http-body", "http-body",
"hyper", "hyper",
"ipnet",
"libc", "libc",
"percent-encoding",
"pin-project-lite", "pin-project-lite",
"socket2 0.6.5", "socket2 0.6.5",
"tokio", "tokio",
@@ -681,6 +704,110 @@ dependencies = [
"tracing", "tracing",
] ]
[[package]]
name = "icu_collections"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513"
dependencies = [
"displaydoc",
"potential_utf",
"utf8_iter",
"yoke",
"zerofrom",
"zerovec",
]
[[package]]
name = "icu_locale_core"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb"
dependencies = [
"displaydoc",
"litemap",
"tinystr",
"writeable",
"zerovec",
]
[[package]]
name = "icu_normalizer"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f"
dependencies = [
"icu_collections",
"icu_normalizer_data",
"icu_properties",
"icu_provider",
"smallvec",
"zerovec",
]
[[package]]
name = "icu_normalizer_data"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0"
[[package]]
name = "icu_properties"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148"
dependencies = [
"displaydoc",
"icu_collections",
"icu_locale_core",
"icu_properties_data",
"icu_provider",
"zerotrie",
"zerovec",
]
[[package]]
name = "icu_properties_data"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa"
[[package]]
name = "icu_provider"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92a7ed671a6aad807a8651a2e1782a6598fda9ce5185dd8158549e95a91c6428"
dependencies = [
"displaydoc",
"icu_locale_core",
"writeable",
"yoke",
"zerofrom",
"zerotrie",
"zerovec",
]
[[package]]
name = "idna"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de"
dependencies = [
"idna_adapter",
"smallvec",
"utf8_iter",
]
[[package]]
name = "idna_adapter"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714"
dependencies = [
"icu_normalizer",
"icu_properties",
]
[[package]] [[package]]
name = "indexmap" name = "indexmap"
version = "1.9.3" version = "1.9.3"
@@ -719,6 +846,12 @@ version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "14c00403deb17c3221a1fe4fb571b9ed0370b3dcd116553c77fa294a3d918699" checksum = "14c00403deb17c3221a1fe4fb571b9ed0370b3dcd116553c77fa294a3d918699"
[[package]]
name = "ipnet"
version = "2.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78"
[[package]] [[package]]
name = "itertools" name = "itertools"
version = "0.12.1" version = "0.12.1"
@@ -800,6 +933,12 @@ version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "litemap"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae"
[[package]] [[package]]
name = "lock_api" name = "lock_api"
version = "0.4.14" version = "0.4.14"
@@ -1080,6 +1219,15 @@ version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "potential_utf"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661"
dependencies = [
"zerovec",
]
[[package]] [[package]]
name = "powerfmt" name = "powerfmt"
version = "0.2.0" version = "0.2.0"
@@ -1279,6 +1427,38 @@ version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "reqwest"
version = "0.12.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"bytes",
"futures-core",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"js-sys",
"log",
"percent-encoding",
"pin-project-lite",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tower 0.5.3",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
]
[[package]] [[package]]
name = "rsasl" name = "rsasl"
version = "2.3.1" version = "2.3.1"
@@ -1377,6 +1557,12 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "ryu"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]] [[package]]
name = "scopeguard" name = "scopeguard"
version = "1.2.0" version = "1.2.0"
@@ -1395,6 +1581,7 @@ version = "0.1.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"prost", "prost",
"reqwest",
"rskafka", "rskafka",
"serde", "serde",
"serde_json", "serde_json",
@@ -1450,6 +1637,18 @@ dependencies = [
"zmij", "zmij",
] ]
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
dependencies = [
"form_urlencoded",
"itoa",
"ryu",
"serde",
]
[[package]] [[package]]
name = "sha2" name = "sha2"
version = "0.10.9" version = "0.10.9"
@@ -1589,6 +1788,20 @@ name = "sync_wrapper"
version = "1.0.2" version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
dependencies = [
"futures-core",
]
[[package]]
name = "synstructure"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]] [[package]]
name = "tantivy" name = "tantivy"
@@ -1823,6 +2036,16 @@ dependencies = [
"time-core", "time-core",
] ]
[[package]]
name = "tinystr"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643"
dependencies = [
"displaydoc",
"zerovec",
]
[[package]] [[package]]
name = "tinyvec" name = "tinyvec"
version = "1.12.0" version = "1.12.0"
@@ -1964,10 +2187,29 @@ dependencies = [
"futures-util", "futures-util",
"pin-project-lite", "pin-project-lite",
"sync_wrapper", "sync_wrapper",
"tokio",
"tower-layer", "tower-layer",
"tower-service", "tower-service",
] ]
[[package]]
name = "tower-http"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
dependencies = [
"bitflags",
"bytes",
"futures-util",
"http",
"http-body",
"pin-project-lite",
"tower 0.5.3",
"tower-layer",
"tower-service",
"url",
]
[[package]] [[package]]
name = "tower-layer" name = "tower-layer"
version = "0.3.3" version = "0.3.3"
@@ -2080,12 +2322,30 @@ version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d"
[[package]]
name = "url"
version = "2.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed"
dependencies = [
"form_urlencoded",
"idna",
"percent-encoding",
"serde",
]
[[package]] [[package]]
name = "utf8-ranges" name = "utf8-ranges"
version = "1.0.5" version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba" checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba"
[[package]]
name = "utf8_iter"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]] [[package]]
name = "uuid" name = "uuid"
version = "1.24.0" version = "1.24.0"
@@ -2138,6 +2398,16 @@ dependencies = [
"wasm-bindgen-shared", "wasm-bindgen-shared",
] ]
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.77"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]] [[package]]
name = "wasm-bindgen-macro" name = "wasm-bindgen-macro"
version = "0.2.127" version = "0.2.127"
@@ -2299,6 +2569,35 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "writeable"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
[[package]]
name = "yoke"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5"
dependencies = [
"stable_deref_trait",
"yoke-derive",
"zerofrom",
]
[[package]]
name = "yoke-derive"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"synstructure",
]
[[package]] [[package]]
name = "zerocopy" name = "zerocopy"
version = "0.8.56" version = "0.8.56"
@@ -2319,6 +2618,60 @@ dependencies = [
"syn 2.0.119", "syn 2.0.119",
] ]
[[package]]
name = "zerofrom"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272"
dependencies = [
"zerofrom-derive",
]
[[package]]
name = "zerofrom-derive"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"synstructure",
]
[[package]]
name = "zerotrie"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f"
dependencies = [
"displaydoc",
"yoke",
"zerofrom",
]
[[package]]
name = "zerovec"
version = "0.11.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94b5c6b5976d66c1d703c4fd17d3f5e43c8cedaacf604961b171adc7130896d8"
dependencies = [
"yoke",
"zerofrom",
"zerovec-derive",
]
[[package]]
name = "zerovec-derive"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47402523226a02bfe5230160dc3ccc089aa6f6f19e7fcbb4e6f824bbb1b4aa62"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.3",
]
[[package]] [[package]]
name = "zmij" name = "zmij"
version = "1.0.23" version = "1.0.23"
+13
View File
@@ -20,6 +20,14 @@ rskafka = "0.6"
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
# tenants.rs's only outbound HTTP call (GET enterprise-auth's
# /internal/active-tenants) -- default-features = false drops TLS
# support entirely, matching every other internal service-to-service
# call in this repo (plain HTTP, network placement is the trust
# boundary, not TLS -- see ingest/internal/grpcserver.
# HTTPTenantResolver's identical posture against the same service).
reqwest = { version = "0.12", default-features = false, features = ["json"] }
anyhow = "1" anyhow = "1"
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
@@ -31,3 +39,8 @@ tonic-build = "0.12"
# Already in the dependency graph transitively (tantivy/tonic-build both # Already in the dependency graph transitively (tantivy/tonic-build both
# pull it in); promoted to a direct dev-dependency for test use. # pull it in); promoted to a direct dev-dependency for test use.
tempfile = "3" tempfile = "3"
# tenants.rs's tests spin up a tiny real TCP server to exercise reqwest
# against, rather than pulling in a mocking crate -- needs tokio
# features the main binary doesn't (edition 2021's resolver keeps these
# out of the release build, test-only).
tokio = { version = "1", features = ["net", "io-util"] }
+29 -14
View File
@@ -62,19 +62,28 @@ binary" needed. The periodic Tantivy commit (`COMMIT_INTERVAL_MS`) now
commits every tenant index that's actually seen a write, plus the commits every tenant index that's actually seen a write, plus the
default index, via `IndexRegistry::commit_all`, not just one index. default index, via `IndexRegistry::commit_all`, not just one index.
**One residual gap, disclosed rather than fixed here**: unlike the read **The active-tenant gap is closed too, the same way the read side closes
side (gated by `enterprise/internal/searchclient`'s `TenantChecker`, it**: `src/tenants.rs`'s `ActiveTenantTracker` polls a new
which refuses to search a tenant that isn't `active` in `rbacstore`) and `GET /internal/active-tenants` endpoint on `enterprise-auth` (this
unlike ClickHouse's write side (`chwriter.Registry`, built from an process has no Postgres access, so unlike `enterprise/internal/
active-tenants-only snapshot at startup, so an unrecognized tenant has searchclient`'s `TenantChecker` — a direct `rbacstore.TenantIsActive`
no writer at all), this consumer's `registry.resolve()` call has no call, since that code runs in `enterprise/` — this needed a network
active-tenant gate — this process has no Postgres access to check call instead), and `consumer.rs` refuses (logs and skips, never falls
against, the same reason `IndexRegistry` couldn't do the mid-provisioning back to the default or another tenant's index) any tagged record whose
check itself before `TenantChecker` was added for the read side. A `tenant_id` isn't in the polled allowlist. Off unless
still-valid (not yet revoked) ingest credential for a tenant that's no `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are both set (see
longer active can cause an index directory to be created for it here. Configuration below) — when they aren't, write-routing behaves exactly
See `src/registry.rs`'s doc comment on `resolve` for the full writeup, as it did before this tracker existed, trusting any syntactically-valid
including why closing it fully isn't scoped yet. `tenant_id`. When they are, startup blocks on the first fetch succeeding
(fail-closed cold start — see `tenants.rs`'s doc comment for why a
partial/degraded startup isn't the safer choice, and for the
last-known-good behavior periodic refresh failures fall back to).
Verified with real HTTP round trips against a hand-rolled TCP test
server (no mocking crate needed for one endpoint) — the Bearer token
actually sent, the initial-fetch-fails-closed path, and an unreachable
server also failing closed. See `src/registry.rs`'s doc comment on
`resolve` for how the mechanism (index lifecycle) and policy (who gets
gated) responsibilities are split.
## Offset tracking: why this isn't a Kafka consumer group ## Offset tracking: why this isn't a Kafka consumer group
@@ -118,6 +127,8 @@ Environment variables (see `src/config.rs`):
| `TENANTS_INDEX_PATH` | `/var/lib/sentry-search/tenants` | Per-tenant index directories live under here, one subdirectory per tenant_id (Phase 4) | | `TENANTS_INDEX_PATH` | `/var/lib/sentry-search/tenants` | Per-tenant index directories live under here, one subdirectory per tenant_id (Phase 4) |
| `OFFSETS_PATH` | `/var/lib/sentry-search/offsets.json` | Offset tracking file | | `OFFSETS_PATH` | `/var/lib/sentry-search/offsets.json` | Offset tracking file |
| `COMMIT_INTERVAL_MS` | `2000` | How often buffered writes become searchable | | `COMMIT_INTERVAL_MS` | `2000` | How often buffered writes become searchable |
| `ENTERPRISE_AUTH_URL` | (empty) | Enables `tenants::ActiveTenantTracker` -- empty means write-routing has no active-tenant gate, same as every deployment before Phase 4. Must be set together with `ENTERPRISE_AUTH_SERVICE_TOKEN` below, or `Config::load` fails |
| `ENTERPRISE_AUTH_SERVICE_TOKEN` | (empty) | RoleService Bearer credential for `GET /internal/active-tenants`, minted via `enterprise-auth -mint-service-token search` |
## Building & testing ## Building & testing
@@ -147,7 +158,11 @@ pure `tenant_id_from_headers` header-extraction helper it uses is
factored out and unit-tested the same way `ingest/consumer`'s Go factored out and unit-tested the same way `ingest/consumer`'s Go
equivalent is, including a guard test equivalent is, including a guard test
(`test_tenant_id_header_key_matches_go`) against the header-key literal (`test_tenant_id_header_key_matches_go`) against the header-key literal
drifting from the Go side's. drifting from the Go side's. `tenants.rs`'s tests genuinely exercise
`reqwest` against a real (if hand-rolled, dependency-free) TCP server —
the actual `Authorization: Bearer` header construction, JSON response
parsing, and both fail-closed paths (a rejected first fetch, an
unreachable server), not a fake HTTP client substituted in.
```sh ```sh
# from the repo root, not search/ # from the repo root, not search/
+39 -4
View File
@@ -13,14 +13,35 @@ pub struct Config {
pub commit_interval: Duration, pub commit_interval: Duration,
/// Phase 4: per-tenant index directories live under here, one /// Phase 4: per-tenant index directories live under here, one
/// subdirectory per tenant_id, opened on demand by /// subdirectory per tenant_id, opened on demand by
/// registry::IndexRegistry -- distinct from `index_path` above, /// registry::IndexRegistry for both the read side (SearchRequest.
/// which stays the single shared index every ingest-written record /// tenant_id) and the write side (consumer.rs, routing on each
/// lands in regardless of tenant (see registry.rs's doc comment and /// record's tenant_id Kafka header) -- `index_path` above stays the
/// /docs/security/threat-model.md's ingest-tenancy caveat). Default /// single shared index an untagged record, or any deployment that
/// never turned on ingest's TenantResolver, still lands in. Default
/// matches the path convention deploy/operator's Tenant controller /// matches the path convention deploy/operator's Tenant controller
/// and enterprise/internal/rbacstore's seeded default data source /// and enterprise/internal/rbacstore's seeded default data source
/// already assume (`/var/lib/sentry-search/tenants/<id>`). /// already assume (`/var/lib/sentry-search/tenants/<id>`).
pub tenants_index_path: PathBuf, pub tenants_index_path: PathBuf,
/// Base URL of enterprise-auth's HTTP API, e.g.
/// `http://enterprise-auth:8082` -- same env var name and "empty
/// means off" shape as ingest/internal/config's own
/// ENTERPRISE_AUTH_URL. When set (together with
/// ENTERPRISE_AUTH_SERVICE_TOKEN below), tenants::ActiveTenantTracker
/// gates consumer.rs's write-routing on a polled active-tenant
/// allowlist -- see that module's doc comment for why this needed a
/// network call instead of direct Postgres access. When unset,
/// write-routing behaves exactly as it did before that tracker
/// existed: any syntactically-valid tenant_id is trusted.
pub enterprise_auth_url: Option<String>,
/// RoleService Bearer credential this process presents to
/// enterprise-auth's GET /internal/active-tenants -- minted via the
/// existing `enterprise-auth -mint-service-token search` (the flag
/// is already generic over caller name, no backend change needed to
/// mint one for a new caller). Required together with
/// enterprise_auth_url above; Config::load fails if exactly one of
/// the two is set, rather than silently running with the tracker
/// half-configured.
pub enterprise_auth_service_token: Option<String>,
} }
impl Config { impl Config {
@@ -29,6 +50,14 @@ impl Config {
.parse() .parse()
.context("COMMIT_INTERVAL_MS must be a number")?; .context("COMMIT_INTERVAL_MS must be a number")?;
let enterprise_auth_url = getenv_opt("ENTERPRISE_AUTH_URL");
let enterprise_auth_service_token = getenv_opt("ENTERPRISE_AUTH_SERVICE_TOKEN");
if enterprise_auth_url.is_some() != enterprise_auth_service_token.is_some() {
anyhow::bail!(
"ENTERPRISE_AUTH_URL and ENTERPRISE_AUTH_SERVICE_TOKEN must be set together, or neither -- got exactly one"
);
}
Ok(Self { Ok(Self {
// Rust's SocketAddr parser needs a full address, unlike Go's // Rust's SocketAddr parser needs a full address, unlike Go's
// net package (ingest/api's ":PORT" convention won't parse // net package (ingest/api's ":PORT" convention won't parse
@@ -49,6 +78,8 @@ impl Config {
"TENANTS_INDEX_PATH", "TENANTS_INDEX_PATH",
"/var/lib/sentry-search/tenants", "/var/lib/sentry-search/tenants",
)), )),
enterprise_auth_url,
enterprise_auth_service_token,
}) })
} }
} }
@@ -56,3 +87,7 @@ impl Config {
fn getenv(key: &str, fallback: &str) -> String { fn getenv(key: &str, fallback: &str) -> String {
std::env::var(key).unwrap_or_else(|_| fallback.to_string()) std::env::var(key).unwrap_or_else(|_| fallback.to_string())
} }
fn getenv_opt(key: &str) -> Option<String> {
std::env::var(key).ok().filter(|v| !v.is_empty())
}
+32 -8
View File
@@ -10,6 +10,7 @@ use crate::config::Config;
use crate::logsv1; use crate::logsv1;
use crate::offsets::OffsetStore; use crate::offsets::OffsetStore;
use crate::registry::IndexRegistry; use crate::registry::IndexRegistry;
use crate::tenants::ActiveTenantTracker;
/// Kafka message header a resolved tenant ID rides in, attached by /// Kafka message header a resolved tenant ID rides in, attached by
/// ingest's gRPC front end. Mirrors `ingest/internal/grpcserver. /// ingest's gRPC front end. Mirrors `ingest/internal/grpcserver.
@@ -30,7 +31,12 @@ const TENANT_ID_HEADER_KEY: &str = "tenant_id";
/// discovered dynamically, since it has to match what /// discovered dynamically, since it has to match what
/// /transport/provision-topics.sh actually created anyway (documented /// /transport/provision-topics.sh actually created anyway (documented
/// cross-component contract, same as the topic name already is). /// cross-component contract, same as the topic name already is).
pub async fn run(cfg: Arc<Config>, registry: Arc<IndexRegistry>, partition_count: i32) -> Result<()> { pub async fn run(
cfg: Arc<Config>,
registry: Arc<IndexRegistry>,
partition_count: i32,
active_tenants: Option<Arc<ActiveTenantTracker>>,
) -> Result<()> {
let client = ClientBuilder::new(cfg.redpanda_brokers.clone()) let client = ClientBuilder::new(cfg.redpanda_brokers.clone())
.build() .build()
.await .await
@@ -65,9 +71,10 @@ pub async fn run(cfg: Arc<Config>, registry: Arc<IndexRegistry>, partition_count
let client = Arc::clone(&client); let client = Arc::clone(&client);
let registry = Arc::clone(&registry); let registry = Arc::clone(&registry);
let offsets = Arc::clone(&offsets); let offsets = Arc::clone(&offsets);
let active_tenants = active_tenants.clone();
let topic = cfg.redpanda_topic.clone(); let topic = cfg.redpanda_topic.clone();
handles.push(tokio::spawn(async move { handles.push(tokio::spawn(async move {
consume_partition(client, topic, partition, start_offset, registry, offsets).await consume_partition(client, topic, partition, start_offset, registry, offsets, active_tenants).await
})); }));
} }
@@ -100,6 +107,7 @@ async fn consume_partition(
start_offset: i64, start_offset: i64,
registry: Arc<IndexRegistry>, registry: Arc<IndexRegistry>,
offsets: Arc<Mutex<OffsetStore>>, offsets: Arc<Mutex<OffsetStore>>,
active_tenants: Option<Arc<ActiveTenantTracker>>,
) -> Result<()> { ) -> Result<()> {
let partition_client = client let partition_client = client
.partition_client(topic.clone(), partition, UnknownTopicHandling::Error) .partition_client(topic.clone(), partition, UnknownTopicHandling::Error)
@@ -143,17 +151,33 @@ async fn consume_partition(
} }
let tenant_id = tenant_id_from_headers(&record_and_offset.record.headers); let tenant_id = tenant_id_from_headers(&record_and_offset.record.headers);
// Fail-closed active-tenant gate (see tenants.rs's doc
// comment) -- only applies to tagged records and only when
// a tracker is actually configured, matching resolve()'s
// own "empty tenant_id always means the default index"
// rule and this codebase's "off unless configured" default
// everywhere else. This is the check registry.rs's `resolve`
// doc comment used to name as missing entirely.
if !tenant_id.is_empty() {
if let Some(tracker) = &active_tenants {
if !tracker.is_active(&tenant_id).await {
tracing::warn!(record_id = %rec.record_id, tenant_id, "skipping record: tenant is not active");
continue;
}
}
}
let index = match registry.resolve(&tenant_id).await { let index = match registry.resolve(&tenant_id).await {
Ok(index) => index, Ok(index) => index,
Err(e) => { Err(e) => {
// Shouldn't normally happen -- ingest/grpcserver only // Shouldn't normally happen -- ingest/grpcserver only
// ever attaches a tenant_id it validated against a // ever attaches a tenant_id it validated against a
// real credential -- but an unsafe/malformed // real credential, and the active-tenant gate above
// tenant_id is a hard skip, never a silent fall-back // already refused anything not currently active when
// to the default or any other tenant's index. See // configured -- but an unsafe/malformed tenant_id is
// registry.rs's doc comment for the one residual gap // a hard skip regardless, never a silent fall-back to
// this consumer doesn't close (no active-tenant // the default or any other tenant's index.
// check, since this process has no Postgres access).
tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "skipping record: failed to resolve tenant index"); tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "skipping record: failed to resolve tenant index");
continue; continue;
} }
+21 -1
View File
@@ -4,6 +4,7 @@ mod grpc;
mod index; mod index;
mod offsets; mod offsets;
mod registry; mod registry;
mod tenants;
pub mod logsv1 { pub mod logsv1 {
tonic::include_proto!("sentry.logs.v1"); tonic::include_proto!("sentry.logs.v1");
@@ -32,6 +33,24 @@ async fn main() -> Result<()> {
let cfg = Arc::new(Config::load().context("loading config")?); let cfg = Arc::new(Config::load().context("loading config")?);
// Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are
// both set (Config::load already rejects exactly one being set).
// Blocks startup entirely on failure, same "fail hard, let the
// orchestrator restart" posture enterprise-ingest's main.go already
// uses when its own required startup fetch (rbacstore.
// ListProvisionedDataSources) fails -- see tenants.rs's doc comment
// for why a partial/degraded startup isn't the safer choice here.
let active_tenants = match (&cfg.enterprise_auth_url, &cfg.enterprise_auth_service_token) {
(Some(url), Some(token)) => {
tracing::info!(url, "active-tenant write-routing gate enabled");
Some(tenants::ActiveTenantTracker::start(url, token).await?)
}
_ => {
tracing::info!("ENTERPRISE_AUTH_URL not set -- write-routing has no active-tenant gate");
None
}
};
let index = Arc::new( let index = Arc::new(
SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?, SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?,
); );
@@ -48,8 +67,9 @@ async fn main() -> Result<()> {
let consumer_cfg = Arc::clone(&cfg); let consumer_cfg = Arc::clone(&cfg);
let consumer_registry = Arc::clone(&registry); let consumer_registry = Arc::clone(&registry);
let consumer_active_tenants = active_tenants.clone();
let consumer_handle = tokio::spawn(async move { let consumer_handle = tokio::spawn(async move {
if let Err(e) = consumer::run(consumer_cfg, consumer_registry, partition_count).await { if let Err(e) = consumer::run(consumer_cfg, consumer_registry, partition_count, consumer_active_tenants).await {
tracing::error!(error = %e, "redpanda consumer exited with error"); tracing::error!(error = %e, "redpanda consumer exited with error");
} }
}); });
+21 -23
View File
@@ -22,29 +22,27 @@ use crate::index::SearchIndex;
/// path every Phase 0-3 deployment, and every untagged record, still /// path every Phase 0-3 deployment, and every untagged record, still
/// uses. /// uses.
/// ///
/// **Known residual gap, disclosed rather than silently accepted**: /// `resolve` itself still has no active-tenant gate of its own -- it
/// unlike the read side (gated by `enterprise/internal/searchclient`'s /// will happily open-or-create an index for any syntactically-valid
/// `TenantChecker`, which refuses to even issue a search for a tenant /// `tenant_id`, active or not. That's deliberate: this struct's job is
/// that isn't `active` in `rbacstore`) and unlike ClickHouse's write /// managing index lifecycles, not policy, the same separation
/// side (`enterprise/internal/chwriter.Registry`, built once at startup /// `clickhousewriter.Writer` (mechanism) vs. `chwriter.Registry`
/// from `rbacstore.ListProvisionedDataSources` -- `active` tenants /// (policy: which tenants get a writer at all) draws on the ClickHouse
/// only, so an unrecognized `tenant_id` has no writer and the whole /// side. The gate lives one layer up, at each caller:
/// batch is refused), this registry's `resolve` has no equivalent gate /// `enterprise/internal/searchclient.TenantChecker` for the read side
/// on the write path: `consumer.rs` calls it directly, with no Postgres /// (refuses to even issue a search for a tenant that isn't `active` in
/// access to check tenant status against, the same reason this /// `rbacstore`, via a direct Postgres-backed query since that code runs
/// module's doc comment used to give for the old read-side gap /// in `enterprise/`), and `consumer.rs`'s `tenants::ActiveTenantTracker`
/// `TenantChecker` was built to close. A syntactically-valid `tenant_id` /// for the write side (a polled allowlist fetched from a new
/// on an ingest credential that's still valid but should have been /// `enterprise-auth` endpoint over HTTP -- `search` is AGPL core with no
/// revoked (deprovisioning does not yet revoke `ingest_credentials` /// Postgres access and no `enterprise/` import allowed, so it needed a
/// rows -- see `/CLAUDE.md`'s Phase 4 non-goals) can therefore cause an /// network boundary instead of an import one, the same shape
/// index directory to be silently created here for a tenant that isn't /// `ingest/internal/grpcserver.TenantResolver` already uses against the
/// really active. The blast radius is narrow -- an orphan, isolated, /// same service). Both gates are optional at this layer -- `resolve`
/// empty-except-for-that-tenant's-own-traffic index directory, not /// itself works identically whether or not either caller happens to
/// cross-tenant data exposure, and only reachable with a real signed /// gate it -- so a future caller that forgets to gate would silently
/// ingest credential, not by an arbitrary caller -- but it is real, not /// reopen this exact class of gap; see `consumer.rs`'s call site for
/// hypothetical. Closing it fully would mean giving `search` (AGPL /// the write side's enforcement.
/// core, no `enterprise/` import allowed) some way to learn which
/// tenants are actually active; not designed yet.
pub struct IndexRegistry { pub struct IndexRegistry {
default_index: Arc<SearchIndex>, default_index: Arc<SearchIndex>,
tenants_root: PathBuf, tenants_root: PathBuf,
+223
View File
@@ -0,0 +1,223 @@
use anyhow::{Context, Result};
use std::collections::HashSet;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::RwLock;
/// How often the tracker re-fetches the active-tenant list after its
/// first successful fetch. Not configurable -- no deployment has needed
/// to tune this yet, and a hardcoded value keeps Config's surface
/// smaller; revisit if that changes.
const REFRESH_INTERVAL: Duration = Duration::from_secs(60);
/// Tracks which tenant_ids are currently `active` in enterprise-auth's
/// `tenants` table, polled from a new `GET /internal/active-tenants`
/// endpoint -- this closes the one gap registry.rs's `resolve` doc
/// comment used to name: `search` (AGPL core) has no Postgres access,
/// so unlike `chwriter.Registry` (an active-tenants-only snapshot built
/// from `rbacstore.ListProvisionedDataSources` at `enterprise-ingest`
/// startup) or the read side (gated by `enterprise/internal/
/// searchclient.TenantChecker`, backed by `rbacstore.TenantIsActive`
/// directly), `consumer.rs`'s write-routing had no allowlist at all --
/// any syntactically-valid `tenant_id` on a still-valid-but-should-
/// have-been-revoked ingest credential could get an index directory
/// created for it.
///
/// Network boundary, not import boundary -- same shape
/// `ingest/internal/grpcserver`'s `TenantResolver` already uses against
/// this exact service, just Rust calling Go instead of Go calling Go,
/// and authenticated the same way `/alerting` authenticates to `/api`:
/// a long-lived RoleService Bearer credential
/// (`enterprise-auth -mint-service-token search`), not a tenant-scoped
/// one -- this tracker proves "I am the search service," never "I may
/// act as tenant X."
///
/// Off unless configured: only constructed when both
/// `ENTERPRISE_AUTH_URL` and `ENTERPRISE_AUTH_SERVICE_TOKEN` are set
/// (see config.rs). When they aren't, `consumer.rs` holds `None` and
/// skips the gate entirely -- every tagged write is routed exactly as
/// it was before this tracker existed, the same "off unless configured"
/// default every other optional integration point in this codebase
/// uses.
pub struct ActiveTenantTracker {
tenants: RwLock<HashSet<String>>,
}
impl ActiveTenantTracker {
/// Blocks until the first fetch succeeds. A cold start with
/// enterprise-auth unreachable must not silently accept every
/// tenant_id it sees -- that's the exact gap this tracker exists to
/// close -- so there is deliberately no empty-set-and-keep-going
/// fallback here; callers should refuse to start the write-routing
/// consumer at all if this returns an error. Once constructed,
/// periodic refreshes are best-effort: a transient failure logs and
/// keeps serving the last-known-good set rather than clearing it
/// (see the spawned task below) -- only the very first fetch is
/// fail-closed-to-refusing-startup.
pub async fn start(base_url: &str, service_token: &str) -> Result<Arc<Self>> {
let client = reqwest::Client::new();
let initial = fetch_active_tenants(&client, base_url, service_token)
.await
.context("fetching initial active-tenant list from enterprise-auth")?;
tracing::info!(count = initial.len(), "loaded initial active-tenant list");
let tracker = Arc::new(Self {
tenants: RwLock::new(initial),
});
let refresh_tracker = Arc::clone(&tracker);
let base_url = base_url.to_string();
let service_token = service_token.to_string();
tokio::spawn(async move {
let mut ticker = tokio::time::interval(REFRESH_INTERVAL);
ticker.tick().await; // fires immediately -- start() already fetched once, skip it
loop {
ticker.tick().await;
match fetch_active_tenants(&client, &base_url, &service_token).await {
Ok(fresh) => {
let count = fresh.len();
*refresh_tracker.tenants.write().await = fresh;
tracing::debug!(count, "refreshed active-tenant list");
}
Err(e) => {
// No staleness ceiling: a prolonged enterprise-auth
// outage means the allowlist just doesn't grow or
// shrink until connectivity resumes, disclosed here
// rather than degrading further (e.g. clearing the
// set, which would stop every tenant's indexing on
// one control-plane blip -- a worse blast radius
// than staleness).
tracing::error!(error = %e, "failed to refresh active-tenant list, keeping last-known-good set");
}
}
}
});
Ok(tracker)
}
pub async fn is_active(&self, tenant_id: &str) -> bool {
self.tenants.read().await.contains(tenant_id)
}
}
#[derive(serde::Deserialize)]
struct ActiveTenantsResponse {
tenant_ids: Vec<String>,
}
async fn fetch_active_tenants(
client: &reqwest::Client,
base_url: &str,
service_token: &str,
) -> Result<HashSet<String>> {
let resp = client
.get(format!("{base_url}/internal/active-tenants"))
.bearer_auth(service_token)
.send()
.await
.context("sending request")?
.error_for_status()
.context("non-2xx response")?
.json::<ActiveTenantsResponse>()
.await
.context("parsing response body")?;
Ok(resp.tenant_ids.into_iter().collect())
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
/// Minimal hand-rolled HTTP/1.1 server -- one dependency-free helper
/// rather than pulling in a mocking crate for the one endpoint this
/// module ever calls. Reads one request, hands it (as raw bytes) to
/// `respond`, writes back exactly what `respond` returns, then
/// closes -- enough to exercise real reqwest request construction
/// (the Bearer header, the URL path) and real response parsing, not
/// a fake client substituted in.
async fn spawn_fake_server(
respond: impl Fn(&str) -> String + Send + Sync + 'static,
) -> String {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let (mut stream, _) = match listener.accept().await {
Ok(v) => v,
Err(_) => return,
};
let mut buf = vec![0u8; 8192];
let n = stream.read(&mut buf).await.unwrap_or(0);
let request = String::from_utf8_lossy(&buf[..n]).to_string();
let response = respond(&request);
let _ = stream.write_all(response.as_bytes()).await;
}
});
format!("http://{addr}")
}
fn json_response(status_line: &str, body: &str) -> String {
format!(
"{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
)
}
#[tokio::test]
async fn start_fetches_and_serves_the_initial_list() {
let base_url = spawn_fake_server(|_req| {
json_response("HTTP/1.1 200 OK", r#"{"tenant_ids":["acme","globex"]}"#)
})
.await;
let tracker = ActiveTenantTracker::start(&base_url, "test-token")
.await
.expect("start should succeed against a healthy fake server");
assert!(tracker.is_active("acme").await);
assert!(tracker.is_active("globex").await);
assert!(!tracker.is_active("initech").await);
}
#[tokio::test]
async fn start_sends_the_bearer_token() {
let base_url = spawn_fake_server(|req| {
if req.contains("authorization: Bearer secret-token") {
json_response("HTTP/1.1 200 OK", r#"{"tenant_ids":["acme"]}"#)
} else {
json_response("HTTP/1.1 401 Unauthorized", r#"{"error":"no credentials presented"}"#)
}
})
.await;
let tracker = ActiveTenantTracker::start(&base_url, "secret-token")
.await
.expect("start should succeed once the fake server sees the right token");
assert!(tracker.is_active("acme").await);
}
#[tokio::test]
async fn start_fails_closed_when_the_first_fetch_fails() {
let base_url = spawn_fake_server(|_req| {
json_response("HTTP/1.1 401 Unauthorized", r#"{"error":"invalid or expired credentials"}"#)
})
.await;
let result = ActiveTenantTracker::start(&base_url, "wrong-token").await;
assert!(
result.is_err(),
"expected start() to fail (not silently start with an empty/permissive allowlist) when the first fetch fails"
);
}
#[tokio::test]
async fn start_fails_closed_when_the_server_is_unreachable() {
// Port 1 is (almost certainly) not listening -- connection refused,
// not a slow timeout, so this test stays fast.
let result = ActiveTenantTracker::start("http://127.0.0.1:1", "any-token").await;
assert!(result.is_err());
}
}