Phase 4: real Tantivy per-tenant isolation (search/src/registry.rs, enterprise/internal/searchclient)

Closes the last named "isolation mechanism" gap: search.proto gains a
tenant_id field on SearchRequest; search/src/registry.rs's IndexRegistry
resolves it to an on-demand-opened, per-tenant Tantivy index (empty
tenant_id keeps today's single default index, so this is purely
additive); enterprise/internal/searchclient sets that field from the
authenticated request identity in ctx, mirroring chrunner's exact
fail-closed "never a parameter" shape. Wired into enterprise-api in
place of the shared api/searchclient.

Unlike the ClickHouse pieces from the previous two commits, this one is
genuinely verified end to end in this environment: Tantivy is an
embedded library, not a networked service, so both the Rust index
registry (cargo test, cargo clippy --all-targets -- -D warnings, both
clean) and the Go client (a real in-process gRPC server) could actually
run. registry.rs's tenant_index_is_isolated_from_default_and_other_tenants
seeds three real indices with the same term and confirms a tenant-scoped
search returns only that tenant's document -- item 3 of the isolation
design doc's verification plan, closed for real, not just written.

With both ClickHouse and Tantivy isolation now built, the single largest
remaining gap is no longer a missing mechanism: it's that nothing forces
or flags whether a deployment actually runs enterprise-api instead of
plain api, and that ingest itself has no tenant concept for either
storage engine (every record still lands in the one shared database/
index no matter what -- undesigned, not just unbuilt). Updated the
threat model, architecture doc, CLAUDE.md, and both READMEs accordingly.
This commit is contained in:
2026-08-13 23:16:22 -07:00
parent 1fab02abd5
commit ba2276aa1a
17 changed files with 696 additions and 168 deletions
+55 -21
View File
@@ -243,11 +243,15 @@ provisions ClickHouse.
## 8. `enterprise-api`: real per-tenant ClickHouse isolation
This is new since this runbook was first written — `enterprise/internal/
tenantprovision` and `enterprise/internal/chrunner` now exist, closing
the headline gap §"Known gaps" below used to describe as completely
unbuilt. It's still a second binary you have to choose to run, though —
see `/docs/security/threat-model.md`'s "Read this first" section.
`enterprise/internal/tenantprovision` and `enterprise/internal/chrunner`
close the ClickHouse half of the headline gap §"Known gaps" below used
to describe as completely unbuilt. It's still a second binary you have
to choose to run, though — see `/docs/security/threat-model.md`'s "Read
this first" section. With OIDC login now built (§3a), a real
`curl -X POST http://localhost:8083/query` walkthrough as a logged-in
tenant is *possible* now, but still needs the manual `tenant_memberships`
bootstrap from §3a — a full end-to-end curl walkthrough isn't included
here yet.
```sh
docker compose build enterprise-api
@@ -257,14 +261,11 @@ docker compose up -d enterprise-api
curl -s http://localhost:8083/healthz
```
There's still no OIDC/SAML login handler and no CLI for minting a human
session token (see `/docs/security/threat-model.md`) -- so a real
`curl -X POST http://localhost:8083/query` walkthrough as tenant acme
isn't possible yet. Confirm isolation end to end against the live stack (this is the same
Confirm isolation end to end against the live stack (this is the same
assertion `enterprise/internal/chrunner/chrunner_test.go`'s
`TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` makes, run here
as an integration test instead of a curl walkthrough since there's no
login flow to drive it through curl yet):
as an integration test instead of a curl walkthrough since a full login
walkthrough isn't scripted yet):
```sh
docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
@@ -285,17 +286,50 @@ pass) and `TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` (item 1,
closed through the actual production code path, not just
tenantprovision's raw grants).
## 9. Tantivy per-tenant isolation — no Docker needed, actually run this one
Unlike everything above, this one doesn't need a live stack at all —
Tantivy is an embedded library, not a networked service, so both halves
(the Rust index registry and the Go client that talks to it) can be
verified with nothing but a local toolchain:
```sh
cd search
cargo build
cargo clippy --all-targets -- -D warnings
cargo test
# expect 14 tests passing, including
# registry::tests::tenant_index_is_isolated_from_default_and_other_tenants
# -- item 3 of /docs/phase-4-isolation-design.md's verification plan.
cd ../enterprise
go test ./internal/searchclient/... -v
# real in-process gRPC server, confirms SearchRequest.tenant_id is set
# correctly and that a request with no/invalid tenant identity is refused.
```
This is the one piece of Phase 4's tenant isolation work that has
**actually been run and confirmed passing** in an environment without
Docker access, alongside `enterprise/internal/loginhandler`'s OIDC
tests (§3a) — both are unusually strong evidence precisely because they
needed no infrastructure this environment lacked.
## Known gaps (do not treat this phase as done without reading these)
Full accounting: `/docs/security/threat-model.md`. Headline items:
- **ClickHouse isolation exists but is opt-in.** `enterprise-api`
(§8) gives real per-tenant ClickHouse isolation, but plain `api`
(still the default in `docker-compose.yml`/`web`'s base URL) has none,
and nothing flags which one a given deployment is actually running.
- **No Tantivy/free-text isolation at all**, regardless of which binary
serves the request -- `enterprise/internal/searchclient` (chrunner's
Tantivy-side sibling) doesn't exist.
- **Both storage engines' isolation exists but is opt-in.**
`enterprise-api` (§8, §9) gives real per-tenant ClickHouse *and*
Tantivy isolation, but plain `api` (still the default in
`docker-compose.yml`/`web`'s base URL) has neither, and nothing flags
which one a given deployment is actually running. This is now the
single largest gap — not a missing mechanism, a missing enforcement/
default.
- **Ingest has no tenant concept for either storage engine.** Every
record `ingest` produces lands in the one shared ClickHouse database
and the one shared Tantivy index no matter what. A newly-provisioned
tenant's storage is real, isolated at query time, and permanently
empty until this changes — undesigned, not just unbuilt.
- **Human SSO login now works for OIDC** (§3a) -- verified with a real
fake IdP, not yet a real external one or a running `enterprise-auth`
container. **SAML login still doesn't exist** -- protocol wiring only,
@@ -305,10 +339,10 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
bootstrap is the only way to grant a logged-in identity access today.
- **No per-resource dashboard grants** (`dashboard_permissions` has a
schema, no handler reads it).
- Two of the four adversarial ClickHouse/Tantivy probes named in
- Three of the four adversarial ClickHouse/Tantivy probes named in
`/docs/phase-4-isolation-design.md`'s verification plan are closed
(§8); the other two (Tantivy cross-tenant search, mid-provisioning-race
handling) are still stubbed as explicitly-skipped tests in
(§8, §9); the last (mid-provisioning-race handling) is still stubbed
as an explicitly-skipped test in
`api/queryapi/tenant_isolation_gap_test.go`.
## Tearing down