Give chwriter.Registry periodic refresh, matching Tantivy's tracker

Closing search's active-tenant gap last commit surfaced a real asymmetry
by comparison: chwriter.Registry's per-tenant writer map was still a
snapshot built once at enterprise-ingest startup with no refresh at all,
while search's new ActiveTenantTracker refreshes every minute. A tenant
deprovisioned after enterprise-ingest started would keep writing
successfully to ClickHouse until the next restart -- a real, disclosed
staleness gap, not matched by anything on the Tantivy side anymore.

Registry.StartRefreshing spawns a goroutine that re-lists active tenants
every minute (dataSourceRefreshInterval, same interval as search's
tracker) via a new SourceLister callback and reconciles the writer map:
opens a connection for a newly-active tenant, closes and removes one no
longer active. New connections are dialed before taking the write lock,
so a slow/unreachable ClickHouse for one newly-active tenant never
blocks WriteBatch's read lock. A refresh failure (lister error, or one
tenant's connection failing to open) logs and leaves the existing map
untouched for that tick -- the same last-known-good posture
ActiveTenantTracker already uses, so a transient rbacstore/Postgres blip
doesn't evict every other tenant's already-working writer.

WriteBatch now takes a read lock and Close takes a write lock -- the
writer map was safe unsynchronized before only because it was immutable
after New() returned; StartRefreshing makes it mutable at runtime.

enterprise-ingest/main.go extracts the existing rbacstore-row-to-
DataSource adaptation into tenantDataSourceLister, reused for both the
initial synchronous load and StartRefreshing's periodic calls, so the
two can't drift into checking different things.

Verified: the lister-error-keeps-last-known-good path is Docker-free
(same "construct a Registry directly, bypass New" trick the existing
fail-closed tests use). The actual add/remove reconciliation against
real ClickHouse connections (TestRefreshAddsNewlyActiveTenant,
TestRefreshRemovesNoLongerActiveTenant) are skip-gated live-ClickHouse
tests, same CHWRITER_TEST_CLICKHOUSE_ADDR convention as this package's
existing integration tests -- not run against a live database in this
environment.

This closes the last disclosed gap from Phase 4's write-routing work:
both storage engines now share the same one-minute active-tenant
staleness bound instead of one being materially staler than the other.
This commit is contained in:
2026-08-14 23:55:40 -07:00
parent 088677643f
commit 2e8ab1ed6a
7 changed files with 379 additions and 90 deletions
+10 -2
View File
@@ -283,8 +283,16 @@ default as everything else optional in this codebase); when they are,
startup blocks on the first fetch succeeding and later refresh failures
keep serving the last-known-good set rather than clearing it. Verified
with real HTTP round trips against a hand-rolled TCP test server in this
environment, no live enterprise-auth needed. **The tenant-picker page is
now built too**:
environment, no live enterprise-auth needed. **This closing move exposed
the ClickHouse side's own gap by comparison** — `chwriter.Registry`'s
writer map was still a startup-only snapshot with no refresh at all, a
real asymmetry once Tantivy's tracker refreshed every minute and
ClickHouse's didn't — so `Registry.StartRefreshing` (new) closes that
too: same one-minute interval, same last-known-good posture on a failed
refresh, opening connections for newly-active tenants and closing ones
no longer active. Both engines now share the same active-tenant
staleness bound instead of one being materially staler than the other.
**The tenant-picker page is now built too**:
`web/src/routes/select-tenant` calls `GET /auth/memberships`/
`POST /auth/select-tenant` via `fetch(..., {credentials: 'include'})`
(new `$lib/api.ts` functions), which needed a second CORS posture