From 216bb5b7112bed634fbf81b3fa26a7c86e20eb0c Mon Sep 17 00:00:00 2001 From: John Coffey Date: Sat, 19 Sep 2026 14:21:54 -0700 Subject: [PATCH] Scale-out storage: implementation status of read replicas (ST-5 to ST-15) Where the routing lives and what a read scope covers, which acceptance tests the container suite runs, what isn't exercised (two nodes, the primary stopped, and the MySQL paths, which are built but unrun), what was settled from the code, and the known limits. --- docs/spec/features/scale-out-storage.md | 52 ++++++++++++++++++------- 1 file changed, 37 insertions(+), 15 deletions(-) diff --git a/docs/spec/features/scale-out-storage.md b/docs/spec/features/scale-out-storage.md index 47e9b75..4567a93 100644 --- a/docs/spec/features/scale-out-storage.md +++ b/docs/spec/features/scale-out-storage.md @@ -529,31 +529,42 @@ and PostgreSQL), to show members of different kinds work together (ST-16). ## Implementation status -Sharded stores built 2026-09-19 from this spec, clean-room, under the -multi-tenancy hand-off brief's rules, in `crates/store/src/backend/scaleout/` -(as decided above), with one variant added to `BlobStore` and -`InMemoryStore` and one arm to each dispatch and build `match`, marked -`inbuxa:`. Read-replica routing waits for per-domain directories (the ST-2 +Built 2026-09-19 from this spec, clean-room, under the multi-tenancy hand-off +brief's rules, in `crates/store/src/backend/scaleout/` (as decided above), +with one variant added to `Store`, `BlobStore` and `InMemoryStore` and one +arm to each dispatch and build `match`, marked `inbuxa:`. Sharded stores +came first; read replicas followed per-domain directories (the ST-2 Decision). -- **ST-1 to ST-4, ST-16 to ST-30:** built, with the limits below. -- **ST-5 to ST-15:** not built yet. Each configured replica is reported at - startup with `store.postgresql-error` or `store.mysql-error` naming it, and - nothing connects to it (the pools the stripped code opened and dropped are - gone). +- **ST-1 to ST-30:** built, with the limits below. +- **Read replicas.** A data store with `readReplicas` becomes + `Store::Replicated`. The read handle of "Interfaces" is a read scope that + the call sites ST-6 names open around a request (`replica_read`), and that + spawned work carries along. Only account data (properties, indexes, + change logs, counters, ACLs, blobs, the search index) is read from a + replica; the registry, in-memory values, the task queue and everything + else go to the primary (ST-5). - **Tests.** `store::scaleout::scaleout_blob_tests` covers tests 2 to 7 on three and four FileSystem members, opening the store directly against the data store. The existing blob suite passes against a sharded store with `BLOB_STORE=Sharded` (three FileSystem members), which is test 8 and the second half of test 2. `scaleout_memory_tests`, built with `redis`, covers tests 20, 22 and 23 over two databases of one Redis server, which the store - treats as two members. Test 1 is the existing store and blob suites - passing unchanged. + treats as two members. `store::replica::replica_tests`, built with + `postgres` and run with `STORE=PostgreSqlReplicated`, runs a primary and a + streaming hot standby in containers and covers tests 9, 10, 12, 13, 14 + and 15. Test 1 is the existing store, blob and protocol suites passing + unchanged. - **Not exercised:** test 3's downloads over JMAP and IMAP after a restart (the same blob reads are checked at the store), test 5's queued delivery (the failing write is checked), test 21 (one of two Redis servers stopped), the `resetRateLimiters` and `removeLock*` maintenance types (the store - operations they use are checked), and tests 9 to 19 (replicas). + operations they use are checked), test 11 (two nodes), test 16 (the + primary stopped), and tests 17 to 19: the MySQL code (GTID and + `Seconds_Behind_Source` lag, the read-only and commit-order checks) is + built but hasn't run against a MySQL replica. Test 9 checks that the + replica served the reads, not the replica's statement log, and test 15 + checks full-text search, not a SQL directory. - **Settled from the code, not a change of intent:** - The FileSystem backend reports any unreadable file as missing, so a FileSystem member that can't be read looks like a miss (ST-17's search @@ -566,12 +577,23 @@ Decision). made by restarting every node settles. - A misplaced blob is logged as `store.unexpected-error`, naming the member it was found on and its home. + - Any write made inside a read scope (a first mailbox read creating the + default mailboxes, an IMAP `FETCH` setting `\Seen`) sends the rest of + the scope to the primary, as ST-6 asks of requests that write. + - A replica that hasn't shown the startup marker within the lag limit is + tried again at each 10-second probe, and left out after six misses: at + startup a replica can still be replaying a burst of writes. + - Shared high-water marks (ST-7, step 2) go through the in-memory store + only when it's Redis; a cluster whose in-memory store is the data store + relies on the state-change broadcasts and ST-8. - **Known limits, not requirements of this spec:** - Placement is the fork's own (ST-16), so an install coming from a sharded upstream deployment reads through ST-17's search (open question). - - The leftover `Store::SQLReadReplica` arm and the store's `enterprise` - feature stay until replica routing lands. + - ST-7's step 4 raises the mark from a JMAP `sinceState` only; IMAP + `CONDSTORE` and `QRESYNC` values and push resumption don't yet. + - The store's `enterprise` Cargo feature stays: other crates' feature + lists name it. ## Observed