Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/jmap/src/api/request.rs

@@ -94,6 +94,10 @@ impl RequestHandler for Server {
             request.method_calls.len(),
         );
 
+        // inbuxa: ST-6: reads before the request's first write may go to a
+        // read replica
+        let mut has_written = false;
+
         for mut call in request.method_calls {
             // Resolve result and id references
             if let Err(error) = response.resolve_references(&mut call.method) {
@@ -126,15 +130,59 @@ impl RequestHandler for Server {
 
                 // Add response
                 let method_name = call.name.as_str();
-                match self
-                    .handle_method_call(
+
+                // inbuxa: ST-6, ST-7: a read before the first write may use a
+                // replica that has every change the client has seen
+                let eligible = !has_written
+                    && matches!(
                         call.method,
-                        call.name,
-                        access_token,
-                        &mut next_call,
-                        session,
+                        RequestMethod::Get(_)
+                            | RequestMethod::Query(_)
+                            | RequestMethod::Changes(_)
+                            | RequestMethod::QueryChanges(_)
+                    );
+                if matches!(
+                    call.method,
+                    RequestMethod::Set(_)
+                        | RequestMethod::Copy(_)
+                        | RequestMethod::ImportEmail(_)
+                        | RequestMethod::UploadBlob(_)
+                ) {
+                    has_written = true;
+                }
+                let presented = match &call.method {
+                    RequestMethod::Changes(changes) => match &changes.since_state {
+                        jmap_proto::types::state::State::Exact(change_id) => {
+                            Some((changes.account_id.document_id(), *change_id))
+                        }
+                        _ => None,
+                    },
+                    _ => None,
+                };
+                let method_call = self.handle_method_call(
+                    call.method,
+                    call.name,
+                    access_token,
+                    &mut next_call,
+                    session,
+                );
+                let result = if eligible {
+                    store::backend::scaleout::replica::replica_read(
+                        access_token.all_ids().map(|account_id| {
+                            (
+                                account_id,
+                                presented
+                                    .filter(|(id, _)| *id == account_id)
+                                    .map_or(0, |(_, change_id)| change_id),
+                            )
+                        }),
+                        method_call,
                     )
                     .await
+                } else {
+                    method_call.await
+                };
+                match result
                 {
                     Ok(mut method_response) => {
                         match &mut method_response {