Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/imap/src/core/client.rs

@@ -93,7 +93,25 @@ impl<T: SessionStream> Session<T> {
 
         let mut requests = requests.into_iter().peekable();
         while let Some(request) = requests.next() {
-            let result = match request.command {
+            // inbuxa: ST-6: commands that only read may be served by a read
+            // replica; any write they make still goes to the primary
+            let replica_accounts = match (&request.command, &self.state) {
+                (
+                    Command::List
+                    | Command::Lsub
+                    | Command::Status
+                    | Command::Search(_)
+                    | Command::Sort(_)
+                    | Command::Fetch(_),
+                    State::Authenticated { data } | State::Selected { data, .. },
+                ) => data
+                    .access_token
+                    .all_ids()
+                    .map(|account_id| (account_id, 0))
+                    .collect::<Vec<_>>(),
+                _ => Vec::new(),
+            };
+            let dispatch = async { match request.command {
                 Command::List | Command::Lsub => self
                     .handle_list(request)
                     .await
@@ -256,6 +274,11 @@ impl<T: SessionStream> Session<T> {
                     .handle_uidbatches(request)
                     .await
                     .map(|_| SessionResult::Continue),
+            } };
+            let result = if replica_accounts.is_empty() {
+                dispatch.await
+            } else {
+                store::backend::scaleout::replica::replica_read(replica_accounts, dispatch).await
             };
 
             match result {

crates/imap/src/op/mod.rs

@@ -62,8 +62,8 @@ impl ToModSeq for u64 {
 macro_rules! spawn_op {
     ($data:expr, $($code:tt)*) => {
         {
-
-        tokio::spawn(async move {
+        // inbuxa: ST-6: the operation keeps its read scope
+        tokio::spawn(store::backend::scaleout::replica::carry(async move {
             let data = &($data);
 
             if let Err(err) = (async {
@@ -73,7 +73,7 @@ macro_rules! spawn_op {
             {
                 let _ = data.write_error(err).await;
             }
-        });
+        }));
 
         Ok(())}
     };