Compat: run them from a copy, and keep our listeners out of the way

Rehearsed the run against a real RocksDB store, and it died at startup
before checking anything: the harness inserts listeners of its own, the
registry keys them by name, and a real server already has a "jmap" and an
"imap". The message was "Primary key conflict on property name with
existing object NetworkListener", which says nothing about what to do.
Under NO_INSERT the harness now calls its listeners compat-jmap and so on,
and the same run gets through to the test's own checks.

run-compat.sh copies the store for each test and removes the copy after,
because several of these write to what they open: monitoring_compat purges
the history it reads and undelete_compat restores what it finds. The source
stays untouched, which matters when it is the only copy of a production
store anyone took that day.

INBUXA runs RocksDB, so a copy is a directory copy. The SQL backends would
need more than this: the harness builds its own container and connects to
fixed local credentials, so it cannot open a dump in place.

docs/spec/compat-tests.md

@@ -78,6 +78,35 @@ It was exercised on 2026-09-19 against the fork's own test server, which
 answers the same JMAP: it recorded 3 tenants with their members, 8 masked
 addresses and 3 archived items, in the shapes above.
 
+## Running them
+
+`tools/fork/run-compat.sh` copies the store for each test and runs it:
+
+```
+tools/fork/run-compat.sh --store /srv/inbuxa-copy/rocks.db \
+    --admin '[email protected]:PASSWORD' --recordings ~/compat
+```
+
+It takes `--only <test name>` for one, and `--keep` to leave each copy
+behind for a post-mortem. A test whose recording is missing is skipped by
+name rather than run against nothing.
+
+**The source copy has to be pristine.** Several of these write to the store
+they open — `monitoring_compat` purges the history it reads,
+`undelete_compat` restores what it finds — and the harness adds its own
+listeners on the way in, so a copy that a run has already touched is not a
+copy of INBUXA's data any more. The script copies from the source for each
+test and removes the copy afterwards, so the source stays clean; take it
+from a stopped server or a snapshot, never from under a running one.
+
+**Why `compat-` listeners appear in the copy.** The harness needs listeners
+on its own ports, and the registry keys listeners by name. A real server
+has its own, and a production listener called `jmap` or `imap` collided
+with the harness's, killing every compat test at startup with `Primary key
+conflict on property "name"` before it checked anything. Under `NO_INSERT`
+the harness now names its listeners `compat-jmap` and so on
+(`tests/src/utils/server.rs`), out of the way of whatever the copy holds.
+
 ## Running one
 
 ```

tests/src/utils/server.rs

@@ -200,11 +200,23 @@ impl TestServerBuilder {
         if protocol == NetworkListenerProtocol::Http {
             self.http_listener_port = port;
         }
+        // inbuxa: with NO_INSERT the store is a copy of a real server's, and
+        // that server has listeners of its own. A production one called
+        // "jmap" or "imap" would collide with ours on the name, which is the
+        // registry's primary key, and every compat test would die here
+        // before it checked anything. Namespace ours out of its way.
+        // The copy must still be pristine: a second run over a copy the
+        // first one wrote to collides with `compat-` instead.
+        let name = if std::env::var("NO_INSERT").is_ok() {
+            format!("compat-{name}")
+        } else {
+            name.to_string()
+        };
         self.insert_object(NetworkListener {
             bind: Map::new(vec![
                 SocketAddr::from_str(&format!("0.0.0.0:{port}")).unwrap(),
             ]),
-            name: name.to_string(),
+            name,
             protocol,
             use_tls: true,
             tls_implicit,

tools/fork/README.md

@@ -32,3 +32,13 @@ tools/fork/record-compat.py --server https://mail.example.org \
     --admin '[email protected]:PASSWORD' --out ./compat \
     --tenant-admin '[email protected]:PASSWORD'
 ```
+
+## run-compat.sh
+
+Runs the `*_compat` tests against a copy of INBUXA's RocksDB store, making
+a fresh copy for each one. See `docs/spec/compat-tests.md`.
+
+```bash
+tools/fork/run-compat.sh --store /srv/inbuxa-copy/rocks.db \
+    --admin '[email protected]:PASSWORD' --recordings ~/compat
+```

tools/fork/run-compat.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: 2026 Coffey Labs
+# SPDX-License-Identifier: AGPL-3.0-only
+#
+# Run the compat tests against a copy of INBUXA's RocksDB store.
+#
+#   tools/fork/run-compat.sh --store /srv/inbuxa-copy/rocks.db \
+#       --admin '[email protected]:PASSWORD' --recordings ~/compat
+#
+# Each test opens $TMPDIR/<its own name>/rocks.db and several of them write:
+# monitoring_compat purges the history it reads, undelete_compat restores
+# what it finds. So every test gets its own copy of the store, made fresh
+# here and removed after, and the source is only ever read.
+#
+# The source must be a copy already: take it from a stopped server or a
+# snapshot, never from under a running one, and never point this at the
+# live store (docs/spec/compat-tests.md).
+set -u
+
+REPO=$(cd "$(dirname "$0")/../.." && pwd)
+STORE= ADMIN= RECORDINGS= ONLY= KEEP=no
+while [ $# -gt 0 ]; do
+    case $1 in
+        --store) STORE=$2; shift 2 ;;
+        --admin) ADMIN=$2; shift 2 ;;
+        --recordings) RECORDINGS=$2; shift 2 ;;   # where record-compat.py wrote its files
+        --only) ONLY=$2; shift 2 ;;               # one test name, e.g. tenant_compat
+        --keep) KEEP=yes; shift ;;                # leave each copy behind for a post-mortem
+        *) echo "run-compat: unknown argument $1" >&2; exit 2 ;;
+    esac
+done
+[ -n "$STORE" ] && [ -n "$ADMIN" ] || {
+    echo "run-compat: --store and --admin are required" >&2; exit 2; }
+[ -d "$STORE" ] || { echo "run-compat: $STORE is not a directory" >&2; exit 2; }
+
+# name | full path | what it needs beyond the store
+TESTS=(
+  "tenant_compat|system::tenant::tenant_compat|INBUXA_COMPAT_EXPECTED=expected.json"
+  "branding_compat|system::branding::branding_compat|"
+  "ai_compat|system::ai::ai_compat|"
+  "monitoring_compat|system::monitoring::monitoring_compat|"
+  "scim_compat|scim::scim_compat|"
+  "per_domain_directory_compat|directory::per_domain::per_domain_directory_compat|"
+  "masked_email_compat|system::masked_email::masked_email_compat|INBUXA_COMPAT_MASKS=masks.json"
+  "undelete_compat|system::undelete::undelete_compat|INBUXA_COMPAT_ARCHIVED=archived.json"
+)
+
+BASE="$REPO/target/tmp/compat"
+mkdir -p "$BASE"
+failed=0 ran=0
+
+for entry in "${TESTS[@]}"; do
+    IFS='|' read -r name path needs <<< "$entry"
+    [ -n "$ONLY" ] && [ "$ONLY" != "$name" ] && continue
+
+    env_extra=()
+    if [ -n "$needs" ]; then
+        variable=${needs%%=*}; file=${needs#*=}
+        if [ -z "$RECORDINGS" ] || [ ! -f "$RECORDINGS/$file" ]; then
+            echo "SKIP $name: needs $variable ($file from record-compat.py)"
+            continue
+        fi
+        env_extra=("$variable=$RECORDINGS/$file")
+    fi
+
+    dir="$BASE/$name"
+    rm -rf "$dir"; mkdir -p "$dir"
+    cp -a "$STORE" "$dir/rocks.db" || { echo "run-compat: copying $STORE failed" >&2; exit 2; }
+
+    echo "=== $name ($(date +%H:%M:%S)) ==="
+    ( cd "$REPO" && env CARGO_TARGET_DIR=target TMPDIR="$BASE" RUST_MIN_STACK=8388608 \
+        STORE=RocksDb NO_INSERT=1 INBUXA_COMPAT_ADMIN="$ADMIN" "${env_extra[@]}" \
+        cargo test -p tests "$path" -- --exact --ignored --nocapture )
+    rc=$?
+    ran=$((ran + 1))
+    [ $rc -ne 0 ] && { failed=$((failed + 1)); echo "FAILED $name (exit $rc)"; }
+    [ "$KEEP" = no ] && rm -rf "$dir"
+done
+
+echo
+echo "ran $ran, failed $failed"
+[ $failed -eq 0 ] || echo "A failure here is a cutover blocker unless compat-tests.md says otherwise."
+exit $((failed > 0))