Files

T

jcoffey-dev 67619f64e9 Cutover: rehearsed, and the Enterprise build can't read the fork's store

The runbook said nothing in it had been rehearsed. Now the sequence has
been, on data made up for the purpose: upstream 0.16.22 in a container as
the install running today, the fork beside it, both unprivileged with
CAP_NET_BIND_SERVICE. It rehearses the sequence, not the data, which is
what the compat tests are for. 27 of 27 checks passed and the rollback
took 1.5 seconds.

The question §"Open" asked about carrying a store back is answered, and
the answer is no. Upstream refuses to start on a store the fork has
opened: "Column families not opened: _". The fork adds one RocksDB column
family for masked email (SUBSPACE_INBUXA = b'_') and opens with
create_missing_column_families, so it creates it on first open; upstream
has no descriptor for it and RocksDB will not open a database holding one
it was not told about.

That makes step 3's "a copy, not a move" load-bearing in a way the step
did not say. One open by the fork is enough: pointing it at the original
even once, to check something, leaves the Enterprise install unable to
start, and there is no rollback after that. It fails loudly and before
reading anything, which is the good version of this failure, but it is
not recoverable.

Two things the rehearsal found that would have wasted time on the day:
memberTenantId does not come down from the domain and is refused on
create, so a tenant "admin" set up the obvious way is a server
administrator and the check passes while proving nothing; and IMAP's
INBOX is not JMAP's account, because mail from an unauthenticated sender
is filed as spam, so the two counts differ before and after alike.

What the rehearsal does not cover is in its README and in §"Open":
systemd and `systemctl disable stalwart` above all, ACME renewal, load,
the front ends, and INBUXA's own data.

2026-09-19 22:23:10 -07:00

2.6 KiB

Raw Blame History

The synthetic cutover rehearsal

docs/spec/cutover.md as a script, against data this makes up rather than INBUXA's. It rehearses the sequence — stop, copy, start the fork beside it, check, roll back — not the data, which is what the compat tests cover (docs/spec/compat-tests.md).

Nothing here touches INBUXA's production server. Two containers stand in for the two installs, both unprivileged with CAP_NET_BIND_SERVICE so the mail ports are bound the way the systemd unit binds them:

	binary	stands for
`cutover-old`	`target/debug/stalwart`	the Enterprise install running today
`cutover-new`	`target/debug/inbuxa`	what it is cut over to

Running it

Needs target/debug/stalwart and target/debug/inbuxa, and Docker.

tools/fork/cutover-rehearsal/phase1.py   # build the old install, record before.json
tools/fork/cutover-rehearsal/phase2.py   # the cutover sequence, and its checks
tools/fork/cutover-rehearsal/phase3.py   # the rollback, and the reverse-compat probe

Phase 1 destroys anything from an earlier run, so the three are repeatable from nothing. State lives under target/cutover/; the containers are named cutover-old, cutover-new and cutover-reverse.

What it covers, and what it does not

Covered, and observed rather than assumed:

the fork opens and serves a store the previous build wrote;
every account signs in with the password it had, over JMAP and over IMAP;
mail is byte-for-byte the same set, and an alias still delivers;
a tenant administrator sees exactly its own accounts and domains, and still cannot read listeners — the thing tenant_compat could not check, because tenantAdmins was empty on the real run;
ports 25, 465 and 993 bound by an unprivileged process;
the STALWART_* → INBUXA_* fallback warnings of SPEC.md §2.5;
the rollback, timed, including what it leaves behind.

Not covered. The rehearsal is containers, not the host:

systemd. inbuxa.service, AmbientCapabilities, and above all systemctl disable stalwart — the step that stops a reboot putting two servers on one set of ports — have no analogue here.
ACME and certificate renewal. This runs with requestTlsCertificate: false, so the certificate on 443 and 993 is a self-signed fallback. Renewal remains the open question it was.
Load. One message at a time, not INBUXA's traffic.
The front ends. INBUXA Admin and ihasmail are not started.
INBUXA's own data. That is compat-tests.md, and a rehearsal on a real snapshot is still owed.

2.6 KiB Raw Blame History

The synthetic cutover rehearsal

Running it

What it covers, and what it does not

2.6 KiB

Raw Blame History