Record that this migrated a production server, and what it took

On 2026-08-25 the tool migrated a live mail server - nine domains, six
accounts, a 2.4 GB RocksDB store - from 0.15.5 to 0.16.19 with 8 seconds of
downtime and every phase green, validation included.

The practice worth copying is the dress rehearsal on a clone that preceded
it. `rehearse` is read-only and stops before applying, which is where a real
migration fails; a clone closes that gap for about six seconds of production
downtime. It caught four faults that would each have reached production, and
three of them only appear against a real instance. The README now describes
how to build one, including the parts that are easy to get wrong: a
consistent copy of a single-writer store, and a guest with no route out so a
copy of a live mail server cannot renew real certificates or deliver real
queued mail.

It also documents what Stalwart's own converter drops without saying so.
migrate_v016.py consumes every acme.* setting, emits nothing, and does not
report them as unmigrated - so a migration reports success while removing
certificate renewal, and nothing looks wrong until the certificate expires
about ninety days later. The AcmeProvider that had to be rebuilt by hand is
recorded in ARCHITECTURE.md, shape confirmed against a live 0.16.19, as the
specification for generating it in the supplemental plan.
This commit is contained in:
2026-08-24 17:15:18 -07:00
parent 019696bce7
commit 3342662b33
2 changed files with 75 additions and 2 deletions
+16
View File
@@ -323,6 +323,22 @@ schema-migration flag is set for the target version, and the plan skips
is intentionally the same engine with a shorter plan, not a separate is intentionally the same engine with a shorter plan, not a separate
code path, so it doesn't rot independently. code path, so it doesn't rot independently.
### 4.6a What the converter drops without saying so
`migrate_v016.py` consumes every `acme.*` setting and emits nothing for it,
and does not list those keys as unmigrated either. The effect is a migration
that reports complete success while quietly removing certificate renewal: the
certificate itself carries over, so nothing looks wrong until it expires
about ninety days later. Observed on a production migration, 2026-08-25.
The supplemental plan already generates what the converter leaves behind for
listeners (§4.6). An `AcmeProvider` generator belongs alongside it. The
object shape is known-good, having been applied to a live 0.16.19 server:
`challengeType` and `renewBefore` are enums (`TlsAlpn01`, `R23`), `contact`
is a value-keyed set rather than a list, and `accountKey`/`accountUri` are
server-set and must be omitted — the server registers a fresh ACME account,
since the v0.15 account key cannot be carried across.
### 4.7 Post-migration validation ### 4.7 Post-migration validation
**What this suite can assert depends on the boundary being crossed, and on **What this suite can assert depends on the boundary being crossed, and on
+59 -2
View File
@@ -13,14 +13,23 @@ Go, standard library only — no external dependencies.
## Status ## Status
Roughly 14,600 lines of Go, stdlib only, of which about 6,300 are tests. Roughly 14,600 lines of Go, stdlib only, of which about 6,300 are tests.
Every phase exists as a package and `run` wires them into a migration that Every phase exists as a package, and on **2026-08-25 this tool migrated a
has been performed end to end. production mail server** — nine domains, six accounts, a 2.4 GB RocksDB
store — from 0.15.5 to 0.16.19 with **8 seconds** of downtime, every phase
green including post-cutover validation.
That run was preceded by a full dress rehearsal on a clone of the same
server, which is the single practice worth copying from this project: it
found four faults that would each have reached production, three of which
only appear against a real instance. See [Rehearse on a clone
first](#rehearse-on-a-clone-first).
| Command | State | | Command | State |
|---|---| |---|---|
| `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan | | `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan |
| `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over | | `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over |
| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes` | | `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes` |
| `stalwart-migrate tenants` | **Works** — read-only; who owns which domain, and what would block a migration |
| `stalwart-migrate status <id>` | **Works** | | `stalwart-migrate status <id>` | **Works** |
| `stalwart-migrate report <id>` | **Works** — prints what validation found for a run | | `stalwart-migrate report <id>` | **Works** — prints what validation found for a run |
@@ -164,6 +173,54 @@ a sandbox and migrated the copy. That proved the store opens, at the cost of
copying it twice — while the half that found every real problem needed no copying it twice — while the half that found every real problem needed no
copy at all. ARCHITECTURE.md §4.9 has the reasoning. copy at all. ARCHITECTURE.md §4.9 has the reasoning.
## Rehearse on a clone first
`rehearse` is read-only and stops short of the half that matters: it converts
your settings but never applies them, and applying is where a real migration
fails. A clone closes that gap, and on a 2.4 GB store it costs about six
seconds of production downtime to build.
1. **Copy the data directory with the service stopped.** RocksDB is
single-writer, so a hot copy may be torn — and a rehearsal on a torn store
fails for reasons production never would, or passes when it should not.
Stop, `cp -a` the data dir to the same disk, start again, archive it
afterwards; the service is down only for the local copy.
2. **Give the guest no route off the host.** A clone of a live mail server
will otherwise renew certificates for your real domains and deliver
whatever is in the outbound queue. In libvirt that means a network with no
`<forward>` element. Verify it from inside the guest rather than assuming.
3. **Run the real thing**: `preflight`, then `run` with `--target-binary`,
`--stalwart-cli` and `--migration-script` pointed at locally staged copies,
since an isolated guest can download nothing.
4. **Snapshot the guest while it is shut down**, so a failed attempt costs
seconds to reset rather than a rebuild.
What that rehearsal caught, none of which the mock could express: a
multi-tenant arrangement v0.16 cannot represent; `--target-binary` never
reaching preflight, so an air-gapped host failed on a release lookup; an
`admin` account that was a config fallback-admin and stopped working the
moment the migration finished; and `tenant-admin` roles the converter does
not restore.
Two things the isolation costs, so they are not mistaken for faults: the
guest cannot fetch the v0.16 web interface, so `/account/` returns 404, and
certificate renewal cannot be exercised at all.
## Stalwart's own converter silently drops ACME
`migrate_v016.py` consumes every `acme.*` setting and emits nothing for them.
They are **not** reported as unmigrated either, so nothing warns you: the
certificate carries over, the provider that renews it does not, and TLS keeps
working until the certificate expires roughly ninety days later.
Check for `acme.*` in your dump before migrating, and recreate an
`AcmeProvider` afterwards if there was one. Note that `accountKey` is
server-set in v0.16, so the existing ACME account cannot be carried over —
the server registers a new one on first issuance.
This tool does not yet generate that object for you. It should: the
supplemental plan already does the equivalent for listeners.
## You need a named admin account before you migrate ## You need a named admin account before you migrate
**A config-file fallback admin will not survive the migration.** If the only **A config-file fallback admin will not survive the migration.** If the only