Say what the field report changed and what it left open

Three real 0.15.5 -> 0.16.19 migrations by @kaya-eu, at 221 GB and on
arm64, are worth recording — with the qualifications that make them
readable. They ran a commit predating the automated Docker cutover and did
that step by hand, so what those runs exercise is preflight, the dumps,
the conversion and the recovery-mode store migration.

Three new sections for what they found:

The store migration may need one more recovery boot. This is open. Two
failures, never both on one run, each cured by one extra recovery-mode
boot, and Stalwart's own "multi-step process" panic says the store
migration is not finished when the single boot exits. Documented as the
manual step rather than left for the next operator to rediscover, and
documented as unfixed rather than implied away.

A certificate that serves HTTPS may not serve the mail ports.
IMAPS/SMTPS/POP3S keep a self-signed certificate until
SystemSettings.defaultCertificateId is set. Written up beside the ACME
section, which has the same shape: something the converter does not carry
and nothing warns about. This tool should generate it and does not yet.

Do not boot recovery mode again afterwards already landed with the
artifacts it argues for; this links the production paragraph to it.

Two status claims were also stale. §4.5 still said the container path was
not reachable from the CLI, which #9 changed, and both it and §7 said no
real Stalwart image had been met — one has now been inspected, which is
how #11's two bugs were found. What remains untested is everything after
the recreate, and the Docker section now says what cutover carries and
how the config reaches the container.
This commit is contained in:
2026-08-29 17:53:38 -07:00
parent 82ddb027e4
commit d0d2954ad1
2 changed files with 98 additions and 11 deletions
+14 -6
View File
@@ -321,11 +321,15 @@ against an already-migrated store.
the previous container again. The `docker inspect` of the container as it the previous container again. The `docker inspect` of the container as it
was is preserved as the `container-definition` artifact before anything was is preserved as the `container-definition` artifact before anything
is replaced, for the same reason the unit file is. is replaced, for the same reason the unit file is.
- **Status: the container path is implemented and not yet reachable from - **Status: the container path is wired end to end and reachable from the
the CLI** - `run` does not pass container options, so a container is CLI** behind `--container-path-unproven` ([#9]). What it inspects and
still refused there. Wiring it up, and lifting preflight's refusal for what it assembles have been checked against a real
the containers it can now handle, is the remaining work in `stalwartlabs/stalwart` image ([#11]); what has not been checked is
[#3](https://github.com/LINUXexpert-org/stalwart-migrator/issues/3). whether the recreated container comes up as the server it was. That is
what would retire the flag.
[#9]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/9
[#11]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/11
- Quota recalculation is the one step allowed to fail without failing the - Quota recalculation is the one step allowed to fail without failing the
phase. Stale counters are an accounting problem; a failed cutover is one phase. Stale counters are an accounting problem; a failed cutover is one
an operator has to respond to by restoring a machine that is otherwise an operator has to respond to by restoring a machine that is otherwise
@@ -848,7 +852,11 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
§4.5 lists exactly which two details are inferred. A smoke test against a §4.5 lists exactly which two details are inferred. A smoke test against a
real 0.16 instance would settle both, and would let this step be promoted real 0.16 instance would settle both, and would let this step be promoted
from "warns on failure" to a hard check. from "warns on failure" to a hard check.
- **Docker is wired end to end and has never met a real Stalwart image.** - **Docker is wired end to end and has never completed a migration against
a real Stalwart image.** A real image has now been inspected, which is
how the inherited-versus-overridden confusion above was found, and how it
emerged that cutover was never handing the new container the migrated
config. What remains unverified is everything after the recreate.
Preflight inspects a container and blocks on what stands in the way; Preflight inspects a container and blocks on what stands in the way;
stage pulls and verifies an image; the recovery cycle runs in a throwaway stage pulls and verifies an image; the recovery cycle runs in a throwaway
container against the live data; cutover recreates the container, container against the live data; cutover recreates the container,
+84 -5
View File
@@ -23,6 +23,25 @@ was preceded by a full dress rehearsal on a clone of the same server, which
is the practice this project most recommends copying: see [Rehearse on a is the practice this project most recommends copying: see [Rehearse on a
clone first](#rehearse-on-a-clone-first). clone first](#rehearse-on-a-clone-first).
**It has also been used at a scale well past that, by someone else.**
[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1)
reported three successful 0.15.5 → 0.16.19 migrations on three servers: a
testing and a production instance, each 16 domains, 55 accounts and roughly
**221 GB** of real mail, and an arm64 home server of 1.4 GB across 181
folders. Read that with two qualifications. They ran a commit predating the
automated Docker cutover, so they performed the cutover by hand — what those
runs exercised is preflight, the dumps, the settings conversion and the
recovery-mode store migration, not the container cutover. And they hit
things worth knowing about before you follow them: an arm64 binary that was
fetched for the wrong architecture (fixed), a store migration that needed
one more recovery-mode boot than the tool performs (**not fixed** — see [The
store migration may need one more recovery
boot](#the-store-migration-may-need-one-more-recovery-boot)), and data loss
from booting recovery mode again *after* a completed
migration (see [Do not boot recovery mode
again](#do-not-boot-recovery-mode-again-afterwards)). Their full report is
[issue #1](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
## Before you start: two things you must fix on the server ## Before you start: two things you must fix on the server
Neither is something this tool can do for you, and both stop a migration Neither is something this tool can do for you, and both stop a migration
@@ -79,10 +98,12 @@ window, and tells you what `run` will and won't carry over.
A containerised Stalwart can be migrated, with two things to know first. A containerised Stalwart can be migrated, with two things to know first.
**The container path has never been run against a real Stalwart image.** **The container path has never completed a migration against a real
Its logic is tested and its refusals are real, but a fake `docker` proves Stalwart image.** What it inspects and what it assembles have now been
only that the right commands are assembled — not that the image reads the checked against one, which is how two problems were found and fixed
config it is handed. `run` refuses a container deployment unless you pass (#11) — but a fake `docker` still proves only that the right commands are
assembled, not that the image reads the config it is handed and comes up
as the server it was. `run` refuses a container deployment unless you pass
`--container-path-unproven`, which is there so nobody reaches it without `--container-path-unproven`, which is there so nobody reaches it without
being told. Rehearse on a clone first; that advice goes double here. being told. Rehearse on a clone first; that advice goes double here.
@@ -104,7 +125,13 @@ rebuilt without its capabilities, its custom network or its device mappings
starts cleanly and is quietly not the server it was. So cutover carries starts cleanly and is quietly not the server it was. So cutover carries
across what it understands (mounts, ports, environment, restart policy, across what it understands (mounts, ports, environment, restart policy,
labels) and refuses outright when it finds anything else, naming what it labels) and refuses outright when it finds anything else, naming what it
found. It also refuses a container whose data is not on a volume — an found — and it asks that question in `preflight`, while the server is still
running, rather than only at cutover after it has stopped. What it carries
is mounts, ports, environment, restart policy, labels, and anything the
container overrides on its image: a `--user`, an `--entrypoint`, a command
of your own. What the container merely *inherits* from its old image is
left to the new one, whose own defaults are the ones that go with it.
It also refuses a container whose data is not on a volume — an
upgrade replaces the container, and the writable layer goes with it — and upgrade replaces the container, and the writable layer goes with it — and
one managed by Docker Compose, because recreating it out from under compose one managed by Docker Compose, because recreating it out from under compose
leaves the container and the compose file disagreeing about what is leaves the container and the compose file disagreeing about what is
@@ -112,6 +139,17 @@ deployed, and the next `compose up` reverts the migration. Compose
deployments are migrated by editing the image tag in the compose file and deployments are migrated by editing the image tag in the compose file and
running `compose up -d`. running `compose up -d`.
**The config.** The converted v0.16 config is written into the host side of
whichever mount covers `--data-dir`, named on the container side, and the
recreated container is started with `--config` pointing at it. It cannot go
anywhere else: cutover recreates a container with the mounts it had and
cannot invent a new one. The official image's own default is
`--config /etc/stalwart/config.json`, which is a *different* volume, so a
container left to that default would come up on whatever the old version
had left there. If your container overrides its command, cutover refuses
rather than merging the two — both are the container's argv and there is no
honest way to guess.
**What it keeps.** The old container is renamed rather than removed, the old **What it keeps.** The old container is renamed rather than removed, the old
image is never pruned, and the container's `docker inspect` is preserved as image is never pruned, and the container's `docker inspect` is preserved as
an artifact before anything is replaced. Together those are the manual an artifact before anything is replaced. Together those are the manual
@@ -296,6 +334,47 @@ the server registers a new one on first issuance.
This tool does not yet generate that object for you. It should: the This tool does not yet generate that object for you. It should: the
supplemental plan already does the equivalent for listeners. supplemental plan already does the equivalent for listeners.
## The store migration may need one more recovery boot
**This one is open, and it is the reason to rehearse on a clone.** The
recovery cycle boots the target version once, replays your settings into
it, and stops. Across three real migrations, two different failures showed
up that one extra recovery-mode boot cured:
- the settings apply failing on its very first object with
`primaryKeyViolation`, where re-running the identical apply against a
fresh recovery boot went straight through; or
- the next normal start panicking with *"Upgrading to version 0.16 is a
multi-step process"*, where booting recovery mode once more, letting it
come up and stopping it cleanly was enough.
Never both on the same run — whichever appeared, one more recovery boot
before the real start got past it. That panic is Stalwart's own, and it
suggests the store migration is not finished when the single boot exits.
The likely fix is a settle boot after the apply, and it is not in yet
because getting an extra recovery boot wrong is its own hazard — see [Do
not boot recovery mode again
afterwards](#do-not-boot-recovery-mode-again-afterwards). If you hit
either, that is the manual step. Reported by
[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
## A certificate that serves HTTPS may not serve the mail ports
A `Certificate` object carried into v0.16 with the right SAN is picked up
by the HTTP listener on its own. **IMAPS, SMTPS and POP3S are not**: they
keep serving a self-signed certificate until `defaultCertificateId` is set
on `SystemSettings` and the server is restarted.
This is the kind of thing that looks fine from a browser and surfaces as a
mail client complaining days later, so check it as part of your
post-migration verification: connect to 993 or 465 and confirm which
certificate you are handed, not just to 443.
This tool does not set it for you. Like the `AcmeProvider` above, it
should, and the supplemental plan is where it belongs. Reported by
[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
## You need a named admin account before you migrate ## You need a named admin account before you migrate
**A config-file fallback admin will not survive the migration.** If the only **A config-file fallback admin will not survive the migration.** If the only