From d0d2954ad15a6019a5ffd40ed098921ded4950a7 Mon Sep 17 00:00:00 2001 From: John Coffey Date: Sat, 29 Aug 2026 17:53:02 -0700 Subject: [PATCH] Say what the field report changed and what it left open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three real 0.15.5 -> 0.16.19 migrations by @kaya-eu, at 221 GB and on arm64, are worth recording — with the qualifications that make them readable. They ran a commit predating the automated Docker cutover and did that step by hand, so what those runs exercise is preflight, the dumps, the conversion and the recovery-mode store migration. Three new sections for what they found: The store migration may need one more recovery boot. This is open. Two failures, never both on one run, each cured by one extra recovery-mode boot, and Stalwart's own "multi-step process" panic says the store migration is not finished when the single boot exits. Documented as the manual step rather than left for the next operator to rediscover, and documented as unfixed rather than implied away. A certificate that serves HTTPS may not serve the mail ports. IMAPS/SMTPS/POP3S keep a self-signed certificate until SystemSettings.defaultCertificateId is set. Written up beside the ACME section, which has the same shape: something the converter does not carry and nothing warns about. This tool should generate it and does not yet. Do not boot recovery mode again afterwards already landed with the artifacts it argues for; this links the production paragraph to it. Two status claims were also stale. §4.5 still said the container path was not reachable from the CLI, which #9 changed, and both it and §7 said no real Stalwart image had been met — one has now been inspected, which is how #11's two bugs were found. What remains untested is everything after the recreate, and the Docker section now says what cutover carries and how the config reaches the container. --- ARCHITECTURE.md | 20 +++++++---- README.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 98 insertions(+), 11 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 4c5645d..d4a1093 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -321,11 +321,15 @@ against an already-migrated store. the previous container again. The `docker inspect` of the container as it was is preserved as the `container-definition` artifact before anything is replaced, for the same reason the unit file is. -- **Status: the container path is implemented and not yet reachable from - the CLI** - `run` does not pass container options, so a container is - still refused there. Wiring it up, and lifting preflight's refusal for - the containers it can now handle, is the remaining work in - [#3](https://github.com/LINUXexpert-org/stalwart-migrator/issues/3). +- **Status: the container path is wired end to end and reachable from the + CLI** behind `--container-path-unproven` ([#9]). What it inspects and + what it assembles have been checked against a real + `stalwartlabs/stalwart` image ([#11]); what has not been checked is + whether the recreated container comes up as the server it was. That is + what would retire the flag. + + [#9]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/9 + [#11]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/11 - Quota recalculation is the one step allowed to fail without failing the phase. Stale counters are an accounting problem; a failed cutover is one an operator has to respond to by restoring a machine that is otherwise @@ -848,7 +852,11 @@ happens to need them. `preflight.DeploymentKind` is a type alias for §4.5 lists exactly which two details are inferred. A smoke test against a real 0.16 instance would settle both, and would let this step be promoted from "warns on failure" to a hard check. -- **Docker is wired end to end and has never met a real Stalwart image.** +- **Docker is wired end to end and has never completed a migration against + a real Stalwart image.** A real image has now been inspected, which is + how the inherited-versus-overridden confusion above was found, and how it + emerged that cutover was never handing the new container the migrated + config. What remains unverified is everything after the recreate. Preflight inspects a container and blocks on what stands in the way; stage pulls and verifies an image; the recovery cycle runs in a throwaway container against the live data; cutover recreates the container, diff --git a/README.md b/README.md index 0fe03c9..b377dd7 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,25 @@ was preceded by a full dress rehearsal on a clone of the same server, which is the practice this project most recommends copying: see [Rehearse on a clone first](#rehearse-on-a-clone-first). +**It has also been used at a scale well past that, by someone else.** +[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1) +reported three successful 0.15.5 → 0.16.19 migrations on three servers: a +testing and a production instance, each 16 domains, 55 accounts and roughly +**221 GB** of real mail, and an arm64 home server of 1.4 GB across 181 +folders. Read that with two qualifications. They ran a commit predating the +automated Docker cutover, so they performed the cutover by hand — what those +runs exercised is preflight, the dumps, the settings conversion and the +recovery-mode store migration, not the container cutover. And they hit +things worth knowing about before you follow them: an arm64 binary that was +fetched for the wrong architecture (fixed), a store migration that needed +one more recovery-mode boot than the tool performs (**not fixed** — see [The +store migration may need one more recovery +boot](#the-store-migration-may-need-one-more-recovery-boot)), and data loss +from booting recovery mode again *after* a completed +migration (see [Do not boot recovery mode +again](#do-not-boot-recovery-mode-again-afterwards)). Their full report is +[issue #1](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1). + ## Before you start: two things you must fix on the server Neither is something this tool can do for you, and both stop a migration @@ -79,10 +98,12 @@ window, and tells you what `run` will and won't carry over. A containerised Stalwart can be migrated, with two things to know first. -**The container path has never been run against a real Stalwart image.** -Its logic is tested and its refusals are real, but a fake `docker` proves -only that the right commands are assembled — not that the image reads the -config it is handed. `run` refuses a container deployment unless you pass +**The container path has never completed a migration against a real +Stalwart image.** What it inspects and what it assembles have now been +checked against one, which is how two problems were found and fixed +(#11) — but a fake `docker` still proves only that the right commands are +assembled, not that the image reads the config it is handed and comes up +as the server it was. `run` refuses a container deployment unless you pass `--container-path-unproven`, which is there so nobody reaches it without being told. Rehearse on a clone first; that advice goes double here. @@ -104,7 +125,13 @@ rebuilt without its capabilities, its custom network or its device mappings starts cleanly and is quietly not the server it was. So cutover carries across what it understands (mounts, ports, environment, restart policy, labels) and refuses outright when it finds anything else, naming what it -found. It also refuses a container whose data is not on a volume — an +found — and it asks that question in `preflight`, while the server is still +running, rather than only at cutover after it has stopped. What it carries +is mounts, ports, environment, restart policy, labels, and anything the +container overrides on its image: a `--user`, an `--entrypoint`, a command +of your own. What the container merely *inherits* from its old image is +left to the new one, whose own defaults are the ones that go with it. +It also refuses a container whose data is not on a volume — an upgrade replaces the container, and the writable layer goes with it — and one managed by Docker Compose, because recreating it out from under compose leaves the container and the compose file disagreeing about what is @@ -112,6 +139,17 @@ deployed, and the next `compose up` reverts the migration. Compose deployments are migrated by editing the image tag in the compose file and running `compose up -d`. +**The config.** The converted v0.16 config is written into the host side of +whichever mount covers `--data-dir`, named on the container side, and the +recreated container is started with `--config` pointing at it. It cannot go +anywhere else: cutover recreates a container with the mounts it had and +cannot invent a new one. The official image's own default is +`--config /etc/stalwart/config.json`, which is a *different* volume, so a +container left to that default would come up on whatever the old version +had left there. If your container overrides its command, cutover refuses +rather than merging the two — both are the container's argv and there is no +honest way to guess. + **What it keeps.** The old container is renamed rather than removed, the old image is never pruned, and the container's `docker inspect` is preserved as an artifact before anything is replaced. Together those are the manual @@ -296,6 +334,47 @@ the server registers a new one on first issuance. This tool does not yet generate that object for you. It should: the supplemental plan already does the equivalent for listeners. +## The store migration may need one more recovery boot + +**This one is open, and it is the reason to rehearse on a clone.** The +recovery cycle boots the target version once, replays your settings into +it, and stops. Across three real migrations, two different failures showed +up that one extra recovery-mode boot cured: + +- the settings apply failing on its very first object with + `primaryKeyViolation`, where re-running the identical apply against a + fresh recovery boot went straight through; or +- the next normal start panicking with *"Upgrading to version 0.16 is a + multi-step process"*, where booting recovery mode once more, letting it + come up and stopping it cleanly was enough. + +Never both on the same run — whichever appeared, one more recovery boot +before the real start got past it. That panic is Stalwart's own, and it +suggests the store migration is not finished when the single boot exits. + +The likely fix is a settle boot after the apply, and it is not in yet +because getting an extra recovery boot wrong is its own hazard — see [Do +not boot recovery mode again +afterwards](#do-not-boot-recovery-mode-again-afterwards). If you hit +either, that is the manual step. Reported by +[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1). + +## A certificate that serves HTTPS may not serve the mail ports + +A `Certificate` object carried into v0.16 with the right SAN is picked up +by the HTTP listener on its own. **IMAPS, SMTPS and POP3S are not**: they +keep serving a self-signed certificate until `defaultCertificateId` is set +on `SystemSettings` and the server is restarted. + +This is the kind of thing that looks fine from a browser and surfaces as a +mail client complaining days later, so check it as part of your +post-migration verification: connect to 993 or 465 and confirm which +certificate you are handed, not just to 443. + +This tool does not set it for you. Like the `AcmeProvider` above, it +should, and the supplemental plan is where it belongs. Reported by +[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1). + ## You need a named admin account before you migrate **A config-file fallback admin will not survive the migration.** If the only