Merge pull request #13 from LINUXexpert-org/docs-from-the-field

Say what the field report changed and what it left open

ARCHITECTURE.md

@@ -321,11 +321,15 @@ against an already-migrated store.
   the previous container again. The `docker inspect` of the container as it
   was is preserved as the `container-definition` artifact before anything
   is replaced, for the same reason the unit file is.
-- **Status: the container path is implemented and not yet reachable from
+- **Status: the container path is wired end to end and reachable from the
-  the CLI** - `run` does not pass container options, so a container is
+  CLI** behind `--container-path-unproven` ([#9]). What it inspects and
-  still refused there. Wiring it up, and lifting preflight's refusal for
+  what it assembles have been checked against a real
-  the containers it can now handle, is the remaining work in
+  `stalwartlabs/stalwart` image ([#11]); what has not been checked is
-  [#3](https://github.com/LINUXexpert-org/stalwart-migrator/issues/3).
+  whether the recreated container comes up as the server it was. That is
+  what would retire the flag.
+
+  [#9]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/9
+  [#11]: https://github.com/LINUXexpert-org/stalwart-migrator/pull/11
 - Quota recalculation is the one step allowed to fail without failing the
   phase. Stale counters are an accounting problem; a failed cutover is one
   an operator has to respond to by restoring a machine that is otherwise
@@ -848,7 +852,11 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
   §4.5 lists exactly which two details are inferred. A smoke test against a
   real 0.16 instance would settle both, and would let this step be promoted
   from "warns on failure" to a hard check.
-- **Docker is wired end to end and has never met a real Stalwart image.**
+- **Docker is wired end to end and has never completed a migration against
+  a real Stalwart image.** A real image has now been inspected, which is
+  how the inherited-versus-overridden confusion above was found, and how it
+  emerged that cutover was never handing the new container the migrated
+  config. What remains unverified is everything after the recreate.
   Preflight inspects a container and blocks on what stands in the way;
   stage pulls and verifies an image; the recovery cycle runs in a throwaway
   container against the live data; cutover recreates the container,

README.md

@@ -23,6 +23,25 @@ was preceded by a full dress rehearsal on a clone of the same server, which
 is the practice this project most recommends copying: see [Rehearse on a
 clone first](#rehearse-on-a-clone-first).
 
+**It has also been used at a scale well past that, by someone else.**
+[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1)
+reported three successful 0.15.5 → 0.16.19 migrations on three servers: a
+testing and a production instance, each 16 domains, 55 accounts and roughly
+**221 GB** of real mail, and an arm64 home server of 1.4 GB across 181
+folders. Read that with two qualifications. They ran a commit predating the
+automated Docker cutover, so they performed the cutover by hand — what those
+runs exercised is preflight, the dumps, the settings conversion and the
+recovery-mode store migration, not the container cutover. And they hit
+things worth knowing about before you follow them: an arm64 binary that was
+fetched for the wrong architecture (fixed), a store migration that needed
+one more recovery-mode boot than the tool performs (**not fixed** — see [The
+store migration may need one more recovery
+boot](#the-store-migration-may-need-one-more-recovery-boot)), and data loss
+from booting recovery mode again *after* a completed
+migration (see [Do not boot recovery mode
+again](#do-not-boot-recovery-mode-again-afterwards)). Their full report is
+[issue #1](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
+
 ## Before you start: two things you must fix on the server
 
 Neither is something this tool can do for you, and both stop a migration
@@ -79,10 +98,12 @@ window, and tells you what `run` will and won't carry over.
 
 A containerised Stalwart can be migrated, with two things to know first.
 
-**The container path has never been run against a real Stalwart image.**
+**The container path has never completed a migration against a real
-Its logic is tested and its refusals are real, but a fake `docker` proves
+Stalwart image.** What it inspects and what it assembles have now been
-only that the right commands are assembled — not that the image reads the
+checked against one, which is how two problems were found and fixed
-config it is handed. `run` refuses a container deployment unless you pass
+(#11) — but a fake `docker` still proves only that the right commands are
+assembled, not that the image reads the config it is handed and comes up
+as the server it was. `run` refuses a container deployment unless you pass
 `--container-path-unproven`, which is there so nobody reaches it without
 being told. Rehearse on a clone first; that advice goes double here.
 
@@ -104,7 +125,13 @@ rebuilt without its capabilities, its custom network or its device mappings
 starts cleanly and is quietly not the server it was. So cutover carries
 across what it understands (mounts, ports, environment, restart policy,
 labels) and refuses outright when it finds anything else, naming what it
-found. It also refuses a container whose data is not on a volume — an
+found — and it asks that question in `preflight`, while the server is still
+running, rather than only at cutover after it has stopped. What it carries
+is mounts, ports, environment, restart policy, labels, and anything the
+container overrides on its image: a `--user`, an `--entrypoint`, a command
+of your own. What the container merely *inherits* from its old image is
+left to the new one, whose own defaults are the ones that go with it.
+It also refuses a container whose data is not on a volume — an
 upgrade replaces the container, and the writable layer goes with it — and
 one managed by Docker Compose, because recreating it out from under compose
 leaves the container and the compose file disagreeing about what is
@@ -112,6 +139,17 @@ deployed, and the next `compose up` reverts the migration. Compose
 deployments are migrated by editing the image tag in the compose file and
 running `compose up -d`.
 
+**The config.** The converted v0.16 config is written into the host side of
+whichever mount covers `--data-dir`, named on the container side, and the
+recreated container is started with `--config` pointing at it. It cannot go
+anywhere else: cutover recreates a container with the mounts it had and
+cannot invent a new one. The official image's own default is
+`--config /etc/stalwart/config.json`, which is a *different* volume, so a
+container left to that default would come up on whatever the old version
+had left there. If your container overrides its command, cutover refuses
+rather than merging the two — both are the container's argv and there is no
+honest way to guess.
+
 **What it keeps.** The old container is renamed rather than removed, the old
 image is never pruned, and the container's `docker inspect` is preserved as
 an artifact before anything is replaced. Together those are the manual
@@ -296,6 +334,47 @@ the server registers a new one on first issuance.
 This tool does not yet generate that object for you. It should: the
 supplemental plan already does the equivalent for listeners.
 
+## The store migration may need one more recovery boot
+
+**This one is open, and it is the reason to rehearse on a clone.** The
+recovery cycle boots the target version once, replays your settings into
+it, and stops. Across three real migrations, two different failures showed
+up that one extra recovery-mode boot cured:
+
+- the settings apply failing on its very first object with
+  `primaryKeyViolation`, where re-running the identical apply against a
+  fresh recovery boot went straight through; or
+- the next normal start panicking with *"Upgrading to version 0.16 is a
+  multi-step process"*, where booting recovery mode once more, letting it
+  come up and stopping it cleanly was enough.
+
+Never both on the same run — whichever appeared, one more recovery boot
+before the real start got past it. That panic is Stalwart's own, and it
+suggests the store migration is not finished when the single boot exits.
+
+The likely fix is a settle boot after the apply, and it is not in yet
+because getting an extra recovery boot wrong is its own hazard — see [Do
+not boot recovery mode again
+afterwards](#do-not-boot-recovery-mode-again-afterwards). If you hit
+either, that is the manual step. Reported by
+[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
+
+## A certificate that serves HTTPS may not serve the mail ports
+
+A `Certificate` object carried into v0.16 with the right SAN is picked up
+by the HTTP listener on its own. **IMAPS, SMTPS and POP3S are not**: they
+keep serving a self-signed certificate until `defaultCertificateId` is set
+on `SystemSettings` and the server is restarted.
+
+This is the kind of thing that looks fine from a browser and surfaces as a
+mail client complaining days later, so check it as part of your
+post-migration verification: connect to 993 or 465 and confirm which
+certificate you are handed, not just to 443.
+
+This tool does not set it for you. Like the `AcmeProvider` above, it
+should, and the supplemental plan is where it belongs. Reported by
+[@kaya-eu](https://github.com/LINUXexpert-org/stalwart-migrator/issues/1).
+
 ## You need a named admin account before you migrate
 
 **A config-file fallback admin will not survive the migration.** If the only