Add cutover; drop rollback in favour of operator-provided recovery

Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5)
is implemented, and the rollback phase is deleted. Recovery from a failed
migration is now explicitly the operator's own snapshot or backup, and out
of scope for this tool.

internal/cutover implements 4.5 as seven checkpointed steps: verify the
staged binary's version, install it, preserve and rewrite the service
definition, reload, start, wait for a healthy JMAP session, recalculate
quotas.

The unit is rewritten in place rather than generated from a template. An
operator's unit carries hardening options, limits and dependencies this
tool has no business having an opinion about, and regenerating it would
silently drop them. It repoints ExecStart (preserving systemd's -@:+!
prefix characters and every argument after the executable), updates
--config, and strips recovery-mode Environment lines - leaving
STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every
restart, forever. It refuses on a unit with no ExecStart, and on an
Environment line mixing a recovery variable with others: a line it only
partly understands is one it must not edit.

Quota recalculation is the one step allowed to fail without failing the
phase. Its wire format is grounded in Stalwart's x:Task schema reference -
Task/set creating one AccountMaintenance per account with maintenanceType
recalculateQuota - but the upgrade guide only documents the WebUI path, so
two details remain inferred and are called out in stalwartapi/task.go:
whether the schema's "read-only" annotation on accountId/maintenanceType
means "immutable after creation", and whether a finished task simply leaves
the queue (TaskStatus documents Pending/Retry/Failed with no success
state). Warning rather than failing is the honest response to that
uncertainty, and stale counters are an accounting problem next to calling
for a restore of a machine that is otherwise migrated and serving mail.

Docker deployments are refused outright: cutting a container over means
pulling an image and recreating it, not swapping a binary.

On removing rollback. The implementation worked and was tested, and it was
removed because restoring bytes correctly is not the hard part. It copied
file contents and permissions and verified every restored file against a
manifest - and did not preserve ownership. Run as root, as this tool
requires, it would have produced a byte-perfect, checksum-verified,
root-owned data directory that Stalwart, running as its own user, could not
open, and it would have reported success. The PostgreSQL path was worse:
pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying
into a database whose tables still exist, and the ON_ERROR_STOP=1 added so
a half-applied restore couldn't be reported as success turned that into a
hard failure. None of it had ever run against a real server. A filesystem
snapshot has none of these failure modes, because it never lost the
metadata to begin with.

So cutover's gate is no longer rollback.CanRollBack but an explicit
RecoveryPointConfirmed acknowledgement. That is an assertion, not a check -
this tool cannot verify someone else's snapshot - and its only value is
that nobody migrates a production mail server having never been asked the
question. Two consequences are accepted deliberately: restoring any
pre-migration recovery point discards mail delivered since, and a failed
migration now stops and reports rather than undoing itself.

What the tool still does to make a manual restore easier: the old binary is
preserved and never deleted, the original service definition is preserved
before the rewrite, the settings and principals dumps stay on disk, and
every artifact path and checksum stays in the checkpoint where `status
<run-id>` can print it.

Also removed: the `confirm` command stub and RollbackWindowClosed, whose
only purpose was closing a rollback window that no longer exists, and
checkpoint.PhaseRollback. Old state.json files still load - JSON ignores
the now-unknown field.

Still open, and recorded in 8: cutover ignores systemd drop-ins, so an
ExecStart or Environment override in stalwart.service.d/*.conf is invisible
to the rewrite - including the recovery variable it exists to strip;
nothing prevents concurrent runs on the same run-id; and nothing in this
repo has ever run against a real Stalwart, real systemd, or a real store.

README.md

@@ -1,17 +1,20 @@
 # stalwart-migrator
 
 In-place upgrade tool for Stalwart Mail Server, 0.15.5 → latest: no data
-loss, a checkpoint at every step so a failure can be undone, and automated
-validation that the server still works afterwards.
+loss, a checkpoint at every step so an interrupted run resumes instead of
+restarting, and automated validation that the server still works afterwards.
+
+**Recovery from a failed migration is your own snapshot or backup — this
+tool does not undo a migration.** See [Recovery is your
+job](#recovery-is-your-job) before using it on anything you care about.
 
 Go, standard library only — no external dependencies.
 
 ## Status
 
-Partially implemented. Roughly 8,400 lines of tested code across backup,
-preflight, checkpointing, validation, recovery and rollback. What's missing
-now is the real cutover phase — the step that actually swaps the binary and
-switches the live service over.
+Partially implemented. Roughly 8,800 lines of tested code. Every phase
+except staging now exists as a package, including cutover, but nothing wires
+them into a production run yet, so `run` still refuses.
 
 | Command | State |
 |---|---|
@@ -19,35 +22,28 @@ switches the live service over.
 | `stalwart-migrate run --dry-run` | **Works** — preflight, real backup, sandboxed trial conversion |
 | `stalwart-migrate run` | **Refuses on purpose** — see below |
 | `stalwart-migrate status <id>` | **Works** |
-| `stalwart-migrate rollback <id>` | **Works** — prints its plan; acts only with `--yes` |
-| `stalwart-migrate confirm <id>` | Not implemented |
 | `stalwart-migrate report <id>` | Not implemented |
 
-**`run` without `--dry-run` deliberately refuses to proceed.** Rollback and
-service control now exist, so the reason has narrowed: what's still missing
-is cutover itself (ARCHITECTURE.md §4.5) — installing the new binary,
-rewriting the service definition, and starting the migrated instance for
-real. `run` stops rather than going partway. That refusal is the correct
+**`run` without `--dry-run` deliberately refuses to proceed.** Cutover
+(ARCHITECTURE.md §4.5) is implemented and tested, but nothing calls it: the
+staging phase (§4.3) and the production pipeline that would run preflight →
+backup → stage → recovery-mode → cutover → validate against real paths don't
+exist yet. `run` stops rather than going partway. That refusal is the correct
 behaviour today, not a bug.
 
-Rollback was built before cutover on purpose: this tool should never be able
-to commit to a change it can't undo. `stalwart-migrate rollback <run-id>`
-resolves what it would do from the run's own checkpoint, prints that plan,
-and touches nothing without `--yes`.
-
 Package state:
 
 | Package | Lines | Tests |
 |---|---|---|
-| `internal/rollback` | 1807 | yes |
-| `internal/backup` | 1805 | yes |
+| `internal/backup` | 1806 | yes |
 | `internal/preflight` | 1329 | yes |
+| `internal/stalwartapi` | 1276 | yes |
+| `internal/cutover` | 1186 | yes |
 | `internal/validate` | 792 | yes |
-| `internal/stalwartapi` | 716 | yes |
 | `internal/recovery` | 702 | yes |
-| `internal/checkpoint` | 559 | yes |
+| `internal/checkpoint` | 556 | yes |
 | `internal/service` | 467 | yes |
-| `internal/plan` | 196 | yes |
+| `internal/plan` | 195 | yes |
 | `internal/config` | stub | — |
 
 ## Why not a shell script
@@ -55,8 +51,9 @@ Package state:
 Stalwart's 0.15 → 0.16 boundary is not a drop-in binary swap: settings move,
 and the data directory has to be migrated rather than merely copied. The
 failure mode that matters is a half-migrated mail store with no way back —
-which is why backup verification, checkpointing, and rollback are the design
-centre rather than conveniences bolted on afterwards.
+which is why backup verification and checkpointing are the design centre
+rather than conveniences bolted on afterwards — and why the tool refuses to
+cut over until you confirm you have a way back.
 
 [`ARCHITECTURE.md`](ARCHITECTURE.md) covers this in full: §1 on why a thin
 wrapper is insufficient, §4 on the migration phases, §5 on the checkpoint
@@ -100,38 +97,60 @@ you care about. Where the plan crosses the 0.15/0.16 boundary it clones that
 verified backup into a disposable sandbox and converts the copy, leaving the
 original untouched.
 
-## Rolling back
+## Recovery is your job
 
-`rollback` is the one command that stops a running mail server and
-overwrites a live data directory, so it never acts on its own reading of the
-situation without showing you that reading first:
+**This tool does not undo a migration.** There is no `rollback` command.
+Recovery from a failed migration is your own snapshot or backup, taken by
+whatever method you already trust and know how to restore — a ZFS, LVM or
+btrfs snapshot, a VM or volume snapshot, or a restorable backup. Choosing
+that method, taking it, and verifying you can actually restore from it is
+out of scope for this tool: it does not take one, does not check that one
+exists, and cannot restore from one.
 
-```sh
-stalwart-migrate rollback <run-id>          # prints the plan, touches nothing
-stalwart-migrate rollback <run-id> --yes    # performs it
-```
+Cutover refuses to start until you confirm a recovery point exists. That
+confirmation is an acknowledgement, not a check — nothing here can verify
+your snapshot. Its only purpose is that nobody migrates a production mail
+server having never been asked the question.
 
-Everything in the plan — which backup, which manifest, which preserved
-binary — comes from that run's own checkpoint, so rolling back days later
-doesn't depend on remembering any of it. What the checkpoint deliberately
-doesn't store (database credentials for an external SQL backend) is what the
-flags are for.
+**Take the snapshot with the service stopped** if you want a clean one. A
+snapshot of a running Stalwart is crash-consistent rather than clean; RocksDB
+will usually recover from its WAL, but "usually" is doing real work in that
+sentence.
 
-The order matters and is not negotiable: the backup is re-verified against
-its manifest **before** the service is stopped, because a corrupt backup is
-survivable while the failed instance is still up and unsurvivable once its
-data directory has been moved aside. Nothing from the failed attempt is
-deleted — the half-migrated data directory and the displaced binary are
-moved to `.failed-<run-id>` names. Afterwards a reduced validation suite runs
-against the *restored* instance (version, reachability, directory counts)
-rather than assuming the restore worked; pass `--admin-url` to get the last
-two, which are skipped without it.
+### Restoring from a snapshot loses mail delivered since
 
-Every step is checkpointed, so a rollback interrupted partway — which is
-exactly when a machine is most likely to be rebooted out from under it —
-resumes where it stopped instead of restarting a destructive sequence from
-the top. Re-running a completed rollback is inert.
+Reverting to any pre-migration recovery point discards mail delivered
+between taking it and restoring it. This is inherent to restoring a point in
+time and this tool cannot solve it — plan your migration window with that in
+mind, and consider holding inbound mail at a secondary MX for the duration
+if the gap matters to you.
 
-FoundationDB installs are refused rather than attempted: the backup phase
-only *starts* an `fdbbackup` job, and restoring one needs `fdbrestore`
-against a quiesced cluster.
+### What the tool does to make a manual restore easier
+
+- **The old binary is preserved**, never deleted, next to the new one as
+  `<binary>.v<old-version>` — so putting things back doesn't depend on
+  re-downloading a specific old release under pressure.
+- **The original service definition is preserved** as `<unit>.pre-<run-id>`
+  before cutover rewrites it, so you aren't reconstructing a unit file from
+  memory.
+- **The settings and principals dumps** taken during backup stay on disk.
+- **Every artifact path and checksum is in the checkpoint**, and
+  `stalwart-migrate status <run-id>` prints exactly which steps completed
+  and which failed — which is the first thing you want when deciding what to
+  restore.
+
+None of this is a substitute for the snapshot. It's what makes the twenty
+minutes after restoring one less unpleasant.
+
+### Why it works this way
+
+An earlier version of this tool implemented rollback itself: it restored the
+filesystem backup, verified every restored file against a manifest, replayed
+SQL dumps, reinstalled the old binary, and re-validated the result. It was
+tested and it looked good. It was removed, because restoring bytes correctly
+is not the hard part — it copied contents and permissions but not
+*ownership*, so run as root it would have produced a byte-perfect,
+checksum-verified, root-owned data directory that Stalwart, running as its
+own user, could not open, and it would have reported success. A filesystem
+snapshot has no such failure mode, because it never lost the metadata to
+begin with. ARCHITECTURE.md §4.8 records the full reasoning.