Replace the sandbox dry run with a read-only rehearsal

`run --dry-run` cloned the data directory into a sandbox, migrated the copy,
booted it, and compared content before and after. Running that design
against a real 0.15.5 instance and a real production settings corpus
retired it:

  * The mechanics were never the risk. Backup, dump, convert and the
    recovery-mode store migration all worked essentially first time.
  * Its final comparison cannot work at all. It needs the migrated sandbox
    to answer an API, and server.listener is not among the settings
    migrate_v016.py carries - so a migrated instance has no listeners and
    answers on nothing. That is the true post-migration state, not a
    sandbox artifact to engineer around.
  * The expensive half bought the least: against a 3.6 GB production store
    it copies the data twice, reading a live mail store, to prove RocksDB
    files copy and recovery mode can open them.

Meanwhile the cheap half found every problem that would have derailed a
real migration - an empty defaultHostname v0.16 rejects, passwords v0.16
refuses to create, and a 12,182-key reconstruction worklist - and needs no
data copy at all.

So `stalwart-migrate rehearse`: preflight, dump, convert, report. It copies
nothing, starts no server, and never writes to the store, so it is safe to
run against production repeatedly without a maintenance window. It needs no
target binary either, since convert is pure Python.

The scratch directory is cleaned up as before, with the rehearsal's two
conclusions lifted out first and recorded as artifacts: export.json (what
will carry over) and unmigrated.txt (what will not). Recording an artifact
whose path was about to be deleted was a bug in the first cut of this;
both now resolve.

`run` keeps its refusal and explains where rehearse went. `--dry-run` is
kept as a flag purely to say what replaced it.

Verified against the smoke VM end to end: rehearsal completes read-only in
seconds and reports 3505 unmigrated settings on a default install,
listeners included.
This commit is contained in:
2026-08-23 20:50:40 -07:00
parent b88724632c
commit a0f846a31b
5 changed files with 487 additions and 381 deletions
+161 -113
View File
@@ -90,7 +90,7 @@ smoke test, not a full migration.
┌─────────────┐ ┌───────────┐ ┌────────────┐ ┌───────────────┐ ┌────────────┐ ┌────────────┐ ┌─────────────┐ ┌───────────┐ ┌────────────┐ ┌───────────────┐ ┌────────────┐ ┌────────────┐
│ PREFLIGHT │──▶│ BACKUP │──▶│ STAGE NEW │──▶│ RECOVERY-MODE │──▶│ CUTOVER │──▶│ VALIDATE │ │ PREFLIGHT │──▶│ BACKUP │──▶│ STAGE NEW │──▶│ RECOVERY-MODE │──▶│ CUTOVER │──▶│ VALIDATE │
│ (checks, │ │ (defense │ │ BINARY + │ │ MIGRATE │ │ (swap, up, │ │ (functional│ │ (checks, │ │ (defense │ │ BINARY + │ │ MIGRATE │ │ (swap, up, │ │ (functional│
dry-run) │ │ in depth) │ │ config │ │ (apply plan) │ │ smoke) │ │ + counts) │ rehearse) │ │ in depth) │ │ config │ │ (apply plan) │ │ smoke) │ │ + counts) │
└─────────────┘ └───────────┘ └────────────┘ └───────────────┘ └────────────┘ └────────────┘ └─────────────┘ └───────────┘ └────────────┘ └───────────────┘ └────────────┘ └────────────┘
│ │ │ │ │ │ │ │ │ │ │ │
└─────────────────┴────────────────┴─── on failure ──┴─────────────────┴──▶ STOP + REPORT └─────────────────┴────────────────┴─── on failure ──┴─────────────────┴──▶ STOP + REPORT
@@ -197,9 +197,17 @@ pre-migration instance is still up, rather than after it isn't.
rules, auth backend config — by diffing the old effective config against rules, auth backend config — by diffing the old effective config against
the new schema and emitting a best-effort JMAP object set for the new schema and emitting a best-effort JMAP object set for
`stalwart-cli apply`. This is flagged clearly as best-effort and included `stalwart-cli apply`. This is flagged clearly as best-effort and included
in the final report for manual review; it's the one part of the in the final report for manual review; silently getting it wrong (rather
documented procedure that's explicitly manual today, and silently getting than flagging it) would be worse than not attempting it.
it wrong (rather than flagging it) would be worse than not attempting it.
**This is no longer an optional enhancement.** Measured against a real
production instance, `migrate_v016.py` migrated 219 of 12,401 settings —
1.8% — leaving 12,182 for the operator to recreate by hand, including
`server.listener`. A migrated instance therefore serves nothing until
somebody rebuilds its listeners, whatever else went right. Something has
to generate that plan; the only question is whether it is this tool,
reviewably, or a human under time pressure during a cutover window.
`unmigrated.txt` (§4.9) is the input it should be built from.
- Stage new systemd unit / Compose file changes without activating them. - Stage new systemd unit / Compose file changes without activating them.
### 4.4 Recovery-mode migration ### 4.4 Recovery-mode migration
@@ -295,6 +303,21 @@ code path, so it doesn't rot independently.
### 4.7 Post-migration validation ### 4.7 Post-migration validation
**What this suite can assert depends on the boundary being crossed, and on
the 0.15/0.16 boundary it is less than this section originally claimed.**
Stalwart 0.15.x reports no per-mailbox message counts at any endpoint, and
the impersonation login 0.16 offers returns 401 there, so there are no
"before" counts to compare against — the before/after message-count
comparison is simply unavailable for the migration this tool exists to
perform. Both versions report per-account used quota, which is captured on
both sides, but §4.5 notes the migration resets quotas to zero pending
recalculation, so it is recorded rather than asserted on. What remains
checkable across the boundary is that every account and every domain
survived, and the reports say so in those words rather than implying a
no-data-loss guarantee that was not measured. See
`internal/validate/content_integrity.go`.
Runs automatically after cutover; failure here stops the run, reports Runs automatically after cutover; failure here stops the run, reports
loudly, and exits non-zero, leaving the operator to decide what to restore loudly, and exits non-zero, leaving the operator to decide what to restore
(§4.8). (§4.8).
@@ -389,66 +412,78 @@ the run and reports, and a human decides what to restore. Both are
deliberate trades for not shipping a recovery path that has never been deliberate trades for not shipping a recovery path that has never been
tested against a real server. tested against a real server.
### 4.9 Dry run ### 4.9 Rehearsal (was: dry run)
`stalwart-migrate run --dry-run` runs the real migration mechanics against a **This section was rewritten after running the previous design against a
disposable sandbox clone of the data, so an operator can get genuine real 0.15.5 instance and a real production settings corpus. What it found
confidence *before* committing to a real cutover — not a simulation that inverted the design's assumptions, so the reasoning is recorded here rather
skips the fragile parts, the actual recovery-mode migration (§4.4) and a than quietly replaced.**
post-migration boot check, just pointed somewhere disposable:
1. **Preflight** (§4.1) runs for real, read-only, against the live instance. The original dry run existed to answer *"will the migration mechanics
2. **Backup** (§4.2) runs for real too, with one exception: work?"* — it cloned the data, ran the real recovery-mode migration against
`SkipBinaryPreservation` is set, so the production binary at the real the clone, booted the result, and compared content before and after. Three
install path is never moved aside. Taking a *consistent* filesystem findings retire that design:
snapshot of an embedded store still means the live service should be
stopped first (the same requirement Stalwart's own export tooling has) —
this tool doesn't automate that stop/start today (no systemd/Docker
control exists yet), so a dry-run without a manual stop first is a
best-effort snapshot of a live, in-use store, and the CLI says so.
3. **Convert**: `migrate_v016.py convert` turns the settings/principals dump
into `config.json` + `export.json`, using the script's own documented
`--patch-paths <old>=<new>` flag to point the generated config at the
sandbox data directory instead of the real one. This is the officially
documented mechanism for exactly this kind of path redirection — the tool
deliberately does not try to rewrite `config.json`'s contents itself,
since depending on its exact schema (which has already changed once,
0.15 → 0.16) is a correctness risk this tool avoids wherever an official
alternative exists.
4. The verified backup copy is cloned again into the sandbox directory
(never reusing the same directory recovery mode is about to mutate as the
one a manual restore would use).
5. **Recovery-mode migration** (§4.4) runs for real against the sandbox:
the actual target binary, actual `STALWART_RECOVERY_MODE=1` boot, actual
`stalwart-cli apply`.
6. **Boot check + content integrity**: the migrated sandbox is started once
more as an ordinary boot (no recovery-mode env vars) and polled until its
HTTP listener answers, confirming the migrated store doesn't just accept
a settings apply but actually comes up cleanly afterward. If preflight
captured a pre-migration snapshot (§4.1, requires `--admin-url`), the
same boot is then used to capture a fresh post-migration snapshot and
compare the two — this is the actual no-data-loss guarantee, not just
"the mechanics ran": every account and mailbox from before must still be
found afterward (matching by exact name, falling back to the part before
`@` since v0.16's own migration rewrites bare usernames to full email
addresses) with an identical message count. A mismatch or a missing
account fails the check. This covers the message-count half of §4.7's
full suite; DKIM/TLS fingerprint checks and a live mail-flow SMTP→IMAP
smoke test are still open.
7. Every byte written by steps 26 (the fs-backup copy, settings/principals
dumps, downloaded `migrate_v016.py`, sandbox clone, and generated
`config.json`/`export.json`) lives under one per-run directory
(`work-dir/<run-id>`), which is removed on *every* exit path - success,
a failed check partway through, or an early refusal - via a deferred
cleanup, not just the happy path. The only thing left behind afterward
is the checkpoint's `state.json` under `--state-dir`: a small structured
success/failure log (which check failed and why), not bulk data.
`--keep-artifacts` opts out for inspecting a failure. Nothing at the real
binary path, the real service, or the real data directory's *contents*
is ever mutated by steps 26 in the first place.
A same-boundary patch bump (§4.6) has no recovery phase to simulate — dry 1. **The mechanics were never the risk.** Backup, settings dump, convert
run for that plan is just preflight + backup. and the recovery-mode store migration all worked essentially first time
against real software. The failures were everywhere else.
2. **The final comparison cannot work, at all.** It needs the migrated
sandbox to answer an API. `server.listener` is not among the settings
`migrate_v016.py` migrates, so a migrated instance has no listeners and
answers on nothing. That is not a sandbox artifact to engineer around —
it is the true post-migration state.
3. **The expensive half buys the least.** Against a 3.6 GB production store
the old flow copies the data twice (backup + sandbox clone, ~11 GB and a
long wait) while reading a live mail store, to prove that RocksDB files
copy correctly and that recovery mode can open them. Real, but modest.
Meanwhile the cheap half — dump, convert, and report what did *not* convert
— is what caught every problem that would have derailed a real migration:
an empty `defaultHostname` that v0.16 rejects, accounts whose passwords
v0.16 refuses to create, and a reconstruction worklist of 12,182 settings.
It needs no data copy at all.
So the phase reduces to the half that earns its cost:
**`stalwart-migrate rehearse`.** Run preflight (§4.1, read-only), dump
settings and principals from the live instance, run `migrate_v016.py
convert`, and report:
- the generated `export.json` plan (what *will* carry over), and
- `unmigrated.txt` (what will *not*, grouped and counted — see §4.3).
That is the whole phase. It copies no data, clones nothing, starts no
server, and never writes to the store — so it is safe to run against
production repeatedly, early and often, without a maintenance window. It
answers the question that actually decides a migration plan: *what will I
have to rebuild by hand, and does my configuration convert at all?*
The sandbox is gone. Cloning the store to run a migration against the copy
proved only that the store migrates and opens — which is worth something,
but not the disk and the wait, and not the risk of reading a live store to
get it. Where that assurance is wanted, rehearse the whole thing on a
throwaway VM restored from a backup, which is what the smoke environment
already does and does better.
Consequences worth stating, since they make this phase much cheaper than
its predecessor:
- **No target binary is needed.** `convert` is pure Python; nothing in
this phase executes a Stalwart binary of either version.
- **No disk headroom is needed.** Nothing is copied. Preflight's
free-space check still runs, but it is anticipating the backup a real
`run` will take, not anything rehearse does — and it says so.
- **Rehearsal performs no content-integrity comparison.** §4.7 explains
why that is unavailable on this boundary regardless of how it is staged.
Artifacts live under `work-dir/<run-id>` and are removed on every exit path
unless `--keep-artifacts` is passed — with one deliberate exception.
`unmigrated.txt` is the operator's reconstruction worklist and is preserved
and checksummed even on a clean run, because deleting it would throw away
the most useful output of the whole exercise.
A same-boundary patch bump (§4.6) needs no settings conversion at all, so
`rehearse` for that plan reports that there is nothing to rehearse.
## 5. State machine / checkpointing ## 5. State machine / checkpointing
@@ -471,21 +506,22 @@ is the same problem.
``` ```
stalwart-migrate preflight [--config PATH] ... # read-only, prints the report stalwart-migrate preflight [--config PATH] ... # read-only, prints the report
stalwart-migrate run --dry-run [--target-binary PATH] ... # implemented — see §4.9 stalwart-migrate rehearse [--keep-artifacts] ... # read-only; see §4.9
[--keep-artifacts] stalwart-migrate run (refused today — see §8)
(without --dry-run: refused today — see §8)
stalwart-migrate status [run-id] # implemented stalwart-migrate status [run-id] # implemented
stalwart-migrate report <run-id> [--json] # not yet implemented stalwart-migrate report <run-id> [--json] # not yet implemented
``` ```
`run` is the only command that mutates anything, and it always starts with `run` is the only command that mutates anything: `rehearse` reads the live
preflight. Nothing in this tool restores a failed migration (§4.8), so there instance and writes only inside its own work directory. `run` always starts
with preflight. Nothing in this tool restores a failed migration (§4.8), so there
is no `rollback` command, and no `confirm` step to close a rollback window is no `rollback` command, and no `confirm` step to close a rollback window
that no longer exists. that no longer exists.
The migration-time artifacts a run leaves behind — the preserved old binary, The migration-time artifacts a run leaves behind — the preserved old binary,
the settings and principals dumps, the preserved service definition, and the settings and principals dumps, the preserved service definition, and
(for the dry-run path) the filesystem copy — are never pruned automatically. and rehearsal's converted plan and unmigrated worklist — are never pruned
automatically.
They're small next to the data directory, they're what a manual restore They're small next to the data directory, they're what a manual restore
reaches for first, and deleting them on a schedule to reclaim disk would be reaches for first, and deleting them on a schedule to reclaim disk would be
the tool making a call that isn't its to make. Flags shown here are the the tool making a call that isn't its to make. Flags shown here are the
@@ -512,10 +548,10 @@ stalwart-migrator/
There's no separate `internal/stage` package: the `convert` half of There's no separate `internal/stage` package: the `convert` half of
`migrate_v016.py` lives in `internal/backup` next to `dump` (same script, `migrate_v016.py` lives in `internal/backup` next to `dump` (same script,
same invocation pattern), and the dry-run sandbox-cloning logic that stands same invocation pattern). The sandbox-cloning logic that used to stand in
in for the rest of §4.3 currently lives directly in `cmd/run.go` rather than for the rest of §4.3 lived directly in `cmd/run.go` and goes away with the
its own package. Now that `internal/cutover` exists, that's the code a real sandbox (§4.9); what §4.3 still needs is the apply-plan generator, which
staging phase would be generalized out of. has no code yet at all.
There's no `internal/rollback` either, and that's a deliberate removal There's no `internal/rollback` either, and that's a deliberate removal
rather than a gap — see §4.8. rather than a gap — see §4.8.
@@ -541,12 +577,13 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
script is Stalwart's, not ours — need a policy for what happens when it script is Stalwart's, not ours — need a policy for what happens when it
changes upstream (re-vendor + re-test before bumping the pin, never changes upstream (re-vendor + re-test before bumping the pin, never
silently float to `main`). silently float to `main`).
- **Best-effort settings apply-plan (§4.3)**: needs real-world testing - **Settings apply-plan (§4.3): now the critical path, not an enhancement.**
against a variety of existing SMTP/routing/spam configs before it's Measured against production, `migrate_v016.py` carries 1.8% of the
trusted un-reviewed; v1 should probably always require operator sign-off settings; `server.listener` is not among them, so a migrated instance
on that specific generated plan even with `--yes` set for everything else. answers on no ports until the rest is rebuilt. Building this from
Not started — dry-run currently only replays what `migrate_v016.py` `unmigrated.txt` is what would make both a meaningful rehearsal and a
itself converts. working cutover possible. It should still require explicit operator
sign-off even with `--yes` set for everything else.
- **Account/mailbox enumeration** (`stalwartapi.Client.AccountSnapshot`): - **Account/mailbox enumeration** (`stalwartapi.Client.AccountSnapshot`):
**implemented**, including per-mailbox message counts. Account count and **implemented**, including per-mailbox message counts. Account count and
domains come from `x:Account/query` + `x:Account/get` against Stalwart's domains come from `x:Account/query` + `x:Account/get` against Stalwart's
@@ -579,40 +616,43 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
the current released version actually exposes — a reminder that "read the the current released version actually exposes — a reminder that "read the
source" and "read what's actually shipped" can disagree, and it's worth source" and "read what's actually shipped" can disagree, and it's worth
checking both before changing already-working code on the strength of one. checking both before changing already-working code on the strength of one.
Preflight now populates `RunState.PreflightSnapshot.MailboxCounts` when **Superseded in part.** That mailbox-count comparison was verified only
`--admin-url` is set, and `validate.BootCheck` now compares it against a against fabricated fixtures, and against a real 0.15.5 source it does not
fresh post-migration snapshot as part of the same boot (§4.9 step 6) — work at all: 0.15.x reports no per-mailbox counts and refuses the
proven end-to-end with a live smoke test that deliberately made the impersonation login, so the "before" side is always empty, and the
"after" instance report fewer messages than the "before" snapshot and comparison used to iterate that empty map and report "all message counts
confirmed the dry run failed loudly with the exact before/after counts, match" — a vacuous pass on the strongest claim this tool makes. It now
rather than just trusting that. **Still open**: preflight/validate always states plainly when counts were not compared (§4.7). Account and domain
attempt every account serially with no sampling/threshold, which could be enumeration against 0.15.x works via its REST principal API.
slow on a large install — `--full-validation`'s sampling idea from §4.7 **Still open**: preflight/validate attempt every account serially with no
hasn't been built yet for this; and DKIM/TLS fingerprint checks plus a sampling, which could be slow on a large install; and DKIM/TLS
live mail-flow SMTP→IMAP smoke test (the rest of §4.7's suite) aren't fingerprint checks plus a live mail-flow SMTP→IMAP smoke test (the rest
implemented. With this done, recovery, backup, dry-run, and account/ of §4.7's suite) aren't implemented.
mailbox snapshotting all work end-to-end, and dry-run's comparison is now
the closest thing to §4.7's actual no-data-loss guarantee this tool has —
the remaining major gap is §4.3 staging and the production pipeline
(below).
- **Cutover is built; nothing wires it into a production run yet.** - **Cutover is built; nothing wires it into a production run yet.**
`internal/cutover` (§4.5) and `internal/service` are implemented and `internal/cutover` (§4.5) and `internal/service` are implemented and
tested. `run` without `--dry-run` still refuses, for one remaining tested against fakes. `run` still refuses, for one remaining reason:
reason: **§4.3 stage doesn't exist**, and neither does the production **§4.3 stage doesn't exist**, and neither does the production pipeline
pipeline that would run preflight → backup → stage → recovery-mode → that would run preflight → backup → stage → recovery-mode → cutover →
cutover → validate against real paths instead of a sandbox. What stage validate against real paths. What stage still needs: downloading and
still needs: downloading and verifying the target binary into a staging verifying the target binary into a staging path
path (`preflight.ResolveRelease` and `backup.DownloadFile` between them (`preflight.ResolveRelease` and `backup.DownloadFile` between them
already have the pieces), running the convert step against real paths already have the pieces), running convert against real paths, and the
rather than the dry-run's patched sandbox ones, and the best-effort settings apply-plan — which, per the measurement above, is what decides
settings apply-plan, which is its own open question below. whether the migrated server serves anything at all.
- **Nothing has ever run against a real Stalwart.** Every test in this - **What has and hasn't been proven against real software.** A smoke VM
repo drives fake `systemctl`, `psql` and `stalwart` binaries and (Debian 13, real Stalwart 0.15.5 under a real systemd unit, RocksDB,
httptest servers. That's sound for logic and ordering and is not seeded accounts and mail) has now exercised preflight, backup, the
evidence about production. One smoke test on a throwaway VM - real settings dump, `migrate_v016.py` convert, and the recovery-mode store
0.15.5, real systemd unit, a few accounts with mail - would settle the migration end to end - and a scrubbed copy of a production settings
quota wire format, systemd drop-in handling, and cutover's unit rewrite corpus has been through the converter. Everything in §4.9's rewrite and
at once. It should happen before §4.3 is wired, not after. most of §8's newer entries came from that, not from reading code.
Still unproven against real software: **cutover** (never executed - its
unit rewrite, service control and quota recalculation are tested only
against fakes), the **`x:Task` quota wire format**, **systemd drop-in**
handling, and anything on a **non-RocksDB backend** or a **Docker**
deployment. Cutover is the gap that matters most, since it is the phase
that mutates production.
- **Quota recalculation is grounded but unproven.** The `x:Task` wire - **Quota recalculation is grounded but unproven.** The `x:Task` wire
format comes from Stalwart's schema reference rather than a live server; format comes from Stalwart's schema reference rather than a live server;
§4.5 lists exactly which two details are inferred. A smoke test against a §4.5 lists exactly which two details are inferred. A smoke test against a
@@ -630,10 +670,18 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
them and refusing. them and refusing.
- **Nothing prevents concurrent runs.** Two invocations against the same - **Nothing prevents concurrent runs.** Two invocations against the same
run-id would both proceed; there's no lock file or equivalent. run-id would both proceed; there's no lock file or equivalent.
- **Dry-run's un-stopped backup snapshot** (§4.9 step 2): a dry-run still - **`rehearse` (§4.9) is designed but not built.** The command is still
backs up a live, in-use store unless the operator stops it manually first. `run --dry-run` with the old sandbox-cloning shape. Building it is mostly
`internal/service` now makes doing this properly possible - dry-run just deletion: the dump, convert and report pieces already exist and work
hasn't been wired to offer it yet. against real instances; what goes away is the backup clone, the sandbox,
the recovery-mode cycle and the boot check. `cmd/run.go`'s sandbox logic
disappears with it, which also removes the reason §7 gives for there
being no `internal/stage` package.
- **Post-migration validation has no reachable instance to validate**
(§4.7/§4.9). Until the apply-plan reconstructs listeners, nothing that
boots from a converted store can answer an API, so "did the migration
preserve the data" cannot be asked of the migrated instance at all. This
is the strongest argument for building the apply-plan first.
## Sources ## Sources
+40 -12
View File
@@ -19,17 +19,20 @@ them into a production run yet, so `run` still refuses.
| Command | State | | Command | State |
|---|---| |---|---|
| `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan | | `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan |
| `stalwart-migrate run --dry-run` | **Works**preflight, real backup, sandboxed trial conversion | | `stalwart-migrate rehearse` | **Works**read-only; converts your settings and reports what won't carry over |
| `stalwart-migrate run` | **Refuses on purpose** — see below | | `stalwart-migrate run` | **Refuses on purpose** — see below |
| `stalwart-migrate status <id>` | **Works** | | `stalwart-migrate status <id>` | **Works** |
| `stalwart-migrate report <id>` | Not implemented | | `stalwart-migrate report <id>` | Not implemented |
**`run` without `--dry-run` deliberately refuses to proceed.** Cutover **`run` deliberately refuses to proceed.** Cutover (ARCHITECTURE.md §4.5) is
(ARCHITECTURE.md §4.5) is implemented and tested, but nothing calls it: the implemented, but nothing calls it: the staging phase (§4.3) and the pipeline
staging phase (§4.3) and the production pipeline that would run preflight that would run preflight → backup → stage → recovery-mode → cutover
backup → stage → recovery-mode → cutover → validate against real paths don't validate against real paths don't exist yet. `run` stops rather than going
exist yet. `run` stops rather than going partway. That refusal is the correct partway. That refusal is the correct behaviour today, not a bug.
behaviour today, not a bug.
**Start with `rehearse` instead.** It is read-only, needs no maintenance
window, and answers the question that actually shapes a migration plan —
see below.
Package state: Package state:
@@ -91,11 +94,36 @@ pre-created writable `/var/lib/stalwart-migrator`. (`run` takes `--work-dir`
for its scratch space, but that is a different directory and does not move for its scratch space, but that is a different directory and does not move
the checkpoint store.) the checkpoint store.)
`run --dry-run` performs a **real backup**, which touches the live data `rehearse` is the next step, and unlike everything else here it is worth
directory — read the caveat the command prints before using it on anything running today — see the next section.
you care about. Where the plan crosses the 0.15/0.16 boundary it clones that
verified backup into a disposable sandbox and converts the copy, leaving the ## Rehearse before you migrate
original untouched.
```sh
stalwart-migrate rehearse --admin-url https://mail.example.com \
--admin-user admin --target 0.16.14
```
It runs preflight, dumps your settings and principals, converts them with
Stalwart's own `migrate_v016.py`, and reports **both halves** of the result:
the apply plan of what will carry over, and the worklist of what will not.
It copies no data, clones nothing, starts no server, and never writes to the
store, so it is safe to run against production repeatedly and without a
maintenance window.
Expect the worklist to be long. Measured against a real production instance,
`migrate_v016.py` carried **219 of 12,401 settings — 1.8%**. The rest,
including `server.listener`, has to be rebuilt by hand; until it is, a
migrated instance answers on no ports at all. Both outputs are preserved
under `<state-dir>/runs/<run-id>/` (`export.json` and `unmigrated.txt`) even
though the rest of the scratch directory is cleaned up, because they are the
conclusions.
This replaced an earlier `run --dry-run` that cloned the data directory into
a sandbox and migrated the copy. That proved the store opens, at the cost of
copying it twice — while the half that found every real problem needed no
copy at all. ARCHITECTURE.md §4.9 has the reasoning.
## Recovery is your job ## Recovery is your job
+4 -1
View File
@@ -23,6 +23,8 @@ func main() {
switch os.Args[1] { switch os.Args[1] {
case "preflight": case "preflight":
err = runPreflight(os.Args[2:]) err = runPreflight(os.Args[2:])
case "rehearse":
err = runRehearse(os.Args[2:])
case "run": case "run":
err = runRun(os.Args[2:]) err = runRun(os.Args[2:])
case "status": case "status":
@@ -45,7 +47,8 @@ func usage() {
commands: commands:
preflight run read-only checks and print the migration plan preflight run read-only checks and print the migration plan
run --dry-run: simulate and validate against a sandbox (real cutover isn't implemented yet) rehearse convert this instance's settings and report what will NOT carry over (read-only)
run perform the migration (not implemented yet - refuses)
status show the state of an in-progress or completed run status show the state of an in-progress or completed run
report print the validation report for a run`) report print the validation report for a run`)
} }
+246
View File
@@ -0,0 +1,246 @@
// SPDX-FileCopyrightText: 2026 LINUXexpert-org
// SPDX-License-Identifier: GPL-3.0-or-later
package main
import (
"context"
"flag"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
"github.com/LINUXexpert-org/stalwart-migrator/internal/plan"
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
)
// runRehearse implements `stalwart-migrate rehearse` (ARCHITECTURE.md §4.9):
// preflight, dump this instance's settings and principals, convert them with
// migrate_v016.py, and report both halves of the result - what will carry
// over, and what will not.
//
// It copies no data, clones nothing, starts no server, and never writes to
// the store, so it is safe to run against production repeatedly and without
// a maintenance window. That is a deliberate narrowing from the sandbox-
// cloning dry run this replaces: cloning the store proved only that the
// store migrates and opens, at the cost of copying the data twice and
// reading a live store to do it, while the half that found every real
// problem - an empty defaultHostname v0.16 rejects, passwords v0.16 refuses,
// and a 12,182-key reconstruction worklist - needs no copy at all.
//
// The worklist is the point. Measured against a real production instance,
// migrate_v016.py carried 219 of 12,401 settings; server.listener was not
// among them, so a migrated instance answers on no ports until an operator
// rebuilds them. Anything that reported such a migration as a success
// without saying so would be actively misleading.
func runRehearse(args []string) (err error) {
fs := flag.NewFlagSet("rehearse", flag.ExitOnError)
binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
configPath := fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory (read-only here; used by preflight's checks)")
containerName := fs.String("container", "stalwart", "docker container name, if applicable")
adminURL := fs.String("admin-url", "", "base URL for the live instance's admin/JMAP API (required)")
adminUser := fs.String("admin-user", "", "admin username")
adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory for the dumps and converted plan (cleaned up afterward - see --keep-artifacts)")
pythonPath := fs.String("python", "python3", "path to python3")
migrationScriptSHA256 := fs.String("migration-script-sha256", "", "pinned sha256 of migrate_v016.py (recommended; the first run prints the hash to pin)")
minFree := fs.Float64("min-free-multiple", 2.0, "free-space multiple preflight checks for; rehearsal itself copies nothing")
keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward")
if err := fs.Parse(args); err != nil {
return err
}
if *adminURL == "" {
return fmt.Errorf("--admin-url is required: rehearsal converts the settings this instance actually has, which means reading them from it")
}
ctx := context.Background()
httpClient := &http.Client{}
if err := os.MkdirAll(*workDir, 0o750); err != nil {
return fmt.Errorf("create work dir %s: %w", *workDir, err)
}
store := checkpoint.NewStore(*stateDir)
rs, err := store.Create("", *targetVersion)
if err != nil {
return fmt.Errorf("create run: %w", err)
}
fmt.Printf("run id: %s\n\n", rs.RunID)
runWorkDir := filepath.Join(*workDir, rs.RunID)
runStateDir := filepath.Join(*stateDir, rs.RunID)
// The rehearsal's two conclusions survive cleanup: the worklist of what
// won't carry over, and the plan of what will. Everything else in the
// work directory is scratch. Recording an artifact that points into a
// directory about to be deleted would leave the checkpoint referring to
// files that aren't there.
keptWorklist := filepath.Join(runStateDir, "unmigrated.txt")
keptPlan := filepath.Join(runStateDir, "export.json")
defer func() {
if _, statErr := os.Stat(runWorkDir); os.IsNotExist(statErr) {
return
}
if *keepArtifacts {
fmt.Printf("\nartifacts kept at %s (--keep-artifacts)\n", runWorkDir)
return
}
if rmErr := os.RemoveAll(runWorkDir); rmErr != nil {
fmt.Fprintf(os.Stderr, "\nwarning: failed to clean up %s: %v (remove it manually)\n", runWorkDir, rmErr)
return
}
fmt.Printf("\ncleaned up %s; the run log is at %s\n", runWorkDir, filepath.Join(runStateDir, "state.json"))
}()
fmt.Println("--- preflight ---")
checker := preflight.New(preflight.Options{
BinaryPath: *binaryPath, ConfigPath: *configPath, DataDir: *dataDir, ContainerName: *containerName,
AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
TargetVersion: *targetVersion, MinFreeMultiple: *minFree, HTTPClient: httpClient,
})
pfReport, err := checker.Run(ctx, store, rs)
fmt.Print(pfReport.String())
if err != nil {
return fmt.Errorf("preflight failed to complete: %w", err)
}
if pfReport.Blocking() {
return fmt.Errorf("preflight found blocking issues - see FAIL lines above")
}
p, err := plan.Decide(rs.SourceVersion, rs.TargetVersion)
if err != nil {
return fmt.Errorf("plan: %w", err)
}
fmt.Printf("\nplan: %s\n", p.Reason)
if !p.CrossesMajorBoundary {
fmt.Println("\nthis is a same-boundary patch upgrade: its settings don't need converting, " +
"so there is nothing to rehearse. A real run would be a binary swap and restart.")
return nil
}
scriptDest := filepath.Join(runWorkDir, "migrate_v016.py")
settingsPath := filepath.Join(runWorkDir, "settings.json")
principalsPath := filepath.Join(runWorkDir, "principals.json")
convertedConfig := filepath.Join(runWorkDir, "config.json")
convertedExport := filepath.Join(runWorkDir, "export.json")
unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
fmt.Println("\n--- dump (read-only) ---")
if _, err := store.RunStep(rs, checkpoint.PhaseBackup, "settings-dump", func() (checkpoint.StepOutcome, error) {
if err := os.MkdirAll(runWorkDir, 0o750); err != nil {
return checkpoint.StepOutcome{}, err
}
sum, err := backup.DownloadFile(ctx, httpClient, backup.DefaultMigrationScriptURL, scriptDest, *migrationScriptSHA256)
if err != nil {
return checkpoint.StepOutcome{}, err
}
pinNote := ""
if *migrationScriptSHA256 == "" {
pinNote = fmt.Sprintf(" (no pin configured - record sha256 %s as --migration-script-sha256 to pin it)", sum)
}
if err := backup.RunSettingsDump(ctx, backup.SettingsDumpOptions{
PythonPath: *pythonPath, ScriptPath: scriptDest, URL: *adminURL,
Username: *adminUser, Password: *adminPassword,
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
}); err != nil {
return checkpoint.StepOutcome{}, err
}
_, settingsSize, err := backup.HashFile(settingsPath)
if err != nil {
return checkpoint.StepOutcome{}, err
}
_, principalsSize, err := backup.HashFile(principalsPath)
if err != nil {
return checkpoint.StepOutcome{}, err
}
return checkpoint.StepOutcome{
Detail: fmt.Sprintf("dumped settings (%d bytes) and principals (%d bytes) from %s%s",
settingsSize, principalsSize, *adminURL, pinNote),
}, nil
}); err != nil {
return fmt.Errorf("settings dump: %w", err)
}
fmt.Println(rs.Outcome(checkpoint.PhaseBackup, "settings-dump").Detail)
fmt.Println("\n--- convert ---")
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "convert-settings", func() (checkpoint.StepOutcome, error) {
if err := backup.RunSettingsConvert(ctx, backup.SettingsConvertOptions{
PythonPath: *pythonPath, ScriptPath: scriptDest,
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
ConfigPath: convertedConfig, OutputPath: convertedExport,
// migrate_v016.py writes unmigrated.txt into its working
// directory; without this it lands wherever the operator
// happened to be, or fails the convert if that isn't writable.
WorkDir: runWorkDir,
}); err != nil {
return checkpoint.StepOutcome{}, err
}
detail := "converted this instance's settings into a v0.16 apply plan"
if report, readErr := backup.ReadUnmigratedReport(unmigratedPath); readErr == nil && report != nil {
detail += fmt.Sprintf("; %d setting(s) will NOT carry over", report.TotalKeys)
}
return checkpoint.StepOutcome{Detail: detail}, nil
}); err != nil {
return fmt.Errorf("convert settings: %w", err)
}
if err := copyFile(convertedExport, keptPlan); err != nil {
fmt.Fprintf(os.Stderr, "warning: couldn't preserve the apply plan: %v\n", err)
} else if sum, size, hashErr := backup.HashFile(keptPlan); hashErr == nil {
rs.RecordArtifact("converted-export", checkpoint.Artifact{Path: keptPlan, SHA256: sum, SizeBytes: size})
fmt.Printf("apply plan: %s (%d bytes) - what WILL carry over\n", keptPlan, size)
}
unmigrated, err := backup.ReadUnmigratedReport(unmigratedPath)
if err != nil {
fmt.Fprintf(os.Stderr, "warning: couldn't read the unmigrated-settings report: %v\n", err)
} else if unmigrated != nil && unmigrated.TotalKeys > 0 {
if err := copyFile(unmigratedPath, keptWorklist); err != nil {
fmt.Fprintf(os.Stderr, "warning: couldn't preserve the worklist: %v\n", err)
} else {
unmigrated.Path = keptWorklist
if sum, size, hashErr := backup.HashFile(keptWorklist); hashErr == nil {
rs.RecordArtifact("unmigrated-settings", checkpoint.Artifact{Path: keptWorklist, SHA256: sum, SizeBytes: size})
}
}
fmt.Printf("\n !! %s\n", unmigrated.Summary(10))
fmt.Println(" These do not carry over. Rebuild them on the migrated instance before it serves mail -")
fmt.Println(" note that server.listener is typically among them, so until you do, it answers on nothing.")
}
if err := store.Save(rs); err != nil {
return fmt.Errorf("save run state: %w", err)
}
fmt.Printf("\nREHEARSAL COMPLETE for run %s. Nothing was modified: no data was copied, no server was started,\n"+
"and the store was never written to.\n", rs.RunID)
return nil
}
// copyFile duplicates src to dst, used to lift the worklist out of the
// scratch directory before it's cleaned up.
func copyFile(src, dst string) error {
if err := os.MkdirAll(filepath.Dir(dst), 0o750); err != nil {
return err
}
in, err := os.Open(src)
if err != nil {
return err
}
defer in.Close()
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o640)
if err != nil {
return err
}
defer out.Close()
if _, err := io.Copy(out, in); err != nil {
return err
}
return out.Sync()
}
+36 -255
View File
@@ -4,275 +4,56 @@
package main package main
import ( import (
"context"
"flag" "flag"
"fmt" "fmt"
"net/http"
"os" "os"
"path/filepath"
"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint" "github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
"github.com/LINUXexpert-org/stalwart-migrator/internal/plan"
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
"github.com/LINUXexpert-org/stalwart-migrator/internal/recovery"
"github.com/LINUXexpert-org/stalwart-migrator/internal/validate"
) )
// runRun implements `stalwart-migrate run`. Only --dry-run is available // runRun implements `stalwart-migrate run`, which refuses.
// today. internal/cutover exists, so the gap is no longer a phase but the //
// pipeline around it: §4.3 staging, and the wiring that would drive // It refuses because §4.3 staging and the pipeline that would drive
// preflight -> backup -> stage -> recovery-mode -> cutover -> validate // preflight -> backup -> stage -> recovery-mode -> cutover -> validate
// against real paths rather than a sandbox. See ARCHITECTURE.md §8. `run` // against real paths don't exist. The phases themselves mostly do:
// without --dry-run refuses rather than doing a migration partway. // internal/preflight, internal/backup, internal/recovery and
// internal/cutover are all implemented, and preflight, backup, the settings
// dump, convert and the recovery-mode store migration have been exercised
// against a real Stalwart 0.15.5. Cutover has not - it has never run
// outside its own tests, and it is the phase that mutates production.
// //
// --dry-run runs preflight and a real backup (see the caveat printed below // What used to live here was `--dry-run`, which cloned the store into a
// about why backup still touches the live data directory), then - if the // sandbox and migrated the copy. That is now `stalwart-migrate rehearse`,
// plan crosses the 0.15/0.16 boundary - clones the verified backup into a // minus the cloning: see ARCHITECTURE.md §4.9 for why the expensive half
// disposable sandbox, converts the settings snapshot to point at that // was dropped rather than fixed.
// sandbox (via migrate_v016.py's own documented --patch-paths mechanism, func runRun(args []string) error {
// not by this tool guessing at config.json's schema), runs the real
// recovery-mode migration against the sandbox, and boots the result
// normally to confirm it comes up. Nothing at the real binary path or the
// real service is ever touched.
//
// Every byte a dry run writes - the fs-backup copy, the settings/principals
// dumps, the downloaded migrate_v016.py, the sandbox clone and its
// config/export files - lives under one per-run directory
// (work-dir/<run-id>) that a deferred cleanup at the bottom of this
// function removes on every exit path: success, a failed check partway
// through, or an early refusal. The only thing left behind afterward is the
// checkpoint's state.json under --state-dir, which is exactly the
// success/failure log a rerun's `status <run-id>` reads - not bulk data.
// --keep-artifacts opts out, for when a failure needs inspecting.
func runRun(args []string) (err error) {
fs := flag.NewFlagSet("run", flag.ExitOnError) fs := flag.NewFlagSet("run", flag.ExitOnError)
binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary") fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
targetBinaryPath := fs.String("target-binary", "", "path to an already-downloaded target-version stalwart binary (required to simulate a major-boundary migration)") fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
configPath := fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file") fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory")
dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory") fs.String("target", "latest", `target Stalwart version, or "latest"`)
containerName := fs.String("container", "stalwart", "docker container name, if applicable") fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
adminURL := fs.String("admin-url", "", "base URL for the live instance's admin/JMAP API (required)") dryRun := fs.Bool("dry-run", false, "removed - see `stalwart-migrate rehearse`")
adminUser := fs.String("admin-user", "", "admin username")
adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory for backups, dumps, and the dry-run sandbox (cleaned up afterward - see --keep-artifacts)")
stalwartCLI := fs.String("stalwart-cli", "stalwart-cli", "path to the stalwart-cli binary")
pythonPath := fs.String("python", "python3", "path to python3")
migrationScriptSHA256 := fs.String("migration-script-sha256", "", "pinned sha256 of migrate_v016.py (recommended; see preflight/backup output for the hash to pin after a first unpinned run)")
recoveryPort := fs.Int("recovery-port", 8080, "port recovery mode's HTTP listener binds, per UPGRADING/v0_16.md's own examples")
minFree := fs.Float64("min-free-multiple", 2.0, "required free disk space as a multiple of the data directory size")
dryRun := fs.Bool("dry-run", false, "simulate and validate the migration against a disposable sandbox, without touching production")
keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward (the fs-backup copy, dumps, and sandbox) - useful for inspecting a failure")
if err := fs.Parse(args); err != nil { if err := fs.Parse(args); err != nil {
return err return err
} }
if !*dryRun { if *dryRun {
return fmt.Errorf( return fmt.Errorf("--dry-run has been replaced by `stalwart-migrate rehearse`, which converts this " +
"real (non-dry-run) migrations aren't available yet: cutover is implemented (ARCHITECTURE.md §4.5), but nothing wires it " + "instance's settings and reports what will and won't carry over. It no longer clones the data " +
"into a production run - the staging phase (§4.3) and the real pipeline don't exist yet, so this command has no path " + "directory: that only proved the store opens, and cost a full copy of it to find out " +
"that touches production. Run with --dry-run to validate the migration mechanics against a disposable sandbox copy " + "(ARCHITECTURE.md §4.9)")
"of your data. Note that when a real run does land, recovery from a failed migration will be your own snapshot or " +
"backup - this tool does not undo a migration (§4.8)",
)
}
if *adminURL == "" {
return fmt.Errorf("--admin-url is required")
} }
ctx := context.Background() fmt.Fprintln(os.Stderr,
httpClient := &http.Client{} "real migrations aren't available yet: the staging phase (ARCHITECTURE.md §4.3) and the pipeline that\n"+
"would drive preflight -> backup -> stage -> recovery-mode -> cutover -> validate don't exist, so this\n"+
if err := os.MkdirAll(*workDir, 0o750); err != nil { "command has no path that touches production.\n\n"+
return fmt.Errorf("create work dir %s: %w", *workDir, err) "Two things worth knowing while you wait:\n"+
} " * `stalwart-migrate rehearse` converts your settings and reports what will NOT carry over. Measured\n"+
" against a production instance that was 98% of them, listeners included - so it decides your\n"+
store := checkpoint.NewStore(*stateDir) " migration plan, and it's safe to run now.\n"+
rs, err := store.Create("", *targetVersion) " * Recovery from a failed migration is your own snapshot or backup. This tool does not undo a\n"+
if err != nil { " migration (§4.8).")
return fmt.Errorf("create run: %w", err) return fmt.Errorf("`run` is not implemented")
}
fmt.Printf("run id: %s\n\n", rs.RunID)
runWorkDir := filepath.Join(*workDir, rs.RunID)
logPath := filepath.Join(*stateDir, rs.RunID, "state.json")
defer func() {
if _, statErr := os.Stat(runWorkDir); os.IsNotExist(statErr) {
return // nothing was ever written (e.g. refused before backup ran)
}
if *keepArtifacts {
fmt.Printf("\nartifacts kept at %s (--keep-artifacts) - remove manually when done inspecting\n", runWorkDir)
return
}
outcome := "succeeded"
if err != nil {
outcome = "failed"
}
if rmErr := os.RemoveAll(runWorkDir); rmErr != nil {
fmt.Fprintf(os.Stderr, "\nwarning: dry run %s, but failed to clean up %s: %v (remove it manually)\n", outcome, runWorkDir, rmErr)
return
}
fmt.Printf("\ndry run %s - cleaned up %s; the run log is at %s\n", outcome, runWorkDir, logPath)
}()
fmt.Println("--- preflight ---")
checker := preflight.New(preflight.Options{
BinaryPath: *binaryPath, ConfigPath: *configPath, DataDir: *dataDir, ContainerName: *containerName,
AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
TargetVersion: *targetVersion, MinFreeMultiple: *minFree, HTTPClient: httpClient,
})
pfReport, err := checker.Run(ctx, store, rs)
fmt.Print(pfReport.String())
if err != nil {
return fmt.Errorf("preflight failed to complete: %w", err)
}
if pfReport.Blocking() {
return fmt.Errorf("preflight found blocking issues - see FAIL lines above")
}
p, err := plan.Decide(rs.SourceVersion, rs.TargetVersion)
if err != nil {
return fmt.Errorf("plan: %w", err)
}
fmt.Printf("\nplan: %s\n", p.Reason)
fmt.Println("\n--- backup ---")
fmt.Println("(dry-run does not stop the live Stalwart service itself - internal/service can now do that, but dry-run " +
"isn't wired to offer it. For a guaranteed-consistent snapshot, stop stalwart before running this; otherwise the " +
"filesystem copy may reflect a live, in-use store. This is unrelated to whether production gets touched - it never does.)")
backupDir := filepath.Join(runWorkDir, "backup")
scriptDest := filepath.Join(runWorkDir, "migrate_v016.py")
settingsPath := filepath.Join(runWorkDir, "settings.json")
principalsPath := filepath.Join(runWorkDir, "principals.json")
backupOpts := backup.Options{
BinaryPath: *binaryPath,
SkipBinaryPreservation: true, // dry-run: never touch the production binary
DataDir: *dataDir,
BackupDir: backupDir,
MigrationScriptSHA256: *migrationScriptSHA256,
ScriptDestPath: scriptDest,
AdminURL: *adminURL,
AdminUser: *adminUser,
AdminPassword: *adminPassword,
SettingsDumpPath: settingsPath,
PrincipalsDumpPath: principalsPath,
PythonPath: *pythonPath,
HTTPClient: httpClient,
}
bkReport, err := backup.Run(ctx, store, rs, backupOpts)
fmt.Print(bkReport.String())
if err != nil {
return fmt.Errorf("backup failed: %w", err)
}
if !p.CrossesMajorBoundary {
fmt.Println("\nthis is a same-boundary patch upgrade: there's no recovery-mode phase to simulate. " +
"preflight and backup above are as far as a dry-run goes for this path - a real run would be a binary swap and restart.")
return nil
}
if *targetBinaryPath == "" {
return fmt.Errorf("--target-binary is required to simulate a major-boundary migration (0.15 -> 0.16 crosses one here)")
}
fmt.Println("\n--- convert (settings -> sandbox config) ---")
sandboxDataDir := filepath.Join(runWorkDir, "sandbox-data")
sandboxConfigPath := filepath.Join(runWorkDir, "sandbox-config.json")
sandboxExportPath := filepath.Join(runWorkDir, "sandbox-export.json")
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "clone-sandbox-data", func() (checkpoint.StepOutcome, error) {
manifest, err := backup.CopyDataDir(backupDir, sandboxDataDir)
if err != nil {
return checkpoint.StepOutcome{}, err
}
return checkpoint.StepOutcome{Detail: fmt.Sprintf("cloned the verified backup (%d files) into the sandbox at %s", len(manifest.Files), sandboxDataDir)}, nil
}); err != nil {
return fmt.Errorf("clone sandbox data: %w", err)
}
fmt.Printf("cloned verified backup into sandbox: %s\n", sandboxDataDir)
unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "convert-settings", func() (checkpoint.StepOutcome, error) {
if err := backup.RunSettingsConvert(ctx, backup.SettingsConvertOptions{
PythonPath: *pythonPath, ScriptPath: scriptDest,
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
ConfigPath: sandboxConfigPath, OutputPath: sandboxExportPath,
PatchPaths: map[string]string{*dataDir: sandboxDataDir},
// Without this the script writes unmigrated.txt into whatever
// directory this command was launched from - or fails outright
// if that isn't writable.
WorkDir: runWorkDir,
}); err != nil {
return checkpoint.StepOutcome{}, err
}
detail := fmt.Sprintf("generated %s and %s, patched to point at the sandbox", sandboxConfigPath, sandboxExportPath)
if report, err := backup.ReadUnmigratedReport(unmigratedPath); err == nil && report != nil && report.TotalKeys > 0 {
detail += fmt.Sprintf("; %d setting(s) were NOT migrated", report.TotalKeys)
}
return checkpoint.StepOutcome{Detail: detail, Extra: unmigratedPath}, nil
}); err != nil {
return fmt.Errorf("convert settings: %w", err)
}
fmt.Println("generated sandbox config.json and export.json")
// What the converter could NOT carry over matters more than what it
// could: against a real instance this is the overwhelming majority of
// the configuration, including the listeners, and an operator who
// doesn't read it will bring up a server that answers on no ports.
unmigrated, err := backup.ReadUnmigratedReport(unmigratedPath)
if err != nil {
fmt.Fprintf(os.Stderr, "warning: couldn't read the unmigrated-settings report: %v\n", err)
} else if unmigrated != nil && unmigrated.TotalKeys > 0 {
if sum, size, hashErr := backup.HashFile(unmigratedPath); hashErr == nil {
rs.RecordArtifact("unmigrated-settings", checkpoint.Artifact{Path: unmigratedPath, SHA256: sum, SizeBytes: size})
if saveErr := store.Save(rs); saveErr != nil {
fmt.Fprintf(os.Stderr, "warning: couldn't record the unmigrated-settings artifact: %v\n", saveErr)
}
}
fmt.Printf("\n !! %s\n", unmigrated.Summary(10))
fmt.Println(" These do not carry over. Recreate them on the migrated instance before it serves mail.")
}
fmt.Println("\n--- recovery-mode migration (against the sandbox) ---")
listenURL := fmt.Sprintf("http://127.0.0.1:%d/", *recoveryPort)
recReport, err := recovery.Run(ctx, store, rs, recovery.Options{
BinaryPath: *targetBinaryPath, ConfigPath: sandboxConfigPath, ListenURL: listenURL,
AdminUser: "admin", ApplyFiles: []string{sandboxExportPath}, CLIBinaryPath: *stalwartCLI,
HTTPClient: httpClient,
})
fmt.Print(recReport.String())
if err != nil {
return fmt.Errorf("recovery-mode migration against the sandbox failed: %w", err)
}
fmt.Println("\n--- boot check (normal boot of the migrated sandbox) ---")
if rs.PreflightSnapshot != nil {
fmt.Println("(comparing against the pre-migration account/mailbox snapshot preflight captured - " +
"this is the actual no-data-loss check, not just a reachability probe)")
} else {
fmt.Println("(no pre-migration snapshot to compare against - preflight couldn't capture one, most likely " +
"because --admin-url wasn't set; only reachability is checked)")
}
valReport, err := validate.Run(ctx, store, rs, validate.BootCheckOptions{
BinaryPath: *targetBinaryPath, ConfigPath: sandboxConfigPath, ListenURL: listenURL, HTTPClient: httpClient,
ContentIntegrityBefore: rs.PreflightSnapshot,
AdminUser: *adminUser,
AdminPassword: *adminPassword,
})
fmt.Print(valReport.String())
if err != nil {
return fmt.Errorf("post-migration validation failed: %w", err)
}
verified := "the migration mechanics succeeded"
if rs.PreflightSnapshot != nil {
verified = "the migration mechanics succeeded AND every account/mailbox message count matched before vs. after"
}
fmt.Printf("\nDRY RUN COMPLETE for run %s: %s, against a disposable sandbox copy of your data. Nothing in production was touched.\n", rs.RunID, verified)
return nil
} }