stalwart-migrator/docs/rehearsal.md

Trying it safely, and rehearsing

How to find out what a migration will do before it does it: running preflight safely, what rehearse reports, and how to rehearse the whole migration on a clone of your server. For the short version, see the README.

Running preflight

preflight is the sensible starting point. Its checks against the Stalwart installation are read-only:

sudo go run ./cmd/stalwart-migrate preflight

It still needs write access, because it records the run as a checkpoint before doing anything else. Without it, you see:

create run: checkpoint: create run directory:
mkdir /var/lib/stalwart-migrator: permission denied

Checkpoints go to /var/lib/stalwart-migrator/runs by default (checkpoint.DefaultBaseDir). Either run as root, pre-create that directory writable, or point --state-dir somewhere writable. run and rehearse also take --work-dir for their scratch space, which is a different directory and does not move the checkpoint store.

Pass --admin-url and --admin-user for the reachability check, and the password with --admin-password or STALWART_MIGRATE_ADMIN_PASSWORD.

What rehearse reports

stalwart-migrate rehearse --admin-url https://mail.example.com \
    --admin-user admin --target 0.16.14

It runs preflight, dumps your settings and principals, converts them with Stalwart's own migrate_v016.py, and reports both halves of the result: the apply plan of what will carry over, and the worklist of what will not.

It copies no data, clones nothing, starts no server, and never writes to the store, so it is safe to run against production repeatedly and without a maintenance window.

The supplemental plan

It also generates a supplemental plan for the part it can rebuild automatically — currently your network listeners, which is the difference between a migrated server that answers and one that doesn't — and reports exactly how much of the worklist that covers (on a test instance: 24 of 3,505 keys, and it says so rather than implying more).

run generates the same supplement and applies it after export.json for you. If you are replaying a rehearsal's plan by hand instead, review it, then apply it after export.json:

stalwart-cli apply --file <state-dir>/<run-id>/supplement.json \
    --url https://mail.example.com

It is applied after export.json rather than merged into it on purpose: the official conversion is the authority on everything it handles, and a generated plan that overlapped it could silently override a correct mapping with a guessed one.

Reading the worklist

The worklist is long but mostly not work, and rehearse says which is which. Measured against a real production instance, migrate_v016.py carried 219 of 12,401 settings. Of the 12,182 it left:

• 8,547 are runtime auto-ban state that repopulates itself
• 3,337 are stock spam-filter and lookup data v0.16 ships its own copies of — restoring v0.15's would revert a year of upstream updates
• ~224 were already carried another way, DKIM signatures included
• ~293 genuinely need your eyes

server.listener is in that third group only because this tool regenerates it for you; without that a migrated instance answers on no ports at all.

What it keeps

The conclusions are preserved under <state-dir>/<run-id>/ — /var/lib/stalwart-migrator/runs/<run-id>/ by default — as export.json, unmigrated.txt and supplement.json, even though the rest of the scratch directory is cleaned up (unless you pass --keep-artifacts).

Why it replaced a dry run

rehearse replaced an earlier run --dry-run that cloned the data directory into a sandbox and migrated the copy. That proved the store opens, at the cost of copying it twice — while the half that found every real problem needed no copy at all. ARCHITECTURE.md §4.9 has the reasoning.

Rehearse on a clone first

rehearse is read-only and stops short of the half that matters: it converts your settings but never applies them, and applying is where a real migration fails. A clone closes that gap, and on a 2.4 GB store it costs about six seconds of production downtime to build.

1. Copy the data directory with the service stopped. RocksDB is single-writer, so a hot copy may be torn — and a rehearsal on a torn store fails for reasons production never would, or passes when it should not. Stop, cp -a the data dir to the same disk, start again, archive it afterwards; the service is down only for the local copy.
2. Give the guest no route off the host. A clone of a live mail server will otherwise renew certificates for your real domains and deliver whatever is in the outbound queue. In libvirt that means a network with no <forward> element. Verify it from inside the guest rather than assuming.
3. Run the real thing: preflight, then run with --target-binary, --stalwart-cli and --migration-script pointed at locally staged copies, since an isolated guest can download nothing.
4. Snapshot the guest while it is shut down, so a failed attempt costs seconds to reset rather than a rebuild.

What it caught

None of these could be expressed by the mock:

• a multi-tenant arrangement v0.16 cannot represent;
• --target-binary never reaching preflight, so an air-gapped host failed on a release lookup;
• an admin account that was a config fallback-admin and stopped working the moment the migration finished;
• tenant-admin roles the converter does not restore.

What isolation costs

Two things the isolation costs, so they are not mistaken for faults: the guest cannot fetch the v0.16 web interface, so /account/ returns 404, and certificate renewal cannot be exercised at all.