A second live attempt failed in the same shape as the first: preflight
clean, settings dumped and converted, then a failure during recovery-mode
migration with the mail server already stopped.
created Tenant (1)
created Domain (9)
create Account restore-13: invalidForeignKey | Object id: Domain#d
migrate_v016.py carries the Tenant and the Domains but emits every Account
with a null tenantId, so the account references a tenant-owned domain while
belonging to no tenant and the foreign key is rejected. That is Stalwart's
converter and there is nothing this tool can do about it: a multi-tenant
install has to be migrated by hand until the converter handles tenants.
What this tool got wrong was the timing. Tenant principals are one API call
away and were readable the entire time the server was running. Preflight now
queries them and fails before anything is touched, with an explanation of
exactly what would otherwise fail and when.
This is the same lesson as the stalwart-cli check: knowable in advance,
discovered after a production mail server had been stopped, twice. Any
dependency of the conversion belongs in preflight, not in the phase that
consumes it.
Also makes the external-tool checks advisory during `rehearse`, which never
invokes stalwart-cli - refusing to run read-only reconnaissance because the
operator lacks a tool that reconnaissance would tell them to get was
backwards.
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
Ran the rehearsal read-only against a live production instance. It
completed, and two preflight checks were wrong in ways a test instance
could not have shown.
1. Store-backend detection missed the config entirely. A config generated
by `stalwart --init` declares `type = "rocksdb"` inside a
`[store.rocksdb]` section, which is what every test fixture here used.
The production config has no section headers at all and declares
`store.rocksdb.type = "rocksdb"` flat. Detection only matched a bare
`type` key, so it reported "no known store backend type found in config"
and left topology.store_backend empty.
That mattered more than a warning suggests: backup.Run treated an
unrecognized backend as a *skip*, so a real run would have continued
with no filesystem or database backup at all - the single artifact that
phase exists to produce, quietly absent. Flat dotted keys are now
detected, and an unrecognized backend is a hard failure rather than a
skip.
2. The cluster warning didn't say where it matched. It fires on any
occurrence of "cluster" anywhere in the config, which is the correct
bias - a missed cluster corrupts a shared store - but on the production
config the only match was inside the value of an unrelated setting,
leaving a whole config to search to establish that. It now names the
location and distinguishes a match in the setting name from one in its
value.
Both verified against the real config: store-backend now reports
"rocksdb (store.rocksdb)", and the cluster warning names the setting,
making a false positive dismissible at a glance.
No production data is in this commit: the fixtures use example.com and the
flat-key shape only. Coverage numbers from that run matched the earlier
scrubbed-corpus measurement exactly.
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two
defects, the second worse than the first.
1. AccountSnapshot could not read the version this tool migrates FROM.
0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns
404 - the JMAP management API and x:Account are 0.16 features. 0.15.x
exposes a REST API at GET /api/principal instead. So preflight's
account-snapshot check warned and moved on, and every run against a real
source instance had no "before" data at all.
AccountSnapshot now dispatches on the capability the session document
advertises - a positive signal, not an inference from a failed call -
and internal/stalwartapi/principal.go implements the 0.15.x REST path,
including its 1-based page/limit pagination so an install larger than
one page isn't silently truncated.
2. With no "before" counts, the content-integrity comparison iterated an
empty map, checked nothing, and reported "all message counts match".
That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it
the actual no-data-loss guarantee - made vacuously, and it would have
passed on a migration that lost every message.
The comparison now derives its account set from whatever the source
could report, verifies every account and domain survived either way, and
carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS
NOT COMPARED ... no-data-loss is NOT verified here" rather than implying
otherwise.
What can and cannot be checked across the 0.15/0.16 boundary, now that a
real server has answered: 0.15.x has no per-mailbox message count at any
endpoint, and the impersonation login 0.16 offers returns 401 there, so
before/after message counts are impossible for the boundary migration this
tool exists for. Both versions do report per-account used quota (usedQuota
in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is
captured on both sides. It is recorded and reported, not asserted on:
4.5 notes the 0.16 migration resets quotas to zero pending recalculation,
so comparing those bytes across the boundary would be a false alarm
generator.
Test servers across preflight, validate and stalwartapi now advertise
urn:stalwart:jmap, since they stand in for 0.16 instances and that
capability is what says so.
Verified end to end against the smoke VM: all nine preflight checks pass,
and the checkpoint records 2 accounts, 1 domain and per-account used quota
where it previously recorded nothing.
GPLv3's "How to Apply These Terms" asks for a notice in each source file;
this is the modern two-line SPDX form of it rather than the full paragraph.
82 files, including tests.
The blank line after the header is load-bearing. In Go a comment block
immediately preceding `package X` becomes the package doc comment, so
without the separator the SPDX lines would be absorbed into the doc for the
eleven packages whose doc.go (or main.go) opens with one, and `go doc` would
print them. Verified it doesn't.
Rename module github.com/johnellis/stalwart-migrator ->
github.com/LINUXexpert-org/stalwart-migrator to match the repository
location, so the module resolves under `go get`.
The initial commit's .gitignore listed the compiled binary as a bare
`stalwart-migrate` pattern, which Git matches at any depth -- so it also
excluded the cmd/stalwart-migrate/ source directory, and the initial
commit shipped without the CLI entrypoint. Anchor the pattern to the
repo root as /stalwart-migrate and add the four missing files.
go build, go vet, and go test ./... all pass.
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with
checkpointed rollback and post-migration validation. Design stage; see
ARCHITECTURE.md.
jcoffey-dev