The README keeps what the tool is, how to install it and the first commands,
and points to the guide on docs.ihasmail.org. Everything else moves, whole,
into docs/ and CONTRIBUTING.md, where it is organized for readers who want
the detail. Where the old README disagreed with the code, the code wins.
The account behind it was renamed from LINUXexpert-org to jcoffey-dev,
and GitHub does not redirect the old name: github.com/sponsors/
LINUXexpert-org answers 404 while the new one answers 200. So the
Sponsor button on this repository has been leading nowhere.
Worth fixing rather than leaving to redirect, because a released
username can be registered by anyone -- a stale link stops being a dead
end and starts being someone else's page.
The repositories moved off LINUXexpert-org. Here that is not a
documentation change: the old organisation was the module path, so it is
declared in go.mod and repeated in every internal import.
Leaving it would have been worse than a stale link. GitHub redirects the
repository, but a go.mod whose module line disagrees with the path it was
fetched from is an error rather than a redirect, so `go get` on the new
address would have failed against the old declaration.
go.mod, 34 files of imports, and the repository links in README and
ARCHITECTURE. go mod tidy leaves go.sum untouched -- no dependency moved,
only our own path.
119 SPDX-FileCopyrightText headers and the README's licence line.
The distinction that matters here: LINUXexpert-org appears in this repository
in two completely different roles. As a copyright holder in the SPDX headers,
which is what changes, and as the GitHub organisation in the module path and 64
import statements, which does not -- the repository still lives at
github.com/LINUXexpert-org/stalwart-migrator, and rewriting that would not be a
licence change, it would break the build.
Both replacements are anchored to their copyright forms, so an import path
cannot match either. Import count is 64 before and after, and go.mod is
untouched.
LICENSE untouched: the FSF's copyright on the GPL text and the "<name of
author>" placeholders are not ours to edit.
go vet, go build and go test all clean.
Preflight's first check ran `--version` on --binary. A container-only host
has no such file, so the check failed, and because it is first, nothing
downstream ever ran — including every container check that exists to
decide whether that container can be migrated at all. The container path
was unreachable on exactly the hosts it is for.
A host that happens to have a binary is the worse case, not the better
one: a stray /usr/local/bin/stalwart from an older install answers
confidently with a version nothing is running, and the whole migration
plan is derived from that number.
The source version now comes from running the image the container is on,
by ID rather than by the tag it was started from, using the same command
and the same fallback stage already uses for the target image — the two
have to agree about what a Stalwart image reports or the source and
target could be read by different rules. Verified against a real
stalwartlabs/stalwart image, not only the fake.
Deployment kind is detected once and shared, rather than asked again by
the check that reports it. Two answers for one run is not a thing this
should be able to produce.
The same reasoning retires preserve-binary on a container: there is
nothing on this host to move aside, and the equivalent is already
guaranteed, since cutover renames the old container and never prunes the
old image. Renaming a stray binary would have preserved something nothing
was running.
Three real 0.15.5 -> 0.16.19 migrations by @kaya-eu, at 221 GB and on
arm64, are worth recording — with the qualifications that make them
readable. They ran a commit predating the automated Docker cutover and did
that step by hand, so what those runs exercise is preflight, the dumps,
the conversion and the recovery-mode store migration.
Three new sections for what they found:
The store migration may need one more recovery boot. This is open. Two
failures, never both on one run, each cured by one extra recovery-mode
boot, and Stalwart's own "multi-step process" panic says the store
migration is not finished when the single boot exits. Documented as the
manual step rather than left for the next operator to rediscover, and
documented as unfixed rather than implied away.
A certificate that serves HTTPS may not serve the mail ports.
IMAPS/SMTPS/POP3S keep a self-signed certificate until
SystemSettings.defaultCertificateId is set. Written up beside the ACME
section, which has the same shape: something the converter does not carry
and nothing warns about. This tool should generate it and does not yet.
Do not boot recovery mode again afterwards already landed with the
artifacts it argues for; this links the production paragraph to it.
Two status claims were also stale. §4.5 still said the container path was
not reachable from the CLI, which #9 changed, and both it and §7 said no
real Stalwart image had been met — one has now been inspected, which is
how #11's two bugs were found. What remains untested is everything after
the recreate, and the Docker section now says what cutover carries and
how the config reaches the container.
The settings and principals dumps, the apply plan and its supplement lived
only in --work-dir, which a successful run deletes. All four are
irreplaceable once the store has been migrated: the dumps can only be
taken from a live pre-migration instance, and the plan is what was
actually replayed. `rehearse` already kept the plan and the supplement, so
the read-only command preserved more of its conclusions than the
destructive one did.
They are now copied into the run's state directory before the store is
touched, recorded as artifacts with checksums, and kept whether or not the
run succeeded and whether or not --keep-artifacts was passed. README
claimed the dumps stayed on disk; now they do.
What made this concrete: an operator who booted recovery mode again after
a completed migration, for an unrelated reason, and found Domain and
Account queries coming back empty on the next start — twice, on two
different servers, and verified as genuinely gone rather than a stale
read. Re-applying that run's export.json and supplement.json against a
fresh recovery boot is what got the server back both times, and they had
those files only because they had thought to pass --keep-artifacts.
Nobody should have to guess that in advance.
The README now says not to boot recovery mode after a migration. That is
Stalwart's behaviour rather than this tool's, but this tool is where an
operator learns the technique, and it said nothing about it being a
one-time step.
Reported by @kaya-eu in #1.
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an
ordinary container reports User "stalwart", Entrypoint
["/usr/local/bin/stalwart"] and Cmd ["--config",
"/etc/stalwart/config.json"] — all three inherited, none of them given.
Two things followed from reading those as the operator's.
A container user was listed as configuration a recreate would drop, so
every container off the official image was refused as unrecreatable. That
refusal lived in cutover, downstream of the stop, the settings conversion
and the store migration: it arrived with mail down and data already
moved, which is the failure issue #1 was filed for. Each of the three is
now compared against `docker image inspect` of the image the container is
on. Inherited values are left to the new image, whose own defaults are
the ones that go with it. Overrides are carried: --user, --entrypoint,
and the rest of an entrypoint as leading argv. Cmd and Entrypoint were
not being read at all, so an overridden one was silently dropped — the
exact loss the unsupported list exists to prevent.
The recreatability question also moved into preflight, while the server
is still running. Cutover asks it again, since the two are separated by
the whole migration, but only one of them can refuse without cost.
The other half: the recreated container is now started with `--config`
pointing at the migrated config in the data volume. Left to the image's
default command it came up on /etc/stalwart/config.json — a different
volume, holding whatever the old version left there — so cutover would
have produced a running server with nothing to do with the migration that
preceded it. An overridden command and that --config are the same argv
and cannot be merged honestly, so a container with one is refused and
told why.
The config is also chowned to whatever owns the data directory, before
the recovery cycle opens it. The image runs as uid 2000 and this tool
writes as root; §4.8 is the standing reminder that byte-perfect and
unreadable is a way to report success.
Found while checking @kaya-eu's field report in #1 against a real image.
Their three manual migrations are where the config step comes from.
The release asset was a constant naming the x86_64 Linux server build, so
an arm64 host downloaded an x86_64 binary and met "exec format error" the
first time anything executed it — which is stage's own version check, so
it failed early and before the service stopped, but with nothing in the
message to say the download had been for the wrong machine.
The asset now follows runtime.GOARCH. Only amd64 and arm64 are selected
for: those are the two with an unambiguous plain gnu server build, while
GOARCH=arm does not say whether a host wants the arm or the armv7
archive. Anything else is refused by name and pointed at --target-binary
rather than falling back to x86_64, which is the bug being fixed.
Matching stays exact — stalwart-foundationdb-aarch64-unknown-linux-gnu is
a substring away from the right answer.
Reported by @kaya-eu, who hit this on an arm64 home server and worked
around it by fetching the aarch64 archive by hand.
Everything the container migration needs has landed a piece at a time and
nothing called any of it. `run` now does: stage pulls and verifies an image
instead of downloading a binary, the recovery cycle launches a throwaway
container against the live container's own mounts, and cutover recreates it.
Preflight's blanket refusal of docker goes with it -- what still refuses is
specific to a container rather than to containers, which is compose and
data that is not on a volume.
It refuses without --container-path-unproven, and that flag is the honest
part of this change. Every test drives a fake docker. That proves the right
commands are assembled and proves nothing about whether a real image reads
the config it is handed -- which is the exact limit ARCHITECTURE.md section
4.8 records about the rollback code that was deleted for being tested only
against fakes. A doc note seemed too quiet for a tool that stops a mail
server, so it is a flag nobody reaches without being told.
The converted config reaches the container through the data volume. It is
written under the host side of whichever mount covers --data-dir and named
on the container side, because cutover recreates a container with the mounts
it had and cannot invent a new one for a config file. --data-dir therefore
names the path inside the container, which preflight already says when it
matches no mount.
PatchPaths stays unused, deliberately. Its documented purpose is pointing a
rehearsal at a sandbox; a real container's dumped settings already carry
container-side paths, because they come from the live server rather than
from a file on this host.
The preflight test that asserted docker was refused outright now asserts
the replacement rather than being deleted -- "docker is allowed through
here" is the thing that would be wrong to regress. Its fixture had to make
--data-dir both a real host directory and one the fake container mounts,
since disk-space stats it and container-data-volume wants it covered.
README gains the container section and, at the top, the note that this is
ihasmail's companion.
A container cannot be edited in place the way a unit file can, so cutting
one over means rebuilding it. That makes silent loss the default failure:
a container recreated without its capabilities, its custom network or its
device mappings starts cleanly and is quietly not the server it was.
Section 4.5 already answers this for a unit file -- it rewrites in place
rather than regenerating, because a generated unit would drop hardening
options this tool has no business having an opinion about, and it refuses
to edit a line it only partly understands. The same rule applies here,
where the whole definition has to be rebuilt: the parts this understands
are carried across, and a container using anything else is refused by name
rather than rebuilt without it. The list of what it looks for is
conservative and not exhaustive, which is the safe direction: docker's
HostConfig has far more fields, and one this does not know about is a
reason not to be recreating that container at all.
The old container is renamed, not removed, and nothing here prunes the old
image. Together they are the container's manual restore path -- one command
starts the previous container again -- which is as close to section 4.2's
preserved binary as a container gets. The inspect output is preserved as an
artifact before anything is replaced, for the reason the unit file is: an
operator putting a machine back by hand should not also be reconstructing
the definition from memory.
Recovery-mode variables are stripped from the recreated container's
environment. Leaving STALWART_RECOVERY_MODE set would recovery-boot on
every restart, which is the same footgun the unit rewrite exists to
prevent.
Run now branches, with the health check and quota recalculation shared:
those ask the same question whatever started the server. The binary path
moved inside an else and is otherwise untouched -- no existing test needed
editing, which is the evidence for that.
The container path is opt-in through Options.Container, so a Docker
deployment without it is still refused exactly as before. Nothing calls it
yet; wiring `run` up and lifting preflight's refusal is what remains of #3,
and ARCHITECTURE.md says so in both places it previously said Docker was
refused outright.
The container deployment's answer to downloading a release: pull the image
the operator named, then ask it what it is. That last part is the point of
the phase, exactly as it is for a binary - the tag, the registry and the
repository name are all assumptions about someone else's publishing
process, and the image's own answer is the only thing that settles what
arrived.
The image is never derived from the running container by swapping its tag.
That derivation is wrong for a digest-pinned image, wrong for a mirror and
wrong for a fork, and being wrong here means pulling the wrong software
into a mail server. It is named in full or the phase refuses.
What comes back is the image's ID rather than the tag it arrived under. A
tag can move between staging and cutover -- that is the whole reason latest
is a hazard -- and running the tag later would run something other than
what was verified here.
VersionFromOutput is exported from preflight so both paths parse a version
identically. Two copies of that regex could disagree about what they
staged, which is a difference nobody would look for.
One caveat recorded rather than hidden: asking an image its version means
running it with --version, which assumes its entrypoint is the server and
passes flags through. That has not been confirmed against a published
Stalwart image, there being none to hand. If the assumption is wrong this
fails loudly with the image's own output rather than staging something
unverified, and the fallback tries the binary by name before giving up.
SkipPull is for a host that loaded the image from a tarball, where a pull
cannot work and its failure would say nothing useful.
The second implementation of the launcher seam: a throwaway container from
the staged image, against the live container's own mounts, with recovery
mode's two environment variables. Everything the cycle does afterwards -
the settings apply, the health check, the boot check - is unchanged, which
was the point of putting the seam there.
Process gained an unexported start(cmd, what) so a container shares its
supervision rather than reimplementing it. For a container the child
process is the `docker run` client, so the captured output is the server's
own: a recovery boot that dies on a bind conflict says so here, where a
detached container would have left it in `docker logs` for nobody.
Stopping is the part that needed care, because the client and the container
are two things and conflating them loses a store. Signals do reach the
container through an attached client, so the ordinary path is fine - but
Process.Stop escalates to SIGKILL when the grace period expires, and
killing the client does not kill the container. It would be left running,
holding the store open, while the run moved on to the next phase believing
it had stopped. So the container is stopped by name with docker's own
timeout, and its absence confirmed afterwards; a container still running
after being told to stop is an error loud enough to halt the run, because
nothing may touch that data until it is gone.
Two refusals rather than defaults. No image, and no mounts - the second
because a recovery container with no mounts would migrate an empty
directory and report success, which is the worst outcome available here.
Eight tests drive a fake docker: the assembled run command, recovery
variables present in recovery mode and absent in an ordinary boot, both
refusals, the stop reaching the container by name, the survived-container
error, an already-removed container being the normal --rm path rather than
a failure, and output capture.
Two races in those tests are worth recording, because both were real and
neither was in the code. Launch returns once the OS has started the client,
which is before the shell it started has run anything, so reading the
invocation log or the output immediately raced them; both now poll. And a
forked `sleep` in the fake outlived its shell, held the output pipe open
and made Wait block for the sleep's full duration - `exec` in the fake
fixes it, and the same shape would affect any child that forks.
Both phases that bring the target version up against a not-yet-migrated
store -- the recovery cycle, and the ordinary boot validate does after it --
constructed a child process from a path on this host directly. That is the
one thing about them packaging changes: a container runs an image against
the data volume instead. Everything either of them is started *for* is
identical afterwards.
So starting it is now a Launcher, returning a Supervised the callers stop
and read output from. BinaryLauncher is today's behaviour and the default
when Options.Launcher is nil, so every existing caller is unchanged -- no
test needed editing, which is the evidence for that rather than a claim
about it.
Deliberately narrower than the interface sketched in issue #3. Stage and
cutover also differ by packaging, but designing their interfaces now would
be designing against a guess: there is no second implementation yet to
shape them, and the shape a container needs is what PR 3 and PR 4 find out.
This seam is different because it already had two callers doing the same
thing for the same reason, so extracting it describes the code rather than
predicting it.
outputSuffix now takes Supervised. It only ever needed Output(), and the
diagnosis it exists to preserve -- the server's own words about a bind
conflict or a rejected config value, which a bare timeout loses -- matters
whatever started the process.
Preflight refuses a container, and until now that was all it said. The two
things that actually decide whether one could be migrated at all were never
looked at, so an operator was refused without being told what to fix or what
a manual migration would involve.
It now inspects the container and reports three things. What image it is
running, tag and digest kept apart because a moved tag makes them disagree
and only one of them says what is really there. Whether compose manages it,
which matters beyond this tool's current refusal: recreating a
compose-managed container out from under compose leaves the container and
the compose file disagreeing about what is deployed, and the next
`compose up` reverts the migration. And whether the data is on a volume at
all -- a container keeping it in its own writable layer loses it when the
container is replaced, and replacing the container is what migrating it
means, so that one is fatal in a way no later phase could recover from.
All three are advisory under rehearse, on the same reasoning that already
made the deployment check advisory there: rehearse never stops or recreates
anything, and an operator doing this by hand needs these facts more than an
automated run does.
Preflight stays read-only. Preserving the container definition as an
artifact belongs with the phase that replaces it, so it goes with cutover
rather than here.
One limitation this surfaced and does not fix: --data-dir naming a path
inside the container breaks the host-side disk-space check, which stats it
locally. Path translation is the next piece of work; the data-volume check
says so when the path matches no mount.
Cutover already refused a container -- recreating one from a new image is
not swapping a binary and rewriting a unit, and this tool does not automate
it -- but it refused from cutover.Run, which run.go calls at line 324. The
service is stopped at line 243. So the sequence on a container was: stop
Stalwart, convert the settings, then discover the deployment cannot be cut
over, return the error, and exit with mail still down. Reported as #1.
The deployment kind is known in preflight, before anything has been touched,
and that is now where it is acted on: docker is a blocking check. rehearse
keeps working -- it never stops the service or cuts over, and telling an
operator what the migration involves is most useful precisely when the tool
cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the
ToolCheckAdvisory it already set for the same reason.
The second half is not docker's alone. Every return between the stop and the
end of cutover returned with the service down; a failed settings conversion
would have done the same to a systemd host. run now registers a restart on
the way out, after the stop rather than before, so it only ever starts
something this tool stopped. It does not claim to have recovered the
migration -- a part-migrated store still needs --resume or the operator's
recovery point -- it removes the narrower failure of exiting on a
foreseeable error while the server it stopped stays stopped.
Someone deciding whether to trust this needs the first fact immediately: it
has migrated a real mail server, and what that cost in downtime. It was
buried in a Status section under a line count.
The two things an operator has to fix before starting - multi-tenancy, and
not migrating as the built-in admin - now sit above everything else too.
Both are changes to the directory rather than flags, so finding them after
booking a maintenance window is the wrong order, and that is exactly how
both were found on the production run.
Status keeps the line counts and the command table and no longer repeats
the paragraph above it.
Both blockers found on the production run were changes to the directory, not
flags, and both were discovered by reading a preflight failure rather than
the documentation. They now lead the README.
Multi-tenancy had no section at all: v0.16 requires a tenant-scoped account
to sit on a domain owned by that same tenant, v0.15 did not, and an install
that is valid today can be unrepresentable tomorrow. Where one tenant's
accounts use a tenant-less domain the conversion repairs it; where two
tenants share a domain nothing can, and it has to be resolved in v0.15.
The admin section predated the preflight check that now refuses a
fallback-admin, and it stopped one step short: the account has to keep its
rights *after* the migration, not merely exist. An account whose admin came
only from `tenant-admin` authenticates afterwards and is still refused every
management operation, and preflight cannot predict that - it cannot know
which roles the converter carries across.
On 2026-08-25 the tool migrated a live mail server - nine domains, six
accounts, a 2.4 GB RocksDB store - from 0.15.5 to 0.16.19 with 8 seconds of
downtime and every phase green, validation included.
The practice worth copying is the dress rehearsal on a clone that preceded
it. `rehearse` is read-only and stops before applying, which is where a real
migration fails; a clone closes that gap for about six seconds of production
downtime. It caught four faults that would each have reached production, and
three of them only appear against a real instance. The README now describes
how to build one, including the parts that are easy to get wrong: a
consistent copy of a single-writer store, and a guest with no route out so a
copy of a live mail server cannot renew real certificates or deliver real
queued mail.
It also documents what Stalwart's own converter drops without saying so.
migrate_v016.py consumes every acme.* setting, emits nothing, and does not
report them as unmigrated - so a migration reports success while removing
certificate renewal, and nothing looks wrong until the certificate expires
about ninety days later. The AcmeProvider that had to be rebuilt by hand is
recorded in ARCHITECTURE.md, shape confirmed against a live 0.16.19, as the
specification for generating it in the supplemental plan.
The dress rehearsal reported MISSING ACCOUNT [email protected] after a
migration that lost nothing: SMTP on the migrated instance accepts mail for
it, while an address that does not exist is refused, so the account is there.
Enumeration is permission-scoped. The "before" snapshot was read by an
account with full rights; the "after" snapshot by one whose admin role the
migration had not carried across - proven one step earlier, where the quota
rebuild was refused as unauthorised. It was shown a fraction of the accounts
and the comparison called the remainder lost.
"Missing" and "not permitted to see" are different findings and only one of
them is about the data. When the migrated instance shows fewer accounts than
existed, the result now says COULD NOT VERIFY and names the reader, rather
than asserting a loss the evidence does not support. It still fails the run:
an unverified migration is not a verified one, and the fix is to re-check
with an admin account, which --resume makes cheap.
Counting accounts is not a strong enough signal to tell the two cases apart,
so it does not try to - it reports the ambiguity instead of guessing.
Also fixes the test fake, which ignored ?types= and returned domains as
individuals, making every account count in these tests wrong.
A dress rehearsal on a clone of production migrated cleanly and then failed
all three post-cutover steps with 401: the health check, the quota rebuild,
and the content comparison. The account was `admin` - a v0.15
[authentication.fallback-admin], not a directory account. v0.16 keeps its
configuration in the store, so the block defining it does not survive, and
the credential stops working the instant the migration completes.
Every check passed beforehand, because on v0.15 that account authenticates
perfectly well. The README has said to use a named account for a while; the
tool now says it too, while nothing has been touched and changing it costs
one flag.
The migration itself is unaffected - it succeeds either way. What is lost is
the ability to verify it afterwards, which is the part worth having.
preflight refuses a layout v0.16 cannot represent, but it reports the problem
rather than the shape of the instance around it. Choosing between giving a
tenant its own domains, moving accounts, or collapsing everything into one
tenant needs the whole picture: which domains have an owner, which accounts
reach across to somebody else's, and which of those the converter would have
repaired by itself.
Read-only, and it names the offending domain against each account that
touches it rather than making the reader join the two lists by eye.
preflight asked the GitHub release API which version it was upgrading to,
so a host with no internet failed there even with the target binary already
on disk - and `run` never passed its own --target-binary down to preflight,
so supplying one did not help. Read the version from the binary instead
when one is given, and fail if it is not the version the run targets.
Found by the first preflight against the production clone, which is
deliberately cut off from the network.
`run` and `rehearse` always fetched migrate_v016.py from GitHub, so a mail
server with no route out could not be migrated at all - an air-gapped host,
or a clone deliberately cut off so it cannot renew certificates or deliver
queued mail for the domains it was copied from. --migration-script takes a
local copy instead, still checked against --migration-script-sha256 when one
is pinned.
Found while staging a production clone for a dress rehearsal: the clone has
no route out on purpose, and that is exactly the property that stops a copy
of a live mail server doing something in the real world.
The post-migration comparison had the two versions counting domains
differently, and yesterday's wiring turned that into a gate: `run` would have
failed a migration that lost nothing.
The 0.15 side added every domain appearing in any account's address on top of
the domain principals - the fallback's own comment says "if the instance has
no explicit domain principals", but the loop ran unconditionally. The 0.16
side did the reverse, listing only domains some account calls its primary,
discarding the full Domain list it had already fetched. An instance with
three declared domains and accounts aliased across nine reported nine before
and three after. INBUXA is exactly that shape, and this was the account/domain
over-count noted as undiagnosed.
Both sides now mean "the domains this server holds". A domain that still goes
missing is reported as a warning rather than failing the run: what the two
versions call a domain differs across this boundary in ways we have now been
caught by once, and a missing account - which is compared with a local-part
fallback and is what actually matters - still fails.
Narrowing OK() also made String() return before printing the domain lines,
so the new warning would have been silent. Caught by its own test.
internal/validate was written and tested and then never called: `run` ended
at cutover, so the tool performed a migration and never confirmed it had
carried the data across, and `report` was an error message pointing at the
package that would have answered.
`run` now compares the migrated instance against the snapshot preflight took
and fails if an account or a domain that existed before is missing from it.
The comparison runs against the service cutover has just started, which is
the instance people will actually use - its real config, its real ports,
under its real service manager - and costs no extra downtime; booting a
second copy inside the maintenance window would. BootCheck stays as the
equivalent for an instance the tool boots itself.
The service is left running on a failure. By that point the store has been
migrated in place, so stopping it undoes nothing, and only the operator can
weigh the finding against their recovery point.
A check that could not run is reported as skipped, never as a pass. Preflight
only captures the "before" when it has an admin URL, and a run without one
has to say it compared nothing rather than imply everything survived - which
is the exact failure ARCHITECTURE.md §4.7 warns about. `report <run-id>`
re-reads the recorded verdict rather than re-checking: run again next week
and you would be asking how the instance looks now, not how it looked when
it was migrated.
§4.7 said validation ran after cutover while the only implementation booted
its own copy, and listed a suite far larger than what exists. It now says
which of the two happens, and which checks are real.
Two places told an operator that `run` is unimplemented and refuses: the
usage text, and the first line of the README's status. Both predate `run`
being wired up, and the README contradicted itself three lines later, where
the command table says it works. Anyone reading either would conclude the
tool cannot migrate.
The rest of that section had drifted too: staging exists as a package, and
the package table was missing it along with applyplan, while the line counts
had aged. Recounted, and the column now says what it is measuring.
`report` really is unimplemented, so the usage text now says so where it is
advertised.
The second production attempt failed during recovery-mode apply, with the
mail server already stopped and the store already at schema v6:
create Account restore-13: invalidForeignKey | Object id: Domain#d
v0.16 requires a tenant-scoped Account to sit on a Domain owned by that
same tenant, for its primary domain and for every alias. v0.15 imposed no
such rule, and migrate_v016.py carries the two facts over independently:
_build_domains sets a domain's memberTenantId only for domains declared as
their own `domain` principal with a `tenant`, while _build_user sets the
account's from the account's own record. A domain that exists only inside
an email address is inferred, gets no tenant, and every tenant-scoped
account using it is then rejected.
Established by reproduction rather than inference: a synthetic v0.15
principal dump, run through the unpatched upstream converter and applied to
a real 0.16.14 in recovery mode, reproduces the error character for
character - the `#d` is the server's own object id for the offending
domain, not a plan client-id. The same harness establishes which directions
are constrained: a tenant-scoped account on a tenant-less domain or on
another tenant's domain is rejected; a global account on a tenant-owned
domain is accepted.
- applyplan.ReconcileDomainTenants repairs the plan between convert and
apply. Where a tenant-less domain is used only by accounts of one
tenant, the domain adopts that tenant - the sole assignment that both
applies and keeps every account. Where accounts genuinely disagree it
changes nothing and reports why, because forcing such a plan through
would mean dropping mailboxes.
- stalwartapi.FetchTenantLayout maps tenant membership over the 0.15 REST
API and predicts the outcome with the same rule the server enforces, so
preflight either warns about the domains that will adopt a tenant or
fails - while the service is still running.
- The plan is parsed generically rather than through the typed Operation.
A real export.json mixes shapes: `create` maps a client-id to an object,
`update` carries a flat one. The typed form failed on the first `update`
line, found by running against actual converter output. Numbers decode
as json.Number so a 10 GiB quota is not rewritten as 1.073741824e+10.
Corrects the record: the previous commit claimed the converter emits every
Account with `tenantId: null` and made preflight refuse every multi-tenant
install on that basis. The field is memberTenantId, the converter does
populate it, and the export had been inspected for a key no version of the
script ever writes. The refusal is now narrowed to what v0.16 genuinely
cannot represent.
The same fix has been prepared for migrate_v016.py upstream. The tool
downloads that script rather than vendoring it, so the repair stays here
until a released version carries it, and is a no-op on a consistent plan.
A second live attempt failed in the same shape as the first: preflight
clean, settings dumped and converted, then a failure during recovery-mode
migration with the mail server already stopped.
created Tenant (1)
created Domain (9)
create Account restore-13: invalidForeignKey | Object id: Domain#d
migrate_v016.py carries the Tenant and the Domains but emits every Account
with a null tenantId, so the account references a tenant-owned domain while
belonging to no tenant and the foreign key is rejected. That is Stalwart's
converter and there is nothing this tool can do about it: a multi-tenant
install has to be migrated by hand until the converter handles tenants.
What this tool got wrong was the timing. Tenant principals are one API call
away and were readable the entire time the server was running. Preflight now
queries them and fails before anything is touched, with an explanation of
exactly what would otherwise fail and when.
This is the same lesson as the stalwart-cli check: knowable in advance,
discovered after a production mail server had been stopped, twice. Any
dependency of the conversion belongs in preflight, not in the phase that
consumes it.
Also makes the external-tool checks advisory during `rehearse`, which never
invokes stalwart-cli - refusing to run read-only reconnaissance because the
operator lacks a tool that reconnaissance would tell them to get was
backwards.
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
The phases have all existed for a while; nothing chained them. The order
here is the one arrived at by performing this migration by hand against a
clone of production before writing it down:
preflight -> stage -> dump -> preserve binary -> STOP ->
convert -> supplement -> recovery-mode migration -> cutover -> START
The dump runs before the stop because it reads settings over the admin API,
and a stopped server has no admin API. Everything from the stop to the end
of cutover is downtime.
internal/stage fills the last missing phase (4.3): resolve the release,
take the x86_64 linux-gnu server build and refuse to substitute another,
verify a pinned checksum if one was given, extract the binary - refusing
any archive entry that isn't a regular file, since a tarball is untrusted
input - and confirm the result reports the version its tag claimed.
Everything upstream of that last check is an assumption about someone
else's release process.
Two gates, separate on purpose. --yes is about intent. --recovery-point-
confirmed is a claim about the world: this tool cannot undo a migration
(4.8) and cannot check whether a snapshot exists, so a run that proceeded
without the operator asserting one would be proceeding on a hope.
Verified end to end against a real Stalwart 0.15.5 with email-style account
names, a named admin account, and seeded mail:
MIGRATION COMPLETE. Mail was down for 6s.
Every cutover step green, including recalculate-quotas ("rebuilt disk
quotas for 2 account(s)") - the first time the x:Task wire format inferred
from Stalwart's schema reference has actually been exercised. It works,
now that endpoint discovery and role restoration make it reachable. After
the migration the named admin still administers, alice logs in with
unchanged credentials to the same four messages, and new SMTP delivery is
accepted.
Both refusal gates were tested, as was the failure path: an apply that
fails leaves the run stopped with the store part-migrated, and the error
says to restore the recovery point rather than restart the old version
against it.
A config-file fallback admin does not survive the migration, and an operator
whose only administrator is one comes out of a successful migration unable
to administer the server. Three verified reasons, all found on real
migrations: v0.16 keeps its config in the store so the old
[authentication.fallback-admin] block stops being read at all;
migrate_v016.py assigns every migrated account the User role regardless of
what it held, so even a directory admin needs its role restored; and the
restoration is refused when an account's local part is ambiguous across
domains, since granting administrator rights to the wrong account is worse
than granting them to none.
jcoffey-dev
jcoffey