The install guide tells people to curl
.../releases/latest/download/<file>
which is a GitHub URL shape. GitLab's equivalent is
/-/releases/permalink/latest/downloads/<path>
but it only resolves for assets that declare direct_asset_path, and the
release job was creating plain links to the package registry. Those carry
the tag in the URL, so they can never be a "latest" link.
Each asset now also declares /binaries/<file>, which is what the docs will
point at. The path is load-bearing: changing it breaks a documented install
command.
Ports .github/workflows/release.yml after the GitHub account was suspended.
Same shape: tag-driven, amd64 and arm64, SOURCE_DATE_EPOCH for reproducible
tarballs, and the same refusal to release a tag that is not an ancestor of
the default branch.
The publishing half had to change. There is no `gh release`, so the
tarballs go to the project's generic package registry and the Release is
created with release-cli pointing at them. The install guide sends people
straight at release asset URLs, so uploading before creating the Release is
deliberate: a Release whose assets 404 is worse than no Release.
The Actions workflow stays in the tree as the reference.
Pushing a v* tag runs .github/workflows/release.yml: vet, test,
govulncheck, then scripts/build-release.sh builds reproducible archives
for linux/amd64 and linux/arm64 with a SHA256SUMS file, and attaches
them to the release. workflow_dispatch takes a tag for a run that never
started. Same shape as ihasmail-oneshot's releases.
Adds a version subcommand, set at build time.
go.mod moves to 1.26.8: the workflow builds with the go.mod version,
and govulncheck finds four standard-library vulnerabilities the tool
reaches in 1.26.5 (GO-2026-6218, GO-2026-6090, GO-2026-5972,
GO-2026-5026), all fixed in 1.26.6.
README installs from the latest release, with building from source as
the alternative; CONTRIBUTING describes how releases are cut.
The README keeps what the tool is, how to install it and the first commands,
and points to the guide on docs.ihasmail.org. Everything else moves, whole,
into docs/ and CONTRIBUTING.md, where it is organized for readers who want
the detail. Where the old README disagreed with the code, the code wins.
The account behind it was renamed from LINUXexpert-org to jcoffey-dev,
and GitHub does not redirect the old name: github.com/sponsors/
LINUXexpert-org answers 404 while the new one answers 200. So the
Sponsor button on this repository has been leading nowhere.
Worth fixing rather than leaving to redirect, because a released
username can be registered by anyone -- a stale link stops being a dead
end and starts being someone else's page.
The repositories moved off LINUXexpert-org. Here that is not a
documentation change: the old organisation was the module path, so it is
declared in go.mod and repeated in every internal import.
Leaving it would have been worse than a stale link. GitHub redirects the
repository, but a go.mod whose module line disagrees with the path it was
fetched from is an error rather than a redirect, so `go get` on the new
address would have failed against the old declaration.
go.mod, 34 files of imports, and the repository links in README and
ARCHITECTURE. go mod tidy leaves go.sum untouched -- no dependency moved,
only our own path.
119 SPDX-FileCopyrightText headers and the README's licence line.
The distinction that matters here: LINUXexpert-org appears in this repository
in two completely different roles. As a copyright holder in the SPDX headers,
which is what changes, and as the GitHub organisation in the module path and 64
import statements, which does not -- the repository still lives at
github.com/LINUXexpert-org/stalwart-migrator, and rewriting that would not be a
licence change, it would break the build.
Both replacements are anchored to their copyright forms, so an import path
cannot match either. Import count is 64 before and after, and go.mod is
untouched.
LICENSE untouched: the FSF's copyright on the GPL text and the "<name of
author>" placeholders are not ours to edit.
go vet, go build and go test all clean.
Preflight's first check ran `--version` on --binary. A container-only host
has no such file, so the check failed, and because it is first, nothing
downstream ever ran — including every container check that exists to
decide whether that container can be migrated at all. The container path
was unreachable on exactly the hosts it is for.
A host that happens to have a binary is the worse case, not the better
one: a stray /usr/local/bin/stalwart from an older install answers
confidently with a version nothing is running, and the whole migration
plan is derived from that number.
The source version now comes from running the image the container is on,
by ID rather than by the tag it was started from, using the same command
and the same fallback stage already uses for the target image — the two
have to agree about what a Stalwart image reports or the source and
target could be read by different rules. Verified against a real
stalwartlabs/stalwart image, not only the fake.
Deployment kind is detected once and shared, rather than asked again by
the check that reports it. Two answers for one run is not a thing this
should be able to produce.
The same reasoning retires preserve-binary on a container: there is
nothing on this host to move aside, and the equivalent is already
guaranteed, since cutover renames the old container and never prunes the
old image. Renaming a stray binary would have preserved something nothing
was running.
Three real 0.15.5 -> 0.16.19 migrations by @kaya-eu, at 221 GB and on
arm64, are worth recording — with the qualifications that make them
readable. They ran a commit predating the automated Docker cutover and did
that step by hand, so what those runs exercise is preflight, the dumps,
the conversion and the recovery-mode store migration.
Three new sections for what they found:
The store migration may need one more recovery boot. This is open. Two
failures, never both on one run, each cured by one extra recovery-mode
boot, and Stalwart's own "multi-step process" panic says the store
migration is not finished when the single boot exits. Documented as the
manual step rather than left for the next operator to rediscover, and
documented as unfixed rather than implied away.
A certificate that serves HTTPS may not serve the mail ports.
IMAPS/SMTPS/POP3S keep a self-signed certificate until
SystemSettings.defaultCertificateId is set. Written up beside the ACME
section, which has the same shape: something the converter does not carry
and nothing warns about. This tool should generate it and does not yet.
Do not boot recovery mode again afterwards already landed with the
artifacts it argues for; this links the production paragraph to it.
Two status claims were also stale. §4.5 still said the container path was
not reachable from the CLI, which #9 changed, and both it and §7 said no
real Stalwart image had been met — one has now been inspected, which is
how #11's two bugs were found. What remains untested is everything after
the recreate, and the Docker section now says what cutover carries and
how the config reaches the container.
The settings and principals dumps, the apply plan and its supplement lived
only in --work-dir, which a successful run deletes. All four are
irreplaceable once the store has been migrated: the dumps can only be
taken from a live pre-migration instance, and the plan is what was
actually replayed. `rehearse` already kept the plan and the supplement, so
the read-only command preserved more of its conclusions than the
destructive one did.
They are now copied into the run's state directory before the store is
touched, recorded as artifacts with checksums, and kept whether or not the
run succeeded and whether or not --keep-artifacts was passed. README
claimed the dumps stayed on disk; now they do.
What made this concrete: an operator who booted recovery mode again after
a completed migration, for an unrelated reason, and found Domain and
Account queries coming back empty on the next start — twice, on two
different servers, and verified as genuinely gone rather than a stale
read. Re-applying that run's export.json and supplement.json against a
fresh recovery boot is what got the server back both times, and they had
those files only because they had thought to pass --keep-artifacts.
Nobody should have to guess that in advance.
The README now says not to boot recovery mode after a migration. That is
Stalwart's behaviour rather than this tool's, but this tool is where an
operator learns the technique, and it said nothing about it being a
one-time step.
Reported by @kaya-eu in #1.
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an
ordinary container reports User "stalwart", Entrypoint
["/usr/local/bin/stalwart"] and Cmd ["--config",
"/etc/stalwart/config.json"] — all three inherited, none of them given.
Two things followed from reading those as the operator's.
A container user was listed as configuration a recreate would drop, so
every container off the official image was refused as unrecreatable. That
refusal lived in cutover, downstream of the stop, the settings conversion
and the store migration: it arrived with mail down and data already
moved, which is the failure issue #1 was filed for. Each of the three is
now compared against `docker image inspect` of the image the container is
on. Inherited values are left to the new image, whose own defaults are
the ones that go with it. Overrides are carried: --user, --entrypoint,
and the rest of an entrypoint as leading argv. Cmd and Entrypoint were
not being read at all, so an overridden one was silently dropped — the
exact loss the unsupported list exists to prevent.
The recreatability question also moved into preflight, while the server
is still running. Cutover asks it again, since the two are separated by
the whole migration, but only one of them can refuse without cost.
The other half: the recreated container is now started with `--config`
pointing at the migrated config in the data volume. Left to the image's
default command it came up on /etc/stalwart/config.json — a different
volume, holding whatever the old version left there — so cutover would
have produced a running server with nothing to do with the migration that
preceded it. An overridden command and that --config are the same argv
and cannot be merged honestly, so a container with one is refused and
told why.
The config is also chowned to whatever owns the data directory, before
the recovery cycle opens it. The image runs as uid 2000 and this tool
writes as root; §4.8 is the standing reminder that byte-perfect and
unreadable is a way to report success.
Found while checking @kaya-eu's field report in #1 against a real image.
Their three manual migrations are where the config step comes from.
The release asset was a constant naming the x86_64 Linux server build, so
an arm64 host downloaded an x86_64 binary and met "exec format error" the
first time anything executed it — which is stage's own version check, so
it failed early and before the service stopped, but with nothing in the
message to say the download had been for the wrong machine.
The asset now follows runtime.GOARCH. Only amd64 and arm64 are selected
for: those are the two with an unambiguous plain gnu server build, while
GOARCH=arm does not say whether a host wants the arm or the armv7
archive. Anything else is refused by name and pointed at --target-binary
rather than falling back to x86_64, which is the bug being fixed.
Matching stays exact — stalwart-foundationdb-aarch64-unknown-linux-gnu is
a substring away from the right answer.
Reported by @kaya-eu, who hit this on an arm64 home server and worked
around it by fetching the aarch64 archive by hand.
Everything the container migration needs has landed a piece at a time and
nothing called any of it. `run` now does: stage pulls and verifies an image
instead of downloading a binary, the recovery cycle launches a throwaway
container against the live container's own mounts, and cutover recreates it.
Preflight's blanket refusal of docker goes with it -- what still refuses is
specific to a container rather than to containers, which is compose and
data that is not on a volume.
It refuses without --container-path-unproven, and that flag is the honest
part of this change. Every test drives a fake docker. That proves the right
commands are assembled and proves nothing about whether a real image reads
the config it is handed -- which is the exact limit ARCHITECTURE.md section
4.8 records about the rollback code that was deleted for being tested only
against fakes. A doc note seemed too quiet for a tool that stops a mail
server, so it is a flag nobody reaches without being told.
The converted config reaches the container through the data volume. It is
written under the host side of whichever mount covers --data-dir and named
on the container side, because cutover recreates a container with the mounts
it had and cannot invent a new one for a config file. --data-dir therefore
names the path inside the container, which preflight already says when it
matches no mount.
PatchPaths stays unused, deliberately. Its documented purpose is pointing a
rehearsal at a sandbox; a real container's dumped settings already carry
container-side paths, because they come from the live server rather than
from a file on this host.
The preflight test that asserted docker was refused outright now asserts
the replacement rather than being deleted -- "docker is allowed through
here" is the thing that would be wrong to regress. Its fixture had to make
--data-dir both a real host directory and one the fake container mounts,
since disk-space stats it and container-data-volume wants it covered.
README gains the container section and, at the top, the note that this is
ihasmail's companion.
A container cannot be edited in place the way a unit file can, so cutting
one over means rebuilding it. That makes silent loss the default failure:
a container recreated without its capabilities, its custom network or its
device mappings starts cleanly and is quietly not the server it was.
Section 4.5 already answers this for a unit file -- it rewrites in place
rather than regenerating, because a generated unit would drop hardening
options this tool has no business having an opinion about, and it refuses
to edit a line it only partly understands. The same rule applies here,
where the whole definition has to be rebuilt: the parts this understands
are carried across, and a container using anything else is refused by name
rather than rebuilt without it. The list of what it looks for is
conservative and not exhaustive, which is the safe direction: docker's
HostConfig has far more fields, and one this does not know about is a
reason not to be recreating that container at all.
The old container is renamed, not removed, and nothing here prunes the old
image. Together they are the container's manual restore path -- one command
starts the previous container again -- which is as close to section 4.2's
preserved binary as a container gets. The inspect output is preserved as an
artifact before anything is replaced, for the reason the unit file is: an
operator putting a machine back by hand should not also be reconstructing
the definition from memory.
Recovery-mode variables are stripped from the recreated container's
environment. Leaving STALWART_RECOVERY_MODE set would recovery-boot on
every restart, which is the same footgun the unit rewrite exists to
prevent.
Run now branches, with the health check and quota recalculation shared:
those ask the same question whatever started the server. The binary path
moved inside an else and is otherwise untouched -- no existing test needed
editing, which is the evidence for that.
The container path is opt-in through Options.Container, so a Docker
deployment without it is still refused exactly as before. Nothing calls it
yet; wiring `run` up and lifting preflight's refusal is what remains of #3,
and ARCHITECTURE.md says so in both places it previously said Docker was
refused outright.
The container deployment's answer to downloading a release: pull the image
the operator named, then ask it what it is. That last part is the point of
the phase, exactly as it is for a binary - the tag, the registry and the
repository name are all assumptions about someone else's publishing
process, and the image's own answer is the only thing that settles what
arrived.
The image is never derived from the running container by swapping its tag.
That derivation is wrong for a digest-pinned image, wrong for a mirror and
wrong for a fork, and being wrong here means pulling the wrong software
into a mail server. It is named in full or the phase refuses.
What comes back is the image's ID rather than the tag it arrived under. A
tag can move between staging and cutover -- that is the whole reason latest
is a hazard -- and running the tag later would run something other than
what was verified here.
VersionFromOutput is exported from preflight so both paths parse a version
identically. Two copies of that regex could disagree about what they
staged, which is a difference nobody would look for.
One caveat recorded rather than hidden: asking an image its version means
running it with --version, which assumes its entrypoint is the server and
passes flags through. That has not been confirmed against a published
Stalwart image, there being none to hand. If the assumption is wrong this
fails loudly with the image's own output rather than staging something
unverified, and the fallback tries the binary by name before giving up.
SkipPull is for a host that loaded the image from a tarball, where a pull
cannot work and its failure would say nothing useful.
The second implementation of the launcher seam: a throwaway container from
the staged image, against the live container's own mounts, with recovery
mode's two environment variables. Everything the cycle does afterwards -
the settings apply, the health check, the boot check - is unchanged, which
was the point of putting the seam there.
Process gained an unexported start(cmd, what) so a container shares its
supervision rather than reimplementing it. For a container the child
process is the `docker run` client, so the captured output is the server's
own: a recovery boot that dies on a bind conflict says so here, where a
detached container would have left it in `docker logs` for nobody.
Stopping is the part that needed care, because the client and the container
are two things and conflating them loses a store. Signals do reach the
container through an attached client, so the ordinary path is fine - but
Process.Stop escalates to SIGKILL when the grace period expires, and
killing the client does not kill the container. It would be left running,
holding the store open, while the run moved on to the next phase believing
it had stopped. So the container is stopped by name with docker's own
timeout, and its absence confirmed afterwards; a container still running
after being told to stop is an error loud enough to halt the run, because
nothing may touch that data until it is gone.
Two refusals rather than defaults. No image, and no mounts - the second
because a recovery container with no mounts would migrate an empty
directory and report success, which is the worst outcome available here.
Eight tests drive a fake docker: the assembled run command, recovery
variables present in recovery mode and absent in an ordinary boot, both
refusals, the stop reaching the container by name, the survived-container
error, an already-removed container being the normal --rm path rather than
a failure, and output capture.
Two races in those tests are worth recording, because both were real and
neither was in the code. Launch returns once the OS has started the client,
which is before the shell it started has run anything, so reading the
invocation log or the output immediately raced them; both now poll. And a
forked `sleep` in the fake outlived its shell, held the output pipe open
and made Wait block for the sleep's full duration - `exec` in the fake
fixes it, and the same shape would affect any child that forks.
Both phases that bring the target version up against a not-yet-migrated
store -- the recovery cycle, and the ordinary boot validate does after it --
constructed a child process from a path on this host directly. That is the
one thing about them packaging changes: a container runs an image against
the data volume instead. Everything either of them is started *for* is
identical afterwards.
So starting it is now a Launcher, returning a Supervised the callers stop
and read output from. BinaryLauncher is today's behaviour and the default
when Options.Launcher is nil, so every existing caller is unchanged -- no
test needed editing, which is the evidence for that rather than a claim
about it.
Deliberately narrower than the interface sketched in issue #3. Stage and
cutover also differ by packaging, but designing their interfaces now would
be designing against a guess: there is no second implementation yet to
shape them, and the shape a container needs is what PR 3 and PR 4 find out.
This seam is different because it already had two callers doing the same
thing for the same reason, so extracting it describes the code rather than
predicting it.
outputSuffix now takes Supervised. It only ever needed Output(), and the
diagnosis it exists to preserve -- the server's own words about a bind
conflict or a rejected config value, which a bare timeout loses -- matters
whatever started the process.
Preflight refuses a container, and until now that was all it said. The two
things that actually decide whether one could be migrated at all were never
looked at, so an operator was refused without being told what to fix or what
a manual migration would involve.
It now inspects the container and reports three things. What image it is
running, tag and digest kept apart because a moved tag makes them disagree
and only one of them says what is really there. Whether compose manages it,
which matters beyond this tool's current refusal: recreating a
compose-managed container out from under compose leaves the container and
the compose file disagreeing about what is deployed, and the next
`compose up` reverts the migration. And whether the data is on a volume at
all -- a container keeping it in its own writable layer loses it when the
container is replaced, and replacing the container is what migrating it
means, so that one is fatal in a way no later phase could recover from.
All three are advisory under rehearse, on the same reasoning that already
made the deployment check advisory there: rehearse never stops or recreates
anything, and an operator doing this by hand needs these facts more than an
automated run does.
Preflight stays read-only. Preserving the container definition as an
artifact belongs with the phase that replaces it, so it goes with cutover
rather than here.
One limitation this surfaced and does not fix: --data-dir naming a path
inside the container breaks the host-side disk-space check, which stats it
locally. Path translation is the next piece of work; the data-volume check
says so when the path matches no mount.
Cutover already refused a container -- recreating one from a new image is
not swapping a binary and rewriting a unit, and this tool does not automate
it -- but it refused from cutover.Run, which run.go calls at line 324. The
service is stopped at line 243. So the sequence on a container was: stop
Stalwart, convert the settings, then discover the deployment cannot be cut
over, return the error, and exit with mail still down. Reported as #1.
The deployment kind is known in preflight, before anything has been touched,
and that is now where it is acted on: docker is a blocking check. rehearse
keeps working -- it never stops the service or cuts over, and telling an
operator what the migration involves is most useful precisely when the tool
cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the
ToolCheckAdvisory it already set for the same reason.
The second half is not docker's alone. Every return between the stop and the
end of cutover returned with the service down; a failed settings conversion
would have done the same to a systemd host. run now registers a restart on
the way out, after the stop rather than before, so it only ever starts
something this tool stopped. It does not claim to have recovered the
migration -- a part-migrated store still needs --resume or the operator's
recovery point -- it removes the narrower failure of exiting on a
foreseeable error while the server it stopped stays stopped.
Someone deciding whether to trust this needs the first fact immediately: it
has migrated a real mail server, and what that cost in downtime. It was
buried in a Status section under a line count.
The two things an operator has to fix before starting - multi-tenancy, and
not migrating as the built-in admin - now sit above everything else too.
Both are changes to the directory rather than flags, so finding them after
booking a maintenance window is the wrong order, and that is exactly how
both were found on the production run.
Status keeps the line counts and the command table and no longer repeats
the paragraph above it.
Both blockers found on the production run were changes to the directory, not
flags, and both were discovered by reading a preflight failure rather than
the documentation. They now lead the README.
Multi-tenancy had no section at all: v0.16 requires a tenant-scoped account
to sit on a domain owned by that same tenant, v0.15 did not, and an install
that is valid today can be unrepresentable tomorrow. Where one tenant's
accounts use a tenant-less domain the conversion repairs it; where two
tenants share a domain nothing can, and it has to be resolved in v0.15.
The admin section predated the preflight check that now refuses a
fallback-admin, and it stopped one step short: the account has to keep its
rights *after* the migration, not merely exist. An account whose admin came
only from `tenant-admin` authenticates afterwards and is still refused every
management operation, and preflight cannot predict that - it cannot know
which roles the converter carries across.
On 2026-08-25 the tool migrated a live mail server - nine domains, six
accounts, a 2.4 GB RocksDB store - from 0.15.5 to 0.16.19 with 8 seconds of
downtime and every phase green, validation included.
The practice worth copying is the dress rehearsal on a clone that preceded
it. `rehearse` is read-only and stops before applying, which is where a real
migration fails; a clone closes that gap for about six seconds of production
downtime. It caught four faults that would each have reached production, and
three of them only appear against a real instance. The README now describes
how to build one, including the parts that are easy to get wrong: a
consistent copy of a single-writer store, and a guest with no route out so a
copy of a live mail server cannot renew real certificates or deliver real
queued mail.
It also documents what Stalwart's own converter drops without saying so.
migrate_v016.py consumes every acme.* setting, emits nothing, and does not
report them as unmigrated - so a migration reports success while removing
certificate renewal, and nothing looks wrong until the certificate expires
about ninety days later. The AcmeProvider that had to be rebuilt by hand is
recorded in ARCHITECTURE.md, shape confirmed against a live 0.16.19, as the
specification for generating it in the supplemental plan.
The dress rehearsal reported MISSING ACCOUNT [email protected] after a
migration that lost nothing: SMTP on the migrated instance accepts mail for
it, while an address that does not exist is refused, so the account is there.
Enumeration is permission-scoped. The "before" snapshot was read by an
account with full rights; the "after" snapshot by one whose admin role the
migration had not carried across - proven one step earlier, where the quota
rebuild was refused as unauthorised. It was shown a fraction of the accounts
and the comparison called the remainder lost.
"Missing" and "not permitted to see" are different findings and only one of
them is about the data. When the migrated instance shows fewer accounts than
existed, the result now says COULD NOT VERIFY and names the reader, rather
than asserting a loss the evidence does not support. It still fails the run:
an unverified migration is not a verified one, and the fix is to re-check
with an admin account, which --resume makes cheap.
Counting accounts is not a strong enough signal to tell the two cases apart,
so it does not try to - it reports the ambiguity instead of guessing.
Also fixes the test fake, which ignored ?types= and returned domains as
individuals, making every account count in these tests wrong.
A dress rehearsal on a clone of production migrated cleanly and then failed
all three post-cutover steps with 401: the health check, the quota rebuild,
and the content comparison. The account was `admin` - a v0.15
[authentication.fallback-admin], not a directory account. v0.16 keeps its
configuration in the store, so the block defining it does not survive, and
the credential stops working the instant the migration completes.
Every check passed beforehand, because on v0.15 that account authenticates
perfectly well. The README has said to use a named account for a while; the
tool now says it too, while nothing has been touched and changing it costs
one flag.
The migration itself is unaffected - it succeeds either way. What is lost is
the ability to verify it afterwards, which is the part worth having.
jcoffey-dev
jcoffey