1684c88877a68e55dd5812ac90851d38bc24817f
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
1684c88877 |
Resolve v0.16 domain ids to names before comparing directories
x:Account.domainId is an internal id on v0.16 ("b"), not a domain name. A
pre-migration snapshot taken from a v0.15 instance records names
("smoke.test"), so the post-migration directory comparison compared ids
against names and would have reported every domain as having vanished -
a false alarm on the check whose whole job is proving nothing was lost.
The client now resolves them with x:Domain/query + x:Domain/get in a single
request, using a JMAP back-reference (RFC 8620 3.7). Confirmed against a
live 0.16.14 before being written:
["x:Domain/get", {"list":[{"name":"smoke.test","id":"b"}]}, "g"]
An id that can't be resolved is kept as-is - a domain that can't be named is
still a domain that exists - but a failure of the resolution call itself is
an error rather than a silent fallback, since quietly comparing ids against
names is precisely the bug being fixed.
Verified against the live migrated instance: the snapshot that reported
domains=[b] now reports domains=[smoke.test], matching what the
pre-migration snapshot recorded.
|
||
|
jcoffey-dev
|
28c0fa57cb |
Restore administrator roles that migrate_v016.py drops
Chased down why a migrated instance had no working administrator. The
account authenticated fine and was refused every management call, and the
cause is that migrate_v016.py assigns every migrated account the User role
regardless of what it held before: an account that was `roles: ["admin"]`
in v0.15 comes out the far side as `roles: {"@type": "User"}`.
Ordinary users were never affected - User is what they had and what they
get - and their credentials, mail and mailboxes survive untouched. It is
specifically administrators who lose their privileges, which is a bad thing
to discover after cutting over.
The v0.16 shape came from the server's own schema document rather than the
published reference: GET /api/schema defines x:UserRoles as a multi-variant
type with variants User, Admin and Custom. Account is itself multi-variant,
so an upsert needs its own "@type" too - without it the server rejects the
operation outright ("upsert entry is missing `@type`").
applyplan.AccountRoleOperations restores roles from the principals dump,
emitting operations only for accounts whose role actually changes.
Rewriting every account would be a much larger blast radius for no benefit.
Where v0.15 listed several roles, admin wins - under-privileging an
administrator locks them out, which is the failure being fixed - and the
collapse is reported rather than done silently, as are roles with no known
v0.16 equivalent.
Verified end to end on the smoke VM: rehearse against the real 0.15.5 put
the role operation in the supplement, applying that supplement to a
migrated 0.16.14 whose admin was broken restored management access
(accounts=3), and alice and bob logged in over IMAPS with unchanged
credentials, read their mail, and accepted new SMTP delivery.
Also recorded: x:Account.domainId returns an internal id on v0.16, not a
domain name, so the post-migration directory comparison would read every
domain as missing. Resolving that needs an x:Domain/get call not yet
confirmed against the binary.
|
||
|
jcoffey-dev
|
3e155fa42c |
Fix three defects a full VM migration exposed
Ran a complete 0.15.5 -> 0.16.14 migration of the smoke VM, driving the
phases in the order the real pipeline will. It worked - all mail intact and
readable afterwards, all ten listeners up, cutover executed for the first
time ever and checkpoint resume exercised - and it exposed three defects.
1. The converted config was installed root-owned while the service runs as
its own user. Stalwart crash-looped 28 times on "Failed to read data
store settings: Permission denied", minutes after the mistake and
nowhere near it. This is the same ownership trap that retired the
rollback implementation, in a new place: writing files as root is the
natural thing for a tool running as root to do, and it is wrong every
time the service is not root.
Cutover now installs the config itself, copying ownership and mode from
the config being replaced.
2. v0.16.14 does not serve /api - the endpoint stalwartapi assumed.
Confirmed against a fully migrated, fully configured, serving instance
rather than a sandbox: /api, /api/principal and /jmap/ all 404. The JMAP
endpoint is the one the session document advertises, which is what RFC
8620 discovery is for.
The client now discovers it, re-basing the advertised path onto the
operator's host: a real instance advertises its canonical public URL
("https://mail.smoke.test/jmap/") which frequently isn't reachable from
where this tool runs. The session is authoritative about the path; the
operator is authoritative about the host.
3. Dispatching on the urn:stalwart:jmap capability was wrong, because
NEITHER version advertises it - not 0.15.5, and not a fully migrated
0.16.14. That sent 0.16 instances down the 0.15 REST path where every
call 404s. The client probes what the instance actually serves instead.
Less elegant than a declared capability, with the advantage of being
true.
Also: a JMAP "forbidden" now explains itself. An account holding the admin
role before the migration was refused x:Account/query afterwards, and a
bare "forbidden" gives an operator nowhere to start. Whether the role
failed to carry or v0.16 wants different permissions was not isolated, and
that question is recorded as open - it gates quota recalculation and any
post-migration validation.
Verified against both live instances: the 0.15.5 reports 3 accounts and its
domain over REST, and the migrated 0.16.14 routes to JMAP, finds the right
endpoint, and returns the explained refusal.
|
||
|
jcoffey-dev
|
3bd694114f |
Generate a v0.16 apply plan for the listeners migrate_v016.py leaves behind
First piece of ARCHITECTURE.md 4.3's apply-plan, and the piece that decides
whether a migrated server answers at all: server.listener is not among the
settings the official converter carries, so a freshly migrated instance
binds nothing. Every other unmigrated setting degrades the server; this one
stops it being a server.
internal/applyplan maps server.listener.* onto x:NetworkListener objects and
reports its own coverage. Against the smoke instance that is 24 of 3,505
unmigrated keys - 0.7% - and the output says 0.7%, listing the largest
groups it did not touch. A plan covering a fraction while implying
completeness would be worse than no plan.
The wire format was confirmed against the binary, not the documentation.
The published schema reference gives NetworkListener.bind as a JSON array;
0.16.14 rejects that outright ("Invalid value for object property.
Properties: bind"). The encoding it accepts is a value-keyed set,
{"[::]:25": true}, found by applying a plan to a live recovery-mode 0.16.14
and reading it back with `stalwart-cli snapshot`. Only mappings confirmed
that way are in DefaultGenerators; managesieve -> manageSieve is the one
protocol whose spelling changes, and an unrecognized protocol is reported
and skipped rather than passed through to fail at apply time.
Operations are upserts matched on name, so a plan can be re-run - an
operator will run it more than once - and the supplement is applied after
export.json rather than merged into it, so a generated mapping can never
override one the official script got right.
Verified end to end: rehearse against a real 0.15.5 generated ten
listeners, `stalwart-cli apply` created all ten on a real 0.16.14 with zero
failures, a snapshot read them back with correct protocols, binds and TLS
flags, and re-applying reported 10 updated / 0 created / 0 failed.
|
||
|
jcoffey-dev
|
a0f846a31b |
Replace the sandbox dry run with a read-only rehearsal
`run --dry-run` cloned the data directory into a sandbox, migrated the copy,
booted it, and compared content before and after. Running that design
against a real 0.15.5 instance and a real production settings corpus
retired it:
* The mechanics were never the risk. Backup, dump, convert and the
recovery-mode store migration all worked essentially first time.
* Its final comparison cannot work at all. It needs the migrated sandbox
to answer an API, and server.listener is not among the settings
migrate_v016.py carries - so a migrated instance has no listeners and
answers on nothing. That is the true post-migration state, not a
sandbox artifact to engineer around.
* The expensive half bought the least: against a 3.6 GB production store
it copies the data twice, reading a live mail store, to prove RocksDB
files copy and recovery mode can open them.
Meanwhile the cheap half found every problem that would have derailed a
real migration - an empty defaultHostname v0.16 rejects, passwords v0.16
refuses to create, and a 12,182-key reconstruction worklist - and needs no
data copy at all.
So `stalwart-migrate rehearse`: preflight, dump, convert, report. It copies
nothing, starts no server, and never writes to the store, so it is safe to
run against production repeatedly without a maintenance window. It needs no
target binary either, since convert is pure Python.
The scratch directory is cleaned up as before, with the rehearsal's two
conclusions lifted out first and recorded as artifacts: export.json (what
will carry over) and unmigrated.txt (what will not). Recording an artifact
whose path was about to be deleted was a bug in the first cut of this;
both now resolve.
`run` keeps its refusal and explains where rehearse went. `--dry-run` is
kept as a flag purely to say what replaced it.
Verified against the smoke VM end to end: rehearsal completes read-only in
seconds and reports 3505 unmigrated settings on a default install,
listeners included.
|
||
|
jcoffey-dev
|
b88724632c |
Capture and surface the settings migrate_v016.py does NOT migrate
Ran the converter against a real production settings corpus pulled from the
instance this tool is being built to migrate (secrets scrubbed on that host;
nothing sensitive transited). The result reframes what "a successful
migration" means:
total settings: 12401
NOT migrated: 12182 (98.2%)
actually migrated: 219 (1.8%)
Stalwart's own script reports this in an unmigrated.txt it writes beside its
output - and this tool was throwing that file away. Worse, RunSettingsConvert
never set cmd.Dir, so the script wrote unmigrated.txt into whatever directory
the operator happened to launch from, or failed the whole convert when that
directory wasn't writable. Both reproduced.
The largest groups left behind on production are spam-filter rules, DNSBLs,
trusted-domain and URL-redirector lookups, queue scheduling and TLS
settings - and server.listener. That last one explains something that had
been puzzling from an earlier smoke run: a freshly migrated 0.16 instance
answered on none of the ports the old one did, and served nothing but
/admin. Its listeners never migrated.
So:
- SettingsConvertOptions gains WorkDir, and the convert runs there. The
report lands somewhere known and an unwritable cwd can't fail the step.
- ReadUnmigratedReport parses it; UnmigratedReport.Summary renders the
largest groups first.
- The dry run records it as an "unmigrated-settings" artifact with a
checksum, notes the count in the checkpoint step, and prints it as a
standing warning rather than a footnote.
ARCHITECTURE 4.3 anticipated a "best-effort apply-plan" gap here. The gap is
not a few stragglers needing review; it is effectively the entire
configuration, and the tool has to say so where nobody can miss it.
Verified against the smoke VM: even a default `stalwart --init` instance
reports 3505 of ~3514 settings unmigrated, listeners included.
|
||
|
jcoffey-dev
|
680e554f23 |
Capture and report the supervised Stalwart process's output
A dry run against a real 0.15.5 instance failed with:
recovery mode did not come up: http://127.0.0.1:8081/ did not become
reachable within 1m0s: connect: connection refused
Stalwart had explained itself immediately - "Failed to bind to [::]:8080:
Address already in use" - into a pipe nothing was reading. Diagnosing a
one-line problem took several rounds because the tool threw away the only
evidence. Anything that reports a supervised process failing has to be able
to say why.
Process now captures the child's combined stdout and stderr into a bounded
buffer (64 KiB, keeping the most recent output, with truncation marked
rather than silent - a dead server's reason is at the end of its log), and
exposes it via Output(). recovery.Run appends it to both the startup-timeout
and settings-apply failures, and validate.BootCheck to its boot failure.
Fixes a second bug found while testing the first: Stop returned early when
Signal reported the process had already exited, so cmd.Wait was never
called. Wait is what reaps the child AND waits for the goroutines copying
its output - so the output was discarded in exactly the case where it
matters most, the server dying on its own. os.ErrProcessDone is now treated
as "already gone, still reap it".
The test reproduces the original failure shape: hold the port, start the
helper, let the health check time out, and assert the child's own bind
error survived. Confirmed against the smoke VM too - the same run now ends
with Stalwart's "Address already in use (os error 98)" printed inside the
tool's error.
|
||
|
jcoffey-dev
|
4568f9abbf |
Capture the pre-migration snapshot from 0.15.x, and stop claiming counts match when none were compared
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two defects, the second worse than the first. 1. AccountSnapshot could not read the version this tool migrates FROM. 0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns 404 - the JMAP management API and x:Account are 0.16 features. 0.15.x exposes a REST API at GET /api/principal instead. So preflight's account-snapshot check warned and moved on, and every run against a real source instance had no "before" data at all. AccountSnapshot now dispatches on the capability the session document advertises - a positive signal, not an inference from a failed call - and internal/stalwartapi/principal.go implements the 0.15.x REST path, including its 1-based page/limit pagination so an install larger than one page isn't silently truncated. 2. With no "before" counts, the content-integrity comparison iterated an empty map, checked nothing, and reported "all message counts match". That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it the actual no-data-loss guarantee - made vacuously, and it would have passed on a migration that lost every message. The comparison now derives its account set from whatever the source could report, verifies every account and domain survived either way, and carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS NOT COMPARED ... no-data-loss is NOT verified here" rather than implying otherwise. What can and cannot be checked across the 0.15/0.16 boundary, now that a real server has answered: 0.15.x has no per-mailbox message count at any endpoint, and the impersonation login 0.16 offers returns 401 there, so before/after message counts are impossible for the boundary migration this tool exists for. Both versions do report per-account used quota (usedQuota in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is captured on both sides. It is recorded and reported, not asserted on: 4.5 notes the 0.16 migration resets quotas to zero pending recalculation, so comparing those bytes across the boundary would be a false alarm generator. Test servers across preflight, validate and stalwartapi now advertise urn:stalwart:jmap, since they stand in for 0.16 instances and that capability is what says so. Verified end to end against the smoke VM: all nine preflight checks pass, and the checkpoint records 2 accounts, 1 domain and per-account used quota where it previously recorded nothing. |
||
|
jcoffey-dev
|
4b0bec8956 |
Add SPDX headers to every Go file
GPLv3's "How to Apply These Terms" asks for a notice in each source file; this is the modern two-line SPDX form of it rather than the full paragraph. 82 files, including tests. The blank line after the header is load-bearing. In Go a comment block immediately preceding `package X` becomes the package doc comment, so without the separator the SPDX lines would be absorbed into the doc for the eleven packages whose doc.go (or main.go) opens with one, and `go doc` would print them. Verified it doesn't. |
||
|
jcoffey-dev
|
6c838e8b70 |
Add GPL-3.0 license
Copyright LINUXexpert-org. LICENSE is the canonical FSF text from gnu.org/licenses/gpl-3.0.txt verbatim, not a reflowed or paraphrased copy, and README gains the standard notice. GPLv3 rather than the AGPLv3 used by cairnobs: this is a command-line tool, so AGPL's network clause - the whole reason to reach for AGPL over GPL - has nothing to attach to here. No per-file SPDX headers, matching this workspace's existing convention. Nothing is vendored: the tool is standard library only, and migrate_v016.py is fetched from Stalwart at runtime rather than redistributed. |
||
|
jcoffey-dev
|
7e04351b0f |
Add cutover; drop rollback in favour of operator-provided recovery
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5) is implemented, and the rollback phase is deleted. Recovery from a failed migration is now explicitly the operator's own snapshot or backup, and out of scope for this tool. internal/cutover implements 4.5 as seven checkpointed steps: verify the staged binary's version, install it, preserve and rewrite the service definition, reload, start, wait for a healthy JMAP session, recalculate quotas. The unit is rewritten in place rather than generated from a template. An operator's unit carries hardening options, limits and dependencies this tool has no business having an opinion about, and regenerating it would silently drop them. It repoints ExecStart (preserving systemd's -@:+! prefix characters and every argument after the executable), updates --config, and strips recovery-mode Environment lines - leaving STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every restart, forever. It refuses on a unit with no ExecStart, and on an Environment line mixing a recovery variable with others: a line it only partly understands is one it must not edit. Quota recalculation is the one step allowed to fail without failing the phase. Its wire format is grounded in Stalwart's x:Task schema reference - Task/set creating one AccountMaintenance per account with maintenanceType recalculateQuota - but the upgrade guide only documents the WebUI path, so two details remain inferred and are called out in stalwartapi/task.go: whether the schema's "read-only" annotation on accountId/maintenanceType means "immutable after creation", and whether a finished task simply leaves the queue (TaskStatus documents Pending/Retry/Failed with no success state). Warning rather than failing is the honest response to that uncertainty, and stale counters are an accounting problem next to calling for a restore of a machine that is otherwise migrated and serving mail. Docker deployments are refused outright: cutting a container over means pulling an image and recreating it, not swapping a binary. On removing rollback. The implementation worked and was tested, and it was removed because restoring bytes correctly is not the hard part. It copied file contents and permissions and verified every restored file against a manifest - and did not preserve ownership. Run as root, as this tool requires, it would have produced a byte-perfect, checksum-verified, root-owned data directory that Stalwart, running as its own user, could not open, and it would have reported success. The PostgreSQL path was worse: pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying into a database whose tables still exist, and the ON_ERROR_STOP=1 added so a half-applied restore couldn't be reported as success turned that into a hard failure. None of it had ever run against a real server. A filesystem snapshot has none of these failure modes, because it never lost the metadata to begin with. So cutover's gate is no longer rollback.CanRollBack but an explicit RecoveryPointConfirmed acknowledgement. That is an assertion, not a check - this tool cannot verify someone else's snapshot - and its only value is that nobody migrates a production mail server having never been asked the question. Two consequences are accepted deliberately: restoring any pre-migration recovery point discards mail delivered since, and a failed migration now stops and reports rather than undoing itself. What the tool still does to make a manual restore easier: the old binary is preserved and never deleted, the original service definition is preserved before the rewrite, the settings and principals dumps stay on disk, and every artifact path and checksum stays in the checkpoint where `status <run-id>` can print it. Also removed: the `confirm` command stub and RollbackWindowClosed, whose only purpose was closing a rollback window that no longer exists, and checkpoint.PhaseRollback. Old state.json files still load - JSON ignores the now-unknown field. Still open, and recorded in 8: cutover ignores systemd drop-ins, so an ExecStart or Environment override in stalwart.service.d/*.conf is invisible to the rewrite - including the recovery variable it exists to strip; nothing prevents concurrent runs on the same run-id; and nothing in this repo has ever run against a real Stalwart, real systemd, or a real store. |
||
|
jcoffey-dev
|
ade9906275 |
Implement internal/rollback and the service control it needs
Rollback was the one phase gating everything else: `run` without --dry-run refused because this tool could not undo a cutover it had committed to. That reason is now gone, and the refusal has narrowed to the fact that there is no real cutover to undo yet. internal/rollback implements ARCHITECTURE.md 4.8 as eight checkpointed steps under PhaseRollback: verify-backup, stop-service, preserve-failed-state, restore-data, restore-binary, restore-service-config, start-service, verify-rollback. Three things depart from what 4.8 specified, each for a reason: - The backup is re-verified against its manifest *before* the service is stopped, which the design didn't call out. Finding a corrupt backup is survivable while the failed instance is still up, and unsurvivable once its data directory has been moved aside. - BuildPlan is separate from Run, so every reason to refuse (closed rollback window, FoundationDB, no recorded backup, unknown deployment kind, missing database credentials) is found before anything is touched. The CLI prints that resolved plan and acts only with --yes. - The restore is re-verified against the same manifest after writing. A restore that put back truncated bytes and reported success would be worse than one that failed outright. Nothing from the failed attempt is deleted: the half-migrated data directory and the displaced binary are moved to .failed-<run-id> names, so a retry after the underlying issue is fixed still has both the evidence and the artifacts. Afterwards a reduced validation suite runs against the *restored* instance (version, reachability, directory counts) rather than assuming the restore worked. internal/service is a new package holding the systemd/Docker control this needs. It's separate rather than living inside internal/rollback because cutover will need the identical operations, and because the commands that can take mail delivery down belong in one auditable place - the same reasoning that makes stalwartapi the only thing speaking JMAP. preflight.DeploymentKind is now a type alias for service.Kind so detection and control can't drift apart. Its Active() reads `systemctl is-active`'s output rather than its exit status: systemctl exits non-zero for every non-active state, so exit-status logic would make "inactive" - the answer a rollback most needs - look like a failure to read the state at all. Also fixes a pre-existing bug in `status`: Go's flag package stops parsing at the first positional argument, so `status <run-id> --state-dir X` looked the run up in the default directory and reported it missing. `rollback` would have inherited the same footgun on a command whose flags decide what gets overwritten. Still open: `confirm` cannot set RollbackWindowClosed. Rollback honours the flag and refuses when it's set, but closing the window is the point of no return for the backups this restores from, so it should land with the retention policy 6 describes rather than before it. Verified end to end against a fake systemd deployment: half-migrated data restored to its original contents, failed state preserved, old binary reinstalled and reporting 0.15.5, unit restarted, and a re-run of the completed rollback inert. |
||
|
jcoffey-dev
|
56f465d8a9 |
docs: rewrite the README around what actually works
The old README was wrong in both directions. It called the project "architected, not yet implemented" while ~6,000 lines of tested code exist across backup, preflight, checkpointing, validation and recovery; and it listed six subcommands as a flat menu when three of them return "not implemented yet" and a fourth refuses unless given --dry-run. Replaced with a per-command and per-package status table, each entry checked against the binary rather than the design doc: rollback, confirm and report were run to confirm they error out, and the package line counts come from the tree. Records why `run` refuses without --dry-run -- internal/rollback is a doc.go and nothing else, and committing to a migration with no working rollback would break the one guarantee the tool exists to make. That refusal is correct behaviour today, not a defect to be filed. Corrects a claim in my own first draft: preflight is not purely read-only. Its checks against the Stalwart install are, but it records the run as a checkpoint first, so it fails without write access to /var/lib/stalwart-migrator -- a compile-time constant with no flag or env override. Found by running it. |
||
|
jcoffey-dev
|
5566eed1c8 |
Fix module path and restore cmd/ omitted by gitignore
Rename module github.com/johnellis/stalwart-migrator -> github.com/LINUXexpert-org/stalwart-migrator to match the repository location, so the module resolves under `go get`. The initial commit's .gitignore listed the compiled binary as a bare `stalwart-migrate` pattern, which Git matches at any depth -- so it also excluded the cmd/stalwart-migrate/ source directory, and the initial commit shipped without the CLI entrypoint. Anchor the pattern to the repo root as /stalwart-migrate and add the four missing files. go build, go vet, and go test ./... all pass. |
||
|
jcoffey-dev
|
719a945d64 |
Initial commit: stalwart-migrator design and scaffolding
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with checkpointed rollback and post-migration validation. Design stage; see ARCHITECTURE.md. |