Implement run: the migration pipeline, end to end

The phases have all existed for a while; nothing chained them. The order
here is the one arrived at by performing this migration by hand against a
clone of production before writing it down:

    preflight -> stage -> dump -> preserve binary -> STOP ->
    convert -> supplement -> recovery-mode migration -> cutover -> START

The dump runs before the stop because it reads settings over the admin API,
and a stopped server has no admin API. Everything from the stop to the end
of cutover is downtime.

internal/stage fills the last missing phase (4.3): resolve the release,
take the x86_64 linux-gnu server build and refuse to substitute another,
verify a pinned checksum if one was given, extract the binary - refusing
any archive entry that isn't a regular file, since a tarball is untrusted
input - and confirm the result reports the version its tag claimed.
Everything upstream of that last check is an assumption about someone
else's release process.

Two gates, separate on purpose. --yes is about intent. --recovery-point-
confirmed is a claim about the world: this tool cannot undo a migration
(4.8) and cannot check whether a snapshot exists, so a run that proceeded
without the operator asserting one would be proceeding on a hope.

Verified end to end against a real Stalwart 0.15.5 with email-style account
names, a named admin account, and seeded mail:

    MIGRATION COMPLETE. Mail was down for 6s.

Every cutover step green, including recalculate-quotas ("rebuilt disk
quotas for 2 account(s)") - the first time the x:Task wire format inferred
from Stalwart's schema reference has actually been exercised. It works,
now that endpoint discovery and role restoration make it reachable. After
the migration the named admin still administers, alice logs in with
unchanged credentials to the same four messages, and new SMTP delivery is
accepted.

Both refusal gates were tested, as was the failure path: an apply that
fails leaves the run stopped with the store part-migrated, and the error
says to restore the recovery point rather than restart the old version
against it.

ARCHITECTURE.md

@@ -829,6 +829,22 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
   - `tenant-admin` has no v0.16 equivalent and is reported as unrestorable
     rather than silently dropped.
 
+- **`run` is built and works.** preflight -> stage -> dump -> preserve ->
+  stop -> convert -> supplement -> recovery-mode -> cutover, checkpointed
+  throughout, verified end to end against a real 0.15.5: mail down for six
+  seconds, users unchanged, and `recalculate-quotas` succeeding for the
+  first time - the `x:Task` wire format inferred from the schema reference
+  turned out to be right, once the endpoint discovery and role restoration
+  made it reachable at all.
+
+  Two gates, deliberately separate: `--yes` is about intent, and
+  `--recovery-point-confirmed` is a claim about the world that this tool
+  cannot verify and must not assume. `internal/stage` (§4.3) fetches the
+  release, refuses to substitute a different build for the one it wants,
+  honours a pinned checksum, and confirms the extracted binary reports the
+  version its tag claims - because everything upstream of that check is an
+  assumption about somebody else's release process.
+
 - **`rehearse` (§4.9) is designed but not built.** The command is still
   `run --dry-run` with the old sandbox-cloning shape. Building it is mostly
   deletion: the dump, convert and report pieces already exist and work

README.md

@@ -20,19 +20,22 @@ them into a production run yet, so `run` still refuses.
 |---|---|
 | `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan |
 | `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over |
-| `stalwart-migrate run` | **Refuses on purpose** — see below |
+| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes` |
 | `stalwart-migrate status <id>` | **Works** |
 | `stalwart-migrate report <id>` | Not implemented |
 
-**`run` deliberately refuses to proceed.** Cutover (ARCHITECTURE.md §4.5) is
+**`run` performs the migration**, in the order
-implemented, but nothing calls it: the staging phase (§4.3) and the pipeline
+preflight → stage → dump → stop → convert → recovery-mode → cutover. It
-that would run preflight → backup → stage → recovery-mode → cutover →
+needs two flags: `--yes` (intent) and `--recovery-point-confirmed` (a claim
-validate against real paths don't exist yet. `run` stops rather than going
+that you have a snapshot or backup you have verified you can restore — this
-partway. That refusal is the correct behaviour today, not a bug.
+tool cannot undo a migration and will not start without it).
 
-**Start with `rehearse` instead.** It is read-only, needs no maintenance
+**Start with `rehearse` first.** It is read-only, needs no maintenance
-window, and answers the question that actually shapes a migration plan —
+window, and tells you what `run` will and won't carry over.
-see below.
+
+Measured on a full migration: the store converts in seconds, and the service
+was down for **6 seconds** end to end. Plan the window around verification,
+not data volume.
 
 Package state:
 

cmd/stalwart-migrate/run.go

@@ -4,56 +4,334 @@
 package main
 
 import (
+	"context"
 	"flag"
 	"fmt"
+	"net/http"
 	"os"
+	"path/filepath"
+	"time"
 
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/applyplan"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
 	"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/cutover"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/plan"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/recovery"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/service"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/stage"
 )
 
-// runRun implements `stalwart-migrate run`, which refuses.
+// runRun implements `stalwart-migrate run`: the real migration.
 //
-// It refuses because §4.3 staging and the pipeline that would drive
+// The phase order is ARCHITECTURE.md §4's, and it was arrived at by
-// preflight -> backup -> stage -> recovery-mode -> cutover -> validate
+// performing this migration by hand against a clone of a production
-// against real paths don't exist. The phases themselves mostly do:
+// instance before it was ever written down as code:
-// internal/preflight, internal/backup, internal/recovery and
-// internal/cutover are all implemented, and preflight, backup, the settings
-// dump, convert and the recovery-mode store migration have been exercised
-// against a real Stalwart 0.15.5. Cutover has not - it has never run
-// outside its own tests, and it is the phase that mutates production.
 //
-// What used to live here was `--dry-run`, which cloned the store into a
+//	preflight -> stage -> dump -> preserve binary -> STOP -> convert ->
-// sandbox and migrated the copy. That is now `stalwart-migrate rehearse`,
+//	generate supplement -> recovery-mode migration -> cutover -> START
-// minus the cloning: see ARCHITECTURE.md §4.9 for why the expensive half
+//
-// was dropped rather than fixed.
+// The dump happens before the service stops, because it reads settings
-func runRun(args []string) error {
+// over the admin API and a stopped server has no admin API. Everything
+// between the stop and the end of cutover is downtime, and on a real 3.6 GB
+// store that stretch was seconds of work - the window is dominated by
+// verification and by however long an operator takes to answer, not by
+// data volume.
+//
+// Two gates, deliberately separate. --yes says "do it"; it is about intent.
+// --recovery-point-confirmed says "I have a snapshot or backup I have
+// verified I can restore"; it is a claim about the world, which this tool
+// cannot check and must not assume. Nothing here can undo a migration -
+// recovery is the operator's (§4.8) - so a run that proceeded without that
+// claim would be proceeding on a hope.
+func runRun(args []string) (err error) {
 	fs := flag.NewFlagSet("run", flag.ExitOnError)
-	fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
+	binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
-	fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
+	configPath := fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
-	fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory")
+	dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory")
-	fs.String("target", "latest", `target Stalwart version, or "latest"`)
+	newConfigPath := fs.String("new-config", "", "where the converted v0.16 config is installed (default: config.json beside --config)")
-	fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
+	unitName := fs.String("unit", "stalwart", "systemd unit name")
-	dryRun := fs.Bool("dry-run", false, "removed - see `stalwart-migrate rehearse`")
+	serviceUnitPath := fs.String("service-unit", "/etc/systemd/system/stalwart.service", "systemd unit file to repoint at the new binary")
+	containerName := fs.String("container", "stalwart", "docker container name, if applicable")
+	adminURL := fs.String("admin-url", "", "base URL for the live instance's admin/JMAP API (required)")
+	adminUser := fs.String("admin-user", "", "admin username - must be a directory account, not a config fallback-admin (see README)")
+	adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
+		"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
+	targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
+	targetBinary := fs.String("target-binary", "", "use an already-downloaded target binary instead of fetching one")
+	stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
+	workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory")
+	pythonPath := fs.String("python", "python3", "path to python3")
+	stalwartCLI := fs.String("stalwart-cli", "stalwart-cli", "path to stalwart-cli (v1.0.2 or later; a separate download from the server)")
+	scriptSHA := fs.String("migration-script-sha256", "", "pinned sha256 of migrate_v016.py")
+	binarySHA := fs.String("target-binary-sha256", "", "pinned sha256 of the target release archive")
+	minFree := fs.Float64("min-free-multiple", 2.0, "required free disk space as a multiple of the data directory size")
+	recalcQuotas := fs.Bool("recalculate-quotas", true, "schedule the post-migration quota rebuild")
+	keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward")
+	yes := fs.Bool("yes", false, "actually perform the migration")
+	recoveryConfirmed := fs.Bool("recovery-point-confirmed", false,
+		"confirm you have a snapshot or backup you have verified you can restore - this tool cannot undo a migration")
 	if err := fs.Parse(args); err != nil {
 		return err
 	}
-
+	if *adminURL == "" {
-	if *dryRun {
+		return fmt.Errorf("--admin-url is required")
-		return fmt.Errorf("--dry-run has been replaced by `stalwart-migrate rehearse`, which converts this " +
+	}
-			"instance's settings and reports what will and won't carry over. It no longer clones the data " +
+	if *newConfigPath == "" {
-			"directory: that only proved the store opens, and cost a full copy of it to find out " +
+		*newConfigPath = filepath.Join(filepath.Dir(*configPath), "config.json")
-			"(ARCHITECTURE.md §4.9)")
 	}
 
-	fmt.Fprintln(os.Stderr,
+	ctx := context.Background()
-		"real migrations aren't available yet: the staging phase (ARCHITECTURE.md §4.3) and the pipeline that\n"+
+	httpClient := &http.Client{}
-			"would drive preflight -> backup -> stage -> recovery-mode -> cutover -> validate don't exist, so this\n"+
+
-			"command has no path that touches production.\n\n"+
+	fmt.Println("This migrates a live mail server in place. It will:")
-			"Two things worth knowing while you wait:\n"+
+	fmt.Printf("  1. check %s, then fetch and verify the %s binary\n", *binaryPath, *targetVersion)
-			"  * `stalwart-migrate rehearse` converts your settings and reports what will NOT carry over. Measured\n"+
+	fmt.Printf("  2. dump settings from %s while it is still running\n", *adminURL)
-			"    against a production instance that was 98% of them, listeners included - so it decides your\n"+
+	fmt.Printf("  3. STOP the service - mail is down from here\n")
-			"    migration plan, and it's safe to run now.\n"+
+	fmt.Printf("  4. convert the settings and migrate the store at %s IN PLACE\n", *dataDir)
-			"  * Recovery from a failed migration is your own snapshot or backup. This tool does not undo a\n"+
+	fmt.Printf("  5. install the new binary at %s and repoint %s\n", *binaryPath, *serviceUnitPath)
-			"    migration (§4.8).")
+	fmt.Printf("  6. start the service and check it answers\n")
-	return fmt.Errorf("`run` is not implemented")
+	fmt.Println("\nThis tool cannot undo any of it. Recovery is your snapshot or backup.")
+
+	if !*recoveryConfirmed {
+		return fmt.Errorf("\nrefusing to start: pass --recovery-point-confirmed once you have a snapshot or backup you have " +
+			"actually verified you can restore. This tool does not take one and cannot check yours")
+	}
+	if !*yes {
+		fmt.Println("\nnothing has been touched. Re-run with --yes to perform this migration.")
+		return nil
+	}
+
+	store := checkpoint.NewStore(*stateDir)
+	rs, err := store.Create("", *targetVersion)
+	if err != nil {
+		return fmt.Errorf("create run: %w", err)
+	}
+	fmt.Printf("\nrun id: %s\n", rs.RunID)
+	runWorkDir := filepath.Join(*workDir, rs.RunID)
+	runStateDir := filepath.Join(*stateDir, rs.RunID)
+	if err := os.MkdirAll(runWorkDir, 0o750); err != nil {
+		return fmt.Errorf("create work dir: %w", err)
+	}
+	defer func() {
+		if *keepArtifacts {
+			fmt.Printf("\nartifacts kept at %s (--keep-artifacts)\n", runWorkDir)
+			return
+		}
+		if rmErr := os.RemoveAll(runWorkDir); rmErr != nil {
+			fmt.Fprintf(os.Stderr, "warning: couldn't clean up %s: %v\n", runWorkDir, rmErr)
+		}
+	}()
+
+	fmt.Println("\n--- preflight ---")
+	pfReport, err := preflight.New(preflight.Options{
+		BinaryPath: *binaryPath, ConfigPath: *configPath, DataDir: *dataDir, ContainerName: *containerName,
+		AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
+		TargetVersion: *targetVersion, MinFreeMultiple: *minFree, HTTPClient: httpClient,
+	}).Run(ctx, store, rs)
+	fmt.Print(pfReport.String())
+	if err != nil {
+		return fmt.Errorf("preflight failed to complete: %w", err)
+	}
+	if pfReport.Blocking() {
+		return fmt.Errorf("preflight found blocking issues - see FAIL lines above")
+	}
+
+	p, err := plan.Decide(rs.SourceVersion, rs.TargetVersion)
+	if err != nil {
+		return fmt.Errorf("plan: %w", err)
+	}
+	fmt.Printf("\nplan: %s\n", p.Reason)
+
+	fmt.Println("\n--- stage ---")
+	staged := *targetBinary
+	if staged == "" {
+		staged = filepath.Join(runWorkDir, "stalwart-"+rs.TargetVersion)
+		if staged, err = stage.Run(ctx, store, rs, stage.Options{
+			TargetVersion: *targetVersion, DestPath: staged, SHA256: *binarySHA, HTTPClient: httpClient,
+		}); err != nil {
+			return fmt.Errorf("stage: %w", err)
+		}
+	}
+	fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-binary").Detail)
+
+	script := filepath.Join(runWorkDir, "migrate_v016.py")
+	settingsPath := filepath.Join(runWorkDir, "settings.json")
+	principalsPath := filepath.Join(runWorkDir, "principals.json")
+	convertedConfig := filepath.Join(runWorkDir, "config.json")
+	convertedExport := filepath.Join(runWorkDir, "export.json")
+	unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
+	supplementPath := filepath.Join(runWorkDir, "supplement.json")
+
+	if p.CrossesMajorBoundary {
+		fmt.Println("\n--- dump (service still up) ---")
+		if _, err := store.RunStep(rs, checkpoint.PhaseBackup, "settings-dump", func() (checkpoint.StepOutcome, error) {
+			if _, err := backup.DownloadFile(ctx, httpClient, backup.DefaultMigrationScriptURL, script, *scriptSHA); err != nil {
+				return checkpoint.StepOutcome{}, err
+			}
+			if err := backup.RunSettingsDump(ctx, backup.SettingsDumpOptions{
+				PythonPath: *pythonPath, ScriptPath: script, URL: *adminURL,
+				Username: *adminUser, Password: *adminPassword,
+				SettingsPath: settingsPath, PrincipalsPath: principalsPath,
+			}); err != nil {
+				return checkpoint.StepOutcome{}, err
+			}
+			return checkpoint.StepOutcome{Detail: "dumped settings and principals"}, nil
+		}); err != nil {
+			return fmt.Errorf("settings dump: %w", err)
+		}
+		fmt.Println("dumped settings and principals")
+	}
+
+	fmt.Println("\n--- preserve the old binary ---")
+	if _, err := store.RunStep(rs, checkpoint.PhaseBackup, "preserve-binary", func() (checkpoint.StepOutcome, error) {
+		preserved, err := backup.PreserveBinary(*binaryPath, rs.SourceVersion)
+		if err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		sum, size, err := backup.HashFile(preserved)
+		if err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		rs.RecordArtifact("old-binary", checkpoint.Artifact{Path: preserved, SHA256: sum, SizeBytes: size})
+		return checkpoint.StepOutcome{Detail: "preserved " + preserved}, nil
+	}); err != nil {
+		return fmt.Errorf("preserve binary: %w", err)
+	}
+	fmt.Println(rs.Outcome(checkpoint.PhaseBackup, "preserve-binary").Detail)
+
+	controller, err := service.New(service.Options{
+		Kind: service.Kind(rs.Topology.DeploymentKind), UnitName: *unitName, ContainerName: *containerName,
+	})
+	if err != nil {
+		return err
+	}
+
+	fmt.Println("\n--- stopping the service: MAIL IS DOWN FROM HERE ---")
+	windowStart := time.Now()
+	if _, err := store.RunStep(rs, checkpoint.PhaseCutover, "stop-service", func() (checkpoint.StepOutcome, error) {
+		if err := controller.Stop(ctx); err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		if err := service.WaitFor(ctx, controller, false, 2*time.Minute); err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		return checkpoint.StepOutcome{Detail: controller.Target() + " is stopped"}, nil
+	}); err != nil {
+		return fmt.Errorf("stop service: %w", err)
+	}
+	fmt.Println(controller.Target(), "stopped")
+
+	if p.CrossesMajorBoundary {
+		fmt.Println("\n--- convert ---")
+		if _, err := store.RunStep(rs, checkpoint.PhaseStage, "convert-settings", func() (checkpoint.StepOutcome, error) {
+			if err := backup.RunSettingsConvert(ctx, backup.SettingsConvertOptions{
+				PythonPath: *pythonPath, ScriptPath: script,
+				SettingsPath: settingsPath, PrincipalsPath: principalsPath,
+				ConfigPath: convertedConfig, OutputPath: convertedExport, WorkDir: runWorkDir,
+			}); err != nil {
+				return checkpoint.StepOutcome{}, err
+			}
+			return checkpoint.StepOutcome{Detail: "converted settings into a v0.16 apply plan"}, nil
+		}); err != nil {
+			return fmt.Errorf("convert settings: %w", err)
+		}
+		unmigrated, readErr := backup.ReadUnmigratedReport(unmigratedPath)
+		if readErr == nil && unmigrated != nil && unmigrated.TotalKeys > 0 {
+			keptWorklist := filepath.Join(runStateDir, "unmigrated.txt")
+			if copyErr := copyFile(unmigratedPath, keptWorklist); copyErr == nil {
+				if sum, size, hashErr := backup.HashFile(keptWorklist); hashErr == nil {
+					rs.RecordArtifact("unmigrated-settings", checkpoint.Artifact{Path: keptWorklist, SHA256: sum, SizeBytes: size})
+				}
+			}
+			fmt.Println(unmigrated.Classify().Summary(keptWorklist))
+		}
+
+		fmt.Println("\n--- supplemental plan ---")
+		applyFiles := []string{convertedExport}
+		if err := buildSupplement(settingsPath, principalsPath, unmigratedPath, supplementPath); err != nil {
+			fmt.Fprintf(os.Stderr, "warning: couldn't generate the supplemental plan: %v\n", err)
+		} else {
+			applyFiles = append(applyFiles, supplementPath)
+		}
+
+		fmt.Println("\n--- recovery-mode migration (the store is migrated IN PLACE) ---")
+		recReport, err := recovery.Run(ctx, store, rs, recovery.Options{
+			BinaryPath: staged, ConfigPath: convertedConfig,
+			ListenURL: "http://127.0.0.1:8080/", AdminUser: "admin",
+			ApplyFiles: applyFiles, CLIBinaryPath: *stalwartCLI,
+			StartupTimeout: 20 * time.Minute, HTTPClient: httpClient,
+		})
+		fmt.Print(recReport.String())
+		if err != nil {
+			return fmt.Errorf("recovery-mode migration failed - the store may be part-migrated and the service is still "+
+				"stopped; restore your recovery point rather than restarting the old version against it: %w", err)
+		}
+	}
+
+	fmt.Println("\n--- cutover ---")
+	configSource := convertedConfig
+	if !p.CrossesMajorBoundary {
+		configSource = "" // a patch bump keeps its existing config
+	}
+	cutReport, err := cutover.Run(ctx, store, rs, cutover.Options{
+		StagedBinaryPath: staged, BinaryPath: *binaryPath,
+		ServiceUnitPath: *serviceUnitPath, ConfigPath: *newConfigPath,
+		ConfigSource: configSource, ConfigOwnerReference: *configPath,
+		Deployment:             service.Options{Kind: service.Kind(rs.Topology.DeploymentKind), UnitName: *unitName, ContainerName: *containerName},
+		RecoveryPointConfirmed: *recoveryConfirmed,
+		AdminURL:               *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
+		HTTPClient: httpClient, RecalculateQuotas: *recalcQuotas && p.CrossesMajorBoundary,
+		StartTimeout: 3 * time.Minute, HealthTimeout: 5 * time.Minute, QuotaTimeout: 30 * time.Minute,
+	})
+	fmt.Print(cutReport.String())
+	if err != nil {
+		return fmt.Errorf("cutover failed: %w", err)
+	}
+
+	fmt.Printf("\nMIGRATION COMPLETE for run %s. Mail was down for %s.\n",
+		rs.RunID, time.Since(windowStart).Round(time.Second))
+	fmt.Printf("Now: confirm you can log in as %s, send and receive a test message, and work through\n", *adminUser)
+	fmt.Printf("the settings that did not carry over: %s\n", filepath.Join(runStateDir, "unmigrated.txt"))
+	return nil
+}
+
+// buildSupplement generates the apply plan for what migrate_v016.py leaves
+// behind - listeners, without which the migrated server binds nothing, and
+// administrator roles, without which nobody can administer it.
+func buildSupplement(settingsPath, principalsPath, unmigratedPath, outPath string) error {
+	settings, err := backup.ReadSettingsDump(settingsPath)
+	if err != nil {
+		return err
+	}
+	unmigrated, err := backup.ReadUnmigratedKeys(unmigratedPath, settings)
+	if err != nil {
+		return err
+	}
+	p, coverage, err := applyplan.Build(settings, unmigrated, applyplan.DefaultGenerators())
+	if err != nil {
+		return err
+	}
+	principals, err := backup.ReadPrincipalsDump(principalsPath)
+	if err != nil {
+		return err
+	}
+	roleOps, _, roleWarnings, err := applyplan.AccountRoleOperations(principals)
+	if err != nil {
+		return err
+	}
+	p.Operations = append(p.Operations, roleOps...)
+	if len(p.Operations) == 0 {
+		return fmt.Errorf("nothing to generate")
+	}
+	if err := p.WriteNDJSON(outPath); err != nil {
+		return err
+	}
+	fmt.Println(coverage.Summary(4))
+	fmt.Printf("  + %d account-role operation(s)\n", len(roleOps))
+	for _, w := range append(coverage.Warnings, roleWarnings...) {
+		fmt.Printf("  warning: %s\n", w)
+	}
+	return nil
 }

internal/preflight/release.go

@@ -75,3 +75,9 @@ func ChecksumAsset(rel *Release) *ReleaseAsset {
 	}
 	return nil
 }
+
+// ReleaseAPIBase returns the release API endpoint, and SetReleaseAPIBase
+// overrides it. Both exist so other packages' tests can point release
+// lookups at a local server instead of the real GitHub API.
+func ReleaseAPIBase() string        { return githubAPIBase }
+func SetReleaseAPIBase(base string) { githubAPIBase = base }

internal/stage/doc.go

@@ -0,0 +1,6 @@
+// SPDX-FileCopyrightText: 2026 LINUXexpert-org
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+// Package stage fetches and verifies the target Stalwart binary, installing it alongside the running one.
+// See ARCHITECTURE.md §4.3 for the design.
+package stage

internal/stage/stage.go

@@ -0,0 +1,227 @@
+// SPDX-FileCopyrightText: 2026 LINUXexpert-org
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+package stage
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
+)
+
+// assetSuffix is the release asset holding a plain Linux x86_64 server
+// binary. Stalwart publishes several builds per release; this deliberately
+// matches only the one, rather than taking the first thing that looks
+// close - picking the FoundationDB build or a musl variant by accident is
+// the kind of mistake that surfaces as a puzzling runtime failure much
+// later.
+const assetSuffix = "stalwart-x86_64-unknown-linux-gnu.tar.gz"
+
+// binaryNameInArchive is the file to extract from that tarball.
+const binaryNameInArchive = "stalwart"
+
+// maxBinaryBytes caps extraction. The 0.16.14 server binary is ~100 MB; a
+// limit an order of magnitude above that stops a malformed or hostile
+// archive from filling the disk while leaving ample room for growth.
+const maxBinaryBytes = 1 << 30
+
+// Options configures staging.
+type Options struct {
+	// TargetVersion is the release to fetch ("0.16.14", or "latest").
+	TargetVersion string
+	// DestPath is where the extracted binary is written. It must not be
+	// the running binary's path: staging installs *alongside*, and cutover
+	// is what moves it into place.
+	DestPath string
+	// SHA256, when set, is the expected checksum of the downloaded archive.
+	// Recommended: the release process does not always publish a checksum
+	// manifest, so pinning is how a second run gets the guarantee the
+	// first one couldn't have.
+	SHA256     string
+	HTTPClient *http.Client
+}
+
+// Run downloads the target release, verifies what it can, extracts the
+// server binary to DestPath, and confirms the binary reports the version
+// that was asked for.
+//
+// That last check is the point of the phase. Everything upstream of it -
+// the tag lookup, the asset name, the archive layout - is an assumption
+// about someone else's release process, and the binary's own --version
+// output is the only thing that actually settles what was fetched. Cutover
+// checks it again before installing, deliberately: this phase and that one
+// can be separated by a long time and an operator's own file management.
+func Run(ctx context.Context, store *checkpoint.Store, rs *checkpoint.RunState, opts Options) (string, error) {
+	if opts.DestPath == "" {
+		return "", fmt.Errorf("stage: no destination path for the target binary")
+	}
+
+	outcome, err := store.RunStep(rs, checkpoint.PhaseStage, "stage-binary", func() (checkpoint.StepOutcome, error) {
+		release, err := preflight.ResolveRelease(ctx, opts.HTTPClient, opts.TargetVersion)
+		if err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		asset := serverAsset(release)
+		if asset == nil {
+			names := make([]string, 0, len(release.Assets))
+			for _, a := range release.Assets {
+				names = append(names, a.Name)
+			}
+			return checkpoint.StepOutcome{}, fmt.Errorf(
+				"stage: release %s publishes no %s asset (found: %s) - this tool stages a Linux x86_64 server build and won't substitute another",
+				release.TagName, assetSuffix, strings.Join(names, ", "))
+		}
+
+		archivePath := opts.DestPath + ".tar.gz"
+		sum, err := download(ctx, opts.HTTPClient, asset.DownloadURL, archivePath)
+		if err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+		defer os.Remove(archivePath)
+
+		if opts.SHA256 != "" && !strings.EqualFold(sum, opts.SHA256) {
+			return checkpoint.StepOutcome{}, fmt.Errorf(
+				"stage: downloaded %s has sha256 %s but %s was expected - refusing to stage a binary that isn't the one that was pinned",
+				asset.Name, sum, opts.SHA256)
+		}
+
+		if err := extractBinary(archivePath, opts.DestPath); err != nil {
+			return checkpoint.StepOutcome{}, err
+		}
+
+		got, err := preflight.DetectVersion(ctx, opts.DestPath)
+		if err != nil {
+			return checkpoint.StepOutcome{}, fmt.Errorf("stage: staged binary at %s won't report its version: %w", opts.DestPath, err)
+		}
+		wanted := strings.TrimPrefix(release.TagName, "v")
+		if got != wanted {
+			return checkpoint.StepOutcome{}, fmt.Errorf(
+				"stage: staged binary reports %s but release %s was fetched - the release asset does not contain what its tag claims",
+				got, release.TagName)
+		}
+
+		pinNote := ""
+		if opts.SHA256 == "" {
+			pinNote = fmt.Sprintf("; no checksum was pinned - record sha256 %s to pin this download for future runs", sum)
+		}
+		return checkpoint.StepOutcome{
+			Detail: fmt.Sprintf("staged %s at %s, which reports %s%s", asset.Name, opts.DestPath, got, pinNote),
+			Extra:  got,
+		}, nil
+	})
+	if err != nil {
+		return "", err
+	}
+	_ = outcome
+	return opts.DestPath, nil
+}
+
+func serverAsset(release *preflight.Release) *preflight.ReleaseAsset {
+	for i := range release.Assets {
+		if release.Assets[i].Name == assetSuffix {
+			return &release.Assets[i]
+		}
+	}
+	return nil
+}
+
+// download fetches url to path and returns its SHA256.
+func download(ctx context.Context, client *http.Client, url, path string) (string, error) {
+	if client == nil {
+		client = &http.Client{}
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return "", err
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return "", fmt.Errorf("stage: download %s: %w", url, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return "", fmt.Errorf("stage: download %s returned %s", url, resp.Status)
+	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
+		return "", fmt.Errorf("stage: create %s: %w", filepath.Dir(path), err)
+	}
+	f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o640)
+	if err != nil {
+		return "", fmt.Errorf("stage: create %s: %w", path, err)
+	}
+	defer f.Close()
+	h := sha256.New()
+	if _, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, maxBinaryBytes)); err != nil {
+		return "", fmt.Errorf("stage: write %s: %w", path, err)
+	}
+	if err := f.Sync(); err != nil {
+		return "", fmt.Errorf("stage: sync %s: %w", path, err)
+	}
+	return hex.EncodeToString(h.Sum(nil)), nil
+}
+
+// extractBinary pulls the server binary out of a release tarball.
+//
+// It searches by base name rather than by a fixed path, because the archive
+// layout is the release process's business and has already varied between
+// products (the separately-versioned CLI ships its binary a directory
+// down). Anything that isn't a regular file is refused rather than
+// followed: a tarball is untrusted input, and an entry that is a symlink or
+// carries a path escaping the destination has no legitimate reason to be
+// there.
+func extractBinary(archivePath, destPath string) error {
+	f, err := os.Open(archivePath)
+	if err != nil {
+		return fmt.Errorf("stage: open %s: %w", archivePath, err)
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		return fmt.Errorf("stage: %s is not gzip: %w", archivePath, err)
+	}
+	defer gz.Close()
+
+	tr := tar.NewReader(gz)
+	for {
+		header, err := tr.Next()
+		if err == io.EOF {
+			return fmt.Errorf("stage: %s contains no %q entry", archivePath, binaryNameInArchive)
+		}
+		if err != nil {
+			return fmt.Errorf("stage: read %s: %w", archivePath, err)
+		}
+		if filepath.Base(header.Name) != binaryNameInArchive {
+			continue
+		}
+		if header.Typeflag != tar.TypeReg {
+			return fmt.Errorf("stage: %q in %s is not a regular file (type %q) - refusing to follow it",
+				header.Name, archivePath, string(header.Typeflag))
+		}
+		out, err := os.OpenFile(destPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
+		if err != nil {
+			return fmt.Errorf("stage: create %s: %w", destPath, err)
+		}
+		defer out.Close()
+		if _, err := io.Copy(out, io.LimitReader(tr, maxBinaryBytes)); err != nil {
+			return fmt.Errorf("stage: extract to %s: %w", destPath, err)
+		}
+		if err := out.Sync(); err != nil {
+			return fmt.Errorf("stage: sync %s: %w", destPath, err)
+		}
+		return nil
+	}
+}
+
+func releaseAPIBase() string        { return preflight.ReleaseAPIBase() }
+func setReleaseAPIBase(base string) { preflight.SetReleaseAPIBase(base) }

internal/stage/stage_test.go

@@ -0,0 +1,254 @@
+// SPDX-FileCopyrightText: 2026 LINUXexpert-org
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+package stage
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
+)
+
+// tarGzWith builds a release-shaped archive containing one entry.
+func tarGzWith(t *testing.T, name, body string, typeflag byte) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	hdr := &tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), Typeflag: typeflag}
+	if typeflag == tar.TypeSymlink {
+		hdr.Size = 0
+		hdr.Linkname = "/etc/passwd"
+	}
+	if err := tw.WriteHeader(hdr); err != nil {
+		t.Fatal(err)
+	}
+	if typeflag == tar.TypeReg {
+		if _, err := tw.Write([]byte(body)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	tw.Close()
+	gz.Close()
+	return buf.Bytes()
+}
+
+// releaseServer stands in for the GitHub release API plus asset hosting.
+func releaseServer(t *testing.T, tag string, archive []byte, assetName string) *httptest.Server {
+	t.Helper()
+	var srv *httptest.Server
+	srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/download") {
+			w.Write(archive)
+			return
+		}
+		json.NewEncoder(w).Encode(map[string]any{
+			"tag_name": tag,
+			"assets": []map[string]any{
+				{"name": "stalwart-foundationdb-x86_64-unknown-linux-gnu.tar.gz", "browser_download_url": srv.URL + "/wrong/download"},
+				{"name": assetName, "browser_download_url": srv.URL + "/right/download"},
+			},
+		})
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+// A fake "binary" that reports a version, so the staged-version check has
+// something real to run.
+func versionScript(v string) string {
+	return fmt.Sprintf("#!/bin/sh\necho 'stalwart %s'\n", v)
+}
+
+func newRun(t *testing.T) (*checkpoint.Store, *checkpoint.RunState) {
+	t.Helper()
+	store := checkpoint.NewStore(filepath.Join(t.TempDir(), "runs"))
+	rs, err := store.Create("0.15.5", "0.16.14")
+	if err != nil {
+		t.Fatal(err)
+	}
+	return store, rs
+}
+
+func withReleaseAPI(t *testing.T, srv *httptest.Server) {
+	t.Helper()
+	// preflight.ResolveRelease reads its base from a package var; point it
+	// at the fake for the duration of the test.
+	old := releaseAPIBase()
+	setReleaseAPIBase(srv.URL)
+	t.Cleanup(func() { setReleaseAPIBase(old) })
+}
+
+func TestRunStagesTheServerBuildAndVerifiesItsVersion(t *testing.T) {
+	archive := tarGzWith(t, "stalwart", versionScript("0.16.14"), tar.TypeReg)
+	srv := releaseServer(t, "v0.16.14", archive, assetSuffix)
+	withReleaseAPI(t, srv)
+	store, rs := newRun(t)
+	dest := filepath.Join(t.TempDir(), "stalwart-0.16.14")
+
+	path, err := Run(context.Background(), store, rs, Options{
+		TargetVersion: "0.16.14", DestPath: dest, HTTPClient: srv.Client(),
+	})
+	if err != nil {
+		t.Fatalf("Run: %v", err)
+	}
+	if path != dest {
+		t.Errorf("path = %q, want %q", path, dest)
+	}
+	info, err := os.Stat(dest)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if info.Mode().Perm()&0o111 == 0 {
+		t.Errorf("staged binary mode = %v, want executable", info.Mode().Perm())
+	}
+	// The archive it must NOT have taken is the FoundationDB build.
+	if _, err := os.Stat(dest + ".tar.gz"); !os.IsNotExist(err) {
+		t.Error("the downloaded archive should be cleaned up after extraction")
+	}
+}
+
+// The release publishes several builds; taking the first plausible one
+// would stage a FoundationDB or musl variant and surface as a puzzling
+// runtime failure much later.
+func TestRunRefusesWhenTheServerBuildIsAbsent(t *testing.T) {
+	archive := tarGzWith(t, "stalwart", versionScript("0.16.14"), tar.TypeReg)
+	srv := releaseServer(t, "v0.16.14", archive, "stalwart-aarch64-unknown-linux-gnu.tar.gz")
+	withReleaseAPI(t, srv)
+	store, rs := newRun(t)
+
+	_, err := Run(context.Background(), store, rs, Options{
+		TargetVersion: "0.16.14", DestPath: filepath.Join(t.TempDir(), "s"), HTTPClient: srv.Client(),
+	})
+	if err == nil {
+		t.Fatal("want a refusal when the x86_64 server build isn't published")
+	}
+	if !strings.Contains(err.Error(), "won't substitute another") {
+		t.Errorf("error %q should say it refuses to substitute a different build", err)
+	}
+}
+
+// The binary's own --version is the only thing that settles what was
+// actually fetched; everything upstream is an assumption about someone
+// else's release process.
+func TestRunRefusesAnArchiveThatDoesNotMatchItsTag(t *testing.T) {
+	archive := tarGzWith(t, "stalwart", versionScript("0.16.9"), tar.TypeReg)
+	srv := releaseServer(t, "v0.16.14", archive, assetSuffix)
+	withReleaseAPI(t, srv)
+	store, rs := newRun(t)
+
+	_, err := Run(context.Background(), store, rs, Options{
+		TargetVersion: "0.16.14", DestPath: filepath.Join(t.TempDir(), "s"), HTTPClient: srv.Client(),
+	})
+	if err == nil {
+		t.Fatal("want a refusal when the asset contains a different version than its tag claims")
+	}
+	if !strings.Contains(err.Error(), "0.16.9") {
+		t.Errorf("error %q should report what the binary actually said", err)
+	}
+}
+
+func TestRunHonoursAPinnedChecksum(t *testing.T) {
+	archive := tarGzWith(t, "stalwart", versionScript("0.16.14"), tar.TypeReg)
+	srv := releaseServer(t, "v0.16.14", archive, assetSuffix)
+	withReleaseAPI(t, srv)
+	store, rs := newRun(t)
+
+	_, err := Run(context.Background(), store, rs, Options{
+		TargetVersion: "0.16.14", DestPath: filepath.Join(t.TempDir(), "s"),
+		SHA256: "0000000000000000000000000000000000000000000000000000000000000000", HTTPClient: srv.Client(),
+	})
+	if err == nil {
+		t.Fatal("want a refusal when the download doesn't match the pin")
+	}
+	if !strings.Contains(err.Error(), "was pinned") {
+		t.Errorf("error %q should say the pin was violated", err)
+	}
+
+	// And the matching pin is accepted.
+	sum := sha256.Sum256(archive)
+	store2, rs2 := newRun(t)
+	if _, err := Run(context.Background(), store2, rs2, Options{
+		TargetVersion: "0.16.14", DestPath: filepath.Join(t.TempDir(), "s"),
+		SHA256: hex.EncodeToString(sum[:]), HTTPClient: srv.Client(),
+	}); err != nil {
+		t.Fatalf("a matching pin should be accepted: %v", err)
+	}
+}
+
+// A tarball is untrusted input.
+func TestExtractRefusesANonRegularEntry(t *testing.T) {
+	dir := t.TempDir()
+	archivePath := filepath.Join(dir, "a.tar.gz")
+	if err := os.WriteFile(archivePath, tarGzWith(t, "stalwart", "", tar.TypeSymlink), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	err := extractBinary(archivePath, filepath.Join(dir, "out"))
+	if err == nil {
+		t.Fatal("want a refusal for a symlink entry, got nil")
+	}
+	if !strings.Contains(err.Error(), "refusing to follow") {
+		t.Errorf("error %q should say it refuses to follow it", err)
+	}
+}
+
+// The separately-versioned CLI ships its binary a directory down, so
+// searching by base name rather than exact path is deliberate.
+func TestExtractFindsTheBinaryInASubdirectory(t *testing.T) {
+	dir := t.TempDir()
+	archivePath := filepath.Join(dir, "a.tar.gz")
+	if err := os.WriteFile(archivePath, tarGzWith(t, "stalwart-x86_64/stalwart", versionScript("0.16.14"), tar.TypeReg), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	dest := filepath.Join(dir, "out")
+	if err := extractBinary(archivePath, dest); err != nil {
+		t.Fatalf("extractBinary: %v", err)
+	}
+	if body, _ := os.ReadFile(dest); !strings.Contains(string(body), "0.16.14") {
+		t.Errorf("extracted %q, want the binary from the subdirectory", body)
+	}
+}
+
+// Re-running a completed stage must not re-download.
+func TestRunIsSkippedOnResume(t *testing.T) {
+	archive := tarGzWith(t, "stalwart", versionScript("0.16.14"), tar.TypeReg)
+	var downloads int
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/download") {
+			downloads++
+			w.Write(archive)
+			return
+		}
+		json.NewEncoder(w).Encode(map[string]any{
+			"tag_name": "v0.16.14",
+			"assets":   []map[string]any{{"name": assetSuffix, "browser_download_url": "http://" + r.Host + "/right/download"}},
+		})
+	}))
+	defer srv.Close()
+	withReleaseAPI(t, srv)
+	store, rs := newRun(t)
+	opts := Options{TargetVersion: "0.16.14", DestPath: filepath.Join(t.TempDir(), "s"), HTTPClient: srv.Client()}
+
+	if _, err := Run(context.Background(), store, rs, opts); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := Run(context.Background(), store, rs, opts); err != nil {
+		t.Fatal(err)
+	}
+	if downloads != 1 {
+		t.Errorf("downloaded %d time(s), want 1 - a resumed run must not re-fetch 100 MB", downloads)
+	}
+}