diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 65359ce..9f997a4 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -834,16 +834,26 @@ happens to need them. `preflight.DeploymentKind` is a type alias for §4.5 lists exactly which two details are inferred. A smoke test against a real 0.16 instance would settle both, and would let this step be promoted from "warns on failure" to a hard check. -- **Docker is implemented but not yet wired to the CLI.** Preflight - inspects a container and reports what stands in the way; stage pulls and - verifies an image; the recovery cycle runs in a throwaway container - against the live data; and cutover recreates the container, refusing one - whose definition it would not carry across intact. What is missing is - `run` passing those options and preflight lifting its refusal for the - containers now handled. Compose stays refused deliberately: recreating a - compose-managed container out from under compose leaves the container and - the compose file disagreeing, and the next `compose up` reverts the - migration. +- **Docker is wired end to end and has never met a real Stalwart image.** + Preflight inspects a container and blocks on what stands in the way; + stage pulls and verifies an image; the recovery cycle runs in a throwaway + container against the live data; cutover recreates the container, + refusing one whose definition it would not carry across intact. Every + test drives a fake `docker`, which proves the right commands are + assembled and proves nothing about whether the image reads the config it + is handed - the same limit §4.8 records about the deleted rollback code, + and the reason `run` refuses a container without + `--container-path-unproven`. A rehearsal on a clone, then a real + migration, is what would retire that flag. + Compose stays refused deliberately: recreating a compose-managed + container out from under compose leaves the container and the compose + file disagreeing, and the next `compose up` reverts the migration. +- **The converted config reaches a container through the data volume.** + `run` writes it under the host side of whichever mount covers + `--data-dir` and names it on the container side, because cutover recreates + a container with the mounts it had and cannot invent a new one for a + config file. That is an inference from how the mounts must line up rather + than something a real deployment has confirmed. - **Cutover ignores systemd drop-ins.** It rewrites only the main unit file, so an `ExecStart` or `Environment` override in `/etc/systemd/system/stalwart.service.d/*.conf` is invisible to it - diff --git a/README.md b/README.md index 9ec88ac..88abd42 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,11 @@ job](#recovery-is-your-job) before using it on anything you care about. Go, standard library only — no external dependencies. +A companion to [**ihasmail**](https://github.com/LINUXexpert-org/ihasmail), +a JMAP-first webmail client for Stalwart. That one is what you read your +mail in; this one is what gets the server underneath it onto a version that +speaks the protocol it needs. + **This has been used to migrate a production mail server.** On 2026-08-25 it took a live server — nine domains, six accounts, a 2.4 GB RocksDB store — from 0.15.5 to 0.16.19 with **8 seconds** of downtime, every phase green @@ -55,7 +60,7 @@ described above. |---|---| | `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan | | `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over | -| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes` | +| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes`. Container deployments additionally need `--container-path-unproven` (see [Docker deployments](#docker-deployments)) | | `stalwart-migrate tenants` | **Works** — read-only; who owns which domain, and what would block a migration | | `stalwart-migrate status ` | **Works** | | `stalwart-migrate report ` | **Works** — prints what validation found for a run | @@ -70,6 +75,49 @@ tool cannot undo a migration and will not start without it). **Start with `rehearse` first.** It is read-only, needs no maintenance window, and tells you what `run` will and won't carry over. +### Docker deployments + +A containerised Stalwart can be migrated, with two things to know first. + +**The container path has never been run against a real Stalwart image.** +Its logic is tested and its refusals are real, but a fake `docker` proves +only that the right commands are assembled — not that the image reads the +config it is handed. `run` refuses a container deployment unless you pass +`--container-path-unproven`, which is there so nobody reaches it without +being told. Rehearse on a clone first; that advice goes double here. + +**Two flags and one convention:** + +- `--target-image` names the image in full, e.g. + `stalwartlabs/stalwart:v0.16.14`. It is never derived from the running + container by swapping the tag — that is wrong for a digest-pinned image, + a mirror or a fork, and being wrong means pulling the wrong software into + a mail server. +- `--container` names the container (default `stalwart`). +- `--data-dir` must name the path **inside** the container, since that is + where its data actually lives. `preflight` says so if it matches none of + the container's mounts. + +**What it refuses, and why.** A container cannot be edited in place the way +a unit file can, so cutting one over means rebuilding it — and a container +rebuilt without its capabilities, its custom network or its device mappings +starts cleanly and is quietly not the server it was. So cutover carries +across what it understands (mounts, ports, environment, restart policy, +labels) and refuses outright when it finds anything else, naming what it +found. It also refuses a container whose data is not on a volume — an +upgrade replaces the container, and the writable layer goes with it — and +one managed by Docker Compose, because recreating it out from under compose +leaves the container and the compose file disagreeing about what is +deployed, and the next `compose up` reverts the migration. Compose +deployments are migrated by editing the image tag in the compose file and +running `compose up -d`. + +**What it keeps.** The old container is renamed rather than removed, the old +image is never pruned, and the container's `docker inspect` is preserved as +an artifact before anything is replaced. Together those are the manual +restore path — see [Recovery is your job](#recovery-is-your-job), which +applies here exactly as it does to a binary install. + Measured on a full migration: the store converts in seconds, and the service was down for **6 seconds** end to end. Plan the window around verification, not data volume. diff --git a/cmd/stalwart-migrate/run.go b/cmd/stalwart-migrate/run.go index 503f475..1973813 100644 --- a/cmd/stalwart-migrate/run.go +++ b/cmd/stalwart-migrate/run.go @@ -9,6 +9,7 @@ import ( "fmt" "net/http" "os" + "path" "path/filepath" "time" @@ -61,6 +62,9 @@ func runRun(args []string) (err error) { "admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)") targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`) targetBinary := fs.String("target-binary", "", "use an already-downloaded target binary instead of fetching one") + targetImage := fs.String("target-image", "", "for a container deployment: the target image, named in full "+ + "(e.g. stalwartlabs/stalwart:v0.16.14). Never guessed from the running container - a derived tag is wrong for a "+ + "digest-pinned image, a mirror or a fork") stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in") workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory") pythonPath := fs.String("python", "python3", "path to python3") @@ -73,6 +77,10 @@ func runRun(args []string) (err error) { keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/ afterward") resume := fs.String("resume", "", "resume an interrupted run by id instead of starting a new one (see `status` for ids)") yes := fs.Bool("yes", false, "actually perform the migration") + containerUnproven := fs.Bool("container-path-unproven", false, + "acknowledge that the container migration path has never been run against a real Stalwart image. Its logic is "+ + "tested and its refusals are real, but a fake docker proves only that the right commands are assembled - not that "+ + "the image reads the config it is handed. Required for a container deployment") recoveryConfirmed := fs.Bool("recovery-point-confirmed", false, "confirm you have a snapshot or backup you have verified you can restore - this tool cannot undo a migration") if err := fs.Parse(args); err != nil { @@ -174,22 +182,79 @@ func runRun(args []string) (err error) { } fmt.Printf("\nplan: %s\n", p.Reason) + // Preflight recorded how this deployment is run, and from here the two + // differ in three places: what gets staged, what the recovery cycle + // launches, and what cutover replaces. Everything between them is the + // same migration. + isContainer := service.Kind(rs.Topology.DeploymentKind) == service.Docker + var containerFacts preflight.ContainerFacts + if isContainer { + if !*containerUnproven { + return fmt.Errorf( + "refusing to start: this is a container deployment, and that path has never been run against a real Stalwart " + + "image. Its logic is tested and its refusals are real, but a fake docker proves only that the right commands " + + "are assembled, not that the image reads the config it is handed. Pass --container-path-unproven if you " + + "accept that, ideally against a clone of production first") + } + if *targetImage == "" { + return fmt.Errorf("refusing to start: a container deployment needs --target-image; it is never guessed from the running container") + } + if containerFacts, err = preflight.InspectContainer(ctx, *containerName); err != nil { + return err + } + } + fmt.Println("\n--- stage ---") staged := *targetBinary - if staged == "" { + stagedImage := "" + switch { + case isContainer: + img, err := stage.RunImage(ctx, store, rs, stage.ImageOptions{ + Image: *targetImage, TargetVersion: rs.TargetVersion, + }) + if err != nil { + return fmt.Errorf("stage: %w", err) + } + stagedImage = img.Ref + fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-image").Detail) + case staged == "": staged = filepath.Join(runWorkDir, "stalwart-"+rs.TargetVersion) if staged, err = stage.Run(ctx, store, rs, stage.Options{ TargetVersion: *targetVersion, DestPath: staged, SHA256: *binarySHA, HTTPClient: httpClient, }); err != nil { return fmt.Errorf("stage: %w", err) } + fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-binary").Detail) + default: + fmt.Println("using the already-staged binary at", staged) } - fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-binary").Detail) script := filepath.Join(runWorkDir, "migrate_v016.py") settingsPath := filepath.Join(runWorkDir, "settings.json") principalsPath := filepath.Join(runWorkDir, "principals.json") + // The converted config has to be readable by whatever boots next. For a + // binary that is any path on this host; for a container it has to be + // somewhere the container already mounts, because cutover recreates it + // with the mounts it had and cannot invent a new one. So it goes inside + // the data volume, written on the host side and named on the container + // side. convertedConfig := filepath.Join(runWorkDir, "config.json") + containerConfigPath := "" + if isContainer { + mount, ok := containerFacts.MountFor(*dataDir) + if !ok { + return fmt.Errorf( + "refusing to start: --data-dir %s is not covered by any of %s's mounts (%s). For a container it must name the "+ + "path *inside* the container, since that is where its data actually lives", + *dataDir, *containerName, preflight.DescribeMounts(containerFacts.Mounts)) + } + hostDir := filepath.Join(mount.Source, "stalwart-migrate") + if err := os.MkdirAll(hostDir, 0o750); err != nil { + return fmt.Errorf("create %s (the host side of %s): %w", hostDir, mount.Destination, err) + } + convertedConfig = filepath.Join(hostDir, "config.json") + containerConfigPath = path.Join(mount.Destination, "stalwart-migrate", "config.json") + } convertedExport := filepath.Join(runWorkDir, "export.json") unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt") supplementPath := filepath.Join(runWorkDir, "supplement.json") @@ -338,12 +403,32 @@ func runRun(args []string) (err error) { } fmt.Println("\n--- recovery-mode migration (the store is migrated IN PLACE) ---") - recReport, err := recovery.Run(ctx, store, rs, recovery.Options{ + recOpts := recovery.Options{ BinaryPath: staged, ConfigPath: convertedConfig, ListenURL: "http://127.0.0.1:8080/", AdminUser: "admin", ApplyFiles: applyFiles, CLIBinaryPath: *stalwartCLI, StartupTimeout: 20 * time.Minute, HTTPClient: httpClient, - }) + } + if isContainer { + // The live container is stopped, so the recovery one takes its + // mounts and publishes recovery mode's listener where the + // health check on this side can reach it. + mounts := make([]recovery.ContainerMount, 0, len(containerFacts.Mounts)) + for _, m := range containerFacts.Mounts { + src := m.Name + if src == "" { + src = m.Source + } + mounts = append(mounts, recovery.ContainerMount{Source: src, Destination: m.Destination, ReadOnly: !m.RW}) + } + recOpts.ConfigPath = containerConfigPath + recOpts.Launcher = recovery.ContainerLauncher{ + Image: stagedImage, Mounts: mounts, + Name: *containerName + "-migrate-recovery", + Publish: []string{"127.0.0.1:8080:8080"}, + } + } + recReport, err := recovery.Run(ctx, store, rs, recOpts) fmt.Print(recReport.String()) if err != nil { return fmt.Errorf("recovery-mode migration failed - the store may be part-migrated and the service is still "+ @@ -361,6 +446,7 @@ func runRun(args []string) (err error) { ServiceUnitPath: *serviceUnitPath, ConfigPath: *newConfigPath, ConfigSource: configSource, ConfigOwnerReference: *configPath, Deployment: service.Options{Kind: service.Kind(rs.Topology.DeploymentKind), UnitName: *unitName, ContainerName: *containerName}, + Container: containerCutover(isContainer, *containerName, stagedImage, runStateDir), RecoveryPointConfirmed: *recoveryConfirmed, AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword, HTTPClient: httpClient, RecalculateQuotas: *recalcQuotas && p.CrossesMajorBoundary, @@ -434,3 +520,13 @@ func buildSupplement(settingsPath, principalsPath, unmigratedPath, outPath strin } return nil } + +// containerCutover is the cutover options for a container deployment, or +// nil for a binary one. Nil is what keeps cutover refusing a container it +// was given no image to recreate from. +func containerCutover(isContainer bool, name, image, preserveDir string) *cutover.ContainerOptions { + if !isContainer { + return nil + } + return &cutover.ContainerOptions{ContainerName: name, StagedImage: image, PreserveDir: preserveDir} +} diff --git a/internal/preflight/checks.go b/internal/preflight/checks.go index 9558134..bde973e 100644 --- a/internal/preflight/checks.go +++ b/internal/preflight/checks.go @@ -200,19 +200,19 @@ func (c *Checker) Run(ctx context.Context, store *checkpoint.Store, rs *checkpoi // automate it - but cutover runs after the service has been // stopped. Refusing there means refusing with mail already down, // which is how a migration attempt turned into an outage. - if kind == DeploymentDocker && !c.opts.DeploymentCheckAdvisory { - return CheckResult{ - Status: StatusFail, - Detail: "detected deployment kind: docker - this tool cannot cut over a container. " + - "Migrating one means pulling the new image and recreating the container, which has to be done by hand; " + - "`rehearse` still works and will tell you what the migration involves", - }, string(kind) - } status := StatusOK - if kind == DeploymentUnknown { + detail := fmt.Sprintf("detected deployment kind: %s", kind) + switch kind { + case DeploymentUnknown: status = StatusWarn + case DeploymentDocker: + // No longer a refusal on its own: a container can be migrated + // now. What still refuses is specific and checked below - + // compose, and data that is not on a volume - because those are + // properties of this container rather than of containers. + detail += " - the container checks below decide whether this one can be migrated" } - return CheckResult{Status: status, Detail: fmt.Sprintf("detected deployment kind: %s", kind)}, string(kind) + return CheckResult{Status: status, Detail: detail}, string(kind) }) if err != nil { return report, err diff --git a/internal/preflight/checks_test.go b/internal/preflight/checks_test.go index 047cbcd..c811f9c 100644 --- a/internal/preflight/checks_test.go +++ b/internal/preflight/checks_test.go @@ -475,7 +475,16 @@ func withFakeDocker(t *testing.T) { // install with a fake docker on PATH. func dockerPreflight(t *testing.T, advisory bool) Report { t.Helper() - withFakeDocker(t) + // disk-space stats DataDir on this host, and container-data-volume + // wants it covered by a mount, so it has to be both: a real directory, + // mounted by the fake container. + dataDir := t.TempDir() + for _, p := range systemdUnitPaths { + if _, err := os.Stat(p); err == nil { + t.Skipf("host has %s, which detection prefers over docker", p) + } + } + fakeInspect(t, inspectDoc(t, nil, []Mount{dataVolume(dataDir)})) counterPath := filepath.Join(t.TempDir(), "invocations") binaryPath := writeFakeBinary(t, "0.15.5", counterPath) @@ -495,7 +504,7 @@ func dockerPreflight(t *testing.T, advisory bool) Report { // only thing that varies: whether stalwart-cli happens to be installed // on the machine running the tests is not what this is testing. report, err := New(Options{ - BinaryPath: binaryPath, ConfigPath: configPath, DataDir: t.TempDir(), + BinaryPath: binaryPath, ConfigPath: configPath, DataDir: dataDir, TargetVersion: "latest", ToolCheckAdvisory: true, DeploymentCheckAdvisory: advisory, }).Run(context.Background(), store, rs) if err != nil { @@ -504,21 +513,25 @@ func dockerPreflight(t *testing.T, advisory bool) Report { return report } -// A container has to be refused here, in preflight, and not later. Cutover -// already refuses it -- but cutover runs after the service has been stopped, -// so refusing there refuses with mail down, which turned an attempted -// migration into an outage. -func TestPreflightBlocksADockerDeployment(t *testing.T) { +// Being a container is no longer a refusal on its own - cutover can +// recreate one now. What refuses is specific to *this* container: compose +// management, and data that is not on a volume. Those live in +// container_test.go, and both still block. +// +// This previously asserted the blanket refusal. It asserts the replacement +// rather than being deleted, because "docker is allowed through here" is +// the thing that would be wrong to regress. +func TestPreflightAllowsAPlainContainerThroughTheKindCheck(t *testing.T) { report := dockerPreflight(t, false) - if !report.Blocking() { - t.Fatalf("expected a blocking report for a docker deployment, got:\n%s", report.String()) + if report.Blocking() { + t.Fatalf("a plain container on a volume should not be blocked:\n%s", report.String()) } var found bool for _, res := range report.Results { if res.Name == "deployment-kind" { found = true - if res.Status != StatusFail { - t.Errorf("deployment-kind status = %q, want %q", res.Status, StatusFail) + if res.Status != StatusOK { + t.Errorf("deployment-kind status = %q, want %q", res.Status, StatusOK) } if !strings.Contains(res.Detail, "docker") { t.Errorf("deployment-kind detail does not mention docker: %q", res.Detail) diff --git a/internal/preflight/container.go b/internal/preflight/container.go index 9ad27ca..77ce68d 100644 --- a/internal/preflight/container.go +++ b/internal/preflight/container.go @@ -300,9 +300,9 @@ func (c *Checker) runContainerChecks(ctx context.Context, runCheck checkFunc) er return CheckResult{Status: status, Detail: fmt.Sprintf( "data dir %s is not covered by any of the container's mounts (%s), so it lives in the writable layer and would "+ "not survive the container being replaced. Check whether --data-dir names the path inside the container", - c.opts.DataDir, describeMounts(facts.Mounts))}, "" + c.opts.DataDir, DescribeMounts(facts.Mounts))}, "" } - return CheckResult{Status: StatusOK, Detail: "container has writable mounts: " + describeMounts(writable)}, "" + return CheckResult{Status: StatusOK, Detail: "container has writable mounts: " + DescribeMounts(writable)}, "" }) return err } @@ -322,7 +322,8 @@ func mountSource(m Mount) string { return m.Source } -func describeMounts(mounts []Mount) string { +// DescribeMounts renders mounts for an operator-facing message. +func DescribeMounts(mounts []Mount) string { if len(mounts) == 0 { return "none" }