Files
stalwart-migrator/internal/rollback/verify.go
T
jcoffey-dev ade9906275 Implement internal/rollback and the service control it needs
Rollback was the one phase gating everything else: `run` without --dry-run
refused because this tool could not undo a cutover it had committed to.
That reason is now gone, and the refusal has narrowed to the fact that
there is no real cutover to undo yet.

internal/rollback implements ARCHITECTURE.md 4.8 as eight checkpointed
steps under PhaseRollback: verify-backup, stop-service,
preserve-failed-state, restore-data, restore-binary,
restore-service-config, start-service, verify-rollback.

Three things depart from what 4.8 specified, each for a reason:

- The backup is re-verified against its manifest *before* the service is
  stopped, which the design didn't call out. Finding a corrupt backup is
  survivable while the failed instance is still up, and unsurvivable once
  its data directory has been moved aside.
- BuildPlan is separate from Run, so every reason to refuse (closed
  rollback window, FoundationDB, no recorded backup, unknown deployment
  kind, missing database credentials) is found before anything is touched.
  The CLI prints that resolved plan and acts only with --yes.
- The restore is re-verified against the same manifest after writing. A
  restore that put back truncated bytes and reported success would be
  worse than one that failed outright.

Nothing from the failed attempt is deleted: the half-migrated data
directory and the displaced binary are moved to .failed-<run-id> names, so
a retry after the underlying issue is fixed still has both the evidence and
the artifacts. Afterwards a reduced validation suite runs against the
*restored* instance (version, reachability, directory counts) rather than
assuming the restore worked.

internal/service is a new package holding the systemd/Docker control this
needs. It's separate rather than living inside internal/rollback because
cutover will need the identical operations, and because the commands that
can take mail delivery down belong in one auditable place - the same
reasoning that makes stalwartapi the only thing speaking JMAP.
preflight.DeploymentKind is now a type alias for service.Kind so detection
and control can't drift apart. Its Active() reads `systemctl is-active`'s
output rather than its exit status: systemctl exits non-zero for every
non-active state, so exit-status logic would make "inactive" - the answer a
rollback most needs - look like a failure to read the state at all.

Also fixes a pre-existing bug in `status`: Go's flag package stops parsing
at the first positional argument, so `status <run-id> --state-dir X` looked
the run up in the default directory and reported it missing. `rollback`
would have inherited the same footgun on a command whose flags decide what
gets overwritten.

Still open: `confirm` cannot set RollbackWindowClosed. Rollback honours the
flag and refuses when it's set, but closing the window is the point of no
return for the backups this restores from, so it should land with the
retention policy 6 describes rather than before it.

Verified end to end against a fake systemd deployment: half-migrated data
restored to its original contents, failed state preserved, old binary
reinstalled and reporting 0.15.5, unit restarted, and a re-run of the
completed rollback inert.
2026-08-23 15:54:48 -07:00

171 lines
6.5 KiB
Go

package rollback
import (
"context"
"fmt"
"net/http"
"sort"
"strings"
"time"
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
"github.com/LINUXexpert-org/stalwart-migrator/internal/stalwartapi"
)
// VerifyOptions configures the reduced validation suite that runs against
// the *restored* instance - ARCHITECTURE.md §4.8 step 5. It's deliberately
// smaller than §4.7's post-migration suite: the question here is "is the
// old instance actually back", not "did a migration preserve everything",
// and every check has to be one that would still pass on a healthy 0.15.5
// install.
type VerifyOptions struct {
BinaryPath string // checked with --version; skipped if empty
ExpectVersion string // the run's recorded source version
AdminURL string
AdminUser string
AdminPassword string
HTTPClient *http.Client
// Snapshot is the pre-migration snapshot preflight captured. When set,
// the restored instance's account count and domains are compared
// against it - the "directory counts" half of §4.8 step 5. Per-mailbox
// message counts are deliberately not re-checked here: the restore is a
// byte-for-byte copy already verified against its manifest, so a
// per-mailbox walk would cost a lot of time on a large install to
// re-answer a question the manifest verification already answered.
Snapshot *checkpoint.PreflightSnapshot
// Timeout bounds how long the reachability check waits for the restored
// service to answer, since it was started moments earlier.
Timeout time.Duration
}
// Verify runs the reduced suite and returns one CheckResult per check.
// It always returns every result, even after a failure, so an operator sees
// the whole picture of a bad rollback in one pass rather than one problem
// at a time. The error is non-nil if any check failed.
func Verify(ctx context.Context, o VerifyOptions) ([]CheckResult, error) {
var results []CheckResult
fail := func(name, format string, args ...any) {
results = append(results, CheckResult{Name: name, Status: StatusFail, Detail: fmt.Sprintf(format, args...)})
}
ok := func(name, format string, args ...any) {
results = append(results, CheckResult{Name: name, Status: StatusOK, Detail: fmt.Sprintf(format, args...)})
}
skip := func(name, detail string) {
results = append(results, CheckResult{Name: name, Status: StatusSkipped, Detail: detail})
}
switch {
case o.BinaryPath == "" || o.ExpectVersion == "":
skip("version", "no binary path or recorded source version to check against")
default:
got, err := preflight.DetectVersion(ctx, o.BinaryPath)
switch {
case err != nil:
fail("version", "couldn't read the restored binary's version: %v", err)
case got != o.ExpectVersion:
fail("version", "restored binary reports %s, but this run started from %s - the rollback did not put the original binary back", got, o.ExpectVersion)
default:
ok("version", "restored binary reports %s, matching the version this run started from", got)
}
}
if o.AdminURL == "" {
skip("reachable", "no --admin-url configured - can't confirm the restored instance answers")
skip("directory-counts", "no --admin-url configured - can't compare the restored directory against the pre-migration snapshot")
return results, resultsError(results)
}
client := &stalwartapi.Client{
BaseURL: o.AdminURL, Username: o.AdminUser, Password: o.AdminPassword, HTTPClient: o.HTTPClient,
}
timeout := o.Timeout
if timeout <= 0 {
timeout = 60 * time.Second
}
if err := waitForPing(ctx, client, timeout); err != nil {
fail("reachable", "restored instance never answered at %s within %s: %v", o.AdminURL, timeout, err)
skip("directory-counts", "skipped because the restored instance isn't reachable")
return results, resultsError(results)
}
ok("reachable", "restored instance answered a JMAP session request at %s", o.AdminURL)
if o.Snapshot == nil {
skip("directory-counts", "this run has no pre-migration snapshot to compare against")
return results, resultsError(results)
}
snap, err := client.AccountSnapshot(ctx)
if err != nil {
fail("directory-counts", "couldn't read the restored instance's directory: %v", err)
return results, resultsError(results)
}
if problems := compareDirectory(o.Snapshot, snap); len(problems) > 0 {
fail("directory-counts", "restored directory doesn't match the pre-migration snapshot: %s", strings.Join(problems, "; "))
} else {
ok("directory-counts", "restored instance has %d account(s) and %d domain(s), matching the pre-migration snapshot",
snap.AccountCount, len(snap.Domains))
}
return results, resultsError(results)
}
// compareDirectory reports every way the restored directory differs from
// the pre-migration snapshot. Unlike the post-migration comparison in
// internal/validate, account names are compared exactly: the v0.16
// migration's bare-username-to-email rewrite is precisely what a rollback
// undoes, so a restored instance that still shows rewritten names has not
// been restored.
func compareDirectory(before *checkpoint.PreflightSnapshot, after *stalwartapi.Snapshot) []string {
var problems []string
if before.AccountCount != after.AccountCount {
problems = append(problems, fmt.Sprintf("%d account(s) before, %d after", before.AccountCount, after.AccountCount))
}
beforeDomains := append([]string(nil), before.Domains...)
afterDomains := append([]string(nil), after.Domains...)
sort.Strings(beforeDomains)
sort.Strings(afterDomains)
if strings.Join(beforeDomains, ",") != strings.Join(afterDomains, ",") {
problems = append(problems, fmt.Sprintf("domains were [%s], now [%s]",
strings.Join(beforeDomains, " "), strings.Join(afterDomains, " ")))
}
return problems
}
// waitForPing polls until the instance accepts an authenticated session
// request or timeout elapses. The service was started seconds ago, so the
// first attempt failing is expected rather than meaningful.
func waitForPing(ctx context.Context, client *stalwartapi.Client, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
var lastErr error
for {
lastErr = client.Ping(ctx)
if lastErr == nil {
return nil
}
if !time.Now().Before(deadline) {
return lastErr
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(500 * time.Millisecond):
}
}
}
func resultsError(results []CheckResult) error {
var failed []string
for _, r := range results {
if r.Status == StatusFail {
failed = append(failed, r.Name)
}
}
if len(failed) == 0 {
return nil
}
return fmt.Errorf("rollback verification failed: %s", strings.Join(failed, ", "))
}