Fix two defects a production-clone dress rehearsal exposed

Streamed a clone of a production store into the smoke VM - 3.6 GB, 12,361
settings, 6 accounts across 9 domains - and migrated it 0.15.5 -> 0.16.14
with the tool's own phases. The migration succeeded. Two defects surfaced
that no smaller instance could have shown, plus one finding worth recording.

1. Account roles broke on production-shaped names. v0.16 stores an account
   as a local part plus a domain reference: a v0.15 account named
   "[email protected]" becomes name "john" with a domainId. The generator
   passed the full address and the server rejected it outright ("Invalid
   email local part"), failing the apply. The smoke instance used bare
   usernames - alice, bob - and never exercised this.

   Fixed to use the local part. And because local parts are unique only
   within a domain - [email protected] and [email protected] both become
   "postmaster" - an ambiguous one is now refused with a warning rather
   than risking an upsert that grants Admin to the wrong account. Verified
   on the clone: the one admin came out with roles {"@type": "Admin"} and
   the other five accounts untouched.

2. Cutover's health check conflated liveness with credentials. A config
   fallback-admin does not survive the migration - v0.16's config is a
   store pointer, so the old [authentication.fallback-admin] block simply
   ceases to exist - so the credentials supplied for the pre-migration
   instance came back 401 on the migrated one, and the check reported the
   service as never having answered. It had answered; it was up and serving
   on all ten ports. Liveness and credentials are now separate: any
   response proves the service is up, and credentials that stopped working
   are a warning that names this cause.

Also recorded: a failed apply leaves the store in bootstrap mode, where
only Bootstrap objects are accessible. A half-applied plan is not a
partially configured server but an unusable one.

Timing, which is the other reason to rehearse: the recovery-mode conversion
of that 3.6 GB store took 2 seconds. A migration window is dominated by
waiting and verification, not data volume.

No production data in this commit; fixtures use example.net and the shapes
involved.
This commit is contained in:
2026-08-23 22:32:17 -07:00
parent 0d83283caa
commit 2ca9522f9a
6 changed files with 245 additions and 21 deletions
+35
View File
@@ -794,6 +794,41 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
them and refusing.
- **Nothing prevents concurrent runs.** Two invocations against the same
run-id would both proceed; there's no lock file or equivalent.
- **A full dress rehearsal has been run against a clone of production** -
the real 3.6 GB store, 12,361 settings, 6 accounts across 9 domains,
streamed into the smoke VM and migrated 0.15.5 -> 0.16.14 with the tool's
own phases. It succeeded, and the timings are the useful part: the
recovery-mode conversion of that store took **2 seconds**, and the whole
sequence from service-stop to service-start was seconds of work. A
migration window is dominated by waiting and verification, not by data
volume - worth knowing before scheduling one around store size.
Four things it found that smaller instances could not:
- **Account roles broke on production-shaped names.** v0.16 stores an
account as a local part plus a domain reference; the generator was
passing v0.15's full address and the server rejected it ("Invalid email
local part"). The smoke instance used bare usernames and never
exercised it. Fixed - and because local parts are unique only within a
domain, an ambiguous one is now refused with a warning rather than
risking Admin landing on the wrong account.
- **A failed apply leaves the store in bootstrap mode.** After a partial
apply the instance answers every management call with "The server is in
bootstrap mode. Only the 'Bootstrap' object type can be accessed until
the bootstrap process is complete." So a half-applied plan is not a
partially configured server, it is an unusable one, which raises the
stakes on apply failures considerably.
- **A config fallback-admin does not survive the migration.** v0.16's
config is a store pointer, so an `[authentication.fallback-admin]`
block in the old config.toml simply ceases to exist. The credentials an
operator supplies for the pre-migration instance therefore stop working
on the migrated one, and cutover's health check - which authenticated -
failed a cutover that had actually succeeded. Liveness and credentials
are now separate questions: any response proves the service is up, and
credentials that no longer work are a warning naming this cause.
- `tenant-admin` has no v0.16 equivalent and is reported as unrestorable
rather than silently dropped.
- **`rehearse` (§4.9) is designed but not built.** The command is still
`run --dry-run` with the old sandbox-cloning shape. Building it is mostly
deletion: the dump, convert and report pieces already exist and work