Files
stalwart-migrator/internal/stalwartapi/principal.go
T
jcoffey-dev e955a41d58 Fix the domain/tenant mismatch that failed the second live migration
The second production attempt failed during recovery-mode apply, with the
mail server already stopped and the store already at schema v6:

    create Account restore-13: invalidForeignKey | Object id: Domain#d

v0.16 requires a tenant-scoped Account to sit on a Domain owned by that
same tenant, for its primary domain and for every alias. v0.15 imposed no
such rule, and migrate_v016.py carries the two facts over independently:
_build_domains sets a domain's memberTenantId only for domains declared as
their own `domain` principal with a `tenant`, while _build_user sets the
account's from the account's own record. A domain that exists only inside
an email address is inferred, gets no tenant, and every tenant-scoped
account using it is then rejected.

Established by reproduction rather than inference: a synthetic v0.15
principal dump, run through the unpatched upstream converter and applied to
a real 0.16.14 in recovery mode, reproduces the error character for
character - the `#d` is the server's own object id for the offending
domain, not a plan client-id. The same harness establishes which directions
are constrained: a tenant-scoped account on a tenant-less domain or on
another tenant's domain is rejected; a global account on a tenant-owned
domain is accepted.

  - applyplan.ReconcileDomainTenants repairs the plan between convert and
    apply. Where a tenant-less domain is used only by accounts of one
    tenant, the domain adopts that tenant - the sole assignment that both
    applies and keeps every account. Where accounts genuinely disagree it
    changes nothing and reports why, because forcing such a plan through
    would mean dropping mailboxes.
  - stalwartapi.FetchTenantLayout maps tenant membership over the 0.15 REST
    API and predicts the outcome with the same rule the server enforces, so
    preflight either warns about the domains that will adopt a tenant or
    fails - while the service is still running.
  - The plan is parsed generically rather than through the typed Operation.
    A real export.json mixes shapes: `create` maps a client-id to an object,
    `update` carries a flat one. The typed form failed on the first `update`
    line, found by running against actual converter output. Numbers decode
    as json.Number so a 10 GiB quota is not rewritten as 1.073741824e+10.

Corrects the record: the previous commit claimed the converter emits every
Account with `tenantId: null` and made preflight refuse every multi-tenant
install on that basis. The field is memberTenantId, the converter does
populate it, and the export had been inspected for a key no version of the
script ever writes. The refusal is now narrowed to what v0.16 genuinely
cannot represent.

The same fix has been prepared for migrate_v016.py upstream. The tool
downloads that script rather than vendoring it, so the repair stays here
until a released version carries it, and is a no-op on a consistent plan.
2026-08-24 00:27:26 -07:00

235 lines
8.6 KiB
Go

// SPDX-FileCopyrightText: 2026 LINUXexpert-org
// SPDX-License-Identifier: GPL-3.0-or-later
package stalwartapi
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strings"
)
// Stalwart 0.15.x - the version this tool migrates *from* - has no
// urn:stalwart:jmap capability and no JMAP management endpoint: POST /api
// returns 404 there. Its management API is REST, at GET /api/principal,
// and that is the only way to enumerate the pre-migration directory.
//
// This was found by running preflight against a real 0.15.5 instance, not
// from documentation: the published schema reference documents 0.16, and
// following it alone produced a tool that could not read the source
// version it exists to migrate. The shapes below were confirmed against
// that live server.
//
// GET /api/principal?types=individual&limit=100&page=1
// {"data":{"items":[{"id":4,"type":"individual","name":"alice",
// "emails":["[email protected]"],"usedQuota":9207}],
// "total":3}}
//
// Two limits of the 0.15 side are worth stating plainly, because they
// decide what the post-migration comparison can actually assert:
//
// - There is no per-mailbox message count anywhere in this API, and the
// impersonation mechanism 0.16 offers (the `<target>%<impersonator>`
// composite login MailboxSnapshot uses) returns 401 on 0.15.5. So a
// pre-migration snapshot cannot carry message counts, and a
// before/after comparison of them is impossible for the boundary
// migration this tool is built for.
// - What both versions do expose per account is used quota in bytes
// (`usedQuota` here, `usedDiskQuota` on 0.16's x:Account). That is a
// real content measure - it moves when mail is lost - so it is what
// the integrity comparison uses across the boundary.
const restPrincipalPageSize = 100
// stalwartManagementCapability is advertised by instances whose management
// API is the JMAP one (0.16+). Its absence is what distinguishes a 0.15.x
// instance, and is a positive signal rather than an inference from a failed
// call.
const stalwartManagementCapability = "urn:stalwart:jmap"
// hasRESTManagement reports whether this instance serves the v0.15.x REST
// management API, by asking it for a single principal.
//
// This replaced a capability check, which cannot work: *neither* version
// advertises urn:stalwart:jmap. A real 0.15.5 doesn't, and a fully
// migrated, fully configured 0.16.14 doesn't either - verified against
// both. Dispatching on the capability sent 0.16 instances down the 0.15
// REST path, where every call 404s.
//
// So the client asks what the instance actually serves instead. 0.15.x
// answers GET /api/principal with a principal list; 0.16.14 returns 404
// for that path and serves JMAP management objects at the endpoint its
// session document advertises. A cheap probe is less elegant than a
// declared capability and has the considerable advantage of being true.
func (c *Client) hasRESTManagement(ctx context.Context) (bool, error) {
endpoint := strings.TrimRight(c.BaseURL, "/") + "/api/principal?limit=1"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return false, err
}
req.SetBasicAuth(c.Username, c.Password)
resp, err := c.httpClient().Do(req)
if err != nil {
return false, fmt.Errorf("stalwartapi: probe %s: %w", endpoint, err)
}
defer resp.Body.Close()
io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<16))
switch resp.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
case http.StatusUnauthorized, http.StatusForbidden:
// The path exists but these credentials can't use it. Say so here
// rather than falling through to the other API and reporting a
// confusing error from there instead.
return false, fmt.Errorf("stalwartapi: %s returned %s - the credentials are not accepted for management operations", endpoint, resp.Status)
default:
return false, nil
}
}
type restPrincipal struct {
ID int `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Emails []string `json:"emails"`
UsedQuota int64 `json:"usedQuota"`
}
type restPrincipalPage struct {
Data struct {
Items []restPrincipal `json:"items"`
Total int `json:"total"`
} `json:"data"`
}
// restPrincipals fetches every principal of the given type, following the
// API's 1-based page/limit pagination rather than assuming one request
// returns everything - an install with more accounts than the page size
// would otherwise be silently truncated, and a truncated "before" snapshot
// would make the post-migration comparison claim more than it checked.
func (c *Client) restPrincipals(ctx context.Context, principalType string) ([]restPrincipal, error) {
var all []restPrincipal
for page := 1; ; page++ {
q := url.Values{}
if principalType != "" {
q.Set("types", principalType)
}
q.Set("limit", fmt.Sprint(restPrincipalPageSize))
q.Set("page", fmt.Sprint(page))
endpoint := strings.TrimRight(c.BaseURL, "/") + "/api/principal?" + q.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.SetBasicAuth(c.Username, c.Password)
resp, err := c.httpClient().Do(req)
if err != nil {
return nil, fmt.Errorf("stalwartapi: list principals: %w", err)
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("stalwartapi: GET %s returned %s: %s", endpoint, resp.Status, strings.TrimSpace(string(body)))
}
if readErr != nil {
return nil, fmt.Errorf("stalwartapi: read principal list: %w", readErr)
}
var parsed restPrincipalPage
if err := json.Unmarshal(body, &parsed); err != nil {
return nil, fmt.Errorf("stalwartapi: parse principal list: %w", err)
}
all = append(all, parsed.Data.Items...)
if len(parsed.Data.Items) == 0 || len(all) >= parsed.Data.Total {
return all, nil
}
}
}
// principalSnapshotREST builds a Snapshot from the 0.15.x REST management
// API. MailboxCounts is deliberately left empty: see this file's opening
// comment for why counts cannot be obtained from a 0.15 instance at all.
func (c *Client) principalSnapshotREST(ctx context.Context) (*Snapshot, error) {
individuals, err := c.restPrincipals(ctx, "individual")
if err != nil {
return nil, err
}
domainPrincipals, err := c.restPrincipals(ctx, "domain")
if err != nil {
return nil, err
}
snap := &Snapshot{
AccountCount: len(individuals),
UsedQuota: make(map[string]int64, len(individuals)),
}
for _, p := range individuals {
snap.UsedQuota[accountKey(p)] = p.UsedQuota
}
domainSet := map[string]bool{}
for _, d := range domainPrincipals {
if d.Name != "" {
domainSet[d.Name] = true
}
}
// Fall back to the domains implied by account addresses if the
// instance has no explicit domain principals.
for _, p := range individuals {
for _, email := range p.Emails {
if at := strings.LastIndex(email, "@"); at >= 0 && at+1 < len(email) {
domainSet[email[at+1:]] = true
}
}
}
for d := range domainSet {
snap.Domains = append(snap.Domains, d)
}
sort.Strings(snap.Domains)
return snap, nil
}
// accountKey identifies an account the same way both API generations can:
// by its primary email address where it has one, falling back to the bare
// login name. The v0.16 migration rewrites bare names into addresses, which
// is exactly why the comparison side matches on local part as well.
func accountKey(p restPrincipal) string {
if len(p.Emails) > 0 && p.Emails[0] != "" {
return p.Emails[0]
}
return p.Name
}
// TenantNames returns the tenant principals on a v0.15.x instance.
//
// Multi-tenancy has to be established before a migration starts. v0.16
// requires a tenant-scoped account to sit on a domain owned by that same
// tenant; v0.15 did not, and Stalwart's converter carries the two facts
// over independently, so an install that is valid today can convert into a
// plan the new server rejects with `invalidForeignKey` on the Domain
// reference - during the recovery-mode apply, with the mail server already
// stopped and the store already at schema v6. See FetchTenantLayout, which
// builds on this to predict that outcome, and applyplan.ReconcileDomainTenants,
// which repairs the plan.
func (c *Client) TenantNames(ctx context.Context) ([]string, error) {
tenants, err := c.restPrincipals(ctx, "tenant")
if err != nil {
return nil, err
}
names := make([]string, 0, len(tenants))
for _, t := range tenants {
if t.Name != "" {
names = append(names, t.Name)
}
}
sort.Strings(names)
return names, nil
}