John's plan, 2026-09-19: stop the Enterprise server and the ihasmail
container, install the fork at its own path, copy the data across, bring up
INBUXA Admin and the new webmail, and every account carries on.
That is a better shape than the in-place swap this draft assumed, because
the rollback becomes a service swap rather than a restore: the old install
and its data are untouched, so going back is stopping one unit and starting
another. What it costs is whatever the fork accepted in between, since the
two stores diverge the moment the fork starts.
It buys one failure the in-place swap couldn't produce: both servers on one
set of ports, each with its own store, if a reboot brings the old unit back.
So the unit is disabled, not just stopped.
Also written down: the front ends' OAuth clients travel inside the store, so
a front end that keeps its client id and redirect URIs keeps working and one
deployed fresh needs them set up, which fails looking like an account
problem when it isn't.
/etc/ufw/user.rules is world-readable, so this needed no privilege after
all. The default input policy is DROP and 8899 is not among the allowed
ports, so pebble's connection to the suite's listener is dropped. That
matches the live run, where twelve probes from a container completed no
handshake while the same openssl reached pebble's own TLS port.
The nc readings that pointed the other way — instant refusals on closed
ports, where a DROP should hang — are still unexplained, and are left on
the page as unexplained rather than quietly dropped, since they are what
sent an earlier pass through this page in the wrong direction.
Nobody has added the allow rule and re-run the suite, so the fix is
written down as a prediction. The cutover draft says the same: the test
failing here is not evidence that renewal works on the host.
Steps 1 to 3 of SPEC §7 are met, so the remaining one is running the fork
as the mail server. The draft covers the sequence on the host, what to
check before letting mail flow, and what to watch in the first week.
Two things it refuses to gloss: rolling back stops being a snapshot restore
the moment the fork accepts a message, because nobody has tested whether
the Enterprise build reads a store the fork has written; and certificate
renewal is the failure that arrives 90 days late and quietly, on the one
path the suites couldn't settle.
Nothing in it has been rehearsed. The rehearsal on a copy is step 2 of
"Before the day", and it is what turns "the data opens" into "the server
runs on it".
Run from a copy of the stopped server's RocksDB store. Eight green lines,
which are worth reading carefully: six carry weight, and masked_email and
undelete carry none, because there are no masked addresses and retention is
off, so they iterate an empty list. tenant_compat checked the tenant, its
quotas and its members, but not what a tenant administrator can see, which
needs a --tenant-admin recording.
ai_compat is the one that might have looked vacuous and isn't: the twelve
LLM_ tags are there with the scores that were observed.
Three things had to be fixed first, each failing all eight identically and
none about the data: listener names, privileged ports, pending tasks. The
next import's copy will bring the same three, so they are written down.
SPEC §7: cutover steps 1 to 3 are met. Step 4 remains.
The run against INBUXA's store hung printing "Waiting for pending task
AcmeRenewal(...)": the copy carries that server's task queue, and a renewal
due in 2026-11 will not come due while a test watches it.
Under NO_INSERT the wait now skips tasks that aren't due and ones that have
permanently failed, which leaves the tasks the test itself caused — a
restore in undelete_compat comes due at once — and gives up after a minute
with the offending task printed. A test that was really waiting on its own
work now fails on its assertion, which says more than a spinner.
Ordinary runs are untouched: system_tests, which waits on tasks throughout,
still passes in 135s.
The first run against INBUXA's store failed all eight tests identically,
before reading a single record: the copy carries that server's listeners on
25, 443, 465, 587, 110, 143, 993 and 995, and nothing in a test run is
root, so each one failed with "Permission denied (os error 13)".
The builder now remembers the listeners it adds, and under NO_INSERT drops
build errors for any it didn't. Every other error still stands, including a
bind failing on one of its own, so this can't hide the case where the
harness's own port is taken.
The copy isn't edited for this: its listeners are simply not what a compat
run needs, and it reaches the server over the compat- ones instead. Checked
that a NO_INSERT run still boots and that scim_tests, which takes the
ordinary path, still passes.
Rehearsed the run against a real RocksDB store, and it died at startup
before checking anything: the harness inserts listeners of its own, the
registry keys them by name, and a real server already has a "jmap" and an
"imap". The message was "Primary key conflict on property name with
existing object NetworkListener", which says nothing about what to do.
Under NO_INSERT the harness now calls its listeners compat-jmap and so on,
and the same run gets through to the test's own checks.
run-compat.sh copies the store for each test and removes the copy after,
because several of these write to what they open: monitoring_compat purges
the history it reads and undelete_compat restores what it finds. The source
stays untouched, which matters when it is the only copy of a production
store anyone took that day.
INBUXA runs RocksDB, so a copy is a directory copy. The SQL backends would
need more than this: the harness builds its own container and connects to
fixed local credentials, so it cannot open a dump in place.
record-compat.py ran against the live server as a server-level
administrator: 8 accounts, which matches the dashboard, so it reached all
of them. One tenant with its quotas and members; no masked addresses, no
archived items.
The empty files are right rather than short. INBUXA has no masked
addresses and retention is off, so masked_email_compat and undelete_compat
iterate an empty list: they pass without comparing anything, which is worth
saying plainly, because a green run from either would otherwise read as
evidence of compatibility. That puts them where scim_compat and
per_domain_directory_compat already sit.
So one recording carries weight, expected.json, and it is made. SPEC §7's
cutover steps 2 and 3 come down to the tenant, the domains and the
accounts until either feature is switched on.
"You are not an owner of account X" is not a permission the server is
withholding; it is how far that identity can see. Answering it with "needs
sysMaskedEmailGet" sends you off to grant something that changes nothing.
A refusal that mentions ownership now says so, and says which account the
run wants: the administrator with the run of the server, with tenant
administrators passed as --tenant-admin.
A recording runs against a server that may not be up again soon, so an
administrator missing one permission shouldn't throw away the whole pass.
Each of the three is recorded on its own now: what the server allows is
written, what it refuses is named at the end with the permission it wants,
and the exit is still non-zero so an incomplete recording can't pass for a
finished one.
A refusal used to print the raw JMAP error. It now reads, for example,
"[email protected] may not x:Tenant/get: You are not authorized to perform
this action (needs sysTenantGet)".
Checked both ways: the refusal path against a stubbed client, where the
other two sections still record; the whole thing against a live test
server, which recorded 3 tenants and 53 masked addresses and exited 0.
A wrong tenant-administrator password failed only once the script had
already enumerated every account, on a server that may not be up for long.
All the identities are now checked against the session endpoint first, and
a refusal names each one that failed, with the two things that usually
explain it: basic authentication wants the account's name rather than its
email address, and an account with two-factor or OAuth-only sign-in needs
an app password. It also gives the curl line to test one on its own.
The unreachable message no longer suggests --insecure for a refused
connection; that hint is now only for a certificate it couldn't verify.
Three of the eight compat tests check INBUXA's data against a recording of
how the Enterprise server read it, and that recording can only be made
while that server is still up. SPEC §7 gives it 45 days from the notice, so
the capture shouldn't wait on the cutover being scheduled.
record-compat.py writes all three files: the tenants with their quotas and
members and what each tenant administrator sees, every masked address and
its state, and every archived item whole, since undelete_compat compares
every property it recorded. It only reads, and refuses to send a method
that isn't /get or /query, because it is the one tool here that runs
against the live server. Queries follow their pages, so a server that caps
one doesn't leave a short recording behind.
Exercised against the fork's own test server, which answers the same JMAP:
3 tenants with members, 8 masked addresses and 3 archived items, each in
the shape its test reads.
Probing further contradicted the previous two commits. During a live run,
when the suite is certainly listening on 8899, twelve handshakes from a
container completed nothing, with or without the ACME ALPN, while the same
openssl in the same container talks to pebble's TLS port and prints its
certificate. A listener that is up but unreachable from a container is what
a ufw DROP looks like. An instant refusal on a closed port, which nc saw
from two images, is not. Both were observed minutes apart.
So the ufw suspicion is neither confirmed nor dismissed, and the page now
says that rather than picking the reading that suits the last probe.
Settling it needs `sudo ufw status verbose` and a listener bound by hand,
neither of which this session could do.
What stands on pebble's own log, and does not depend on any of this: it
runs validations and marks the authorizations invalid, so "never validates,
stays pending" was wrong.
The previous commit called the bridge open on the strength of one nc run.
A later openssl s_client against the same closed port hung for its whole
timeout instead of reporting the refusal nc had just seen. Repeating the nc
test from a second image, with a control port and two closed ports, agreed
with the first: immediate refusal, which is not what a DROP looks like. The
openssl behavior is still unexplained, so the claim now carries what was
measured and the anomaly beside it.
The ALPN probe is recorded as proving nothing, for the same reason: it
hangs against a port with nothing behind it, so its silence during a
renewal says nothing about acme-tls/1.
Pebble's log is untouched by any of this: it validates and marks the
authorizations invalid, which is what makes the old explanation wrong.
The page blamed ufw for blocking the docker bridge, and said pebble never
validates, so the authorizations stay pending. All three are wrong.
From a container on the ACME network the host answers on both gateway
addresses: port 22 connects, and 8899 refuses at once with nothing
listening, where a DROP would hang. No firewall rule was read or changed to
establish that. Pebble's own log shows 20 validation attempts in the
regression run, five for each of the four tls.org names, each ending in
"INVALID by completed challenge". The challenges are answered and refused.
So the order goes invalid, no certificate is issued, and the test unwraps a
None. What's left to explain is the TLS-ALPN handshake. The responder is
intact and listen.rs picks it per connection from has_acme_tls_challenge,
which is computed when the network config is parsed, while the test adds
its provider after boot. That's written down as a hypothesis, not a
finding: it hasn't been tested.
None of the eight had ever executed, so all eight ran against an empty store
with synthetic inputs. The plumbing works: the documented JSON shapes parse,
and NO_INSERT stops each one before the harness touches the store, which a
sentinel file in each store directory confirmed — it survived every run,
including the one launched without NO_INSERT.
Two things the runbook got wrong, both of which would have cost a day on the
day the copy exists:
- TMPDIR is the copy's parent, not the copy. The harness opens
$TMPDIR/<test name>, so a TMPDIR pointing at the copy gets an empty store
created beside it and the test calls INBUXA's data missing.
- masked_email_compat and undelete_compat need INBUXA_COMPAT_MASKS and
INBUXA_COMPAT_ARCHIVED, which only the tests' doc comments mentioned.
Every run ended on a 401 raised as "Missing list in response", which reads
as INBUXA's data being wrong when the login is what's wrong. Each test now
authenticates once first and names the variable that failed.
Every one of the 14 was `#[cfg(not(feature = "enterprise"))]` on the arm the
fork always compiles: the Enterprise arms went with the import, and nothing
turns the feature on. Removing the attribute leaves the same code, now
unconditional, in 11 files.
Two of them looked like behavior worth checking before touching: the
`validate_tenant_quota` stub that always passes, and the refusal to cancel a
pending DestroyAccount task. The stub is vestigial — the rebuilt
multi-tenancy enforces quotas in `crates/features/src/tenancy/quota.rs` for
those objects and more — and the refusal is undelete's open question, which
this change leaves exactly as it was.
The binary builds with no new warnings, and `system_tests` and `jmap_tests`,
which cover the touched registry, task-manager and auth paths, both pass.
The feature definitions stay in the manifests, inert: taking them out would
widen every sync's diff for nothing.
The feature held the shared tests of features the fork hadn't rebuilt yet.
All nine are built and the last gate came off with per-domain directories,
so the feature was defined and documented but gated nothing.
The crate builds and holds the same tests without it: 113 by default, 118
with postgres, mysql and redis. SPEC §2.2b keeps its account of the first
import and now says when the gates came off, so a spec that still describes
a suite as gated reads as the record of its own date, which is what
monitoring.md's gate table already calls itself.
Ran it single-threaded on RocksDb: 87 passed, 3 failed, 23 ignored, all 113
tests a default build holds. lmtp_delivery passed this time, which is what
the note about queue timing under a sequential run predicted, so the four
documented failures are three. The ACME suite logged no 400 at all, so the
renewal fix holds; it still ends on the certificate that never came,
because pebble can't validate across the bridge with ufw up.
The line it replaces claimed 86 passed, 4 failed, 27 ignored from the same
command. That totals 117, and no feature set of this tree produces 117 —
113 by default, 118 with all three backends — with no test added or removed
since. Said so rather than presenting the two as a trend.
A few of upstream's dual-licensed files carry code from other projects
under MIT or BSD terms. The fork redistributes it, so their licenses
require the notices to travel with it. THIRD-PARTY.md reproduces them.
strip.py now reads the stripped tree's comments for another copyright
holder, another license, or a note that code came from somewhere else, and
names any file THIRD-PARTY.md doesn't cover. It reports, never fails: the
notice goes in with the merge that brings the release in.
On v0.16.22 it finds 14 files, all of them covered. The rest of the report
is byte-for-byte what the committed one says, so the scan disturbs nothing
it already did.
Re-ran all nine on containers removed beforehand, one suite at a time.
All nine pass. mysql_replica_position_tests failed the first time on test
18's lag assertion, with the pair half a minute old and still catching up,
and passed on a second run against the same containers; it fails before
the point where it changes the replica's settings, so it leaves nothing to
restore. Noted both, with each suite's time.
The table's STORE column said "default" for five suites, which reads as
"leave it unset". The harness has no default: it panics with "Missing or
invalid store type" before the suite starts. They run on RocksDb, as the
regression does, so the column now names it.
The renewal loop re-posted the challenge every time it polled and found the
authorization still pending. RFC 8555 section 7.5.1 has the client post a
challenge once to say it is ready and then poll; a server that has already
moved the challenge to "processing" refuses a second post, and pebble
answers 400 malformed, which failed the whole renewal.
Verified against pebble: the 400s are gone and the client polls. The suite
still can't finish on this machine, because pebble never reaches the test
server to validate the challenge; that path is the environment, and the
runbook now says so.
`cargo test -p tests` runs none of the fork's own feature suites except
what's inside `system_tests`: SCIM, per-domain directories, the sharded
stores and the four replica suites are all ignored, each needing a
container, a STORE the harness only builds on request, or both. The
runbook lists what each one needs, and the container-reuse trap.
`allowScimProvisioning` is cached with the domain as DOMAIN_FLAG_SCIM, but
it wasn't among the fields whose change drops the cached entry, so flipping
the flag changed nothing until something else evicted the domain. SCIM-60
says the change takes effect without a restart.
Found by the two acceptance checks that were written but never called from
the driver, so neither had ever run: `authority` (SCIM-58 to SCIM-60,
through `synchronize_account` itself) and `rate_limits` (SCIM-14). Both are
wired in now, and `scim_tests` passes with them.
Two long-standing flakes in system_tests, both timing:
- security: the ban period was set to one second, but the test makes a
hundred more requests before it checks that a valid password from the
banned address is refused, so under load the ban had already expired.
Five seconds, and the expiry check sleeps six.
- task: a task scheduled one second out was queried for straight away,
and under load the query landed after the task manager had run and
removed it. Three seconds.
Seven consecutive system_tests runs, neither recurred.
docs/spec/compat-tests.md lists each compat test, what it needs, what it
checks and what a failure means, and says plainly that they run against a
copy only, since the monitoring one purges the history it reads. The
scale-out and per-domain statuses record the acceptance tests that now
run.
A token the OIDC directory rejects is an authentication failure, so it
counts toward the ban; a network, provider or configuration fault stays
an error and doesn't. Before, a rejected token was an error too, so bad
tokens never led to a ban.
The Keycloak container now imports a second realm, so test 10 checks
/api/discover and the PACC record answer with each domain's own provider.
Test 18 checks that eight sign-ins during an outage don't ban the client,
while bad tokens do.
A FETCH with CHANGEDSINCE presents its mod-sequence to the read scope, so
a replica must have that change before it answers, as a JMAP sinceState
already did. replica_cluster_tests covers test 11: with the replica's
replay paused, a write on one node is read back through a second store
with its own marks, sharing through Redis.
Composite stores nest store futures deeply enough to pass rustc's default
query depth once both postgres and redis are compiled in, so the server
crates raise their recursion limit.
Two source-and-replica pairs run in containers: one replicating with
GTIDs, one by binary log position. mysql_replica_tests covers test 17
(tests 9, 10 and 12 with GTIDs) and mysql_replica_position_tests covers
test 18 (lag from Seconds_Behind_Source, and a replica whose account
lacks REPLICATION CLIENT getting no reads) and test 19 (a parallel
replica without replica_preserve_commit_order left out at startup).
The lag reader now takes Seconds_Behind_Source whatever numeric type the
server returns, accepts the older column name, and says in the log why it
gave up measuring.
Where the routing lives and what a read scope covers, which acceptance
tests the container suite runs, what isn't exercised (two nodes, the
primary stopped, and the MySQL paths, which are built but unrun), what
was settled from the code, and the known limits.
A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.
In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.
Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.
replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.
per_domain_directory_compat, ignored, checks a copy of INBUXA's data has
no directory, no server default and no domain with its own directory, as
observed. The status names where the rules live, which suite covers each
acceptance test and how far, what was settled from the code, and the
known limits. SCIM's status notes its test 5 now passes.
directory_tests now runs a new oidc module in place of the removed one,
with Keycloak as example.org's own directory: first sign-in creates the
account with its name and group, an existing account is reused, a token
named as another user is refused, password sign-in is refused, forged
JWTs (HS256, unknown kid, another issuer, expired) are refused, an OIDC
address is a recipient only once an administrator creates it, and sync
can't pass a tenant's account limit. scim_oidc_tests, deferred until
this feature, passes.
A write to x:Directory or Authentication reloads the directories at once,
here and across the cluster. A directory that fails to open is logged as
a warning against its id and becomes unavailable; before, it was a build
error, and any build error stopped every later reload from applying.
per_domain_directory_tests covers acceptance tests 1, 3, 4, 6, 7, 9, 11,
15 and 19 over SQL directories on SQLite files: each domain against its
own directory and the default, no fallback to internal passwords, a
directory answering for another directory's domain, aliases and groups
dropped, recipients through the directory and a 4xx while it's down, app
passwords while it's down, password changes refused and then allowed
after a move to the internal directory, linked directories, tenant
foreign keys, and changes taking effect without a reload.
The two lookups every caller uses now honor Domain.directoryId, then the
server default, then the internal directory, so sign-in, bearer routing,
recipient lookup, discovery, the PACC record and the refusal of password
changes on external accounts all follow the domain. A directoryId, or a
server default, naming a directory that doesn't exist is unavailable,
never the internal directory.
A directory speaks only for the domains it serves: an account it returns
on another directory's domain is refused, for sign-in and recipients
alike, and aliases and group claims on such domains are dropped with a
warning. A bearer token must belong to the user the client names, or the
name must be one of its aliases with alias sign-in allowed. Accounts and
groups that sync creates pass the tenant checks, limits included.
Where the code lives, which suite covers each acceptance test, what isn't
exercised, what was settled from the code, and that read-replica routing
(ST-5 to ST-15) waits for per-domain directories.
A Sharded blob store places each blob on xxh3(key) mod N, over the whole
key; reads fall back to the other members, so blobs placed under an
earlier member list stay readable, and deletes find them wherever they
are. The member list is recorded in the data store (secrets left out):
added or reordered members are a warning, a missing one refuses to open.
Blobs are compressed and marked before they reach a member. A Sharded
in-memory or lookup store sends each key to its home Redis member, and
prefix deletes and purges to all; a node whose member list differs from
the recorded one logs an error and runs on. Members are checked for
duplicates and must all open.
Until read-replica routing is built, each configured replica is reported
at startup instead of being silently ignored, and nothing connects to it.
The new scaleout_blob_tests covers tests 2 to 7, and the existing blob
suite passes against three FileSystem members (BLOB_STORE=Sharded).
Where each part lives, which suite covers each acceptance test, test 5
deferred to per-domain directories and test 31 unrun until a copy of
INBUXA's data, what was settled from the code, and the known limits.
A principal's key without unlimitedRequests gets 429 with Retry-After
over the limit, while its unlimited key still works. scim_compat, ignored,
checks a copy of INBUXA's data reads back as observed: no domain open to
SCIM and no externalId.
scim2-client builds its models from /Schemas, so meta is described there
as the mapping tables give it, without lastModified. The tester container
now shares the host's network, so a host firewall that drops the Docker
bridge doesn't block the test server. With SCIM_CONFORMANCE=1, the
scim2-client lifecycle (12 steps), the eight replayed Okta, Keycloak and
Entra payloads, and scim2-tester (errors only for its generated
non-address userName) all pass.
The push router records which account each subscription belongs to, and
a new Revoke event drops every subscription the account holds, so its
IMAP IDLE, JMAP event streams and WebSockets on this node end. Other
users' subscriptions to its shared mailboxes stay. Test 28 now checks an
open IDLE is ended, and that the account's password over HTTP and its
own API key, both already cached, are refused on the next request.
On such a domain, sign-in sync creates no account (the person gets an
ordinary authentication failure), changes nothing on an existing one,
and creates no group from a groups claim. Without the flag sync works as
before, and turning it off hands accounts back to sync with no restart.
Checked through synchronize_account itself; acceptance test 5 does the
same over OIDC once per-domain directories exist.
Every SCIM operation becomes the x:Account get, query or set JMAP makes,
as the service principal, so permissions, tenant scope and limits,
address uniqueness and account destruction are enforced in one place.
Discovery is anonymous; everything else takes an API key as a bearer
token and nothing else. Domains open to SCIM carry a flag in the domain
cache. Filters take eq and and, answered from the account indexes, with
unindexed attributes checked on at most 200 candidates. Cursors are
stateless, HMAC-sealed under the server key. PATCH applies to the
resource in memory and saves it as a PUT, so it is all or nothing.
Groups get an address from their display name on the principal's
domain; membership is written on each user.
Every write emits one of five new scim.* events (ids 637 to 641), also
added to the packaged schema. The helpers the surviving SCIM suites
import are rebuilt from the spec; scim_tests runs the new acceptance
suite and the surviving tenant isolation suite, and both pass.
The filter parser reads all of RFC 7644's grammar, so a server supporting
only eq and and can name the construct it refuses. PATCH paths take a
schema URN prefix, sub-attributes and value filters.
jcoffey-dev