90 Commits
Author SHA1 Message Date
jcoffey-dev d264429298 Cutover: the old install is retired, and that changes what a rollback is
John, 2026-09-20: committed, remove it. Done the same day as the cutover and
before the certificate-renewal gate this page proposed, which is the
operator's call to make and is recorded as such.

Archived first and the archive verified off-host by checksum, then the tree,
the unit and its drop-in removed. The stalwart user stays: redis-server runs
as it, which step 0's pgrep had already shown and which is exactly the kind
of thing that makes "remove the service user" a bad reflex.

Two things the doing taught, both for migration.md. The unit does not live
inside the tree it manages, so an archive of /opt/stalwart alone is not a
restorable rollback -- stalwart.service and its drop-in had to be saved
separately, and a tool that archives before retiring has to take them too.
And retiring changes what a rollback means: up to that moment it was a
service swap against a store still on disk, minutes and no restore; after
it, an untar, a chown, a unit to reinstate and a webmail image that is no
longer on the host. Still possible, slower, and no longer what the Rollback
section describes.
2026-09-20 01:58:34 -07:00
jcoffey-dev 80972695b9 Cutover: the run, 2026-09-20 — 78 seconds, and what it found
INBUXA is on the fork. The window was 78 seconds, 2.2 G of store copied in
1.4, mail queued at senders and nothing lost, both front ends up within the
hour, and the rollback never needed.

The page stops being a plan and becomes the record of one, which is what
migration.md is built from. Written up by kind rather than in order, because
nobody reading it later wants the chronology.

What step 0 was worth, most of all. Reading `systemctl cat stalwart` before
touching anything found a network namespace nothing in this document knew
about, and that was two failures rather than one: a collision with nginx on
443, loud and quickly understood, and egress from the wrong address, which
would have cost the provider's port 25 exemption and failed outbound mail
at every receiver with no local symptom and no logs to find it in.

What the copied store brought with it, three times in three guises: a
tracer still writing to the old tree, Stalwart's own web interface being
served from the registry's Application entries, and — from the other
direction — a front end configured by copying variable names the fork had
renamed. A front end reporting healthy is not a front end talking to the
right server; the health check passed while it pointed at example.com.

What this document had wrong: `systemctl mask` cannot mask a unit that
lives in /etc/systemd/system; there is no "let mail flow" gate, because the
fork takes port 25 as it starts and the free-rollback window closes there;
and IMAP's INBOX is not JMAP's account, so the counts differ before and
after alike.

And the bug it found, which only exists when §5.3 is followed: INBUXA Admin
hosted off the mail server cannot fetch its schema, because that response
was publicly cacheable and immutable for a year while its CORS headers vary
by origin. Fixed in 7c4add8.

The Open section loses the two the run settled and gains the two it
created, and names the ACME date: ~28 October, because R12 renews at the
halfway point and nothing brings that forward.
2026-09-20 01:47:10 -07:00
jcoffey-dev 0117500d85 Cutover: the sheet to follow at the terminal on the night
cutover.md is the reasoning and is too long to read at 2am. This is the
same sequence as commands, for this install: eight mailboxes, two people
and a printer.

That scale settles three things the general plan leaves open. The store is
small enough that the two-pass rsync buys nothing, so it is one cp inside
the window and a simpler sequence when it matters. "Every account still
works" is two sign-ins. And a reboot inside the window is affordable, which
is the only honest proof that the fork comes up on boot and the old unit
does not — is-enabled says what is configured, a reboot says what happens.

The rollback leads with chattr -i, because step 7's guard stops the
Enterprise build exactly as it stops the fork, and finding that out during
a rollback costs the worst ten minutes of the night.

And it names the printer as its own check. It is the one user that cannot
report a fault: a hardcoded credential and an old TLS stack, of the kind a
stricter default quietly refuses. The people will phone; the printer will
just stop, and nobody will notice for a fortnight.
2026-09-19 23:08:19 -07:00
jcoffey-dev 8c3450dffa Migration: copy in two passes, and time the second one
Taken from the cutover page, where the reasoning is worked out: with no
filesystem snapshot to take, a single copy inside the window makes every
byte downtime. A first pass while the server still serves moves the bulk
and is deliberately inconsistent; a delta pass after the process has exited
makes it consistent and moves little, because a RocksDB store is mostly
immutable SST files.

That is the difference between a window proportional to the store and one
proportional to the delta, which is the number this tool exists to
advertise. Also carried over: hand the copy to the user the fork runs as,
and make the original unwritable before the fork starts, since the old
server's store lock was the only thing holding that line until it stopped.
2026-09-19 22:58:30 -07:00
jcoffey-dev 7196f9dda2 Specs: when the source goes out, and who owes the offer
AGPL section 13 starts at the cutover, not at the announcement: the fork is
a modified AGPL program and its users reach it over a network. Settled
today that the source is released after the cutover, with the links live
then, and in the meantime the server's users are the operator's household,
so the people owed an offer and the people holding the repository are the
same people.

Anyone migrating their own server inherits that obligation on their first
day and has no such overlap, so the migration tool says so at the end of a
successful run instead of leaving it to be discovered.
2026-09-19 22:58:30 -07:00
jcoffey-dev efe0b01bac Specs: what may carry the Stalwart name, and when it stops
Two decisions §2 implied but never settled.

§2.4 said no "Stalwart" in UI text; §2.6 requires the startup banner, the
JMAP implementation string and OpenTelemetry's service.version to name the
base. Taken literally, the first would strip exactly what the second exists
to keep. Version and build metadata are now exempt, with the distinction
written down: §2.4's first bullet governs identity, the new one governs
provenance.

A second bullet covers material outside the product. The name appears with
its trademark attribution; the fork relationship is stated once in the
provenance or license section; the migration path names the server it
migrates from, because an operator searching for it has to find it. The base
version stays out of taglines, page titles and social previews, where it
reads as a source identifier rather than a fact. No comparison in either
direction: what INBUXA offers is stated on its own terms.

§2.6 said the base drops out of the version string "when it happens", which
left someone judging the moment. The trigger is now the first release that
isn't a rebase on an upstream tag. Because the reason for publishing the
base is one-way store conversion, and that outlives the string, the
amendment routes it to the upgrade documentation rather than letting it go.

§8 no longer asks whether the fork follows upstream's version numbers. §2.6
answered that on 2026-09-18: it has its own.
2026-09-19 22:53:28 -07:00
jcoffey-dev 04a5a8bdc2 Specs: features 6 to 9 are built, and the table hadn't caught up
Monitoring, SCIM, scale-out storage and per-domain directories each say
"Built 2026-09-19" in their own implementation-status sections, and the
code is where those sections say it is. The §4 table still listed them as
specs awaiting a build, which made the whole feature set look half-finished
to anyone reading the table alone.

Each row now names where the feature landed, as rows 1 to 5 already did.
Monitoring and per-domain directories sit outside crates/features, so their
rows name the paths rather than the crate.

§2.2b already recorded that all nine were rebuilt by 2026-09-19 and every
pending-rebuild gate came off. Only the table lagged.
2026-09-19 22:53:28 -07:00
jcoffey-dev 848bfeda4e Migration: name the four ways this goes badly, not just that it might
The section warned in general and so warned about nothing. An operator
reading "it can fail in ways it cannot undo" learns less than one reading
that the window is usually longer than guessed, that opening the source
store with the new server ends the rollback permanently, that a rollback
after mail has flowed does not bring that mail with it, and that a
certificate which stops renewing says nothing for ninety days. Each of
those has been measured or seen; each has something the operator can do
about it.

Also sharpens the part that matters most and is easiest to get wrong: the
old install is a service safety net, not a data one. One copy, same
machine, one moment. A backup is a copy elsewhere that has been restored
from, and anyone who cannot say when they last restored one does not yet
know whether they have one.

And replaces the flat line about nobody else being responsible with what
it was trying to say: the operator carries the outcome, because this is
software running against a server it has never seen, holding data somebody
else depends on.
2026-09-19 22:52:12 -07:00
jcoffey-dev 4ce5fc6f68 Migration: say what the operator is carrying, before the tool moves anything
Asked for by John, 2026-09-19. A tool that stops somebody's mail server
should say so while there is still time to stop it, rather than leaving the
licence to have said it in a file nobody opens. AGPL-3.0 §15 and §16
already disclaim warranty and liability and this narrows neither; it is the
same thing at the moment it is useful.

Specific rather than blanket, because a blanket one protects less and helps
nobody: what the tool does to the server, what a rollback does not return,
and that backups and recovery are the operator's. Keeping the source
install is not a backup — it is one copy, on one machine, of one moment,
and the same disk failure takes both.

Paired with what the tool does to earn the trust it is asking for, because
that is the half that reduces the friction: a dry run the real run refuses
to start without, never writing to the source, verification before mail
flows with automatic rollback, the old install kept, and every phase timed.
--yes skips the prompt, not the dry run.
2026-09-19 22:48:50 -07:00
jcoffey-dev 3e69b6139f Cutover: the old install is a reference, kept until someone says otherwise
John, 2026-09-19, on both counts. The old install is kept, shut down, not
removed: its unit installed and disabled, its store read-only, started
again if a rollback is ever wanted. When it stops being worth the disk the
tool asks — keep or delete — rather than deciding, because it does not
remove the thing its own rollback depends on.

And the new stack depends on nothing in it. That is the shape's purpose:
/opt/stalwart is a reference, everything needed is copied to new paths, and
when it goes nothing notices. Nothing in the fork works against that —
inbuxa.service substitutes its own prefix, no path names the old tree, and
certificates and ACME keys are in the registry inside the store — so a
dependency, if one appears, was made by hand during the move.

Which is worth proving rather than asserting, and reversibly: nothing open
under the old tree, then rename it and leave it a day under real traffic.
Deleting proves the same thing and cannot be undone.

Two corrections this forces. The rollback has to make the original store
writable again first: the guard of step 3 blocks the Enterprise build
exactly as it blocks the fork, and finding that out during a rollback is
the worst time. And the copy has to be chowned — rsync -a preserves
ownership, so it arrives owned by the old service user while the unit runs
as User=inbuxa.

Also drops the stale "untested" wording about carrying the data back. It
was tested; it is impossible.
2026-09-19 22:45:09 -07:00
jcoffey-dev 7c27bae4f3 Cutover: ext4, so copy in two passes and time the second one
The mail host is ext4 (John, 2026-09-19). There is no filesystem snapshot
to take, so the sequence as written puts the whole store inside the
downtime: stop, copy everything, start.

An rsync before the stop and a second one after it moves the bulk while
mail is still flowing and leaves only the delta in the window. The first
pass is knowingly inconsistent and exists only as a warm-up; the second,
once the process has actually exited, is what makes the copy consistent.
A RocksDB store suits this, being mostly immutable SST files: what changes
between the passes is the WAL, the MANIFEST and any compaction output.

Step 3 now says so, and says to time both during the rehearsal, because
the second pass is the window and nobody knows yet how long it is.
2026-09-19 22:41:59 -07:00
jcoffey-dev bfda639d73 Cutover: the fork must not live on /opt/stalwart, which is to be removed
John, 2026-09-19: the fork takes a copy of the config rather than pointing
at the old one, and /opt/stalwart goes away once the migration is
confirmed.

That turns step 4's store path from a free choice into a constraint. The
step said an existing install keeps whatever its configuration names, which
is true of the server and no longer true of this migration: nothing the
fork runs on may sit under a directory that is going to be deleted. Worth
checking before starting rather than after removing.

Removing it strands nothing else. ACME account keys and issued certificates
are written to the registry, inside the store, so they came across with the
copy; /opt/stalwart holds the old binary, its config and its data and
nothing the fork reads.

What it does end is the rollback, so the new section says when. The
rollback stops being one within hours anyway — after mail has flowed,
going back means losing what arrived since — so the real question is how
long to keep a cold copy of the pre-cutover state. The gate is the first
certificate renewal, which is the one check in "The first week" whose
failure would send anyone back; forcing a renewal closes it in a day
rather than ninety. Archive the store off-host before removing the
directory.
2026-09-19 22:36:51 -07:00
jcoffey-dev daaa06a5fb Cutover: stopping the old server is what removes the guard, so add one
Step 2 stops stalwart.service and step 4 points the fork at a store path.
Between those two moments nothing protects the original, and the rehearsal
had already shown that one open by the fork costs the rollback for good.

What protects it until then turns out to be the running server itself:
RocksDB refuses a second opener with "While lock file: LOCK: Resource
temporarily unavailable". So the intuition that a service shutdown prevents
the mistake is backwards — the shutdown is what enables it.

Measured, in probe_guard.py, in the three states that matter: held by the
running server, the fork is refused and the rollback is intact; stopped but
read-only, the fork is refused while rotating its own log and the Enterprise
build still starts on it afterwards; stopped and writable, the fork opens,
adds its column family, and upstream never starts again.

So step 3 now makes the original read-only as soon as the copy is taken,
which turns a discipline problem into a one-line one, and the answered
section carries the table.
2026-09-19 22:34:45 -07:00
jcoffey-dev 67619f64e9 Cutover: rehearsed, and the Enterprise build can't read the fork's store
The runbook said nothing in it had been rehearsed. Now the sequence has
been, on data made up for the purpose: upstream 0.16.22 in a container as
the install running today, the fork beside it, both unprivileged with
CAP_NET_BIND_SERVICE. It rehearses the sequence, not the data, which is
what the compat tests are for. 27 of 27 checks passed and the rollback
took 1.5 seconds.

The question §"Open" asked about carrying a store back is answered, and
the answer is no. Upstream refuses to start on a store the fork has
opened: "Column families not opened: _". The fork adds one RocksDB column
family for masked email (SUBSPACE_INBUXA = b'_') and opens with
create_missing_column_families, so it creates it on first open; upstream
has no descriptor for it and RocksDB will not open a database holding one
it was not told about.

That makes step 3's "a copy, not a move" load-bearing in a way the step
did not say. One open by the fork is enough: pointing it at the original
even once, to check something, leaves the Enterprise install unable to
start, and there is no rollback after that. It fails loudly and before
reading anything, which is the good version of this failure, but it is
not recoverable.

Two things the rehearsal found that would have wasted time on the day:
memberTenantId does not come down from the domain and is refused on
create, so a tenant "admin" set up the obvious way is a server
administrator and the check passes while proving nothing; and IMAP's
INBOX is not JMAP's account, because mail from an unauthenticated sender
is filed as spam, so the two counts differ before and after alike.

What the rehearsal does not cover is in its README and in §"Open":
systemd and `systemctl disable stalwart` above all, ACME renewal, load,
the front ends, and INBUXA's own data.
2026-09-19 22:23:10 -07:00
jcoffey-dev f338ddf57d Specs: point a stock ihasmail at the migrated server afterwards
Asked for by John, 2026-09-19. Public ihasmail is Stalwart-facing and knows
nothing about INBUXA, so running an unmodified one against the migrated
server checks something the fork's own suites cannot: that a client written
for upstream still works.

Each difference it finds is one of two things, and the point is to say
which: a regression against upstream's contract, which the fork's tests
would not catch because they test the fork; or a feature that now expects
INBUXA's own front ends, which belongs in the contract and the release
notes rather than in a user's surprise.

After mail is flowing, not as a gate. It informs the contract; it doesn't
block a cutover.
2026-09-19 21:36:10 -07:00
jcoffey-dev 7d468ad22e Specs: a migration tool, with rollback as a path rather than an appendix
INBUXA's cutover is the first run of something other operators will want:
an existing Stalwart server becoming an INBUXA one with nothing re-entered
and nothing re-issued. Accounts, passwords, app passwords, OAuth sessions,
aliases, tenants, DNS records and provider settings, certificates and ACME
state, Sieve scripts, the queue and the mail all live in the store, so a
migration that copies the store carries them.

Offered beside the fresh-install workflow, which has different questions to
ask, so §6.1 now names both.

It rolls back, which is where it parts company with stalwart-migrator:
that one upgrades in place and says outright it cannot undo a migration.
This one never writes to what it migrates from, so going back is stopping
one service and starting another. Rollback is automatic when verification
fails, available on demand while the old install stands, honest about the
mail that stays behind, and never points the old server at the store the
fork has written.

Every phase is timed, and the number to advertise is the downtime, phases
2 to 7, not the total that preflight and the copy dominate. The report
writes both as JSON so a release note quotes something measured.
2026-09-19 21:34:11 -07:00
jcoffey-dev 92aed3df21 Cutover: side by side, with the old install left standing
John's plan, 2026-09-19: stop the Enterprise server and the ihasmail
container, install the fork at its own path, copy the data across, bring up
INBUXA Admin and the new webmail, and every account carries on.

That is a better shape than the in-place swap this draft assumed, because
the rollback becomes a service swap rather than a restore: the old install
and its data are untouched, so going back is stopping one unit and starting
another. What it costs is whatever the fork accepted in between, since the
two stores diverge the moment the fork starts.

It buys one failure the in-place swap couldn't produce: both servers on one
set of ports, each with its own store, if a reboot brings the old unit back.
So the unit is disabled, not just stopped.

Also written down: the front ends' OAuth clients travel inside the store, so
a front end that keeps its client id and redirect URIs keeps working and one
deployed fresh needs them set up, which fails looking like an account
problem when it isn't.
2026-09-19 21:31:50 -07:00
jcoffey-dev 63fe315457 Specs: ufw is in the way of the ACME test, and the rules say so
/etc/ufw/user.rules is world-readable, so this needed no privilege after
all. The default input policy is DROP and 8899 is not among the allowed
ports, so pebble's connection to the suite's listener is dropped. That
matches the live run, where twelve probes from a container completed no
handshake while the same openssl reached pebble's own TLS port.

The nc readings that pointed the other way — instant refusals on closed
ports, where a DROP should hang — are still unexplained, and are left on
the page as unexplained rather than quietly dropped, since they are what
sent an earlier pass through this page in the wrong direction.

Nobody has added the allow rule and re-run the suite, so the fix is
written down as a prediction. The cutover draft says the same: the test
failing here is not evidence that renewal works on the host.
2026-09-19 21:27:40 -07:00
jcoffey-dev 26759a1847 Specs: a draft runbook for the cutover
Steps 1 to 3 of SPEC §7 are met, so the remaining one is running the fork
as the mail server. The draft covers the sequence on the host, what to
check before letting mail flow, and what to watch in the first week.

Two things it refuses to gloss: rolling back stops being a snapshot restore
the moment the fork accepts a message, because nobody has tested whether
the Enterprise build reads a store the fork has written; and certificate
renewal is the failure that arrives 90 days late and quietly, on the one
path the suites couldn't settle.

Nothing in it has been rehearsed. The rehearsal on a copy is step 2 of
"Before the day", and it is what turns "the data opens" into "the server
runs on it".
2026-09-19 21:24:25 -07:00
jcoffey-dev f646f2ec3a Compat: all eight pass against a copy of INBUXA's data
Run from a copy of the stopped server's RocksDB store. Eight green lines,
which are worth reading carefully: six carry weight, and masked_email and
undelete carry none, because there are no masked addresses and retention is
off, so they iterate an empty list. tenant_compat checked the tenant, its
quotas and its members, but not what a tenant administrator can see, which
needs a --tenant-admin recording.

ai_compat is the one that might have looked vacuous and isn't: the twelve
LLM_ tags are there with the scores that were observed.

Three things had to be fixed first, each failing all eight identically and
none about the data: listener names, privileged ports, pending tasks. The
next import's copy will bring the same three, so they are written down.

SPEC §7: cutover steps 1 to 3 are met. Step 4 remains.
2026-09-19 21:19:56 -07:00
jcoffey-dev bfd2784819 Compat: the copy's pending tasks aren't this run's to wait for
The run against INBUXA's store hung printing "Waiting for pending task
AcmeRenewal(...)": the copy carries that server's task queue, and a renewal
due in 2026-11 will not come due while a test watches it.

Under NO_INSERT the wait now skips tasks that aren't due and ones that have
permanently failed, which leaves the tasks the test itself caused — a
restore in undelete_compat comes due at once — and gives up after a minute
with the offending task printed. A test that was really waiting on its own
work now fails on its assertion, which says more than a spinner.

Ordinary runs are untouched: system_tests, which waits on tasks throughout,
still passes in 135s.
2026-09-19 21:17:21 -07:00
jcoffey-dev 01c5503a19 Compat: a copy's own listeners aren't ours to bind
The first run against INBUXA's store failed all eight tests identically,
before reading a single record: the copy carries that server's listeners on
25, 443, 465, 587, 110, 143, 993 and 995, and nothing in a test run is
root, so each one failed with "Permission denied (os error 13)".

The builder now remembers the listeners it adds, and under NO_INSERT drops
build errors for any it didn't. Every other error still stands, including a
bind failing on one of its own, so this can't hide the case where the
harness's own port is taken.

The copy isn't edited for this: its listeners are simply not what a compat
run needs, and it reaches the server over the compat- ones instead. Checked
that a NO_INSERT run still boots and that scim_tests, which takes the
ordinary path, still passes.
2026-09-19 20:58:49 -07:00
jcoffey-dev 1da75e986e Compat: run them from a copy, and keep our listeners out of the way
Rehearsed the run against a real RocksDB store, and it died at startup
before checking anything: the harness inserts listeners of its own, the
registry keys them by name, and a real server already has a "jmap" and an
"imap". The message was "Primary key conflict on property name with
existing object NetworkListener", which says nothing about what to do.
Under NO_INSERT the harness now calls its listeners compat-jmap and so on,
and the same run gets through to the test's own checks.

run-compat.sh copies the store for each test and removes the copy after,
because several of these write to what they open: monitoring_compat purges
the history it reads and undelete_compat restores what it finds. The source
stays untouched, which matters when it is the only copy of a production
store anyone took that day.

INBUXA runs RocksDB, so a copy is a directory copy. The SQL backends would
need more than this: the harness builds its own container and connects to
fixed local credentials, so it cannot open a dump in place.
2026-09-19 20:21:05 -07:00
jcoffey-dev d407509f59 Compat: what the recording holds, and what two of the tests now prove
record-compat.py ran against the live server as a server-level
administrator: 8 accounts, which matches the dashboard, so it reached all
of them. One tenant with its quotas and members; no masked addresses, no
archived items.

The empty files are right rather than short. INBUXA has no masked
addresses and retention is off, so masked_email_compat and undelete_compat
iterate an empty list: they pass without comparing anything, which is worth
saying plainly, because a green run from either would otherwise read as
evidence of compatibility. That puts them where scim_compat and
per_domain_directory_compat already sit.

So one recording carries weight, expected.json, and it is made. SPEC §7's
cutover steps 2 and 3 come down to the tenant, the domains and the
accounts until either feature is switched on.
2026-09-19 20:10:50 -07:00
jcoffey-dev fd7747ef21 Fork tooling: record what the compat tests compare against
Three of the eight compat tests check INBUXA's data against a recording of
how the Enterprise server read it, and that recording can only be made
while that server is still up. SPEC §7 gives it 45 days from the notice, so
the capture shouldn't wait on the cutover being scheduled.

record-compat.py writes all three files: the tenants with their quotas and
members and what each tenant administrator sees, every masked address and
its state, and every archived item whole, since undelete_compat compares
every property it recorded. It only reads, and refuses to send a method
that isn't /get or /query, because it is the one tool here that runs
against the live server. Queries follow their pages, so a server that caps
one doesn't leave a short recording behind.

Exercised against the fork's own test server, which answers the same JMAP:
3 tenants with members, 8 masked addresses and 3 archived items, each in
the shape its test reads.
2026-09-19 18:42:25 -07:00
jcoffey-dev 2a127d6b9a Specs: the ACME network question is open, not answered
Probing further contradicted the previous two commits. During a live run,
when the suite is certainly listening on 8899, twelve handshakes from a
container completed nothing, with or without the ACME ALPN, while the same
openssl in the same container talks to pebble's TLS port and prints its
certificate. A listener that is up but unreachable from a container is what
a ufw DROP looks like. An instant refusal on a closed port, which nc saw
from two images, is not. Both were observed minutes apart.

So the ufw suspicion is neither confirmed nor dismissed, and the page now
says that rather than picking the reading that suits the last probe.
Settling it needs `sudo ufw status verbose` and a listener bound by hand,
neither of which this session could do.

What stands on pebble's own log, and does not depend on any of this: it
runs validations and marks the authorizations invalid, so "never validates,
stays pending" was wrong.
2026-09-19 18:10:22 -07:00
jcoffey-dev 261175aae5 Specs: say only what the ACME probes support
The previous commit called the bridge open on the strength of one nc run.
A later openssl s_client against the same closed port hung for its whole
timeout instead of reporting the refusal nc had just seen. Repeating the nc
test from a second image, with a control port and two closed ports, agreed
with the first: immediate refusal, which is not what a DROP looks like. The
openssl behavior is still unexplained, so the claim now carries what was
measured and the anomaly beside it.

The ALPN probe is recorded as proving nothing, for the same reason: it
hangs against a port with nothing behind it, so its silence during a
renewal says nothing about acme-tls/1.

Pebble's log is untouched by any of this: it validates and marks the
authorizations invalid, which is what makes the old explanation wrong.
2026-09-19 18:06:10 -07:00
jcoffey-dev 4336251cea Specs: the ACME failure isn't ufw, and pebble does validate
The page blamed ufw for blocking the docker bridge, and said pebble never
validates, so the authorizations stay pending. All three are wrong.

From a container on the ACME network the host answers on both gateway
addresses: port 22 connects, and 8899 refuses at once with nothing
listening, where a DROP would hang. No firewall rule was read or changed to
establish that. Pebble's own log shows 20 validation attempts in the
regression run, five for each of the four tls.org names, each ending in
"INVALID by completed challenge". The challenges are answered and refused.

So the order goes invalid, no certificate is issued, and the test unwraps a
None. What's left to explain is the TLS-ALPN handshake. The responder is
intact and listen.rs picks it per connection from has_acme_tls_challenge,
which is computed when the network config is parsed, while the test adds
its provider after boot. That's written down as a hypothesis, not a
finding: it hasn't been tested.
2026-09-19 17:57:52 -07:00
jcoffey-dev 8839078a2b Compat tests: dry-run the harness, and say so when the admin can't log in
None of the eight had ever executed, so all eight ran against an empty store
with synthetic inputs. The plumbing works: the documented JSON shapes parse,
and NO_INSERT stops each one before the harness touches the store, which a
sentinel file in each store directory confirmed — it survived every run,
including the one launched without NO_INSERT.

Two things the runbook got wrong, both of which would have cost a day on the
day the copy exists:

- TMPDIR is the copy's parent, not the copy. The harness opens
  $TMPDIR/<test name>, so a TMPDIR pointing at the copy gets an empty store
  created beside it and the test calls INBUXA's data missing.
- masked_email_compat and undelete_compat need INBUXA_COMPAT_MASKS and
  INBUXA_COMPAT_ARCHIVED, which only the tests' doc comments mentioned.

Every run ended on a 401 raised as "Missing list in response", which reads
as INBUXA's data being wrong when the login is what's wrong. Each test now
authenticates once first and names the variable that failed.
2026-09-19 17:51:42 -07:00
jcoffey-dev 29d9263071 One edition: the last enterprise gates come out of shared code (SPEC 2.3)
Every one of the 14 was `#[cfg(not(feature = "enterprise"))]` on the arm the
fork always compiles: the Enterprise arms went with the import, and nothing
turns the feature on. Removing the attribute leaves the same code, now
unconditional, in 11 files.

Two of them looked like behavior worth checking before touching: the
`validate_tenant_quota` stub that always passes, and the refusal to cancel a
pending DestroyAccount task. The stub is vestigial — the rebuilt
multi-tenancy enforces quotas in `crates/features/src/tenancy/quota.rs` for
those objects and more — and the refusal is undelete's open question, which
this change leaves exactly as it was.

The binary builds with no new warnings, and `system_tests` and `jmap_tests`,
which cover the touched registry, task-manager and auth paths, both pass.
The feature definitions stay in the manifests, inert: taking them out would
widen every sync's diff for nothing.
2026-09-19 17:44:20 -07:00
jcoffey-dev dac1808bbd Tests: retire pending-rebuild, which nothing has been gated behind for a day
The feature held the shared tests of features the fork hadn't rebuilt yet.
All nine are built and the last gate came off with per-domain directories,
so the feature was defined and documented but gated nothing.

The crate builds and holds the same tests without it: 113 by default, 118
with postgres, mysql and redis. SPEC §2.2b keeps its account of the first
import and now says when the gates came off, so a spec that still describes
a suite as gated reads as the record of its own date, which is what
monitoring.md's gate table already calls itself.
2026-09-19 17:28:14 -07:00
jcoffey-dev 08b3210cff Specs: the regression after the ACME fix, 87 passed and 3 failed
Ran it single-threaded on RocksDb: 87 passed, 3 failed, 23 ignored, all 113
tests a default build holds. lmtp_delivery passed this time, which is what
the note about queue timing under a sequential run predicted, so the four
documented failures are three. The ACME suite logged no 400 at all, so the
renewal fix holds; it still ends on the certificate that never came,
because pebble can't validate across the bridge with ufw up.

The line it replaces claimed 86 passed, 4 failed, 27 ignored from the same
command. That totals 117, and no feature set of this tree produces 117 —
113 by default, 118 with all three backends — with no test added or removed
since. Said so rather than presenting the two as a trend.
2026-09-19 17:21:58 -07:00
jcoffey-dev 9f444d2458 Fork tooling: the third-party code upstream carries, listed and checked
A few of upstream's dual-licensed files carry code from other projects
under MIT or BSD terms. The fork redistributes it, so their licenses
require the notices to travel with it. THIRD-PARTY.md reproduces them.

strip.py now reads the stripped tree's comments for another copyright
holder, another license, or a note that code came from somewhere else, and
names any file THIRD-PARTY.md doesn't cover. It reports, never fails: the
notice goes in with the merge that brings the release in.

On v0.16.22 it finds 14 files, all of them covered. The rest of the report
is byte-for-byte what the committed one says, so the scan disturbs nothing
it already did.
2026-09-19 16:59:36 -07:00
jcoffey-dev adfa22c817 Specs: the nine ignored suites re-run, and the STORE each one needs
Re-ran all nine on containers removed beforehand, one suite at a time.
All nine pass. mysql_replica_position_tests failed the first time on test
18's lag assertion, with the pair half a minute old and still catching up,
and passed on a second run against the same containers; it fails before
the point where it changes the replica's settings, so it leaves nothing to
restore. Noted both, with each suite's time.

The table's STORE column said "default" for five suites, which reads as
"leave it unset". The harness has no default: it panics with "Missing or
invalid store type" before the suite starts. They run on RocksDb, as the
regression does, so the column now names it.
2026-09-19 16:54:27 -07:00
jcoffey-dev 309835be1d ACME: post a challenge once, then poll the authorization
The renewal loop re-posted the challenge every time it polled and found the
authorization still pending. RFC 8555 section 7.5.1 has the client post a
challenge once to say it is ready and then poll; a server that has already
moved the challenge to "processing" refuses a second post, and pebble
answers 400 malformed, which failed the whole renewal.

Verified against pebble: the 400s are gone and the client polls. The suite
still can't finish on this machine, because pebble never reaches the test
server to validate the challenge; that path is the environment, and the
runbook now says so.
2026-09-19 16:28:15 -07:00
jcoffey-dev ce12659138 Specs: what a plain regression leaves failing, and why none of it is ours 2026-09-19 16:15:09 -07:00
jcoffey-dev e492b7875a Specs: how to run the suites a plain regression doesn't reach
`cargo test -p tests` runs none of the fork's own feature suites except
what's inside `system_tests`: SCIM, per-domain directories, the sharded
stores and the four replica suites are all ignored, each needing a
container, a STORE the harness only builds on request, or both. The
runbook lists what each one needs, and the container-reuse trap.
2026-09-19 16:14:44 -07:00
jcoffey-dev 5cc28784df SCIM: turning a domain's SCIM authority off takes effect at once (SCIM-60)
`allowScimProvisioning` is cached with the domain as DOMAIN_FLAG_SCIM, but
it wasn't among the fields whose change drops the cached entry, so flipping
the flag changed nothing until something else evicted the domain. SCIM-60
says the change takes effect without a restart.

Found by the two acceptance checks that were written but never called from
the driver, so neither had ever run: `authority` (SCIM-58 to SCIM-60,
through `synchronize_account` itself) and `rate_limits` (SCIM-14). Both are
wired in now, and `scim_tests` passes with them.
2026-09-19 16:14:38 -07:00
jcoffey-dev cee4fd6bc6 Specs: how to run the eight compat tests on a copy of INBUXA's data, and the statuses of tests 10, 11, 17 to 19
docs/spec/compat-tests.md lists each compat test, what it needs, what it
checks and what a failure means, and says plainly that they run against a
copy only, since the monitoring one purges the history it reads. The
scale-out and per-domain statuses record the acceptance tests that now
run.
2026-09-19 15:11:04 -07:00
jcoffey-dev 216bb5b711 Scale-out storage: implementation status of read replicas (ST-5 to ST-15)
Where the routing lives and what a read scope covers, which acceptance
tests the container suite runs, what isn't exercised (two nodes, the
primary stopped, and the MySQL paths, which are built but unrun), what
was settled from the code, and the known limits.
2026-09-19 14:21:54 -07:00
jcoffey-dev a635b490ec Per-domain directories: implementation status and the compat check (DIR-1 to DIR-32, test 20)
per_domain_directory_compat, ignored, checks a copy of INBUXA's data has
no directory, no server default and no domain with its own directory, as
observed. The status names where the rules live, which suite covers each
acceptance test and how far, what was settled from the code, and the
known limits. SCIM's status notes its test 5 now passes.
2026-09-19 10:51:41 -07:00
jcoffey-dev b080fa0736 Scale-out storage: implementation status of the sharded stores (ST-1 to ST-4, ST-16 to ST-30)
Where the code lives, which suite covers each acceptance test, what isn't
exercised, what was settled from the code, and that read-replica routing
(ST-5 to ST-15) waits for per-domain directories.
2026-09-19 10:21:30 -07:00
jcoffey-dev e7efa91cbc Scale-out storage decision: sharded stores first, read-replica routing after per-domain directories, with configured replicas reported at startup meanwhile (ST-2) 2026-09-19 10:01:34 -07:00
jcoffey-dev 00b2eb6f4b SCIM: implementation status (SCIM-1 to SCIM-61)
Where each part lives, which suite covers each acceptance test, test 5
deferred to per-domain directories and test 31 unrun until a copy of
INBUXA's data, what was settled from the code, and the known limits.
2026-09-19 09:59:53 -07:00
jcoffey-dev 2d3f8251c5 SCIM decision: a groups value equal to the current membership in a POST or PUT body is accepted and ignored, since Okta sends groups: [] on create (SCIM-28) 2026-09-19 09:04:25 -07:00
jcoffey-dev 4b389b6b4e SCIM decisions: build before per-domain directories, with the OIDC authority test deferred; build to the spec without the profile drafts or vendor docs (SCIM-61, open questions 12 and 13) 2026-09-19 08:56:53 -07:00
jcoffey-dev f107443b29 Monitoring: implementation status, and the compat test for INBUXA's data (test 26)
monitoring_compat, ignored, checks the observed settings against a copy
of INBUXA's data, reads the old history without an error, and purges it.
The status names where each part lives, which suite covers each
acceptance test, the tests not exercised, and the known limits.
2026-09-19 08:54:21 -07:00
jcoffey-dev 143a10fcf0 Monitoring decision: a live token may be reused within its 60 seconds, since INBUXA Admin reconnects with the same token (MON-23) 2026-09-19 08:34:03 -07:00
jcoffey-dev 715f219528 Monitoring: correct the storage decision; history uses the AGPL telemetry key classes with the registry's encoding, and INBUXA's old history can't be read 2026-09-19 08:12:11 -07:00
jcoffey-dev 3ea242f7fb Monitoring decisions before implementation: live streams and stored histograms match INBUXA Admin, and history is stored as x:Trace and x:Metric registry items (MON-4, MON-20, MON-22, MON-23) 2026-09-19 08:10:27 -07:00