From c2f74b4e1aac4f59e1142bcbb7dd78d1275b3eb8 Mon Sep 17 00:00:00 2001 From: John Coffey Date: Fri, 4 Sep 2026 21:02:35 -0700 Subject: [PATCH] Put apply-then-verify in v1, and specify it It was a defensible cut while a canary might have caught a bad rollout partway through. With no canary it is the only thing that recovers a host without an operator noticing, and "total evaluation should make it unreachable" is what every crash-loop was before it happened. Specified rather than named. A two-field file is written and fsynced before rules are applied: the version being attempted, and whether it is trying, good or quarantined. The rule set itself is never written, which is what keeps the crash-loop-not-strand property -- an agent that loses the file still boots clean and re-syncs. A version found still "trying" at boot is quarantined, the agent starts with no rules, and it reports the quarantined version so the failure is visible rather than merely survived. A different version clears the quarantine, because pushing new rules is the correction. Three failure modes decided instead of discovered. An agent that cannot write the file logs once and runs with total evaluation alone: degrading to "no rules" would punish every read-only deployment for a failure that has not happened, and the backstop is best-effort by construction rather than by accident. An agent killed for an unrelated reason quarantines a blameless rule set, which is a deliberate false positive -- the alternative is claiming to distinguish "died because of the rules" from "died while they happened to be loaded", which it cannot do honestly. And a rule set fatal on only some hosts quarantines per host, which is the closest thing to a canary this design has: the first host to hit it reports while the rest carry on. Also states that the conformance corpus cannot test any of this, so nobody tries. The corpus pins rule semantics -- records in, records out. Process death and file state across restarts are not expressible that way and need agent-side tests driving a real process through crash and restart. Signed-off-by: John Coffey --- docs/phase-8-processing-design.md | 98 +++++++++++++++++++++++++++---- 1 file changed, 86 insertions(+), 12 deletions(-) diff --git a/docs/phase-8-processing-design.md b/docs/phase-8-processing-design.md index 664b332..d75d713 100644 --- a/docs/phase-8-processing-design.md +++ b/docs/phase-8-processing-design.md @@ -1,10 +1,14 @@ # Phase 8 processing design: rules, where they run, and how they arrive -> **Status:** Design, drafted 2026-09-05. **Not approved and not -> implemented.** Nothing in Phase 8 is started. This is a proposal to -> argue with — the sections marked **Open** are genuine decisions, not -> rhetorical ones. If implementation shows this is wrong somewhere, fix -> this doc in the same change rather than letting them drift. +> **Status:** Design, drafted 2026-09-05, **not implemented** — nothing +> in Phase 8 is built. Four decisions are settled and dated: the rule +> shape, total evaluation with apply-then-verify, `regex-lite` on the +> agent, and shipping without a canary. Two remain genuinely open — rule +> placement, and what `aggregate_count` emits. The specification itself +> is the conformance corpus in [`/processing`](../processing/README.md); +> this document is a summary of it and loses any argument between them. +> If implementation shows this is wrong somewhere, fix this doc in the +> same change rather than letting them drift. ## Why this design, in one paragraph @@ -90,10 +94,79 @@ itself** — a few bytes, and a host that cannot write even that simply loses the backstop and keeps total evaluation. The agent must degrade to "no rules" on a write failure, never to "apply anyway". -**Open:** whether the backstop is worth its complexity in the first -release, given total evaluation should make it unreachable. My view is -yes — "should be unreachable" is what every crash-loop was before it -happened — but it is a defensible cut for a v1. +**Decided 2026-09-05: apply-then-verify ships in v1.** + +It was a defensible cut while a canary might have backstopped a bad +rollout. With no canary (Decision 4), it is the only thing that recovers +a host without an operator noticing, and "total evaluation should make +it unreachable" is what every crash-loop was before it happened. + +### What persists, exactly + +A single small file next to the agent's config, holding two fields: + +``` +override_version the version stamp the agent is currently attempting +state "trying" | "good" | "quarantined" +``` + +The rule set itself is **never** written. That is what keeps the +crash-loop-not-strand property intact: an agent that loses this file, or +never had it, still boots clean and re-syncs. + +The sequence on receiving a rule set with a new version: + +1. Write `{version, "trying"}` and fsync **before** applying anything. +2. Apply the rules. +3. On the next successful check-in, rewrite as `{version, "good"}`. + +On boot, the agent reads the file: + +- absent, or `good` — normal start, apply whatever the next check-in + returns. +- `trying` — the previous process died while carrying that version. + Rewrite as `{version, "quarantined"}`, start with **no rules**, and + report the quarantined version on the next check-in so it is visible + rather than merely survived. +- `quarantined` — keep refusing that exact version. Any *different* + version clears the quarantine and is tried normally, because the + operator pushing a new rule set is the correction. + +### Failure modes, decided rather than discovered + +**The agent cannot write the file** (read-only image, full disk). It +logs once, runs with total evaluation alone, and applies rules normally. +Degrading to "no rules" would punish every read-only deployment for a +failure that has not happened; degrading to "apply anyway" is what the +mechanism already does minus the recovery. The backstop is best-effort +by construction, and the doc should not pretend otherwise. + +**The agent dies for an unrelated reason** while carrying a good rule +set — OOM from something else, a host reboot, a `kill -9`. It +quarantines a blameless rule set. This is a false positive by design: +the alternative is distinguishing "died because of the rules" from "died +while the rules happened to be loaded", which the agent cannot do +honestly. One unnecessary quarantine, visible on the Agents page and +cleared by re-pushing, is a much better error than one missed real one. + +**A rule set is fatal only on some hosts** — a pattern that behaves +badly against data only one host sees. Each agent quarantines +independently, which is the correct behaviour and also the closest thing +to a canary this design has: the first host to hit it quarantines and +reports while the others carry on. + +### The conformance corpus cannot test this + +Worth stating so nobody tries. The corpus is records in, records out — +it pins rule *semantics*. Apply-then-verify is agent lifecycle +behaviour: process death, file state across restarts, and what gets +reported on the next check-in. None of that is expressible as an input +record and an expected output record. + +It needs its own tests, on the agent side, driving a real process +through crash and restart — closer in shape to how +`agent-management-design.md`'s restart command was verified live than to +anything in `/processing`. ## Decision 2: the rule shape @@ -278,9 +351,10 @@ release cannot do that, it is not finished. ## Open questions, collected -1. Is apply-then-verify in v1, or is total evaluation alone enough? - **Harder to cut than it looked** — with no canary (decided below), - total evaluation is the only thing between a bad rule and every host. +1. ~~Apply-then-verify in v1?~~ **Decided 2026-09-05:** yes. With no + canary, it is the only thing that recovers a host unattended. The + persisted state, its failure modes, and why the conformance corpus + cannot cover it are specified in Decision 1. 2. ~~Regex on the agent?~~ **Decided 2026-09-05:** `regex-lite` on the agent, full `regex` at ingest, corpus limited to their common syntax. 3. ~~Canary rollout?~~ **Decided 2026-09-05:** no gate, ship without it;