regex-lite on the agent, full regex at ingest, conformance corpus
limited to the syntax both accept. The agent had no regex dependency at
all and the full crate is a megabyte-plus against a binary whose pitch
is that it is small and static. The rejected alternative, regex
ingest-side only, would have given up redacting PII before it leaves the
host -- the one capability that most needs to be on the agent, and most
of the reason on-host processing exists at all.
Checked rather than assumed: every pattern the corpus uses compiles and
behaves under regex-lite, including named captures and replace_all
replacing every occurrence, which is what mask needs and what a
redaction stopping at the first hit would get wrong. Both engines also
resolve alternation leftmost-first, confirmed by running the same
pattern through regex-lite and Go's regexp and getting the same output.
A case now pins that, since it is the kind of semantic two independent
implementations can differ on silently.
No canary gate. An edit reaches every matching agent at once, and this
design had suggested that might have to block the feature. It does not,
but the risk is accepted rather than waved away: total evaluation is the
real mitigation, a fatal rule set crash-loops rather than strands
because overrides are never persisted, apply-then-verify makes that
self-healing, and applied_override_version already shows the blast
radius. What is genuinely given up is the ability to stop a bad rollout
partway through -- everything else shortens the outage without
preventing the rule reaching every host first.
That raises the stakes on apply-then-verify, which is still open.
Without a canary, total evaluation stops being a nice property of a
well-designed DSL and becomes the only thing between a bad rule and
every host at once, so cutting the backstop from v1 is a harder call
than it looked when it was written down.
Signed-off-by: John Coffey <[email protected]>
The design argues the conformance suite is the specification and should
be built before either implementation, since two hand-written
implementations of one language diverge unless something shared pins
them. This is that suite: 38 cases in /processing, a language-neutral
top-level directory for the same reason /proto is one -- the Rust agent
and the Go ingest tier both consume the definition and neither owns it.
Nothing executes the cases, because neither implementation exists. A
stdlib-only validator checks the corpus stays well-formed and runs in
CI: known actions, addressable fields, compilable patterns, names
matching filenames, and no case depending on record_id, which is
withheld so the question of whether an ingest-side rule can see one
stays open. The validator was checked against seven deliberately broken
cases before being trusted, since "38/38 valid" means nothing from a
validator that cannot fail.
Writing the cases first has already paid for itself twice.
It forced two determinism decisions the prose had left vague, both of
which a conformance suite cannot avoid answering. Sampling is
counter-based rather than random: random is statistically nicer and
impossible to assert on. Windows are measured on record timestamps
rather than wall-clock, which makes replay deterministic and, not
incidentally, makes backfill behave correctly where a wall-clock window
would not.
And it made the missing aggregate_count answer concrete. The design
does not say what that action emits, or what a query not expecting a
synthetic record sees. Rather than invent one by writing cases, the
validator rejects any case using it, so the design question has to be
answered before the behaviour can be frozen by accident.
The corpus includes the acceptance case from real measured data: the
two processes that account for roughly 60% of a real workstation's
journal volume, and the one kernel message worth keeping.
Signed-off-by: John Coffey <[email protected]>
jcoffey-dev