3db885059891fcd8a689111286ce1ee928c76539
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
ec4b860ba8 |
Specify what aggregate_count emits
The last unanswered action, and the only one whose output is not the input with edits -- it emits a record that never existed, which is why it was deferred twice. It emits the window's first record unchanged, tagged with cairnobs.aggregated, cairnobs.count, and the observed window bounds. That follows the convention the agent already uses for heartbeat and host-metrics records rather than inventing a second synthetic-record mechanism, and keeping the first record intact means a reader sees a real example of what was collapsed instead of an invented summary. It tags even when the count is one. Emitting a bare record there would be tidier and would make cairnobs.count present only sometimes, so summing it silently breaks on quiet windows. window_last is the last record that actually contributed, never window_start + window_ms, because a window flushed early must not claim an end that never happened. Specifying it surfaced a problem the other nine actions do not have. Windows are measured on record time, so a window can only be closed by a later record arriving. suppress_duplicates never has anything pending; aggregate_count holds state, so a matching stream that goes quiet leaves its aggregate unemitted indefinitely -- data loss dressed as latency. Emission therefore has a second trigger, end of stream, which the corpus defines as an implicit flush after the last input and which production gets from the batch flush. The cost is stated rather than hidden: window_ms becomes a maximum, not a guarantee, and one burst can produce more than one aggregate. And it has a consequence nobody should meet in production first: stats count undercounts aggregated data silently, so every panel and alert counting rows changes meaning the moment a rule aggregates the data behind it. Nothing here fixes that. The correct idiom is summing cairnobs.count; teaching the query layer to do it automatically is a Phase 2 change to the IR, recorded as the open question this decision leaves in its place rather than quietly inherited. Six cases added, corpus at 45. The validator's unspecified-action guard stays in place with an empty set, still rejecting anything added to it. Signed-off-by: John Coffey <[email protected]> |
||
|
jcoffey-dev
|
02ebbc86a1 |
Settle Phase 8's regex and rollout questions
regex-lite on the agent, full regex at ingest, conformance corpus limited to the syntax both accept. The agent had no regex dependency at all and the full crate is a megabyte-plus against a binary whose pitch is that it is small and static. The rejected alternative, regex ingest-side only, would have given up redacting PII before it leaves the host -- the one capability that most needs to be on the agent, and most of the reason on-host processing exists at all. Checked rather than assumed: every pattern the corpus uses compiles and behaves under regex-lite, including named captures and replace_all replacing every occurrence, which is what mask needs and what a redaction stopping at the first hit would get wrong. Both engines also resolve alternation leftmost-first, confirmed by running the same pattern through regex-lite and Go's regexp and getting the same output. A case now pins that, since it is the kind of semantic two independent implementations can differ on silently. No canary gate. An edit reaches every matching agent at once, and this design had suggested that might have to block the feature. It does not, but the risk is accepted rather than waved away: total evaluation is the real mitigation, a fatal rule set crash-loops rather than strands because overrides are never persisted, apply-then-verify makes that self-healing, and applied_override_version already shows the blast radius. What is genuinely given up is the ability to stop a bad rollout partway through -- everything else shortens the outage without preventing the rule reaching every host first. That raises the stakes on apply-then-verify, which is still open. Without a canary, total evaluation stops being a nice property of a well-designed DSL and becomes the only thing between a bad rule and every host at once, so cutting the backstop from v1 is a harder call than it looked when it was written down. Signed-off-by: John Coffey <[email protected]> |
||
|
jcoffey-dev
|
7fabc6a067 |
Build the Phase 8 conformance corpus
The design argues the conformance suite is the specification and should be built before either implementation, since two hand-written implementations of one language diverge unless something shared pins them. This is that suite: 38 cases in /processing, a language-neutral top-level directory for the same reason /proto is one -- the Rust agent and the Go ingest tier both consume the definition and neither owns it. Nothing executes the cases, because neither implementation exists. A stdlib-only validator checks the corpus stays well-formed and runs in CI: known actions, addressable fields, compilable patterns, names matching filenames, and no case depending on record_id, which is withheld so the question of whether an ingest-side rule can see one stays open. The validator was checked against seven deliberately broken cases before being trusted, since "38/38 valid" means nothing from a validator that cannot fail. Writing the cases first has already paid for itself twice. It forced two determinism decisions the prose had left vague, both of which a conformance suite cannot avoid answering. Sampling is counter-based rather than random: random is statistically nicer and impossible to assert on. Windows are measured on record timestamps rather than wall-clock, which makes replay deterministic and, not incidentally, makes backfill behave correctly where a wall-clock window would not. And it made the missing aggregate_count answer concrete. The design does not say what that action emits, or what a query not expecting a synthetic record sees. Rather than invent one by writing cases, the validator rejects any case using it, so the design question has to be answered before the behaviour can be frozen by accident. The corpus includes the acceptance case from real measured data: the two processes that account for roughly 60% of a real workstation's journal volume, and the one kernel message worth keeping. Signed-off-by: John Coffey <[email protected]> |