c60028aad1b86cf052545994a692aa96e3b79f1e
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
7cc2fd8c78 |
Draft the Phase 8 processing design
Starts from the distribution channel and the safety invariant it protects, and derives the language from them, rather than designing a rule language and asking later how to ship it. Four decisions proposed. A rule is a matcher plus ordered typed actions, with no expressions and nothing resembling eval -- less expressive than Cribl on purpose, and the only shape that can be pushed to ten thousand hosts and audited by reading it. Total evaluation is the primary safety guarantee, with apply-then-verify as a backstop. One spec with two implementations means a language-neutral conformance suite is the specification and should be built first, not last. Distribution reuses DesiredOverride, following extra_file_paths as the precedent for a repeated field. It also corrects something #21 got wrong. That change said a fatal rule would strand an agent the way a corrupted ingest endpoint does. Reading apply_override's actual semantics, overrides live only in the running process's memory and are never written to disk, so a restarted agent boots clean and re-syncs -- a fatal rule set crash-loops rather than strands, and the agent keeps checking in, so it stays correctable. That is a much better failure mode, and it was acquired by accident: the "don't persist" choice was made to avoid filesystem writes on read-only images, not for safety. This design promotes it to a constraint, since persisting overrides later would silently convert every crash-loop into a strand. positioning.md is corrected to match rather than left disagreeing. Five open questions are left open rather than answered to look decisive, the sharpest being that there is no staged rollout today: an edit reaches every matching agent at once, which for executable rules is the difference between breaking one host and breaking all of them. The v1 acceptance test is real data, not a fixture: two processes on the maintainer's own workstation account for 308 of 325 journal entries in five minutes, and a suppress rule should remove about 60% of that host's volume. Signed-off-by: John Coffey <[email protected]> |
||
|
jcoffey-dev
|
18a55ccd5a |
Realign the roadmap with what is already built
Three of the roadmap's claims were contradicted by the repository itself. Fleet management was Phase 11, "Planned", and positioning.md said config still flowed to the agent from the host rather than from the platform. agent-management-design.md has recorded the opposite for some time: punch list complete, verified live, with central authoring, versioning, rollout on the next check-in, observation, and a restart command a real agent picks up and acts on. There is no Phase 11 now. Its remainder is either already named there -- stop/uninstall, per-host multi-row alerting, a rule-per-host generator -- or belongs to Phase 8, since distributing rules is the one genuinely new thing the mechanism has to carry, and a rule language nobody can push to a fleet is not worth having. That inverts the old ordering argument, which put fleet last on the grounds that it manages configuration the earlier phases define. Sound reasoning; the world went the other way and built the mechanism first. Recorded rather than quietly dropped, because the instinct behind it is a good one that happened not to apply. Retention was listed as a question Phase 10 would finally have to answer. Half of it is answered: api/logretention serves operator-driven preview and delete with an owner-only per-agent floor. What is missing is an automatic TTL, so Phase 10 owns tiering and automatic TTL rather than retention from nothing. The rule-language recommendation is now settled rather than proposed, and not on its own authority: DesiredOverride is already a closed typed shape that cannot carry arbitrary code, so no channel exists that would deliver JavaScript to an agent even if the language argument had gone the other way. That surfaced a requirement nothing had written down. Agent management rests on an invariant it states outright -- every editable field degrades behaviour without cutting off the agent's ability to receive the next correction. Processing rules break it: a rule that panics or loops strands the agent exactly the way a corrupted ingest endpoint would, across every host it reached first. Phase 8 now owes either total evaluation or apply-then-verify with rollback, chosen deliberately rather than discovered mid-rollout. Signed-off-by: John Coffey <[email protected]> |
||
|
jcoffey-dev
|
50dc3f7ee3 |
Write down the naming contrast, because it argues the position
Splunk is from spelunking: caving, a lamp, feeling your way along in the dark. An honest description of search-driven investigation -- powerful with expertise, unforgiving without it. Cribl is from cribble, to sift, from Latin cribrum, a sieve; the same root gives engraving its maniere criblee, the dotted ground punched into a plate. Both senses land together: the data is a medium to be worked and thinned on the way through. A name about the material, not the destination. A cairn is a stack of stones on open ground, where the path is not obvious, doing one job -- somebody came this way, and this is the way. Three of its properties map onto things this project already does rather than things it claims. It is left by whoever went first for whoever comes next, which is the runbook culture and the reason every phase records what was actually run including the failures. Anyone passing adds to it, which is AGPLv3 throughout and an egress path that helps data leave. And you can see it from a distance in daylight, which is a legible query language, an AI that explains rather than divines, and a plan that publishes what has not been proven. Written into positioning.md rather than kept as a marketing note because it is a reason the position coheres, not decoration on top of it. Also noted there that it should not be turned into a slogan. |
||
|
jcoffey-dev
|
d49ddb943e |
Add the AI axis: plain English as an option, analysis as the end state
Cost is the argument against Splunk and control is the argument against Cribl. AI is the third, and the difference there is not a feature comparison, it is where the model runs. Plain-English querying shipped in Phase 7 and stays an option rather than a replacement for writing a query: every generated query compiles through the same IR and executor as a hand-written one, with the same tenant scoping, cost guardrails and audit logging. The model suggests, it does not get a private path to the data. AI-assisted analysis and explanation is the end state and is not built. Authoring answers "how do I ask this"; the valuable question is "what does this mean" -- what changed in a result set, why an alert fired and what preceded it, summarising an incident from the records around it. Recorded on the status page as an end-state goal rather than a numbered phase, because it is a property the product keeps rather than a thing to finish and tick off. Local is the non-negotiable part, and it is worth stating as position rather than as a bullet: the default runs qwen2.5-coder through Ollama on the customer's own hardware, Apache-2.0 weights chosen so Phase 6's licence work survives contact with the model, and the cloud adapter is opt-in and off by default. Logs are the most sensitive unstructured data most organisations hold -- credentials in stack traces, customer identifiers, internal topology -- so an assistant that reads them is either running where the data already is, or it is a data-egress decision wearing a helpful interface. The constraint it imposes is stated too, because it bounds what can be promised: a 7B model on a customer's hardware will not match a frontier model, and the honest claim is not that it is as clever but that it is good enough at a bounded task and runs somewhere you control. Analysis features have to be designed to that budget rather than assuming an API is one call away. |
||
|
jcoffey-dev
|
e57486add5 |
Position against Cribl as well as Splunk, and say what that costs us
Splunk and Cribl are not the same competitor and the claim is not the same claim twice. Splunk is the destination and Cairn OBS replaces it, which is what phases 0-7 were for. Cribl is the road: routing, reduction, enrichment, redaction and replay on the way to wherever data is going. Cairn OBS is already a road in shape -- agent, Redpanda, ingest -- and exposes none of a pipeline's controls. The agent cannot filter, drop, sample, mask or re-route anything; ingest normalises a schema and writes it; there is exactly one destination and it is us. Reconciling the two turns up something a cost-led project has to face rather than paper over: most people buy Cribl because Splunk is expensive per gigabyte, so being genuinely cheap per gigabyte removes the main reason to buy Cribl in front of us. That makes the strongest pitch "one system where there were two" rather than "we are also a pipeline vendor" -- but that pitch only survives a buyer if we also do the four things people buy a pipeline for that are not about spend: routing to several destinations, redacting before data leaves the network, archive and replay, and not being locked to one analytics vendor. Those are about control, which is better ground anyway: cost advantages get matched and architectural ones do not. The consequence is uncomfortable and is written down as a decision rather than left to be discovered: competing with Cribl means being able to send data to S3, Splunk HEC, Elastic, OTLP and Kafka -- building features whose purpose is to help data leave this platform. A project that refuses lock-in in its licence and then builds it into its egress would be lying about itself. Four phases follow, ordered so each pays for itself: processing (8), routing (9), archive and replay (10), fleet (11). Processing without routing still shrinks what is stored; routing without processing forwards everything and helps nobody. One design decision is called out now because it collides with a non-negotiable constraint. Cribl's rule language is JavaScript, and embedding a JS engine in a statically-linked musl agent would end "no glibc runtime deps" as a claim. The recommendation is a declarative rule DSL -- matchers and typed actions, no arbitrary code -- deliberately less expressive, small enough to audit and safe to push to ten thousand hosts. The retention/TTL question in architecture.md is no longer deferrable and now says so: Phase 10 asks it from the other side. |