It was a defensible cut while a canary might have caught a bad rollout
partway through. With no canary it is the only thing that recovers a
host without an operator noticing, and "total evaluation should make it
unreachable" is what every crash-loop was before it happened.
Specified rather than named. A two-field file is written and fsynced
before rules are applied: the version being attempted, and whether it is
trying, good or quarantined. The rule set itself is never written, which
is what keeps the crash-loop-not-strand property -- an agent that loses
the file still boots clean and re-syncs. A version found still "trying"
at boot is quarantined, the agent starts with no rules, and it reports
the quarantined version so the failure is visible rather than merely
survived. A different version clears the quarantine, because pushing new
rules is the correction.
Three failure modes decided instead of discovered. An agent that cannot
write the file logs once and runs with total evaluation alone: degrading
to "no rules" would punish every read-only deployment for a failure that
has not happened, and the backstop is best-effort by construction rather
than by accident. An agent killed for an unrelated reason quarantines a
blameless rule set, which is a deliberate false positive -- the
alternative is claiming to distinguish "died because of the rules" from
"died while they happened to be loaded", which it cannot do honestly.
And a rule set fatal on only some hosts quarantines per host, which is
the closest thing to a canary this design has: the first host to hit it
reports while the rest carry on.
Also states that the conformance corpus cannot test any of this, so
nobody tries. The corpus pins rule semantics -- records in, records out.
Process death and file state across restarts are not expressible that
way and need agent-side tests driving a real process through crash and
restart.
Signed-off-by: John Coffey <[email protected]>
regex-lite on the agent, full regex at ingest, conformance corpus
limited to the syntax both accept. The agent had no regex dependency at
all and the full crate is a megabyte-plus against a binary whose pitch
is that it is small and static. The rejected alternative, regex
ingest-side only, would have given up redacting PII before it leaves the
host -- the one capability that most needs to be on the agent, and most
of the reason on-host processing exists at all.
Checked rather than assumed: every pattern the corpus uses compiles and
behaves under regex-lite, including named captures and replace_all
replacing every occurrence, which is what mask needs and what a
redaction stopping at the first hit would get wrong. Both engines also
resolve alternation leftmost-first, confirmed by running the same
pattern through regex-lite and Go's regexp and getting the same output.
A case now pins that, since it is the kind of semantic two independent
implementations can differ on silently.
No canary gate. An edit reaches every matching agent at once, and this
design had suggested that might have to block the feature. It does not,
but the risk is accepted rather than waved away: total evaluation is the
real mitigation, a fatal rule set crash-loops rather than strands
because overrides are never persisted, apply-then-verify makes that
self-healing, and applied_override_version already shows the blast
radius. What is genuinely given up is the ability to stop a bad rollout
partway through -- everything else shortens the outage without
preventing the rule reaching every host first.
That raises the stakes on apply-then-verify, which is still open.
Without a canary, total evaluation stops being a nice property of a
well-designed DSL and becomes the only thing between a bad rule and
every host at once, so cutting the backstop from v1 is a harder call
than it looked when it was written down.
Signed-off-by: John Coffey <[email protected]>
The design argues the conformance suite is the specification and should
be built before either implementation, since two hand-written
implementations of one language diverge unless something shared pins
them. This is that suite: 38 cases in /processing, a language-neutral
top-level directory for the same reason /proto is one -- the Rust agent
and the Go ingest tier both consume the definition and neither owns it.
Nothing executes the cases, because neither implementation exists. A
stdlib-only validator checks the corpus stays well-formed and runs in
CI: known actions, addressable fields, compilable patterns, names
matching filenames, and no case depending on record_id, which is
withheld so the question of whether an ingest-side rule can see one
stays open. The validator was checked against seven deliberately broken
cases before being trusted, since "38/38 valid" means nothing from a
validator that cannot fail.
Writing the cases first has already paid for itself twice.
It forced two determinism decisions the prose had left vague, both of
which a conformance suite cannot avoid answering. Sampling is
counter-based rather than random: random is statistically nicer and
impossible to assert on. Windows are measured on record timestamps
rather than wall-clock, which makes replay deterministic and, not
incidentally, makes backfill behave correctly where a wall-clock window
would not.
And it made the missing aggregate_count answer concrete. The design
does not say what that action emits, or what a query not expecting a
synthetic record sees. Rather than invent one by writing cases, the
validator rejects any case using it, so the design question has to be
answered before the behaviour can be frozen by accident.
The corpus includes the acceptance case from real measured data: the
two processes that account for roughly 60% of a real workstation's
journal volume, and the one kernel message worth keeping.
Signed-off-by: John Coffey <[email protected]>
Starts from the distribution channel and the safety invariant it
protects, and derives the language from them, rather than designing a
rule language and asking later how to ship it.
Four decisions proposed. A rule is a matcher plus ordered typed actions,
with no expressions and nothing resembling eval -- less expressive than
Cribl on purpose, and the only shape that can be pushed to ten thousand
hosts and audited by reading it. Total evaluation is the primary safety
guarantee, with apply-then-verify as a backstop. One spec with two
implementations means a language-neutral conformance suite is the
specification and should be built first, not last. Distribution reuses
DesiredOverride, following extra_file_paths as the precedent for a
repeated field.
It also corrects something #21 got wrong. That change said a fatal rule
would strand an agent the way a corrupted ingest endpoint does. Reading
apply_override's actual semantics, overrides live only in the running
process's memory and are never written to disk, so a restarted agent
boots clean and re-syncs -- a fatal rule set crash-loops rather than
strands, and the agent keeps checking in, so it stays correctable. That
is a much better failure mode, and it was acquired by accident: the
"don't persist" choice was made to avoid filesystem writes on read-only
images, not for safety. This design promotes it to a constraint, since
persisting overrides later would silently convert every crash-loop into
a strand. positioning.md is corrected to match rather than left
disagreeing.
Five open questions are left open rather than answered to look decisive,
the sharpest being that there is no staged rollout today: an edit
reaches every matching agent at once, which for executable rules is the
difference between breaking one host and breaking all of them.
The v1 acceptance test is real data, not a fixture: two processes on the
maintainer's own workstation account for 308 of 325 journal entries in
five minutes, and a suppress rule should remove about 60% of that host's
volume.
Signed-off-by: John Coffey <[email protected]>
Three of the roadmap's claims were contradicted by the repository
itself.
Fleet management was Phase 11, "Planned", and positioning.md said config
still flowed to the agent from the host rather than from the platform.
agent-management-design.md has recorded the opposite for some time:
punch list complete, verified live, with central authoring, versioning,
rollout on the next check-in, observation, and a restart command a real
agent picks up and acts on. There is no Phase 11 now. Its remainder is
either already named there -- stop/uninstall, per-host multi-row
alerting, a rule-per-host generator -- or belongs to Phase 8, since
distributing rules is the one genuinely new thing the mechanism has to
carry, and a rule language nobody can push to a fleet is not worth
having.
That inverts the old ordering argument, which put fleet last on the
grounds that it manages configuration the earlier phases define. Sound
reasoning; the world went the other way and built the mechanism first.
Recorded rather than quietly dropped, because the instinct behind it is
a good one that happened not to apply.
Retention was listed as a question Phase 10 would finally have to
answer. Half of it is answered: api/logretention serves operator-driven
preview and delete with an owner-only per-agent floor. What is missing
is an automatic TTL, so Phase 10 owns tiering and automatic TTL rather
than retention from nothing.
The rule-language recommendation is now settled rather than proposed,
and not on its own authority: DesiredOverride is already a closed typed
shape that cannot carry arbitrary code, so no channel exists that would
deliver JavaScript to an agent even if the language argument had gone
the other way.
That surfaced a requirement nothing had written down. Agent management
rests on an invariant it states outright -- every editable field
degrades behaviour without cutting off the agent's ability to receive
the next correction. Processing rules break it: a rule that panics or
loops strands the agent exactly the way a corrupted ingest endpoint
would, across every host it reached first. Phase 8 now owes either total
evaluation or apply-then-verify with rollback, chosen deliberately
rather than discovered mid-rollout.
Signed-off-by: John Coffey <[email protected]>
Cairn OBS is self-hosted, and the way to separate two environments is to
run two installations rather than two tenants inside one. Tenancy is the
wrong boundary for that, on three counts this repository demonstrates
rather than assumes: chwriter.WriteBatch is all-or-nothing across
tenants, so one tenant's failure stalls offset progress for every other;
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT puts every tenant's data behind a
single superuser credential, as docker-compose.yml's own comment says;
and one binary with one set of migrations moves every tenant together,
which is the opposite of what separate environments are for. A whole
installation idles at about 1.3 GB, so the sharing buys nothing.
The project led with multi-tenant RBAC in the README banner and in
PROJECT-SPEC's goal statement. Both now say what it is instead:
self-hosted. "Open-core" goes with them -- it was already inaccurate,
since CONTRIBUTING states there is no feature gate and no paid tier, and
with enterprise/ off the roadmap there will not be one.
A second identity provider comes off the list of things standing between
this and production-ready. SSO belongs to enterprise/, and a self-hosted
deployment is not waiting on it. Terraform's tenant/RBAC resources move
from "disclosed future work" to not planned.
Nothing is scrubbed from the record. Phase 4 stays shipped, its runbook
stays, and its known gaps stay stated -- rewriting that history would
contradict the candour the Status section is built on. enterprise/ stays
in the tree, AGPLv3 and working, as the answer to a question this
project is not asking.
Signed-off-by: John Coffey <[email protected]>
Local login is implemented, wired through api, alerting and web, and
undiscoverable. No compose file turns it on, the Helm chart sets none
of its variables, and no markdown in the repository mentions
-seed-admin, LOCAL_AUTH_ENABLED or local login at all. The only way to
find it is to read cmd/api/main.go's authorizer switch.
Enabling it in docker-compose.yml is not the answer: a plain
`docker compose up` has no authentication, and every Phase 0-3 runbook
verifies the pipeline with bare curl against /query. Turning login on
by default would break the project's own documented verification.
So it's an opt-in overlay instead.
Four settings have to agree, and only one of them is obviously about
login. Each fails differently and none of the failures name the cause:
the route 404s, or the browser refuses the request before sending it,
or login returns 200 and every later request is anonymous because the
cookie was never stored, or the same symptom again from the opposite
end because the bundle never attaches it. That is what the new document
is mostly for.
The Helm chart still has no local-login support. Recorded in the
document as a gap rather than papered over.
Signed-off-by: John Coffey <[email protected]>
Phase 2 unified the two query languages behind one endpoint and renamed
the request field, and the runbooks were never updated. Following them
today does not work:
{"sql": ...} -> 400 query must not be empty
POST /api :8080/search -> 404, the route no longer exists
Both appear in the Phase 0 and Phase 1 runbooks and in the
windows-fixture README. That matters more than a normal doc typo,
because status.md cites the Phase 0 runbook as the record of how Phase
0 was verified -- so the documented verification procedure is one
nobody can re-run as written.
The Phase 1 step is rewritten rather than search-and-replaced: it
checked the SQL and full-text paths against two different endpoints,
and its exit criterion (the same record_id from both) now has to be
expressed against /query twice, once with SQL and once with a bare
word.
Phase 0's expected output for SELECT 1 also gained a warnings field
since it was written.
Every command here was run against a live stack before being written
down, including confirming both paths return the same record_id.
Signed-off-by: John Coffey <[email protected]>
The sidebar decided which auth mode was live from enterpriseAuthBase,
so any deployment with VITE_ENTERPRISE_AUTH_BASE_URL set rendered the
enterprise block -- and compose sets it unconditionally, so the tenant
picker can exist. On a single-tenant stack with local login on, that
meant the local block could never render: no username, no role, no
Change password, no Log out, and in their place a "Sign in" link
pointing at enterprise-auth's OIDC route, which is disabled unless
OIDC_ISSUER_URL is configured. A dead link where the account controls
should be.
The build-time flag was never the right thing to ask. api registers
/auth/* only when LOCAL_AUTH_ENABLED is set and ENTERPRISE_AUTH_URL is
not, so the frontend cannot know the mode from its own build args --
the two can disagree, and here they did. getLocalSession already
distinguishes 'disabled' (a 404 from /auth/session) from null (a 401,
logged out); the sidebar collapsed both to null and threw the answer
away. It now keeps that distinction and branches on it, so the mode
comes from what the server actually serves.
Logged out under local auth, the sidebar previously rendered no auth
block at all -- no way back to the login page from the nav. It now
offers Sign in, pointing at /login.
Neither block renders until the probe lands, so nothing flashes the
wrong mode on load.
Signed-off-by: John Coffey <[email protected]>
Deleting your own user succeeded, and logged you out doing it:
local_sessions.user_id is ON DELETE CASCADE, so the delete took the
caller's own live session with it. Nothing refused this. The
last-owner guard is the only thing in the path, and it passes cleanly
as soon as a second owner exists -- which is exactly the state you are
in just after creating one.
The way back in was then whatever other account happened to exist, and
-seed-admin could not help: it skipped whenever *any* local user was
present, so the command documented as the way to create an
administrator refused precisely when there was no usable one, because
some other account still existed. It now asks whether the admin
account itself is missing, which is what its own help text always
claimed, and what makes it useful as recovery rather than only as
first-run bootstrap.
TestCanDeleteAnOwnerWhenAnotherRemains signed in as admin1 and deleted
admin1, asserting 204 -- it encoded the lockout as intended behaviour.
It now deletes the other owner, which is what it meant to cover, and a
new test holds the refusal in place.
runSeedAdmin takes a small interface so the bootstrap path is tested
without a Postgres pool; it had no tests before.
Signed-off-by: John Coffey <[email protected]>
The commerce generators shipped with currency=GBP on every order, refund,
authorisation and chargeback -- twelve places, all of them a default
nobody chose. Dollars is the convention everywhere else.
Only the label changes. The SKU prices stay as they are: a $489 task
chair and a $629 standing desk are as plausible as the pound figures
were, and moving them would have shifted average order value on the
demo's dashboards for no reason other than tidiness.
Nothing else referenced the currency -- no dashboard panel and no alert
rule filters or groups on it -- so this is the whole change.
Splunk is from spelunking: caving, a lamp, feeling your way along in the
dark. An honest description of search-driven investigation -- powerful
with expertise, unforgiving without it.
Cribl is from cribble, to sift, from Latin cribrum, a sieve; the same root
gives engraving its maniere criblee, the dotted ground punched into a
plate. Both senses land together: the data is a medium to be worked and
thinned on the way through. A name about the material, not the
destination.
A cairn is a stack of stones on open ground, where the path is not
obvious, doing one job -- somebody came this way, and this is the way.
Three of its properties map onto things this project already does rather
than things it claims. It is left by whoever went first for whoever comes
next, which is the runbook culture and the reason every phase records
what was actually run including the failures. Anyone passing adds to it,
which is AGPLv3 throughout and an egress path that helps data leave.
And you can see it from a distance in daylight, which is a legible query
language, an AI that explains rather than divines, and a plan that
publishes what has not been proven.
Written into positioning.md rather than kept as a marketing note because
it is a reason the position coheres, not decoration on top of it. Also
noted there that it should not be turned into a slogan.
The Status section went from a table of "Shipped" straight into three
paragraphs of caveats, with nothing in between telling a reader what
standard was being applied. Read cold, that is a project confessing. Read
with the standard stated first, it is a project that refuses to call
something done because the tests pass.
So the section now opens by saying what stage this is -- pre-1.0, no
production workload -- and what "shipped" means here: verified against
real infrastructure with a runbook recording how, including what the
verification could not reach. And it says plainly why the caveats are
long, which is that they are disclosed rather than discovered. A shorter
Status section would not mean a more finished product, only a less
careful one.
It closes with what would actually close the gap: a second IdP, a real
cluster, the Windows agent on Windows, sustained load, and somebody
else's data. None of that is research, it is time on real infrastructure
-- which makes it the list a pilot works through, and makes a 1.0 tag the
wrong next milestone to reach for.
Written because the risk of a public repo at this stage is not a
competitor reading the roadmap, it is a prospect reading unusual honesty
as immaturity. The fix for that is context, not privacy.
Cost is the argument against Splunk and control is the argument against
Cribl. AI is the third, and the difference there is not a feature
comparison, it is where the model runs.
Plain-English querying shipped in Phase 7 and stays an option rather than
a replacement for writing a query: every generated query compiles through
the same IR and executor as a hand-written one, with the same tenant
scoping, cost guardrails and audit logging. The model suggests, it does
not get a private path to the data.
AI-assisted analysis and explanation is the end state and is not built.
Authoring answers "how do I ask this"; the valuable question is "what
does this mean" -- what changed in a result set, why an alert fired and
what preceded it, summarising an incident from the records around it.
Recorded on the status page as an end-state goal rather than a numbered
phase, because it is a property the product keeps rather than a thing to
finish and tick off.
Local is the non-negotiable part, and it is worth stating as position
rather than as a bullet: the default runs qwen2.5-coder through Ollama on
the customer's own hardware, Apache-2.0 weights chosen so Phase 6's
licence work survives contact with the model, and the cloud adapter is
opt-in and off by default. Logs are the most sensitive unstructured data
most organisations hold -- credentials in stack traces, customer
identifiers, internal topology -- so an assistant that reads them is
either running where the data already is, or it is a data-egress decision
wearing a helpful interface.
The constraint it imposes is stated too, because it bounds what can be
promised: a 7B model on a customer's hardware will not match a frontier
model, and the honest claim is not that it is as clever but that it is
good enough at a bounded task and runs somewhere you control. Analysis
features have to be designed to that budget rather than assuming an API
is one call away.
Splunk and Cribl are not the same competitor and the claim is not the
same claim twice. Splunk is the destination and Cairn OBS replaces it,
which is what phases 0-7 were for. Cribl is the road: routing, reduction,
enrichment, redaction and replay on the way to wherever data is going.
Cairn OBS is already a road in shape -- agent, Redpanda, ingest -- and
exposes none of a pipeline's controls. The agent cannot filter, drop,
sample, mask or re-route anything; ingest normalises a schema and writes
it; there is exactly one destination and it is us.
Reconciling the two turns up something a cost-led project has to face
rather than paper over: most people buy Cribl because Splunk is expensive
per gigabyte, so being genuinely cheap per gigabyte removes the main
reason to buy Cribl in front of us. That makes the strongest pitch "one
system where there were two" rather than "we are also a pipeline vendor"
-- but that pitch only survives a buyer if we also do the four things
people buy a pipeline for that are not about spend: routing to several
destinations, redacting before data leaves the network, archive and
replay, and not being locked to one analytics vendor. Those are about
control, which is better ground anyway: cost advantages get matched and
architectural ones do not.
The consequence is uncomfortable and is written down as a decision rather
than left to be discovered: competing with Cribl means being able to send
data to S3, Splunk HEC, Elastic, OTLP and Kafka -- building features whose
purpose is to help data leave this platform. A project that refuses
lock-in in its licence and then builds it into its egress would be lying
about itself.
Four phases follow, ordered so each pays for itself: processing (8),
routing (9), archive and replay (10), fleet (11). Processing without
routing still shrinks what is stored; routing without processing forwards
everything and helps nobody.
One design decision is called out now because it collides with a
non-negotiable constraint. Cribl's rule language is JavaScript, and
embedding a JS engine in a statically-linked musl agent would end "no
glibc runtime deps" as a claim. The recommendation is a declarative rule
DSL -- matchers and typed actions, no arbitrary code -- deliberately less
expressive, small enough to audit and safe to push to ten thousand hosts.
The retention/TTL question in architecture.md is no longer deferrable and
now says so: Phase 10 asks it from the other side.
The estate could show an operator their infrastructure and had nothing to
say to the business paying for it. Two storefronts and the gateway behind
both fix that: Magento on two hosts, WooCommerce on one, and pay-01
carrying authorisations with amount, gateway and decline reason. Orders,
revenue, average order value, where checkout loses people and why a card
was refused now come out of the same log lines the operators are already
reading, which is the argument for not running a separate metrics stack
beside this one.
Two platforms rather than one deliberately. Magento and WooCommerce write
about the same events differently, so a panel that groups by service
instead of assuming a single shape is the honest way to build one -- and
the demo shows that rather than describing it.
Order totals are built from a basket of real SKUs at real prices rather
than drawn from a distribution, so average order value moves the way one
actually moves. Declines rise during the seeded outage window alongside
the 5xx rate, because whatever fails requests fails authorisations too.
Twenty-seven new alert rules, thresholds calibrated against what the
fleet actually emits -- measured on the demo's own week of history rather
than guessed. A rule set at the average fires constantly and one set an
order of magnitude above it never fires; these sit two to three times the
steady-state rate, so they are quiet in normal operation and true during
the diurnal peak or the seeded incident. Four are absence rules, because
a domain controller or a storefront going silent is not a threshold
question.
Six new dashboards: fleet health, golden signals, security posture,
capacity and storage, commerce, payments.
Three limits of the query language found the hard way and worth writing
down, because each was discovered by a panel failing rather than by
reading: `dc()` does not exist -- the functions are count, sum, avg, min
and max; `or` is not supported between structured filters, so a panel
spanning tiers filters on the attribute they share and groups by service;
and dashboards refuse raw SQL outright. The validator run over all 169
panels and 38 rules now checks every one of those, plus stages, viz types
and comparators.
Two faults, both found by running the reset against the demo rather than
by reading it, and both fixed on the box before this commit existed.
The reset raced its own alerting container. Writing .env changes
alerting's environment, so `docker compose up -d alerting` recreates it
-- and the next line posted notification targets to it with no wait. On a
busy box that lost: curl returned nothing, json.load threw on an empty
string, and set -e killed the script. The damage is in the ordering:
`docker compose down -v` runs near the top, so any failure after it
leaves the public demo up, empty, and with the simulator still stopped,
because the unit is only restarted on the last line. It has been
surviving nightly on timing alone. It now polls /healthz for up to 60
seconds and fails loudly before the seed rather than after the wipe.
Five dashboard panels were raw ClickHouse SQL, which dashboards refuse:
validatePanel rejects query_language "sql" outright, because the
time-range picker is injected as leading query terms and a SELECT has
nowhere to put them. They were written that way because the pipe syntax
has no time bucketing -- no timechart, no bin -- so a genuine time series
is not available to a dashboard panel at all. Each is now the breakdown
the panel was actually asking for: kubelet events by kind and host, DNS
by type and result, queue depth by queue and host, IIS by site and host,
HAProxy by backend and balancer.
Both mistakes were mine and both were avoidable by reading: the rule
about pipe-syntax-only dashboards is stated in terraform/README.md, and I
had already read the line that says it.
Verified on the demo: 50 hosts, 24 services, 1,922,128 records, all 13
dashboards and all 11 alert rules applied, simulator active. `system`
appears on exactly 31 hosts, which is the Linux count -- no Windows host
was given a journald stream.
`go build ./...` inside a main package writes its binary beside the
source, and the fleet commit's `git add -A` took it: fa7c6c0 carries
hack/demo-simulator/demo-simulator at 18,756,186 bytes. Nothing needs it
in the repo -- the demo box gets a binary built for linux/amd64 and
pushed to /home/john/cairnobs-demo/bin/, not this one -- and it landed on
main before anybody noticed.
Removing it here stops the working tree carrying it and stops it growing
with every rebuild, but it does not remove it from history: the blob is
in fa7c6c0 for good unless main is rewritten, so every clone pays 18 MB
for a file no clone uses. Worth deciding deliberately rather than by
default; this change is the half that needs no decision.
The .gitignore entries name all five main packages under hack/ whose
directory name is also their binary name, not just the one that got
caught, since the same `go build ./...` in any of them produces the same
artefact and the next one would land the same way. Checked: none of the
other four is currently tracked, and no other binary over 1 MB is.
Twelve hosts running six services read as somebody's side project. Fifty
hosts across twenty-one services read as an estate, which is what a
visitor is trying to see themselves in.
Thirty-one Linux, eighteen Windows, one Linux host whose agent is gone.
The proportions are the point: Windows now carries Active Directory, IIS,
SQL Server, Exchange, file shares, Remote Desktop, print, WSUS and SCCM
rather than appearing as a Security channel on one box. Linux gains a
load-balancer tier, an outbound proxy, MySQL beside Postgres, RabbitMQ,
Elasticsearch, three Kubernetes nodes, CI, Vault, OpenLDAP, BIND and a
backup server.
Fifteen new generators, each writing what the real daemon writes --
HAProxy's timing quintuple, MySQL slow-query blocks, W3C extended format
for IIS, kubelet PLEG lines, BIND query logging with NXDOMAIN, Squid's
TCP_DENIED, SQL Server deadlock and I/O-stall messages -- with the
structured fields carried in attributes so both halves of the query
language have something to work on.
Three things found while doing it, each of which would have shipped as a
quiet wrongness:
linuxHosts() decided Windows by `service == "eventlog"`. That held while
eventlog was the only Windows role; with IIS and SQL Server on Windows it
would have given every one of them a journald system stream -- sshd and
UFW lines on a Windows box. It now decides from `os`.
worker-02 lost its filling disk when the fleet was rewritten, which is
the story worker-disk-filling thresholds on. Restored at its original
rate, with a comment saying why it cannot move.
Five dashboard panels were written as `timechart`, which this query
language does not have -- its stages are where/stats/sort/fields/head/
tail. They are raw ClickHouse SQL now, which the reference recommends for
exactly this, and a `pie` panel was dropped before it shipped because the
web's VizType union has no such member even though the API accepts one.
Five new dashboards: platform/Kubernetes, directory and DNS, messaging
and search, the Windows server estate, and edge/proxy. Every panel was
checked against the language's real stage list and the web's real viz
types.
Volume roughly quadruples: about 316 records/minute at rate-scale 1, and
1.9M per nightly reset at the demo's own settings against about 0.5M
before. ClickHouse will not notice; reset time and disk on the demo box
might, so the README says so and names RATE_SCALE as the lever.
The status file and the README both still said Phase 4 was not shipped
because the environment had lost Docker and database access partway
through, and that only the audit-logging guarantees had been confirmed
against a live database. That stopped being true some time ago.
phase-4-runbook.md records the opposite in detail: Docker access came
back, a real docker-compose stack ran with real ClickHouse and Postgres
and two provisioned tenants, a local kind cluster ran the Helm chart end
to end, and both SSO protocols were verified against a real Auth0 tenant
with full browser round trips. Eight real bugs came out of that, six from
compose and two from the chart's first real install -- none of them
findable without the infrastructure.
PROJECT-SPEC.md sends readers to status.md and tells them to read it
before assuming a capability works end to end, so the one file that is
meant to be authoritative was the one understating the project by the
widest margin.
Correcting it matters more now than it would have last week, because the
evidence cannot be regenerated: proto.cairnobs.org and the VPS under it
were retired on 2026-09-04, taking the mTLS CA, the server certificate
and six enrolled agents with them. The runbooks are what is left.
Three gaps are now stated rather than implied:
The prototype is gone, so none of this can be re-run today without
building one. The DNS was kept for that; the certificates deliberately
were not.
demo.cairnobs.org is live and is not evidence for Phase 4. It runs
COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says
nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy
demo proving multi-tenancy is exactly the wrong inference to leave
available.
SSO has been tried against one IdP and one local kind cluster, not two
IdPs and not a production-grade cluster.
The Terraform entry now names its cause instead of pointing at another
file: alerting exposes no PUT for rules or targets, and neither
rulestore.Store nor notifystore.Store has an Update method to wire one
to, so Terraform destroys and recreates -- which resets alert_state and
delivery-log continuity.
The status file and the README both still said Phase 4 was not shipped
because the environment had lost Docker and database access partway
through, and that only the audit-logging guarantees had been confirmed
against a live database. That stopped being true some time ago.
phase-4-runbook.md records the opposite in detail: Docker access came
back, a real docker-compose stack ran with real ClickHouse and Postgres
and two provisioned tenants, a local kind cluster ran the Helm chart end
to end, and both SSO protocols were verified against a real Auth0 tenant
with full browser round trips. Eight real bugs came out of that, six from
compose and two from the chart's first real install -- none of them
findable without the infrastructure.
PROJECT-SPEC.md sends readers to status.md and tells them to read it
before assuming a capability works end to end, so the one file that is
meant to be authoritative was the one understating the project by the
widest margin.
Correcting it matters more now than it would have last week, because the
evidence cannot be regenerated: proto.cairnobs.org and the VPS under it
were retired on 2026-09-04, taking the mTLS CA, the server certificate
and six enrolled agents with them. The runbooks are what is left.
Three gaps are now stated rather than implied:
The prototype is gone, so none of this can be re-run today without
building one. The DNS was kept for that; the certificates deliberately
were not.
demo.cairnobs.org is live and is not evidence for Phase 4. It runs
COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says
nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy
demo proving multi-tenancy is exactly the wrong inference to leave
available.
SSO has been tried against one IdP and one local kind cluster, not two
IdPs and not a production-grade cluster.
The Terraform entry now names its cause instead of pointing at another
file: alerting exposes no PUT for rules or targets, and neither
rulestore.Store nor notifystore.Store has an Update method to wire one
to, so Terraform destroys and recreates -- which resets alert_state and
delivery-log continuity.
proto.cairnobs.org and its three subdomains were deleted from Cloudflare
on 2026-09-04 when the prototype was retired and its host destroyed. The
names, types, TTLs and the grey-cloud setting existed only in that zone,
so with the zone entries gone there was nothing left to rebuild from --
and the next prototype would have been reconstructed by guesswork.
Named retired-prototype-dns.json rather than proto-*: proto/ in this
repository is protobuf, and a file in deploy/ starting with proto- reads
as generated definitions rather than a retired environment.
Records only. The mTLS material is deliberately not here: the CA and
server certificate lived on the destroyed host and the six field agents
were uninstalled with their client certs the same day. The next
prototype issues its own CA and re-enrols against it, which the file
says so nobody plans around reusing something that no longer exists.
Dependabot alerts were switched on for this repo today and reported 12 open
findings. Ten are fixed here; the other two are addressed below.
gRPC 1.83.0 -> 1.83.1, in all nine modules that require it. This is
GHSA-vp52-pcj8-j9qc / CVE-2026-84304, heap memory exhaustion via HTTP/2 DATA
frame fragmentation, affecting <= 1.83.0. It matters more than the version
delta suggests: ingest/ is a gRPC listener deliberately exposed to the internet
on :4317, so a remote OOM is reachable. mTLS narrows that to holders of a
client certificate, which is why this was not an emergency, but the fix is one
patch release away and there is no reason to carry it.
golang.org/x/oauth2 0.21.0 -> 0.27.0 in deploy/operator, an indirect
dependency (GHSA-6v2p-p543-phr9). enterprise/ was already past it at 0.36.0.
npm cookie 0.6.0 -> 0.7.2, via an overrides entry rather than a dependency
bump. @sveltejs/kit requires ^0.6.0 and still does at 2.70.3, the latest
release, so there is no version of kit that resolves this on its own -- an
override is the only route that does not involve waiting on upstream.
Three incidental changes came out of `go mod tidy` and are not mine:
genproto/googleapis/rpc moved forward as a transitive of the new grpc; pgx/v5
was reclassified from indirect to direct in enterprise/, which is simply
correct, since audit.go and cmd/enterprise-auth import it; and the proto
replace directive shuffled between require blocks at the same version.
The twelfth finding, lru (GHSA-rhfx-m35p-ff5j), is not fixed and is not
fixable here -- see the note in the pull request. It is CVSS 0, a Stacked
Borrows soundness issue in IterMut, and reaching a patched version means
tantivy 0.22 -> 0.26, which is a search engine migration rather than a
dependency bump.
Verified: all ten Go modules build, 40 test packages pass, the web app builds
and svelte-check reports 0 errors across 288 files.
Unlike the other four projects, cairnobs had no copyright statement to change.
The Licensing section named the licence and pointed at LICENSE, but never said
who holds the copyright, and there were no SPDX headers in the source either.
That is a gap rather than a style choice: an AGPL project with no stated holder
leaves the question open, and the licence's obligations are written around
there being one. So this ADDS a line rather than replacing one.
The two LINUXexpert-org references here are github.com issue and discussion
links in CONTRIBUTING.md -- the repository's real path, unchanged.
LICENSE untouched. Its only copyright is the FSF's on the AGPL text.
The repository has carried only a LICENSE and a README, which scores 42%
on GitHub's community profile against ihasmail's 85% and, more to the
point, leaves someone who finds a vulnerability with nowhere to send it
but a public issue.
SECURITY.md is the one that matters. It names cross-tenant data exposure
as the highest-severity class, since that is the guarantee this project
actually sells and the one hardest to be sure of. It also puts the two
knowingly unverified areas in front of a reporter -- Phase 4 beyond its
audit-logging guarantees, and the Windows agent, which has never run on
Windows -- so a finding there is not mistaken for a regression in a path
that was ever verified. And it declares the demo credential out of scope
explicitly: it is published on purpose, prefilled on the login page and
baked into the web bundle, so a report about it costs everyone time.
CONTRIBUTING.md is written for this tree rather than adapted loosely from
the other one: the storage/query split, PostgreSQL being control-plane
only, the enterprise/ module boundary core never imports across, and the
fact that the whole repository is AGPLv3 with no feature gate. Setup is
the compose path from the README, including COMPOSE_PROFILES, so the two
cannot drift into disagreeing.
CODE_OF_CONDUCT.md is Contributor Covenant 2.0, copied from ihasmail with
one fix: that copy never filled in the contact placeholder, so its
reporting sentence ends at "responsible for enforcement at ." -- a code of
conduct with no way to report anything. Worth fixing there too.
Signed-off-by: John Coffey <[email protected]>
The charter file carried a tool-specific name while being the repository's own
document: mission, non-negotiable constraints, the pinned stack, repo
conventions and phase status, cited as authority by thirty files across the
agent, api, deploy, docs, search and terraform trees.
PROJECT-SPEC.md says what it is. All 42 references are updated in the same
commit, including the relative link in docs/status.md, so nothing points at a
filename that no longer exists.
web/nginx.conf ended its try_files chain in an unconditional /200.html, so
every path the site does not have -- /wp-login.php, /.env, a typo'd inbound
link -- came back as the SPA shell with a success status. It now answers 404,
which needs nginx to know which routes exist: most it infers from the build
output, but dynamic routes and ones that never opted into prerendering have no
file on disk and are listed by hand.
Those hand-maintained lists drift, and the drift is invisible until it ships:
vite dev and npm run preview route from the client manifest and never read
nginx.conf, so a new dynamic route works everywhere a developer would look and
404s in production. hack/check-web-routes.sh compares the lists against
web/src/routes, and a workflow runs it. Its own workflow rather than another
job on license-compliance.yml, which already carries one unrelated check.
Also turns absolute_redirect off. With nginx's default the trailing-slash
canonicaliser reconstructs the origin from its own listen port, so a request
for https://demo.cairnobs.org/settings/ was answered with
Location: http://127.0.0.1:3000/settings -- the container's internal address,
unreachable from the client, and downgraded to http on the way. Verified by
curl against the built image; it was latent here before the canonicaliser
existed too, through the directory redirect on /dev.
GitHub's social card wants a 1280x640 raster and the brand package ships
none. hero-grid.svg alone is the wrong source: it shows the mark with no
project name, which is fine as a splash but wrong for a card that
several platforms render without any accompanying repo title.
Derived from two files already in the package rather than redrawn --
hero-grid.svg with its cairn group dropped for the background, and the
package's own logo-horizontal-dark.svg composited on top so mark-to-
wordmark spacing stays the designer's. The full recipe is recorded in
web/src/lib/assets/README.md beside the existing note for the other
derived asset, including why the crop is 96px off the top and 32px off
the bottom rather than centred (the composition sits low and would crowd
the bottom edge).
Also noted there: the wide gaps in the wordmark are intended spacing,
not a missing font -- the glyphs are outlined paths with no font-family
in the file -- and re-running the recipe matches pixel for pixel but not
byte for byte, since PNG writes a timestamp chunk.
Not applied yet. GitHub exposes Settings -> General -> Social preview
only on public repositories, and the section is absent while this one is
private.
Coffey Labs
jcoffey-dev