Ports license-compliance.yml, security-scan.yml, processing-conformance.yml
and web-routes.yml after the GitHub account was suspended and Actions
stopped being reachable.
The matrices become parallel:matrix over the same crate and module
directories, and cargo-deny-action becomes cargo-deny installed in the job,
since it has no GitLab equivalent. The command lines are unchanged,
including the allowed-licence and ignore lists: that is licence policy, not
configuration, and it must not drift silently.
One real difference: check-tenant-boundary.sh and check-web-routes.sh shell
out to go. The Actions runner happened to have a toolchain on PATH so the
workflow never said so; here they run on the Go image.
The Actions workflows stay in the tree as the reference.
Dependabot merges update lockfiles but never the inventory, so it had
drifted: gridstack was recorded at 11.5.1 and vite at 8.2.1, the agent
still listed tonic 0.12 and toml 0.8, and alerting -- named in the
audit's methodology -- had no rows at all.
Regenerated with the audit's own tools against this tree: cargo deny
list for agent and search, go-licenses csv per Go module, license-checker
for web. 52 versions bumped, 71 rows added, 55 gone, 786 -> 802 rows.
Everything added is permissive (MIT, Apache-2.0, BSD-3-Clause, Zlib or
an OR of those), so nothing is newly flagged; flagged rows go 36 -> 35
because ryu (Apache-2.0 OR BSL-1.0) left search's graph with reqwest
0.13. No bumped crate changed its license.
Built on the rustls update, so agent records rustls 0.23.45 and no
longer lists rustls-pemfile.
As last time, rows whose name and version are unchanged are left
byte-for-byte alone -- 679 of them -- so the diff is only the real
change. That also keeps every hand-written row: the segmentio/asm MIT-0
confirmations, the fonts, the favicon, and the three Docker images.
First-party github.com/cairnobs/cairnobs packages are left out, as
before. New rows follow the conventions the existing ones use, which
the regeneration reproduced exactly for every unchanged Rust and npm
row.
Signed-off-by: John Coffey <[email protected]>
rustls 0.23.43 accepted TLS 1.3 handshake messages sent at the wrong
encryption level when they followed a key-changing message in the same
record, where RFC 8446 requires the connection to be terminated. The
transcript is still authenticated, so a handshake cannot be altered, but
a peer could send in plaintext what should be encrypted. It reaches the
agent through tonic -> tokio-rustls, on its mTLS gRPC link to ingest.
Fixed in 0.23.45, inside the existing range, so only the lockfile moves.
The advisory was published after main last passed, so the agent's
cargo-deny advisories job failed on every open PR, including ones that
never touched Rust.
Also drops the RUSTSEC-2025-0134 ignore from agent/deny.toml. Its own
reason said to delete it once tonic stopped pulling in rustls-pemfile;
tonic 0.14 no longer does, and cargo-deny warned the advisory was not
encountered.
Signed-off-by: John Coffey <[email protected]>
The demo host holds a copy of this tree rather than a checkout, so
updating it means pushing, and the push has to leave the host's own
files alone. Done by hand on 2026-09-10 that went wrong: the exclude
list named .env and both compose files, and missed
`hack/dev-certs/out/` -- gitignored, generated per machine. ingest
mounts that directory, so after the rebuild it served a certificate
signed by the developer's CA while the host's agent simulator still
trusted the host's own, and all 54 agents failed to check in with
`x509: certificate signed by unknown authority` until the simulator was
restarted.
So this syncs `git ls-files` rather than everything-minus-a-list.
Anything gitignored is then unsendable by construction, which is the
whole category the hand-written list kept missing.
Two tracked files are still held back, and they need naming because the
rule above does not cover them:
docker-compose.yml the host binds every published port to 127.0.0.1
by editing this in place; `ports:` cannot be
overridden from an override file, since Compose
concatenates list keys instead of replacing them.
.env the host's carries ALERTING_SERVICE_TOKEN, which
the repo's does not.
It also restarts the simulator. That runs on the host outside Compose,
reads its mTLS material once at startup, and is exactly what nothing
else would have restarted.
Dry run by default; --yes to apply, and it refuses a dirty tree then.
#38 cannot be made green from this side. No @sveltejs/kit release
accepts TypeScript 7: 2.70.3, the current latest, peers
`^5.3.3 || ^6.0.0`, and even the 3.0.0-next.27 prerelease peers
`^6.0.0`. svelte-check caps at 6 as well. `npm ci` stops on ERESOLVE
before a line is compiled, so there is nothing to fix here -- the block
is upstream.
Left open, Dependabot reopens it every week for as long as that stays
true. So the major is ignored for typescript in /web only, with the
condition for removing it written next to it. Minor and patch updates
still come through, and every other package is untouched.
#31 and #36 bump tonic-build alone to 0.14 and break the build outright:
0.14 moved prost codegen out to tonic-prost-build, so
`tonic_build::configure()` no longer exists. Both repositories also pin
`tonic = "0.12"` as the runtime beside it, and codegen from 0.14 against
a 0.12 runtime would be incoherent even if it compiled.
So the whole set moves together, in agent and in search:
tonic 0.12 -> 0.14
tonic-build 0.12 -> tonic-prost-build 0.14 (build-dependency)
prost 0.13 -> 0.14
tonic-prost new runtime dependency
Two consequences worth naming. The generated code now reaches for
`tonic_prost::ProstCodec`, so tonic-prost has to be a real dependency
rather than something the build script pulls in. And prost had to move
with it: tonic-prost 0.14 wants prost 0.14, so leaving ours at 0.13
left the generated types deriving a `Message` trait from a different
prost than the one the codec demanded.
tonic's `tls` feature is gone, replaced by one feature per crypto
provider. `tls-ring` is the like-for-like choice: it is the rustls-plus-
ring pairing `tls` used to mean, and the agent hands its own CA and
identity to ClientTlsConfig for mTLS, so it needs no root store.
The call sites did not change at all -- transport::{Certificate,
Channel, ClientTlsConfig, Identity} and include_proto! are all still
where they were.
Verified by building both, since no CI job compiles Rust: agent and
search check clean, 34 agent tests and 24 search tests pass.
gridstack 13 changed `GridStack.init` to return `GridStack | null` where
11 always handed one back. The dashboard held the result in a
`GridStack | undefined` and called `.on('change')` on it straight after,
so svelte-check stopped on two errors: null is not undefined, and the
value is possibly neither.
Coalesced to undefined so the declared type stays as it was, then
guarded before the listener is attached. `gridEl` is already checked at
the top of the function, so a null here should not occur -- but the type
allows it, and a dashboard that quietly stops persisting drags beats one
that throws inside an effect.
Nothing else in the repository touches gridstack: one import of
`GridStack` and its stylesheet, in this file.
Worth noting `vite build` passes either way. Only `npm run check` sees
this, and no CI job runs it -- the two majors would have gone in
looking clean.
Two Dependabot PRs are stuck behind the same number.
#35 raises the go directive to 1.26.0 in six modules, because
golang.org/x/crypto v0.56.0 requires it -- x/crypto tracks the two most
recent Go releases and 0.56 dropped 1.25. A module that says 1.26 cannot
be built by the 1.25 this repository pins in two places, so that PR
fails every Go job.
#29 raises actions/setup-go to v7, which sets GOTOOLCHAIN=local. With
that set, `go install golang.org/x/vuln/cmd/govulncheck@latest` cannot
quietly fetch a newer toolchain, and stops with
golang.org/x/[email protected] requires go >= 1.26.0 (running go 1.25.14)
Under setup-go v5 the same install succeeded by downloading 1.26 behind
our backs, which is its own reason to be on 1.26 deliberately instead.
So: security-scan's go-version and all eight Dockerfiles move together,
1.25 -> 1.26. Nothing else needs to. A newer toolchain builds an older
directive happily, so this stands on its own before #35 lands, and the
go.mod files stay where they are here.
Checked by building rather than by reading: the api and ingest images
both build on golang:1.26-alpine, and api, ingest and enterprise still
`go build ./...` clean against their existing 1.25 directives.
The account behind it was renamed from LINUXexpert-org to jcoffey-dev,
and GitHub does not redirect the old name: github.com/sponsors/
LINUXexpert-org answers 404 while the new one answers 200. So the
Sponsor button on this repository has been leading nowhere.
Worth fixing rather than leaving to redirect, because a released
username can be registered by anyone -- a stale link stops being a dead
end and starts being someone else's page.
GHSA-2v4p-qf9q-27wj is a panic in gRPC-Go's xDS routing interceptor: a
request arriving with neither `:authority` nor `Host` indexes an empty
slice, the per-RPC goroutine does not recover, and the process dies.
High, and nine alerts, because nine go.mod files pin the same version --
eight directly, terraform indirectly.
Nothing here was reachable. The interceptor is installed by
`xds.NewGRPCServer`, which this repo never calls: the one production
server is `grpc.NewServer(grpc.Creds(...))` in ingest/internal/grpcserver
and the only other is a plain one in a searchclient test. That is also
why security-scan has been green throughout -- govulncheck reports on
reachability and found nothing on 1.83.1, while Dependabot reports on
version ranges and found nine. Both were right.
Taken anyway: it is a patch release, and the next advisory in this
dependency may well land somewhere we do reach.
`go mod tidy` carried the indirect requirements grpc 1.83.2 asks for --
x/net, x/text, x/sys and friends. No CI job builds or tests Go here, so
all nine modules were built locally and api, ingest and enterprise
tested with -count=1, since a cached pass would not have exercised the
new version.
The dependabot.yml is the other half. There was no config, so nothing
opened a PR against any of this. Go majors stay out of the group, being
import path changes rather than bumps.
Dependabot #10: lru's IterMut violates Stacked Borrows, fixed in 0.16.3.
lru was transitive through tantivy 0.22.1, which pins lru ^0.12.0, so
there was no in-range fix -- cargo update -p lru locks nothing. The
advisory was also not reachable: tantivy calls only get, put, len,
peek_lru and new on its LruCache, never iter_mut. Upgrading rather than
dismissing because it is early enough that carrying four versions of
drift costs more than paying it now, and the alert then closes on its
own evidence rather than on an argument.
lru resolves to 0.16.4, past the patch line.
One API change across the four releases. TopDocs no longer implements
Collector on its own -- an ordering has to be chosen rather than
defaulted into. order_by_score() is exactly what bare TopDocs did in
0.22, so result order is preserved rather than quietly changed, which
matters for a search endpoint whose contract is "most relevant first".
Index compatibility checked rather than assumed, since a format change
would have meant a reindex for every existing deployment. Against a live
index of 1757 documents written by 0.22: the service opened it without
error, a document indexed hours earlier by 0.22 is still findable, new
documents written by 0.26 are findable, and a phrase query spans both.
No migration needed.
The compliance inventory is regenerated for the new graph: 17 crates
added, 5 gone, 20 bumped, and three duplicate-version entries collapsed
where the graph no longer needs two. Nothing newly flagged -- every
addition is permissive -- and cargo-deny check licenses, which is the
gate CI actually runs, passes. Rows for crates whose name and version
are unchanged are left byte-for-byte alone, so the diff shows the real
change rather than 200 rows of SPDX term reordering.
Signed-off-by: John Coffey <[email protected]>
The last unanswered action, and the only one whose output is not the
input with edits -- it emits a record that never existed, which is why
it was deferred twice.
It emits the window's first record unchanged, tagged with
cairnobs.aggregated, cairnobs.count, and the observed window bounds.
That follows the convention the agent already uses for heartbeat and
host-metrics records rather than inventing a second synthetic-record
mechanism, and keeping the first record intact means a reader sees a
real example of what was collapsed instead of an invented summary.
It tags even when the count is one. Emitting a bare record there would
be tidier and would make cairnobs.count present only sometimes, so
summing it silently breaks on quiet windows. window_last is the last
record that actually contributed, never window_start + window_ms,
because a window flushed early must not claim an end that never
happened.
Specifying it surfaced a problem the other nine actions do not have.
Windows are measured on record time, so a window can only be closed by a
later record arriving. suppress_duplicates never has anything pending;
aggregate_count holds state, so a matching stream that goes quiet leaves
its aggregate unemitted indefinitely -- data loss dressed as latency.
Emission therefore has a second trigger, end of stream, which the corpus
defines as an implicit flush after the last input and which production
gets from the batch flush. The cost is stated rather than hidden:
window_ms becomes a maximum, not a guarantee, and one burst can produce
more than one aggregate.
And it has a consequence nobody should meet in production first: stats
count undercounts aggregated data silently, so every panel and alert
counting rows changes meaning the moment a rule aggregates the data
behind it. Nothing here fixes that. The correct idiom is summing
cairnobs.count; teaching the query layer to do it automatically is a
Phase 2 change to the IR, recorded as the open question this decision
leaves in its place rather than quietly inherited.
Six cases added, corpus at 45. The validator's unspecified-action guard
stays in place with an empty set, still rejecting anything added to it.
Signed-off-by: John Coffey <[email protected]>
It was a defensible cut while a canary might have caught a bad rollout
partway through. With no canary it is the only thing that recovers a
host without an operator noticing, and "total evaluation should make it
unreachable" is what every crash-loop was before it happened.
Specified rather than named. A two-field file is written and fsynced
before rules are applied: the version being attempted, and whether it is
trying, good or quarantined. The rule set itself is never written, which
is what keeps the crash-loop-not-strand property -- an agent that loses
the file still boots clean and re-syncs. A version found still "trying"
at boot is quarantined, the agent starts with no rules, and it reports
the quarantined version so the failure is visible rather than merely
survived. A different version clears the quarantine, because pushing new
rules is the correction.
Three failure modes decided instead of discovered. An agent that cannot
write the file logs once and runs with total evaluation alone: degrading
to "no rules" would punish every read-only deployment for a failure that
has not happened, and the backstop is best-effort by construction rather
than by accident. An agent killed for an unrelated reason quarantines a
blameless rule set, which is a deliberate false positive -- the
alternative is claiming to distinguish "died because of the rules" from
"died while they happened to be loaded", which it cannot do honestly.
And a rule set fatal on only some hosts quarantines per host, which is
the closest thing to a canary this design has: the first host to hit it
reports while the rest carry on.
Also states that the conformance corpus cannot test any of this, so
nobody tries. The corpus pins rule semantics -- records in, records out.
Process death and file state across restarts are not expressible that
way and need agent-side tests driving a real process through crash and
restart.
Signed-off-by: John Coffey <[email protected]>
regex-lite on the agent, full regex at ingest, conformance corpus
limited to the syntax both accept. The agent had no regex dependency at
all and the full crate is a megabyte-plus against a binary whose pitch
is that it is small and static. The rejected alternative, regex
ingest-side only, would have given up redacting PII before it leaves the
host -- the one capability that most needs to be on the agent, and most
of the reason on-host processing exists at all.
Checked rather than assumed: every pattern the corpus uses compiles and
behaves under regex-lite, including named captures and replace_all
replacing every occurrence, which is what mask needs and what a
redaction stopping at the first hit would get wrong. Both engines also
resolve alternation leftmost-first, confirmed by running the same
pattern through regex-lite and Go's regexp and getting the same output.
A case now pins that, since it is the kind of semantic two independent
implementations can differ on silently.
No canary gate. An edit reaches every matching agent at once, and this
design had suggested that might have to block the feature. It does not,
but the risk is accepted rather than waved away: total evaluation is the
real mitigation, a fatal rule set crash-loops rather than strands
because overrides are never persisted, apply-then-verify makes that
self-healing, and applied_override_version already shows the blast
radius. What is genuinely given up is the ability to stop a bad rollout
partway through -- everything else shortens the outage without
preventing the rule reaching every host first.
That raises the stakes on apply-then-verify, which is still open.
Without a canary, total evaluation stops being a nice property of a
well-designed DSL and becomes the only thing between a bad rule and
every host at once, so cutting the backstop from v1 is a harder call
than it looked when it was written down.
Signed-off-by: John Coffey <[email protected]>
The design argues the conformance suite is the specification and should
be built before either implementation, since two hand-written
implementations of one language diverge unless something shared pins
them. This is that suite: 38 cases in /processing, a language-neutral
top-level directory for the same reason /proto is one -- the Rust agent
and the Go ingest tier both consume the definition and neither owns it.
Nothing executes the cases, because neither implementation exists. A
stdlib-only validator checks the corpus stays well-formed and runs in
CI: known actions, addressable fields, compilable patterns, names
matching filenames, and no case depending on record_id, which is
withheld so the question of whether an ingest-side rule can see one
stays open. The validator was checked against seven deliberately broken
cases before being trusted, since "38/38 valid" means nothing from a
validator that cannot fail.
Writing the cases first has already paid for itself twice.
It forced two determinism decisions the prose had left vague, both of
which a conformance suite cannot avoid answering. Sampling is
counter-based rather than random: random is statistically nicer and
impossible to assert on. Windows are measured on record timestamps
rather than wall-clock, which makes replay deterministic and, not
incidentally, makes backfill behave correctly where a wall-clock window
would not.
And it made the missing aggregate_count answer concrete. The design
does not say what that action emits, or what a query not expecting a
synthetic record sees. Rather than invent one by writing cases, the
validator rejects any case using it, so the design question has to be
answered before the behaviour can be frozen by accident.
The corpus includes the acceptance case from real measured data: the
two processes that account for roughly 60% of a real workstation's
journal volume, and the one kernel message worth keeping.
Signed-off-by: John Coffey <[email protected]>
Starts from the distribution channel and the safety invariant it
protects, and derives the language from them, rather than designing a
rule language and asking later how to ship it.
Four decisions proposed. A rule is a matcher plus ordered typed actions,
with no expressions and nothing resembling eval -- less expressive than
Cribl on purpose, and the only shape that can be pushed to ten thousand
hosts and audited by reading it. Total evaluation is the primary safety
guarantee, with apply-then-verify as a backstop. One spec with two
implementations means a language-neutral conformance suite is the
specification and should be built first, not last. Distribution reuses
DesiredOverride, following extra_file_paths as the precedent for a
repeated field.
It also corrects something #21 got wrong. That change said a fatal rule
would strand an agent the way a corrupted ingest endpoint does. Reading
apply_override's actual semantics, overrides live only in the running
process's memory and are never written to disk, so a restarted agent
boots clean and re-syncs -- a fatal rule set crash-loops rather than
strands, and the agent keeps checking in, so it stays correctable. That
is a much better failure mode, and it was acquired by accident: the
"don't persist" choice was made to avoid filesystem writes on read-only
images, not for safety. This design promotes it to a constraint, since
persisting overrides later would silently convert every crash-loop into
a strand. positioning.md is corrected to match rather than left
disagreeing.
Five open questions are left open rather than answered to look decisive,
the sharpest being that there is no staged rollout today: an edit
reaches every matching agent at once, which for executable rules is the
difference between breaking one host and breaking all of them.
The v1 acceptance test is real data, not a fixture: two processes on the
maintainer's own workstation account for 308 of 325 journal entries in
five minutes, and a suppress rule should remove about 60% of that host's
volume.
Signed-off-by: John Coffey <[email protected]>
Three of the roadmap's claims were contradicted by the repository
itself.
Fleet management was Phase 11, "Planned", and positioning.md said config
still flowed to the agent from the host rather than from the platform.
agent-management-design.md has recorded the opposite for some time:
punch list complete, verified live, with central authoring, versioning,
rollout on the next check-in, observation, and a restart command a real
agent picks up and acts on. There is no Phase 11 now. Its remainder is
either already named there -- stop/uninstall, per-host multi-row
alerting, a rule-per-host generator -- or belongs to Phase 8, since
distributing rules is the one genuinely new thing the mechanism has to
carry, and a rule language nobody can push to a fleet is not worth
having.
That inverts the old ordering argument, which put fleet last on the
grounds that it manages configuration the earlier phases define. Sound
reasoning; the world went the other way and built the mechanism first.
Recorded rather than quietly dropped, because the instinct behind it is
a good one that happened not to apply.
Retention was listed as a question Phase 10 would finally have to
answer. Half of it is answered: api/logretention serves operator-driven
preview and delete with an owner-only per-agent floor. What is missing
is an automatic TTL, so Phase 10 owns tiering and automatic TTL rather
than retention from nothing.
The rule-language recommendation is now settled rather than proposed,
and not on its own authority: DesiredOverride is already a closed typed
shape that cannot carry arbitrary code, so no channel exists that would
deliver JavaScript to an agent even if the language argument had gone
the other way.
That surfaced a requirement nothing had written down. Agent management
rests on an invariant it states outright -- every editable field
degrades behaviour without cutting off the agent's ability to receive
the next correction. Processing rules break it: a rule that panics or
loops strands the agent exactly the way a corrupted ingest endpoint
would, across every host it reached first. Phase 8 now owes either total
evaluation or apply-then-verify with rollback, chosen deliberately
rather than discovered mid-rollout.
Signed-off-by: John Coffey <[email protected]>
Cairn OBS is self-hosted, and the way to separate two environments is to
run two installations rather than two tenants inside one. Tenancy is the
wrong boundary for that, on three counts this repository demonstrates
rather than assumes: chwriter.WriteBatch is all-or-nothing across
tenants, so one tenant's failure stalls offset progress for every other;
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT puts every tenant's data behind a
single superuser credential, as docker-compose.yml's own comment says;
and one binary with one set of migrations moves every tenant together,
which is the opposite of what separate environments are for. A whole
installation idles at about 1.3 GB, so the sharing buys nothing.
The project led with multi-tenant RBAC in the README banner and in
PROJECT-SPEC's goal statement. Both now say what it is instead:
self-hosted. "Open-core" goes with them -- it was already inaccurate,
since CONTRIBUTING states there is no feature gate and no paid tier, and
with enterprise/ off the roadmap there will not be one.
A second identity provider comes off the list of things standing between
this and production-ready. SSO belongs to enterprise/, and a self-hosted
deployment is not waiting on it. Terraform's tenant/RBAC resources move
from "disclosed future work" to not planned.
Nothing is scrubbed from the record. Phase 4 stays shipped, its runbook
stays, and its known gaps stay stated -- rewriting that history would
contradict the candour the Status section is built on. enterprise/ stays
in the tree, AGPLv3 and working, as the answer to a question this
project is not asking.
Signed-off-by: John Coffey <[email protected]>
Local login is implemented, wired through api, alerting and web, and
undiscoverable. No compose file turns it on, the Helm chart sets none
of its variables, and no markdown in the repository mentions
-seed-admin, LOCAL_AUTH_ENABLED or local login at all. The only way to
find it is to read cmd/api/main.go's authorizer switch.
Enabling it in docker-compose.yml is not the answer: a plain
`docker compose up` has no authentication, and every Phase 0-3 runbook
verifies the pipeline with bare curl against /query. Turning login on
by default would break the project's own documented verification.
So it's an opt-in overlay instead.
Four settings have to agree, and only one of them is obviously about
login. Each fails differently and none of the failures name the cause:
the route 404s, or the browser refuses the request before sending it,
or login returns 200 and every later request is anonymous because the
cookie was never stored, or the same symptom again from the opposite
end because the bundle never attaches it. That is what the new document
is mostly for.
The Helm chart still has no local-login support. Recorded in the
document as a gap rather than papered over.
Signed-off-by: John Coffey <[email protected]>
Phase 2 unified the two query languages behind one endpoint and renamed
the request field, and the runbooks were never updated. Following them
today does not work:
{"sql": ...} -> 400 query must not be empty
POST /api :8080/search -> 404, the route no longer exists
Both appear in the Phase 0 and Phase 1 runbooks and in the
windows-fixture README. That matters more than a normal doc typo,
because status.md cites the Phase 0 runbook as the record of how Phase
0 was verified -- so the documented verification procedure is one
nobody can re-run as written.
The Phase 1 step is rewritten rather than search-and-replaced: it
checked the SQL and full-text paths against two different endpoints,
and its exit criterion (the same record_id from both) now has to be
expressed against /query twice, once with SQL and once with a bare
word.
Phase 0's expected output for SELECT 1 also gained a warnings field
since it was written.
Every command here was run against a live stack before being written
down, including confirming both paths return the same record_id.
Signed-off-by: John Coffey <[email protected]>
The sidebar decided which auth mode was live from enterpriseAuthBase,
so any deployment with VITE_ENTERPRISE_AUTH_BASE_URL set rendered the
enterprise block -- and compose sets it unconditionally, so the tenant
picker can exist. On a single-tenant stack with local login on, that
meant the local block could never render: no username, no role, no
Change password, no Log out, and in their place a "Sign in" link
pointing at enterprise-auth's OIDC route, which is disabled unless
OIDC_ISSUER_URL is configured. A dead link where the account controls
should be.
The build-time flag was never the right thing to ask. api registers
/auth/* only when LOCAL_AUTH_ENABLED is set and ENTERPRISE_AUTH_URL is
not, so the frontend cannot know the mode from its own build args --
the two can disagree, and here they did. getLocalSession already
distinguishes 'disabled' (a 404 from /auth/session) from null (a 401,
logged out); the sidebar collapsed both to null and threw the answer
away. It now keeps that distinction and branches on it, so the mode
comes from what the server actually serves.
Logged out under local auth, the sidebar previously rendered no auth
block at all -- no way back to the login page from the nav. It now
offers Sign in, pointing at /login.
Neither block renders until the probe lands, so nothing flashes the
wrong mode on load.
Signed-off-by: John Coffey <[email protected]>
Deleting your own user succeeded, and logged you out doing it:
local_sessions.user_id is ON DELETE CASCADE, so the delete took the
caller's own live session with it. Nothing refused this. The
last-owner guard is the only thing in the path, and it passes cleanly
as soon as a second owner exists -- which is exactly the state you are
in just after creating one.
The way back in was then whatever other account happened to exist, and
-seed-admin could not help: it skipped whenever *any* local user was
present, so the command documented as the way to create an
administrator refused precisely when there was no usable one, because
some other account still existed. It now asks whether the admin
account itself is missing, which is what its own help text always
claimed, and what makes it useful as recovery rather than only as
first-run bootstrap.
TestCanDeleteAnOwnerWhenAnotherRemains signed in as admin1 and deleted
admin1, asserting 204 -- it encoded the lockout as intended behaviour.
It now deletes the other owner, which is what it meant to cover, and a
new test holds the refusal in place.
runSeedAdmin takes a small interface so the bootstrap path is tested
without a Postgres pool; it had no tests before.
Signed-off-by: John Coffey <[email protected]>
The commerce generators shipped with currency=GBP on every order, refund,
authorisation and chargeback -- twelve places, all of them a default
nobody chose. Dollars is the convention everywhere else.
Only the label changes. The SKU prices stay as they are: a $489 task
chair and a $629 standing desk are as plausible as the pound figures
were, and moving them would have shifted average order value on the
demo's dashboards for no reason other than tidiness.
Nothing else referenced the currency -- no dashboard panel and no alert
rule filters or groups on it -- so this is the whole change.
Splunk is from spelunking: caving, a lamp, feeling your way along in the
dark. An honest description of search-driven investigation -- powerful
with expertise, unforgiving without it.
Cribl is from cribble, to sift, from Latin cribrum, a sieve; the same root
gives engraving its maniere criblee, the dotted ground punched into a
plate. Both senses land together: the data is a medium to be worked and
thinned on the way through. A name about the material, not the
destination.
A cairn is a stack of stones on open ground, where the path is not
obvious, doing one job -- somebody came this way, and this is the way.
Three of its properties map onto things this project already does rather
than things it claims. It is left by whoever went first for whoever comes
next, which is the runbook culture and the reason every phase records
what was actually run including the failures. Anyone passing adds to it,
which is AGPLv3 throughout and an egress path that helps data leave.
And you can see it from a distance in daylight, which is a legible query
language, an AI that explains rather than divines, and a plan that
publishes what has not been proven.
Written into positioning.md rather than kept as a marketing note because
it is a reason the position coheres, not decoration on top of it. Also
noted there that it should not be turned into a slogan.
The Status section went from a table of "Shipped" straight into three
paragraphs of caveats, with nothing in between telling a reader what
standard was being applied. Read cold, that is a project confessing. Read
with the standard stated first, it is a project that refuses to call
something done because the tests pass.
So the section now opens by saying what stage this is -- pre-1.0, no
production workload -- and what "shipped" means here: verified against
real infrastructure with a runbook recording how, including what the
verification could not reach. And it says plainly why the caveats are
long, which is that they are disclosed rather than discovered. A shorter
Status section would not mean a more finished product, only a less
careful one.
It closes with what would actually close the gap: a second IdP, a real
cluster, the Windows agent on Windows, sustained load, and somebody
else's data. None of that is research, it is time on real infrastructure
-- which makes it the list a pilot works through, and makes a 1.0 tag the
wrong next milestone to reach for.
Written because the risk of a public repo at this stage is not a
competitor reading the roadmap, it is a prospect reading unusual honesty
as immaturity. The fix for that is context, not privacy.
Cost is the argument against Splunk and control is the argument against
Cribl. AI is the third, and the difference there is not a feature
comparison, it is where the model runs.
Plain-English querying shipped in Phase 7 and stays an option rather than
a replacement for writing a query: every generated query compiles through
the same IR and executor as a hand-written one, with the same tenant
scoping, cost guardrails and audit logging. The model suggests, it does
not get a private path to the data.
AI-assisted analysis and explanation is the end state and is not built.
Authoring answers "how do I ask this"; the valuable question is "what
does this mean" -- what changed in a result set, why an alert fired and
what preceded it, summarising an incident from the records around it.
Recorded on the status page as an end-state goal rather than a numbered
phase, because it is a property the product keeps rather than a thing to
finish and tick off.
Local is the non-negotiable part, and it is worth stating as position
rather than as a bullet: the default runs qwen2.5-coder through Ollama on
the customer's own hardware, Apache-2.0 weights chosen so Phase 6's
licence work survives contact with the model, and the cloud adapter is
opt-in and off by default. Logs are the most sensitive unstructured data
most organisations hold -- credentials in stack traces, customer
identifiers, internal topology -- so an assistant that reads them is
either running where the data already is, or it is a data-egress decision
wearing a helpful interface.
The constraint it imposes is stated too, because it bounds what can be
promised: a 7B model on a customer's hardware will not match a frontier
model, and the honest claim is not that it is as clever but that it is
good enough at a bounded task and runs somewhere you control. Analysis
features have to be designed to that budget rather than assuming an API
is one call away.
Splunk and Cribl are not the same competitor and the claim is not the
same claim twice. Splunk is the destination and Cairn OBS replaces it,
which is what phases 0-7 were for. Cribl is the road: routing, reduction,
enrichment, redaction and replay on the way to wherever data is going.
Cairn OBS is already a road in shape -- agent, Redpanda, ingest -- and
exposes none of a pipeline's controls. The agent cannot filter, drop,
sample, mask or re-route anything; ingest normalises a schema and writes
it; there is exactly one destination and it is us.
Reconciling the two turns up something a cost-led project has to face
rather than paper over: most people buy Cribl because Splunk is expensive
per gigabyte, so being genuinely cheap per gigabyte removes the main
reason to buy Cribl in front of us. That makes the strongest pitch "one
system where there were two" rather than "we are also a pipeline vendor"
-- but that pitch only survives a buyer if we also do the four things
people buy a pipeline for that are not about spend: routing to several
destinations, redacting before data leaves the network, archive and
replay, and not being locked to one analytics vendor. Those are about
control, which is better ground anyway: cost advantages get matched and
architectural ones do not.
The consequence is uncomfortable and is written down as a decision rather
than left to be discovered: competing with Cribl means being able to send
data to S3, Splunk HEC, Elastic, OTLP and Kafka -- building features whose
purpose is to help data leave this platform. A project that refuses
lock-in in its licence and then builds it into its egress would be lying
about itself.
Four phases follow, ordered so each pays for itself: processing (8),
routing (9), archive and replay (10), fleet (11). Processing without
routing still shrinks what is stored; routing without processing forwards
everything and helps nobody.
One design decision is called out now because it collides with a
non-negotiable constraint. Cribl's rule language is JavaScript, and
embedding a JS engine in a statically-linked musl agent would end "no
glibc runtime deps" as a claim. The recommendation is a declarative rule
DSL -- matchers and typed actions, no arbitrary code -- deliberately less
expressive, small enough to audit and safe to push to ten thousand hosts.
The retention/TTL question in architecture.md is no longer deferrable and
now says so: Phase 10 asks it from the other side.
The estate could show an operator their infrastructure and had nothing to
say to the business paying for it. Two storefronts and the gateway behind
both fix that: Magento on two hosts, WooCommerce on one, and pay-01
carrying authorisations with amount, gateway and decline reason. Orders,
revenue, average order value, where checkout loses people and why a card
was refused now come out of the same log lines the operators are already
reading, which is the argument for not running a separate metrics stack
beside this one.
Two platforms rather than one deliberately. Magento and WooCommerce write
about the same events differently, so a panel that groups by service
instead of assuming a single shape is the honest way to build one -- and
the demo shows that rather than describing it.
Order totals are built from a basket of real SKUs at real prices rather
than drawn from a distribution, so average order value moves the way one
actually moves. Declines rise during the seeded outage window alongside
the 5xx rate, because whatever fails requests fails authorisations too.
Twenty-seven new alert rules, thresholds calibrated against what the
fleet actually emits -- measured on the demo's own week of history rather
than guessed. A rule set at the average fires constantly and one set an
order of magnitude above it never fires; these sit two to three times the
steady-state rate, so they are quiet in normal operation and true during
the diurnal peak or the seeded incident. Four are absence rules, because
a domain controller or a storefront going silent is not a threshold
question.
Six new dashboards: fleet health, golden signals, security posture,
capacity and storage, commerce, payments.
Three limits of the query language found the hard way and worth writing
down, because each was discovered by a panel failing rather than by
reading: `dc()` does not exist -- the functions are count, sum, avg, min
and max; `or` is not supported between structured filters, so a panel
spanning tiers filters on the attribute they share and groups by service;
and dashboards refuse raw SQL outright. The validator run over all 169
panels and 38 rules now checks every one of those, plus stages, viz types
and comparators.
Two faults, both found by running the reset against the demo rather than
by reading it, and both fixed on the box before this commit existed.
The reset raced its own alerting container. Writing .env changes
alerting's environment, so `docker compose up -d alerting` recreates it
-- and the next line posted notification targets to it with no wait. On a
busy box that lost: curl returned nothing, json.load threw on an empty
string, and set -e killed the script. The damage is in the ordering:
`docker compose down -v` runs near the top, so any failure after it
leaves the public demo up, empty, and with the simulator still stopped,
because the unit is only restarted on the last line. It has been
surviving nightly on timing alone. It now polls /healthz for up to 60
seconds and fails loudly before the seed rather than after the wipe.
Five dashboard panels were raw ClickHouse SQL, which dashboards refuse:
validatePanel rejects query_language "sql" outright, because the
time-range picker is injected as leading query terms and a SELECT has
nowhere to put them. They were written that way because the pipe syntax
has no time bucketing -- no timechart, no bin -- so a genuine time series
is not available to a dashboard panel at all. Each is now the breakdown
the panel was actually asking for: kubelet events by kind and host, DNS
by type and result, queue depth by queue and host, IIS by site and host,
HAProxy by backend and balancer.
Both mistakes were mine and both were avoidable by reading: the rule
about pipe-syntax-only dashboards is stated in terraform/README.md, and I
had already read the line that says it.
Verified on the demo: 50 hosts, 24 services, 1,922,128 records, all 13
dashboards and all 11 alert rules applied, simulator active. `system`
appears on exactly 31 hosts, which is the Linux count -- no Windows host
was given a journald stream.
`go build ./...` inside a main package writes its binary beside the
source, and the fleet commit's `git add -A` took it: fa7c6c0 carries
hack/demo-simulator/demo-simulator at 18,756,186 bytes. Nothing needs it
in the repo -- the demo box gets a binary built for linux/amd64 and
pushed to /home/john/cairnobs-demo/bin/, not this one -- and it landed on
main before anybody noticed.
Removing it here stops the working tree carrying it and stops it growing
with every rebuild, but it does not remove it from history: the blob is
in fa7c6c0 for good unless main is rewritten, so every clone pays 18 MB
for a file no clone uses. Worth deciding deliberately rather than by
default; this change is the half that needs no decision.
The .gitignore entries name all five main packages under hack/ whose
directory name is also their binary name, not just the one that got
caught, since the same `go build ./...` in any of them produces the same
artefact and the next one would land the same way. Checked: none of the
other four is currently tracked, and no other binary over 1 MB is.
Twelve hosts running six services read as somebody's side project. Fifty
hosts across twenty-one services read as an estate, which is what a
visitor is trying to see themselves in.
Thirty-one Linux, eighteen Windows, one Linux host whose agent is gone.
The proportions are the point: Windows now carries Active Directory, IIS,
SQL Server, Exchange, file shares, Remote Desktop, print, WSUS and SCCM
rather than appearing as a Security channel on one box. Linux gains a
load-balancer tier, an outbound proxy, MySQL beside Postgres, RabbitMQ,
Elasticsearch, three Kubernetes nodes, CI, Vault, OpenLDAP, BIND and a
backup server.
Fifteen new generators, each writing what the real daemon writes --
HAProxy's timing quintuple, MySQL slow-query blocks, W3C extended format
for IIS, kubelet PLEG lines, BIND query logging with NXDOMAIN, Squid's
TCP_DENIED, SQL Server deadlock and I/O-stall messages -- with the
structured fields carried in attributes so both halves of the query
language have something to work on.
Three things found while doing it, each of which would have shipped as a
quiet wrongness:
linuxHosts() decided Windows by `service == "eventlog"`. That held while
eventlog was the only Windows role; with IIS and SQL Server on Windows it
would have given every one of them a journald system stream -- sshd and
UFW lines on a Windows box. It now decides from `os`.
worker-02 lost its filling disk when the fleet was rewritten, which is
the story worker-disk-filling thresholds on. Restored at its original
rate, with a comment saying why it cannot move.
Five dashboard panels were written as `timechart`, which this query
language does not have -- its stages are where/stats/sort/fields/head/
tail. They are raw ClickHouse SQL now, which the reference recommends for
exactly this, and a `pie` panel was dropped before it shipped because the
web's VizType union has no such member even though the API accepts one.
Five new dashboards: platform/Kubernetes, directory and DNS, messaging
and search, the Windows server estate, and edge/proxy. Every panel was
checked against the language's real stage list and the web's real viz
types.
Volume roughly quadruples: about 316 records/minute at rate-scale 1, and
1.9M per nightly reset at the demo's own settings against about 0.5M
before. ClickHouse will not notice; reset time and disk on the demo box
might, so the README says so and names RATE_SCALE as the lever.
The status file and the README both still said Phase 4 was not shipped
because the environment had lost Docker and database access partway
through, and that only the audit-logging guarantees had been confirmed
against a live database. That stopped being true some time ago.
phase-4-runbook.md records the opposite in detail: Docker access came
back, a real docker-compose stack ran with real ClickHouse and Postgres
and two provisioned tenants, a local kind cluster ran the Helm chart end
to end, and both SSO protocols were verified against a real Auth0 tenant
with full browser round trips. Eight real bugs came out of that, six from
compose and two from the chart's first real install -- none of them
findable without the infrastructure.
PROJECT-SPEC.md sends readers to status.md and tells them to read it
before assuming a capability works end to end, so the one file that is
meant to be authoritative was the one understating the project by the
widest margin.
Correcting it matters more now than it would have last week, because the
evidence cannot be regenerated: proto.cairnobs.org and the VPS under it
were retired on 2026-09-04, taking the mTLS CA, the server certificate
and six enrolled agents with them. The runbooks are what is left.
Three gaps are now stated rather than implied:
The prototype is gone, so none of this can be re-run today without
building one. The DNS was kept for that; the certificates deliberately
were not.
demo.cairnobs.org is live and is not evidence for Phase 4. It runs
COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says
nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy
demo proving multi-tenancy is exactly the wrong inference to leave
available.
SSO has been tried against one IdP and one local kind cluster, not two
IdPs and not a production-grade cluster.
The Terraform entry now names its cause instead of pointing at another
file: alerting exposes no PUT for rules or targets, and neither
rulestore.Store nor notifystore.Store has an Update method to wire one
to, so Terraform destroys and recreates -- which resets alert_state and
delivery-log continuity.
Dependabot alerts were switched on for this repo today and reported 12 open
findings. Ten are fixed here; the other two are addressed below.
gRPC 1.83.0 -> 1.83.1, in all nine modules that require it. This is
GHSA-vp52-pcj8-j9qc / CVE-2026-84304, heap memory exhaustion via HTTP/2 DATA
frame fragmentation, affecting <= 1.83.0. It matters more than the version
delta suggests: ingest/ is a gRPC listener deliberately exposed to the internet
on :4317, so a remote OOM is reachable. mTLS narrows that to holders of a
client certificate, which is why this was not an emergency, but the fix is one
patch release away and there is no reason to carry it.
golang.org/x/oauth2 0.21.0 -> 0.27.0 in deploy/operator, an indirect
dependency (GHSA-6v2p-p543-phr9). enterprise/ was already past it at 0.36.0.
npm cookie 0.6.0 -> 0.7.2, via an overrides entry rather than a dependency
bump. @sveltejs/kit requires ^0.6.0 and still does at 2.70.3, the latest
release, so there is no version of kit that resolves this on its own -- an
override is the only route that does not involve waiting on upstream.
Three incidental changes came out of `go mod tidy` and are not mine:
genproto/googleapis/rpc moved forward as a transitive of the new grpc; pgx/v5
was reclassified from indirect to direct in enterprise/, which is simply
correct, since audit.go and cmd/enterprise-auth import it; and the proto
replace directive shuffled between require blocks at the same version.
The twelfth finding, lru (GHSA-rhfx-m35p-ff5j), is not fixed and is not
fixable here -- see the note in the pull request. It is CVSS 0, a Stacked
Borrows soundness issue in IterMut, and reaching a patched version means
tantivy 0.22 -> 0.26, which is a search engine migration rather than a
dependency bump.
Verified: all ten Go modules build, 40 test packages pass, the web app builds
and svelte-check reports 0 errors across 288 files.
Unlike the other four projects, cairnobs had no copyright statement to change.
The Licensing section named the licence and pointed at LICENSE, but never said
who holds the copyright, and there were no SPDX headers in the source either.
That is a gap rather than a style choice: an AGPL project with no stated holder
leaves the question open, and the licence's obligations are written around
there being one. So this ADDS a line rather than replacing one.
The two LINUXexpert-org references here are github.com issue and discussion
links in CONTRIBUTING.md -- the repository's real path, unchanged.
LICENSE untouched. Its only copyright is the FSF's on the AGPL text.
The repository has carried only a LICENSE and a README, which scores 42%
on GitHub's community profile against ihasmail's 85% and, more to the
point, leaves someone who finds a vulnerability with nowhere to send it
but a public issue.
SECURITY.md is the one that matters. It names cross-tenant data exposure
as the highest-severity class, since that is the guarantee this project
actually sells and the one hardest to be sure of. It also puts the two
knowingly unverified areas in front of a reporter -- Phase 4 beyond its
audit-logging guarantees, and the Windows agent, which has never run on
Windows -- so a finding there is not mistaken for a regression in a path
that was ever verified. And it declares the demo credential out of scope
explicitly: it is published on purpose, prefilled on the login page and
baked into the web bundle, so a report about it costs everyone time.
CONTRIBUTING.md is written for this tree rather than adapted loosely from
the other one: the storage/query split, PostgreSQL being control-plane
only, the enterprise/ module boundary core never imports across, and the
fact that the whole repository is AGPLv3 with no feature gate. Setup is
the compose path from the README, including COMPOSE_PROFILES, so the two
cannot drift into disagreeing.
CODE_OF_CONDUCT.md is Contributor Covenant 2.0, copied from ihasmail with
one fix: that copy never filled in the contact placeholder, so its
reporting sentence ends at "responsible for enforcement at ." -- a code of
conduct with no way to report anything. Worth fixing there too.
Signed-off-by: John Coffey <[email protected]>
The charter file carried a tool-specific name while being the repository's own
document: mission, non-negotiable constraints, the pinned stack, repo
conventions and phase status, cited as authority by thirty files across the
agent, api, deploy, docs, search and terraform trees.
PROJECT-SPEC.md says what it is. All 42 references are updated in the same
commit, including the relative link in docs/status.md, so nothing points at a
filename that no longer exists.
web/nginx.conf ended its try_files chain in an unconditional /200.html, so
every path the site does not have -- /wp-login.php, /.env, a typo'd inbound
link -- came back as the SPA shell with a success status. It now answers 404,
which needs nginx to know which routes exist: most it infers from the build
output, but dynamic routes and ones that never opted into prerendering have no
file on disk and are listed by hand.
Those hand-maintained lists drift, and the drift is invisible until it ships:
vite dev and npm run preview route from the client manifest and never read
nginx.conf, so a new dynamic route works everywhere a developer would look and
404s in production. hack/check-web-routes.sh compares the lists against
web/src/routes, and a workflow runs it. Its own workflow rather than another
job on license-compliance.yml, which already carries one unrelated check.
Also turns absolute_redirect off. With nginx's default the trailing-slash
canonicaliser reconstructs the origin from its own listen port, so a request
for https://demo.cairnobs.org/settings/ was answered with
Location: http://127.0.0.1:3000/settings -- the container's internal address,
unreachable from the client, and downgraded to http on the way. Verified by
curl against the built image; it was latent here before the canonicaliser
existed too, through the directory redirect on /dev.
GitHub's social card wants a 1280x640 raster and the brand package ships
none. hero-grid.svg alone is the wrong source: it shows the mark with no
project name, which is fine as a splash but wrong for a card that
several platforms render without any accompanying repo title.
Derived from two files already in the package rather than redrawn --
hero-grid.svg with its cairn group dropped for the background, and the
package's own logo-horizontal-dark.svg composited on top so mark-to-
wordmark spacing stays the designer's. The full recipe is recorded in
web/src/lib/assets/README.md beside the existing note for the other
derived asset, including why the crop is 96px off the top and 32px off
the bottom rather than centred (the composition sits low and would crowd
the bottom edge).
Also noted there: the wide gaps in the wordmark are intended spacing,
not a missing font -- the glyphs are outlined paths with no font-family
in the file -- and re-running the recipe matches pixel for pixel but not
byte for byte, since PNG writes a timestamp chunk.
Not applied yet. GitHub exposes Settings -> General -> Social preview
only on public repositories, and the section is absent while this one is
private.
The horizontal logo already carries the wordmark, so "# Cairn OBS"
directly beneath it repeated the same words twice in the first inch of
the page. Removed, and the tagline and licence line centred under the
logo so the header reads as one block.
The bold and the link are written as <strong>/<a> rather than markdown:
GitHub does not process markdown inside a raw HTML block, so **...** and
[...](...) would have rendered as literal asterisks and brackets.
Nothing linked to the removed heading, and the #licensing anchor it
referenced comes from `## Licensing`, which is untouched.
Uses the brand assets already in the repo rather than adding a copy --
web/src/lib/assets/logo-horizontal-{light,dark}.svg are the same files
NavSidebar renders, so the README cannot drift from the app's logo.
Wrapped in <picture> with a prefers-color-scheme source, since the two
variants differ by the colour they are drawn *for*, not by name: -light
carries #111315 text for light backgrounds, -dark carries #F2F2F2 for
dark ones. A single file would have been near-invisible on one of
GitHub's two themes. Both have transparent backgrounds -- the only
<rect> in each is a small design element, not a fill -- so neither
variant shows a card behind it.
reset-demo.sh carried EVALUATOR_PASSWORD as a literal, which put a
working service-account password in the repository. Anyone who could
read the source could log in as alerting-evaluator against the live demo
and mint an ALERTING_SERVICE_TOKEN whenever they liked -- so rotating
the token was theatre while the password that mints it stayed published.
Nothing needed it to be a constant. The account exists only to mint that
token a few lines later, and `docker compose down -v` earlier in the
same script has already destroyed the previous account, so the value
never outlives a single reset. Generated per run instead.
DEMO_PASSWORD directly above is deliberately left alone: it is a public
demo credential, prefilled on the login page and baked into the web
bundle at build time from the demo host's compose override. Randomising
it would break the demo's own login form, and it protects nothing.
The runbook opened with `ALTER USER default IDENTIFIED BY ...` and
claimed the image entrypoint reads CLICKHOUSE_PASSWORD only when it
initialises an empty data volume. Both claims were wrong, and following
them would have failed at the first command.
`SELECT name, storage FROM system.users` reports `default | users_xml`:
the user is defined by a config file, and ClickHouse rejects ALTER USER
against that storage. There is no in-database password to change. The
password lives in /etc/clickhouse-server/users.d/default-user.xml, which
the entrypoint regenerates on *every* container start -- that path is in
the container filesystem, and only /var/lib/clickhouse is a volume. The
file's mtime matches the container's StartedAt to the second, before and
after a rotation.
So the env var is the source of truth and recreating the container is
what applies it -- close to the inverse of what was written.
/opt/sentry's own docker-compose.override.yml header had recorded the
correct mechanism during the 2026-08-19 rotation. The runbook was
drafted without reading it. Noted at the top of the new Verification
status section, since a runbook that was wrong once should say so.
Procedure rewritten around what was actually executed against proto on
2026-08-23 and verified: back up first, generate hex on the box so the
value never reaches a terminal or transcript, sed only indented KEY:
value lines so the header comment is untouched, `docker compose config
-q` before touching anything live, then recreate. Verification now
checks that the OLD password is rejected -- read from the backup rather
than retyped -- not merely that the new one works.
Rollback rewritten to restore the backup, since there is no ALTER to
reverse. Every shell block syntax-checked with bash -n.
The last failing job in the security scan, and one that only became
visible once fail-fast stopped cancelling it.
Same class as the instant advisory handled in search/deny.toml:
unmaintained, not vulnerable. rustls-pemfile's repository was archived
in August 2025; there is no CVE, and the advisory itself says "No safe
upgrade is available". It arrives via cairnobs-agent -> tonic 0.12.3 ->
rustls-pemfile 2.2.0, so the migration to rustls-pki-types' PemObject
API is tonic's to make, not ours. The final rustls-pemfile release is a
thin wrapper over that same rustls-pki-types code, so the archived crate
is not running different logic from its replacement.
Recorded with a note to delete the entry at the next tonic upgrade
rather than let it ossify.
Two unrelated causes, plus one that hid the others.
govulncheck: setup-go used go-version-file, so it installed exactly what
each go.mod pins -- `go 1.25.0` -- and then reported 28 CVEs in that
release's standard library (crypto/x509 quadratic name-constraint
parsing, GO-2025-4007, and friends), all fixed in 1.25.3. None of it
described anything we ship: every Dockerfile builds FROM
golang:1.25-alpine, a floating tag that resolves to the newest 1.25.x,
so the binaries already had the fixes. The go directive is a minimum
language version, not a statement about which toolchain to audit with.
Track the floating 1.25 line instead, and the scan matches production.
Confirmed by running govulncheck against a patched toolchain locally:
deploy/operator reports 0 vulnerabilities and exits 0.
cargo-deny: RUSTSEC-2024-0384, `instant` is unmaintained. A maintenance
advisory rather than a vulnerability -- no CVE, nothing to patch -- and
it arrives transitively via tantivy 0.22.1 -> measure_time 0.8.3, so it
cannot be dropped without moving off the pinned Tantivy. The advisory's
substance does not apply here anyway: instant papers over
std::time::Instant being missing on wasm, and search builds native musl.
Ignored in search/deny.toml with that reasoning recorded and a note to
delete the entry at the next Tantivy upgrade rather than let it ossify.
Both matrices now set fail-fast: false. Only two of the twelve jobs
actually failed; the other nine were cancelled, which made a two-cause
failure look like a total collapse and hid every finding but the first.
Written because rotating this value is not the env-var edit it looks
like. The official image's entrypoint consumes CLICKHOUSE_PASSWORD only
when initialising an empty data volume, so on a running box editing the
override and restarting leaves ClickHouse on the old password while the
clients present the new one -- every query 403s. ALTER USER has to come
first.
Records what was verified against the live boxes rather than inferred:
the default user really does hold ACCESS MANAGEMENT and CLUSTER ON *.*
WITH GRANT OPTION; ClickHouse publishes only to loopback there, so the
credential is not network-reachable; and proto and demo do not share a
password, so they rotate independently.
Notes that four compose services declare the credential, not the three
visible in `docker ps` -- clickhouse-migrate is a one-shot that has
already exited, and missing it in the override leaves the next
`docker compose up` failing on a stale password.
The note called this password "not a real secret ... purely to satisfy
this image's login gate for local/homelab use". That was accurate when
written, and stopped being accurate a phase later, in the same file:
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 was added directly below it for
Phase 4's per-tenant provisioning, which promoted the default user to a
ClickHouse superuser. `SHOW GRANTS FOR default` on a running instance
returns ACCESS MANAGEMENT and CLUSTER ON *.* WITH GRANT OPTION -- read
and write over every tenant's logs, plus the ability to mint more users.
Nothing about the deployment changes here; this is a comment-only edit.
It matters because the old wording actively told a reader the value was
safe to treat casually, which is how such a value ends up pasted into a
ticket or a screenshot.
Also records what the surrounding text did not: that rotation is not an
env-var edit, since the image's entrypoint consumes this only at
volume-init time, and which three services actually carry it.
Sweeps the references that carry no runtime coupling, and fixes one that
turned out to be a real bug rather than stale branding.
Docker network: sentry_default -> cairnobs_default across 23 runbook and
test-header `docker run` commands. Compose derives the network from the
directory name, so this lands together with renaming the working copy to
cairnobs/ -- the two are only correct as one change.
Stale references corrected: four Dockerfile "repo root (sentry/)"
headers; .env pointing at the long-renamed deploy/helm/sentry/ chart;
five Helm comments describing the topic as sentry.logs.raw when all four
code paths have defaulted to cairnobs.logs.raw for some time; an
absolute /home/john/Projects/sentry/ path in the operator's package doc,
now repo-relative; the hand-written Tenant CRD description in both of
its identical copies, whose Go source already said Cairn OBS.
Migration 0043 repoints the default tenant's data source. 0026 seeded it
with ('sentry', '/var/lib/sentry-search') to match what
api/internal/config then defaulted to; the rebrand later moved those
defaults to "cairnobs" and /var/lib/cairnobs-search without moving the
already-applied row, leaving the default tenant naming a ClickHouse
database nothing writes to. Scoped to the exact stale values so it is a
no-op on any deployment that set them deliberately. 0026's comment is
annotated as superseded; its applied SQL is untouched.
Deliberately not included: the gRPC wire packages (sentry.logs.v1,
sentry.agent.v1) and proto/sentry/ import paths, which cannot change
without a lockstep agent/server upgrade; the Helm chart's
sentry_metadata database and sentry role, which need a real Postgres
migration on existing deployments; and the compliance audit records in
docs/compliance/, which are a dated historical record.
go build, go vet, and go test pass for ingest and deploy/operator.
The rebrand renamed the Terraform provider binary to
terraform-provider-cairnobs (terraform/main.go, and the build command in
terraform/README.md), but .gitignore still listed the old
terraform-provider-sentry. The pattern no longer matched, so a locally
built provider binary would have shown up as untracked and could have
been committed by a `git add -A`.
Verified by building the provider and confirming git check-ignore now
matches it.
CLAUDE.md was doing two jobs: durable repo conventions, and a ~500-line
phase-by-phase status narrative that duplicates the per-phase runbooks
and goes stale the moment a phase ships.
Keep mission, constraints, pinned stack, conventions, and "when in doubt"
in CLAUDE.md (572 -> 78 lines). Move the phase record verbatim to
docs/status.md, prefaced with a summary table and the known verification
gaps. Content is byte-identical; nothing was reworded or dropped.
Link both directions, and point the README's status section at the new
file.
Overview, architecture, layout, local-dev quickstart, and phase status
for Cairn OBS. Records the Phase 4 verification gap and the unverified
Windows agent path rather than presenting them as shipped.
Swaps every brand asset the web UI imports for the v2 package: faceted
stones with per-stone gradients and an ambient glow, replacing v1's flat
four-colour paths. Raster favicons (16/32/48/180/512) are the package's
own renders at the sizes already referenced, so the layout head is
unchanged.
Two things the swap needed beyond copying files in:
- v2 ships no light-background stacked lockup, but the landing page picks
one on theme. Derived it from the stacked-dark file the same way the
package derives its own horizontal-light variant: same stone geometry
and gradients, glow dropped, wordmark on light-surface ink/accent.
- The stacked lockup is 360x320, no longer square; the landing page hard
set an 11rem square, which stretched the wordmark. Height is auto now.
The wordmark is outlined rather than left as live text. Upstream sets it
in JetBrains Mono, but these load via <img src> and are inlined as data
URIs, and an <img>-loaded SVG cannot reach the page's @font-face rules --
so neither JetBrains Mono nor our self-hosted Overpass Mono applied and
the wordmark rendered in each visitor's default monospace, changing shape
by platform. web/scripts/outline-wordmark.py converts every <text> to
paths set in Overpass Mono Bold, the font the app already ships, so the
lockup is pixel-stable and agrees with the UI's own type. Re-run it after
refreshing the package, which will ship live <text> again.
The four lockups now exceed Vite's 4KB inline threshold and ship as
separate hashed immutable assets instead of data URIs duplicated across
chunks. Unused package variants (icons, wordmarks, hero) are carried for
completeness and imported nowhere, so they are not bundled.
Two things that made reading logs harder than it needed to be.
Query input: the API accepts an absolute time only if it is quoted AND
carries an explicit offset, so someone reading logs in America/Denver
who wanted "9am today" had to convert to UTC in their head and remember
the quotes. Now a time typed without an offset is read as wall-clock
time in that reader's display timezone and converted to the instant it
names; anything with an explicit offset is taken at its word, and
relative ranges never depended on a zone.
This widens what's accepted rather than reinterpreting anything -- every
naive form now handled is one the parser rejects outright today, so no
query that works now can change meaning. The conversion happens before a
query is sent *or saved*: a stored dashboard range becomes an explicit
instant, because storing "2026-08-22 10:00" would mean 10am in whatever
zone each viewer sat in, and one shared dashboard would show two people
two different windows.
It also fixes two bugs that predate the timezone work. injectTimeRange
emitted absolute values unquoted, which the parser rejects -- so zooming
a time-series chart into a range, and clicking a chart to drill down,
both produced a syntax error on every panel. Both fed an ISO string
straight into that unquoted path.
Width: the query page's 64rem cap is gone, so the query bar and results
table use the whole window -- a log table is the widest thing in this
app and that cap was the horizontal scrolling. Prose keeps a readable
measure, since full-width paragraphs are harder to read, not easier.
Ambiguous local times -- the hour that repeats when clocks go back, the
hour skipped when they go forward -- resolve to one instant. That is
inherent to naming a moment by wall clock; an explicit offset sidesteps
it. Documented at the conversion.
Everything stays UTC: ingest still records Unix nanoseconds, ClickHouse
still stores UTC, every API response is still RFC3339 with a Z, and
queries are evaluated exactly as before. This changes only how those
instants are written on screen, so two people in two timezones looking
at one log line see the same instant written two ways -- never two
different lines, and never a different sort order.
Where the preference lives differs by deployment, and the three cases
are genuinely different products rather than one with fallbacks:
- Local login: server-side per named user (display_timezone on users,
PUT /auth/timezone), so it follows the person across browsers and
survives logout. Self-service at the RoleViewer floor, same as the
password change -- a viewer is the role most likely to be *only*
reading logs, so gating it higher would make it useless.
- Public demo: sessionStorage, so every new session starts at UTC. A
shared account's visitors have nothing to do with each other.
- Neither: localStorage, since there's no per-user record to write to.
api/cmd/api/main.go now imports time/tzdata. The image is
distroless/static with no /usr/share/zoneinfo, so LoadLocation would
otherwise reject every real zone name and the validation would refuse
every valid input.
Two details worth knowing when reading $lib/time.ts. Sub-second digits
are copied verbatim from the source string rather than round-tripped
through a JS Date, which is millisecond-precision and would silently
drop six digits of a ClickHouse nanosecond timestamp; expanding a result
row shows the localized value and the full-precision UTC original
together. And chart axes format their own labels, because ECharts'
type: 'time' axis renders in the browser's zone with no override --
which today puts a chart's clock out of step with the table beside it.
Timestamps are detected by value, not by column name: query output is
arbitrary, so a column called "timestamp" holding something else must
not be mangled, and `stats max(timestamp) as newest` must still be
formatted.
Verified against real zones including both sides of a DST boundary
(America/New_York at -05:00 in January, -04:00 in July), a half-hour
offset, and date rollover.
Two demo-only affordances, both off by default everywhere else.
The login form starts with the demo's read-only account already in both
fields, so a visitor doesn't need credentials handed to them out of
band. It's a build-time opt-in: the web image is built with
VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD, and the page prefills only when
it has both, so a deployment that sets neither -- every deployment
except the demo -- gets the ordinary empty form, and a half-configured
one can't leave a password next to an empty username box.
This does bake a password into a static bundle, which is fine for
exactly this case and nothing else: a Viewer-role account on a
deployment whose database is wiped and reseeded nightly. api.ts says so
next to the export, so nobody later points these at an account that can
do something.
The home page then explains what a visitor is actually looking at --
synthetic data from a simulated fleet, a nightly reset that discards
anything they change, and the features that are deliberately limited
(read-only account, alerts that notify a placeholder webhook, no
time-series charts). Gated on the same signal as the prefill rather than
a second flag that could drift out of sync with it.
Also carries the landing page's light/dark logo swap, which touches the
same file.
The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.
/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.
Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.
Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.
/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.
No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
Pushes synthetic cairnobs.metrics/cairnobs.heartbeat-tagged records
straight to ingest so a demo deployment's Hosts page and per-host detail
page have CPU/memory/disk to show. Neither benchmark-fixture nor
windows-fixture emits those two attributes -- only the real agent does
-- so nothing else could populate that page.
Superseded by /hack/demo-simulator in the next commit, which covers the
same records as part of a whole fleet; kept because it is still the
smallest thing that fills the Hosts page on its own.
`stats count by host` with no leading filter is documented as valid in
/docs/query-language-reference.md, but every pipe-stage keyword is also
a valid bare identifier, so the parser read it as a base_search of four
ANDed free-text terms ("stats", "count", "by", "host") -- matching
nothing, and returning an empty result rather than an error, which is
the worst of both outcomes for anyone typing it.
Recognizing a leading stage keyword up front and skipping straight to
stage-parsing fixes it with no planner or executor change: q.Base stays
its zero value, and compileBoolExpr already treats zero terms as
match-everything. The comparator lookahead keeps a genuine field named
`where`/`stats` (`where=foo`) parsing as a filter, as before.
The rename commit before this one covered module paths and the obvious
user-facing strings; this is the rest of it -- the places where "sentry"
was a default value, a filename, or a picture rather than a word in a
sentence.
Defaults that changed: CLICKHOUSE_DATABASE (sentry -> cairnobs),
POSTGRES_DATABASE (sentry_metadata -> cairnobs_metadata), and
POSTGRES_USERNAME (sentry -> cairnobs), across api/alerting/ingest and
the enterprise binaries, plus the compose files and migrate scripts that
create those objects. These are *defaults*, so a deployment that sets
them explicitly is unaffected -- but any deployment relying on the old
defaults must have its environment updated before it picks this up, or
it will come up pointing at a database that doesn't exist.
Also: the light-mode logo variants (the dark ones existed alone, so the
landing page and sidebar rendered a dark mark on a light background),
regenerated favicons, and the docs/README/threat-model prose that still
said Sentry.
Full rebrand across cosmetic branding, code identifiers, and
infrastructure/data-plane naming, using the supplied Cairn OBS logo
package. Cosmetic: favicon/logo swap (also closes a stale license-audit
finding -- the old favicon was SvelteKit's unreplaced scaffold logo),
new centered welcome landing page, larger/legible sidebar logo, page
titles, CLAUDE.md/README/docs prose.
Code identifiers: Go module path github.com/sentry/sentry ->
github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc
regenerated); Rust crates sentry-agent/sentry-parser/sentry-search ->
cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully
renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type,
env vars); every session/auth cookie name; agent config paths and
Windows service identity.
Deliberately preserved: the gRPC wire protocol's protobuf packages
(sentry.logs.v1, sentry.agent.v1) and their Go import directory
(proto/sentry/...) -- renaming the wire-level package would break every
currently-deployed agent binary (confirmed two real hosts, including
mail.inbuxa.com, are actively streaming through this exact contract)
until rebuilt and redeployed in lockstep with an ingest cutover. Only
the Go module path wrapping the generated code changes.
Infrastructure: every docker-compose container name (root and three
component-level compose files); the Helm chart (directory, Chart.yaml,
named-template helpers, all templates, values.yaml image repos);
Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML
files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd
package. Caught and fixed real path-coupling bugs along the way: the
Helm chart's search/ingest volume mounts and the dev-only-credential
detection constant vs. docker-compose.yml's literal values had to move
together or a security warning would have silently stopped firing.
Data plane: Postgres database sentry_metadata -> cairnobs_metadata and
role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka
topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups.
Source-level defaults, docker-compose.yml, and every migrate.sh/
provision script default updated together; already-applied migration
files left untouched per this repo's immutable-migration convention.
Verified at every layer: all 13 Go modules build/vet/test clean, both
Rust workspaces (agent, search) build/clippy/test clean, npm run check/
build clean, docker compose config validates on all four compose files.
Live-verified against a real docker stack multiple times through this
work, including a final fresh-volume run confirming the actual renamed
Postgres database/role, ClickHouse database, and Kafka topic all work
end to end with a real login and query, zero console errors.
Root cause: handleHosts and partitionTargets both declared their result
slices with `var`, so an empty result (no logs old enough yet, or every
requested target blocked by a floor) marshaled to JSON `null` instead of
`[]` on fields without `omitempty`. The frontend's `.length` access on
that `null` threw mid-render, which is why this shipped with the spinner
stuck forever instead of the empty state ever painting -- production is
freshly deployed with nothing yet older than the default 30-day cutoff,
so every user hit this on first load.
Also replaces the static "Loading hosts…" text with the existing
shimmer Skeleton component for real visual feedback, and adds `?? []`
fallbacks in api.ts as a second line of defense.
api/localauth now enforces every rule of the requested matrix, each
checked inside the handler beyond RegisterRoutes' floor:
- At least one owner must always exist -- handleDeleteUser and
handleSetRole both refuse an operation that would leave zero
owners (wouldRemoveLastOwner, backed by new store method
CountUsersWithRole), whether the caller is admin or owner.
- Owner can create/delete any role, including another owner (subject
to the above). Admin can only create/delete viewer or editor --
GET/POST /auth/users and DELETE .../{id} moved from RoleOwner to
RoleAdmin floor, with an inner check narrowing what an admin
caller specifically may target.
- Only a user can change their own password -- new POST
/auth/password (RoleViewer floor, i.e. every role) requires the
caller's current password (verified via new store method
GetPasswordHashByID) and is now the only path to changing your
own, including for an owner. The existing admin-reset endpoint
(POST /auth/users/{id}/reset-password, also moved to RoleAdmin
floor) now refuses id == the caller's own ID, and refuses an
owner target unless the caller is themselves an owner -- "admin
can change any password except an owner's; owner can change any
password, even another owner's."
- Role reassignment (PUT .../{id}/role) stays owner-only, unchanged
beyond the last-owner guard above.
New web/src/routes/account page (linked from NavSidebar next to "Log
out", visible to every local-auth role) is the self-service password
change UI. /users now mirrors the server's per-row restrictions
client-side (disabled role selects/delete/reset buttons with an
explanatory title, a restricted role list on the create form) so an
admin never sees an action that would just 403 -- the server remains
the actual authority.
Verified live against real Postgres and in the browser: the full
matrix via curl (owner creating a second owner, admin blocked from
creating/deleting/resetting admin or owner accounts, last-owner delete
and demote both blocked, admin resetting non-owner passwords,
self-target reset rejected, self-service change with wrong/right
current password), plus the actual /users page rendering correctly
restricted for an admin session and a full change-password round trip
through the real UI ending in a forced re-login with the new password.
logs rows carry a real per-record `service` (nginx, smtp, ufw, ...) --
already true of the schema (storage/migrations/0001) and wire protocol,
not something this feature invents. Both the deletion picker and the
retention floor now operate on (host, service) pairs instead of whole
hosts, so an operator can delete just one noisy log type from an agent
without touching everything else it ships, and can protect one service
(e.g. keep smtp a year) longer than the rest of that host's default.
api/agents.ConfigOverride gains ServiceLogRetentionDays (map[string]int),
owner-only to change like LogRetentionDays -- a service listed there
overrides the host's LogRetentionDays default for that service only.
Agent config page gets a matching "Per-service log retention overrides"
add/remove list next to the existing host-level field.
api/logretention: Store's count/delete now take []HostService and build
a ClickHouse tuple IN ((?,?),...) over (host, service); AgentRetentionStore.
FloorsByHost returns each host's default plus its per-service map, with
HostFloor.Effective(service) resolving which one applies. preview/delete
moved from GET/DELETE-with-query-params to POST-with-JSON-body (a list of
targets needs a real body, not a repeated compound query param), and
partitionTargets checks the floor per target so one protected service
never blocks deleting a different, unprotected one in the same request.
Settings' Log retention section is a two-level picker now: each host
row (with a "select all services" checkbox and its default floor badge)
expands to its services, each with its own count and effective
protected-days badge.
Verified live against real ClickHouse/Postgres and in-browser: a host
with a 7-day default plus a 365-day smtp override -- deleting nginx+
smtp+ufw together correctly removed nginx and ufw, left smtp's 10
records untouched, and confirmed via a follow-up owner delete that
bypassing the floor works. Also verified the full click-through (add a
service override on the agent page, see it reflected in Settings'
picker, select/preview/cancel) and confirmed no regression from the
prior host-only version's tests.
api/logretention no longer deletes wholesale by age alone: a new
GET /logs/retention/hosts lists every host with matching records (plus
any configured retention floor), and preview/delete now require an
explicit, non-empty host list -- there is no "omitted host means every
host" shortcut server-side. Store's count/delete statements are
host-scoped (host IN (...)); Handler.partitionHosts checks the floor
per host instead of one global max, so a floor on one host never
blocks acting on other hosts requested in the same call. A request
that ends up fully or partially blocked still returns 200 with
blocked_hosts explaining why, rather than rejecting the whole call.
Settings' Log retention section is a host picker now: checkboxes with
per-host counts and a "protected Nd" badge where a floor applies,
"select all/none", and a confirm panel that names exactly which hosts
will be affected and which were skipped and why.
Verified live against real ClickHouse/Postgres and in-browser: three
hosts seeded, one protected by a 90-day floor -- a scoped delete
correctly removed the two open hosts' records, left the protected
host's untouched, and the response/UI both named it as skipped. Also
fixed a real spacing bug in the result message caught during that
browser pass (an adjacent {expr}{#if} with no source whitespace
between them rendered with no space either).
api/agents.ConfigOverride gains LogRetentionDays: a per-agent setting
edited on the same remote-config page as extra_file_paths, but unlike
every other field there it's central-policy metadata api/logretention
reads, never something the agent process itself sees. Any change to
it -- setting, raising, lowering, or clearing -- requires RoleOwner,
not just RoleAdmin: the whole point of the field is a floor an admin
can't move, so an admin able to freely edit it would defeat that.
api/logretention now checks the largest LogRetentionDays configured
across any agent (AgentRetentionStore, new) before every preview/delete:
a non-owner's request is rejected with a clear 403 if it would reach
into that protected window. An owner always bypasses it, matching "make
the log retention override any attempts to delete logs by anyone other
than owner role."
Verified live end-to-end: owner sets a 90-day floor on an agent, admin
is blocked deleting anything newer than that (both preview and delete),
allowed beyond it, and owner bypasses it entirely -- confirmed against
real ClickHouse data, not just the fake-backed unit tests. Also caught
and fixed a real pre-existing latent bug while verifying in-browser: a
type="number" Input's bind:value becomes an actual JS number once a
user types into it (only the initial value is a string), which broke a
bare .trim() call on the new field.
New api/logretention package: GET /logs/retention/preview and
DELETE /logs/retention, both gated to RoleAdmin (Owner satisfies it
too), issue purpose-built parameterized statements against ClickHouse's
logs table (a count and a synchronous ALTER TABLE ... DELETE mutation)
rather than routing through querylang/executor's SELECT-only SQLRunner.
Settings gets a new "Log retention" section, visible only to an owner
or admin, that previews how many records a chosen age cutoff would
remove before showing an explicit confirm/cancel panel -- no delete
happens without that second step.
Scoped to core's single-tenant ClickHouse table; enterprise/'s
per-tenant routing and Tantivy's lack of a bulk-delete primitive are
disclosed gaps in api/logretention/store.go's doc comment, not silently
assumed to already work.
Move user management out of Settings into its own /users page (nav-gated
to owners), let an owner type a specific password on reset instead of
always generating a random one, and add role reassignment via a new
PUT /auth/users/{id}/role endpoint. Role changes revoke the target's
existing sessions, same as a password reset, so a demoted user can't
keep acting under a stale, higher-privileged session.
This is a large squashed commit covering two batches of prior uncommitted
work plus a full security-audit remediation pass, kept together because
go.mod/go.sum and several shared files (main.go, handler.go) were touched
by both and splitting risked non-building intermediate commits.
Features (built earlier, previously uncommitted):
- Local username/password login for single-tenant deployments with no
SSO configured (api/localauth, alerting/internal/sessioncheck,
sentryctl users, web/src/routes/login, metadata migrations 0040/0041).
- Remotely-editable additional log file paths for agents, on top of
their existing primary source (api/agents, agent/sentry-agent
extra-file-path diffing, web agent config UI).
- IPv4/IPv6 addresses reported alongside other host system metrics.
Security audit remediation (this pass, all live-verified in production):
- Critical: block ClickHouse SSRF table functions (url/remote/file/s3/...)
in the raw-SQL query escape hatch.
- High: deny sensitive paths and require Admin to add agent
extra_file_paths (Editor could previously point an agent at /etc/shadow
or an SSH key); alerting webhook targets now validate against
internal/metadata/loopback addresses, both at creation and send time;
alerting's session middleware now enforces an Editor+ floor on
mutating requests instead of "any authenticated session"; bumped
goxmldsig to close a SAML signature-verification bypass (GO-2026-4753).
- Medium: per-IP login rate limiting; security response headers
(HSTS/CSP/nosniff/X-Frame-Options/Referrer-Policy/Permissions-Policy)
on web/nginx.conf; a DevCredentialWarnings check in every Go service's
config loader, logging loudly at startup if a deployment is still on
docker-compose.yml's literal dev-only credentials; dependency bumps
(golang.org/x/text, grpc, x/net, quick-xml, h2) across every affected
Go module and both Rust crates, including a previously-uncovered x/net
vulnerability in deploy/operator; a new security-scan.yml CI workflow
running cargo-deny/govulncheck/npm-audit, mirroring the existing
license-compliance.yml matrix shape.
- Low: removed sentryctl's plaintext --password flag (shell
history/`ps` exposure) in favor of stdin and a --password-stdin flag
for reset-password's optional specific-password path; a dummy bcrypt
comparison closes a login response-time username-enumeration
side-channel.
sentryctl agents list|get, config get|set|clear, restart -- same
list/get shape as dashboards/alerts, plus a config sub-subcommand
mirroring dashboards' permissions since an override has its own
lifecycle distinct from the agent resource itself.
config set is the one command with real logic: since PUT
/agents/{host}/config replaces the whole stored override rather than
patching individual fields, it fetches the agent's current effective
config first and merges only the flags actually passed on top of it,
mirroring the web UI's edit form logic in Go. restart requires
explicit confirmation (interactive y/N or --yes) and refuses on
non-interactive stdin without --yes, the same posture cmd_query.go's
--nl/--execute already established for anything that changes what's
running.
Live-verified the merge logic specifically, since it's the part most
likely to hide a real bug: setting one field on a clean agent
correctly carried forward its other reported values, and a second
config set call correctly carried forward the first call's override
rather than resetting it to the reported baseline. config clear and
restart --yes both round-tripped against a live agent, with the
restart picked up and acted on within one check-in cycle.
This closes out the agent-management punch list (restart, fleet-wide
alerting, this CLI surface) -- see /docs/agent-management-design.md.
No code changes -- alert rules already accepted query_language: "sql"
with zero validation restricting it to the pipe syntax, and the web
UI's rule-creation form already auto-detects SQL vs. pipe syntax via
the shared QueryBar component. This was simply never exercised in
this specific combination before.
Live-verified with three agents: a threshold rule on
count(DISTINCT host) against an expected fleet size correctly
evaluated ok with all three healthy, then correctly fired when one
was killed and its heartbeat rows aged out of the window -- one rule
covering a whole named group of hosts instead of one rule per host.
Documented as an honest aggregate check, not true per-host alerting:
that would need the alerting engine's own per-group state tracking,
already named a Phase 3 non-goal for the whole engine, not something
specific to agents -- explicitly out of scope here rather than
quietly built as a side effect.
Extends the existing CheckIn RPC with a one-shot AgentCommand
(restart only -- stop/uninstall need real per-platform OS
service-manager integration and stay deliberately out of scope),
delivered at-most-once: cleared the instant it's handed to the agent
in a response, since a restarting agent's process is gone before it
could ever confirm receipt. On restart, the agent flushes whatever's
buffered, aborts its source task, and exits cleanly, relying entirely
on the host's own service manager to bring it back up.
Issuing a command is gated at RoleAdmin (stricter than config
editing's RoleEditor) and logged into the same audit_log table Phase
7's AI interactions use, via a new agent_command event type.
A real bug was found and fixed during live verification: the first
implementation tried to atomically read-and-clear pending_command in
a single INSERT...ON CONFLICT statement using a sibling CTE
referenced only from RETURNING, on the assumption that Postgres
evaluates every part of a WITH query against one pre-statement
snapshot. That's wrong specifically for FOR UPDATE, which always
reads the latest row version including one written earlier in the
same statement -- confirmed empirically (a restart command was
always coming back empty even when genuinely pending, so the agent
never received it). Fixed by splitting into two real, ordered
statements inside one explicit transaction.
See /docs/agent-management-design.md's "Lifecycle commands" section.
-time-spread spreads generated records' timestamps uniformly at random
across [now-spread, now] instead of all landing at ~now, for building a
demo/exploration dataset with a real time axis (0, the default,
preserves the original all-at-now volume-benchmark behavior).
-include-fatal adds a low-frequency FATAL severity to the mix, off by
default so the volume benchmark's existing severity distribution is
unchanged unless asked for.
Extends the heartbeat mechanism with a second gRPC service on the same
mTLS channel (AgentControl.CheckIn, agent-initiated on the existing
heartbeat ticker -- still push-only, no inbound port on any agent) so
an agent reports its running config and can pick up an operator-set
override. A new web UI section (/agents) lists every agent that's
checked in, shows its reported config, and lets an operator edit a
narrow, deliberately-scoped subset remotely: batch/heartbeat tuning,
and (journald sources only) the unit filter.
TLS material and the ingest endpoint are never reportable or remotely
editable, by proto shape rather than a validation rule -- a bad or
malicious edit there could permanently strand an agent or redirect
where its logs go, unlike every other editable field, which only
degrades behavior.
An override lives only in the agent's memory (agent.toml is never
rewritten) and re-syncs on the agent's own schedule; changing the
journald filter aborts and respawns the source task since there's no
other way to change what's being tailed. Building the hot-reload path
surfaced a real, independent, pre-existing bug: shutdown was using
poll_timeout(), which only drains once flush_interval has elapsed,
silently dropping anything buffered more recently on every graceful
shutdown that landed between flushes -- fixed with a new unconditional
Batcher::flush_all(), now used at both shutdown and hot-reload.
Verified live end-to-end against a real stack: an edited heartbeat
interval changed a running agent's actual send cadence within one
check-in cycle (confirmed by the real timestamps landing in
ClickHouse), and an edited journald filter triggered a real source
restart, both reflected back in the next reported-config snapshot.
See /docs/agent-management-design.md.
Agents now send an independent "still alive" record on a configurable
schedule (seconds/minutes/hours, [heartbeat] in agent.toml), separate
from real log traffic and tagged with a sentry.heartbeat attribute.
No new wire protocol -- it's an ordinary record through the same
PushBatch RPC/mTLS identity every log line already uses. Unavailability
alerting reuses the existing absence-condition alert rule type
unchanged; no new alerting code was needed. See
/docs/agent-heartbeat-monitoring.md for the design and how to build the
alert rule.
While verifying the alert rule live, found that the query language's
lexer never treated '-' as part of an identifier, so any unquoted
hyphenated filter value -- including the reference doc's own canonical
example, `host!=host-03` -- failed to parse at all. Fixed in
api/internal/querylang/lexer/lexer.go with regression tests; a leading
'-' still lexes as its own token so earliest=-1h/sort -count are
unaffected.
Adds a self-hosted (Ollama, qwen2.5-coder) model provider abstraction
with a pluggable opt-in cloud adapter, schema grounding, and a shared
cost/safety guard every AI-suggested query is assessed against --
compiling to and executing through the same unchanged Phase 2 IR/
compiler and Phase 4 tenant scoping as a hand-written query, no
parallel execution path.
Track A (built into the query bar): inline ghost-text autocomplete,
"Explain this query", "Fix this query" with a diff view, and a
rule-based "Optimize" suggestion. Track B: natural-language-to-query
translation, always a separate review step from execution, with
`sentryctl query --nl` requiring explicit confirmation to run.
Every accepted/dismissed translate-fix-optimize interaction is logged
into the same append-only audit_log table Phase 4 built.
Two real product bugs were found and fixed via live browser
verification (a Svelte effect re-running on every keystroke that
silently cancelled the ghost-text debounce; a ghost-text widget
positioned at document offset 0 instead of the cursor), and a real
costguard logic bug (unbounded-aggregation vs. raw-row) was caught by
its own test suite. New integration tests wire a real Ollama client
through the real HTTP handler against a mock server matching Ollama's
wire contract (hack/mock-ollama), keeping model-quality verification
out of CI as a disclosed, periodic human-run check instead.
See /docs/phase-7-ai-design.md and /docs/phase-7-runbook.md.
Full dependency inventory across Rust/Go/npm plus Docker base images
and vendored assets (776 rows, 502 unique deps), classified against
AGPLv3 compatibility with real citations rather than assumptions.
enterprise/ relicensed from its commercial-license stub to AGPLv3,
matching core -- the one real flag (Redpanda's BSL 1.1) was evaluated
against primary sources and accepted as-is rather than triggering a
broker swap. CI enforcement wired up (.github/workflows/license-
compliance.yml, this repo's first CI workflow), a root LICENSE file
added, and every doc/comment referencing the old commercial-license
boundary updated to describe it as architectural only.
See /docs/compliance/ for the full report, inventory, and policy.
design-system.md was still describing a mid-Phase-5 state (charting,
dashboard panels, query/search, and alerting UI all listed as "not
built yet"); added sections for all of them plus a real Accessibility
section, and fixed color-token values that had drifted from the actual
tokens.css since the contrast fixes.
phase-5-runbook.md documents what was actually verified against a live
docker-compose stack with real seeded data, including the five real
bugs that live-verification caught -- two of them backend bugs with no
connection to the frontend redesign, only surfaced because getting
real dashboard/alert data required exercising write paths nothing had
exercised since Phase 4's tenant_id migrations landed.
CLAUDE.md gets Phase 5's exit criteria, matching every prior phase's
"what done looks like" section.
Phase 4 added a NOT NULL tenant_id column to both tables (migrations
0022/0023, backfilled via a join through alert_rules.id), but Store's
Create and ApplyTransition were never updated to populate it on new
inserts -- every existing row already had a value from the backfill,
which is exactly why this went uncaught: nothing created a *new* rule
against a Phase-4-or-later database until now. Every alert rule
creation since those migrations landed was silently broken.
Create's alert_state insert now passes rule.TenantID explicitly.
ApplyTransition only receives a rule ID, not a full Rule, so its
delivery_log insert resolves tenant_id via a subquery against
alert_rules. Confirmed against a live stack: rule creation, evaluation,
firing, and a real delivery attempt all completed end to end.
AlertStatePill reuses the log-severity color tiers instead of a second
color vocabulary: ok -> quiet, pending -> warn, firing -> critical.
DeliveryTimeline reframes the existing delivery_log data (no new
backend fields) as a vertical timeline -- "why didn't I get paged" is a
chronological question a flat table answered less directly. Rules list
sorts firing-first. The list table's trailing actions column had a
bare empty <th>, which axe-core flags (empty-table-header) -- fixed
with visually-hidden text via app.css's shared .sr-only utility.
QueryEditor.svelte wraps CodeMirror 6, not a hand-rolled
textarea-plus-overlay highlighter -- autocomplete needs real
cursor-aware popup positioning a plain textarea can't give. language.ts
is a StreamLanguage tokenizer for the pipe grammar; its token() function
must return real @lezer/highlight tag names looked up by string
('controlKeyword', 'operatorKeyword', 'name.function' for tag+modifier
pairs) -- a custom Tag.define() looks plausible but silently highlights
nothing. completions.ts is context-aware: stage keywords after `|`,
stats functions after `stats`, field names elsewhere.
A two-way-binding race between the editor's updateListener and an
external-sync $effect could drop characters on rapid/bulk input --
fixed with a lastEmitted guard so the sync effect only reacts to
genuinely external value changes, not its own echoes.
ResultsTable gets sortable columns (a real <button> in the <th>, so
sorting is keyboard-operable for free), resizable columns
(pointer-drag, deliberately mouse-only -- the resize handle stays out
of the tab order, same as most apps treat column resize), and
expandable rows. The row-expand affordance was originally a bare `<tr
onclick>` with no keyboard equivalent at all; fixed with
tabindex/role="button"/aria-expanded and an Enter/Space handler.
AddToDashboardModal lets a query built on the Search page become a
saved panel without hand-copying the query string.
Drag-and-drop grid stays on GridStack (already a Phase 3 dependency --
no new library needed). PanelEditor.svelte (a Modal) replaces the old
inline add-panel form: a debounced live preview reuses PanelViz
directly, so the preview is pixel-identical to what renders on save
instead of drifting from a separate preview renderer. Dashboards list
and detail pages get EmptyState/Skeleton for empty/loading states
instead of a blank panel or a raw error string, and panel titles are
now clickable buttons that open the editor.
Five chart types on ECharts (modular imports, not the full bundle):
TimeSeriesChart (multi-series, legend toggle), BarChart (incl.
stacked), SingleStat (big number + sparkline + trend), Heatmap, TopN.
Shared interactions: tooltips, dataZoom feeding the global time-range
picker, click-to-drill-into-query (drilldown.ts strips a panel's query
to its pre-stats filter and appends the clicked series/x-value as a
new filter term -- no backend change needed).
pivot.ts reshapes the query language's existing {columns, rows} tabular
output into per-series chart data client-side -- `stats count by
service, timestamp` already returns "long" rows, so multi-series
support needed zero query-language changes. theme.ts reads real
computed CSS custom properties so charts render in the active theme's
actual colors, with an SSR_FALLBACK for adapter-static's prerender pass
where `document` doesn't exist.
heatmap is the one narrow, justified backend change: a new VizType
needed to feed a new visualization, not a new query capability. Three
places had to change together, not two -- api/dashboards/types.go's
validator, web/src/lib/api.ts's union (previous commit), and the
dashboard_panels table's viz_type CHECK constraint
(migrations/0035_add_heatmap_viz_type.sql), which mirrors the Go
validator and doesn't update itself.
/dev/charts (unlisted, dev-only) is a synthetic fixture/perf-test route:
confirmed 50ms first-two-frames render time on a production build
against a 30,006-row/6-series stress case, and a 211,975-byte gzipped
chart chunk -- both real measurements behind the ECharts-over-
Observable-Plot-or-D3 choice, not estimates.
Button, Input, Select, Badge, SeverityBadge, Table, Card, Modal,
Tooltip, Tabs, Skeleton, EmptyState -- a shared library so pages stop
hand-rolling markup per page (web/src/lib/components/ui, barrel export
in index.ts). Modal and CommandPalette are built on native <dialog>
for a real focus trap, Escape-to-close, and top-layer stacking instead
of hand-rolling those. Tabs uses roving tabindex with arrow-key nav.
NavSidebar replaces the old top-nav with a persistent sidebar
(Search/Dashboards/Alerts/Data Sources/Settings), a live tenant
indicator (api.ts's new getCurrentSession(), a client for the
already-existing POST /internal/authorize -- zero new backend surface),
theme/density quick-toggles, and a command-palette hint. Collapses to
an off-canvas drawer under 860px. CommandPalette (Cmd/Ctrl+K) indexes
the five static destinations plus live-fetched dashboards/alert rules.
Data Sources is a new, honestly-scoped placeholder page (one data
source per tenant today, no UI needed yet). Settings and Select-tenant
are re-tokened onto the new component library. +layout.svelte's content
wrapper is a plain <div>, not a second <main> -- every page already
renders its own top-level <main>.
Self-hosted variable fonts (Overpass/Overpass Mono, OFL-licensed) --
no CDN dependency for the app to render correctly. Dark is the literal
default in tokens.css (:root defines it directly; light is the
override via both prefers-color-scheme and an explicit data-theme),
not a retrofit. Severity tokens collapse OTel's seven severities to
five visual tiers (severity.ts); their translucent -bg variants and
light-mode warn's base color are already tuned for WCAG AA contrast
against an opaque surface, not just the plain page background --
verified with real axe-core runs during the accessibility pass, see
docs/design-system.md.
theme.svelte.ts/density.svelte.ts persist to localStorage and expose
getter/setter functions wrapping module-level $state (Svelte 5's
shared-state-module pattern -- a directly exported $state doesn't
preserve reactivity across modules). app.html's inline script applies
both before first paint to avoid a flash of the wrong theme/density.
ECharts backs the new charting layer, the @codemirror packages back
the query editor's syntax highlighting and autocomplete, and axe-core
(dev-only) drives the automated accessibility sweep. Drops uplot, which
the chart rebuild replaces entirely.
kind/kubectl/helm were installed without root and a real local cluster
ran the full two-tenant walkthrough end to end: both acme and globex
reached Tenant.status.phase: Active with real generated ClickHouse
credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's
verification-status framing from "not verified against a live cluster"
to what's actually true now, fixes both READMEs' helm install
--include-crds (a helm template-only flag, never valid for install),
documents the two chart bugs this run found (see the previous commit)
and the mandatory ingest TLS Secret step, and updates the threat model's
summary table and "No general multi-cluster orchestration" residual-risk
note accordingly. This closes the last remaining gap in
/docs/phase-4-runbook.md.
templates/enterprise-auth.yaml never set POSTGRES_ADDR/DATABASE/
USERNAME/PASSWORD at all -- enterprise-auth silently fell back to its
localhost:5432 default and could never actually reach Postgres,
crash-looping forever. Fixed to match api.yaml's existing pattern
(Service DNS name + Secret-sourced password), plus a wait-for-postgres
initContainer for the same startup-ordering reason api.yaml has one.
templates/clickhouse.yaml was missing CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
-- same real bug docker-compose.yml had, now fixed there too: the
official image's default user lacks CREATE USER privilege without it,
so tenantprovision's -provision-tenant could never actually provision a
tenant through this chart.
Neither of these had ever been caught before because this chart had
never been installed against a real cluster -- both surfaced and were
fixed running the full "Trying the two-tenant example" walkthrough
against a real kind cluster, ending with both tenants reaching
status.phase: Active and real generated ClickHouse credentials in their
Secrets, closing /docs/phase-4-runbook.md's last remaining gap.
Auth0's SAML2 Web App addon (the same dev tenant §3a used) stood in as
a real SAML IdP, over a genuine self-signed TLS proxy in front of
enterprise-auth (required, not optional, for SAML's SameSite=None
cookie). Full round trip confirmed: real signed assertion, audience/
destination/signature validation, correct multi-membership handling,
and POST /internal/authorize returning the selected tenant/role.
Updates the runbook's verification status, §3b, and the threat model's
"Read this first" finding and summary table to reflect this and the
isSecureRequest fix it found. §7/§11's live-cluster steps (no
kind/kubectl in this environment) are now the only remaining gap in the
entire runbook.
Every auth cookie loginhandler.go sets (OIDC state, SAML request,
pending-login, session) decided Secure from r.TLS != nil alone --
correct only if enterprise-auth terminates TLS itself, which it never
does (it's a plain http.Server, same as every other service here). In
any real deployment, TLS is terminated at a reverse proxy/ingress in
front of it, so r.TLS is nil at this process even over a genuinely
HTTPS client connection.
Found live: SAML's request-tracking cookie is SameSite=None (required,
since the ACS POST is cross-site from the IdP's origin), which the
cookie spec requires to be paired with Secure. Behind a real
TLS-terminating nginx proxy, the cookie came back without Secure and
Chrome silently dropped it -- breaking the SAML login flow entirely, not
just weakening it. Fixed with isSecureRequest(r), which also checks
X-Forwarded-Proto: https -- not a new trust boundary, since this handler
already assumes it sits behind exactly this kind of proxy, never
directly internet-facing.
A free Auth0 developer tenant was wired into enterprise-auth via a
local-only docker-compose.override.yml and driven through a real
browser: login correctly failed closed with no tenant membership while
still creating the users row, then succeeded after -grant-membership-*
and issued a real session. With a second real membership granted, the
multi-membership path landed on the real /select-tenant page, rendered
both real tenants with correct roles via a real credentialed
cross-origin request to the real enterprise-auth container, and
selecting either one issued a session that POST /internal/authorize
confirmed matched. Updates the runbook's top-level verification status,
§3a, and §12, plus the threat model's "Read this first" finding and
summary table to reflect what's now genuinely confirmed versus what
still needs SAML's real IdP (§3b) or a real cluster (§7/§11).
Only VITE_API_BASE_URL had a matching ARG/ENV pair; docker-compose.yml's
build args for VITE_ALERTING_API_BASE_URL and
VITE_ENTERPRISE_AUTH_BASE_URL were silently dropped by Docker (an
undeclared --build-arg is dropped, not an error). enterpriseAuthBase
came out undefined in the built bundle, so the tenant-picker page threw
"enterprise-auth is not configured" against a real running container
even though docker-compose.yml looked correct. The other two vars masked
this because web/src/lib/api.ts's apiBase/alertingBase both have
hardcoded fallbacks that happen to match the intended values.
Found while wiring a real Auth0 developer tenant into enterprise-auth to
close §3a/§12's remaining "real external IdP" gap. Also gitignores
docker-compose.override.yml, since that's where such real credentials
belong for local testing -- never committed.
Docker access became available and §§1-10, 10a, 13, 14, and most of §8
of the runbook have now genuinely been run against a real docker-
compose stack, not just documented as a procedure to run. Rewrites the
runbook's "Verification status" section and the threat model's "Read
this first" finding to describe what was actually confirmed (including
the six real bugs this pass found and fixed) versus what's still gated
on a real external IdP (§3a/§3b/§12) or a real Kubernetes cluster
(§7/§11's live-cluster halves). Also fixes two stale runbook commands:
§5a/§6's docker run mounts needed the repo root, not just enterprise/,
for the same replace-directive reason the Dockerfile fix does; §14 cited
a chwriter test name that never existed.
main.go registered GET /healthz explicitly, on top of the one
queryapi.Handler.RegisterRoutes already registers -- net/http's
ServeMux panics on a duplicate pattern, so enterprise-api could never
actually start. Every previous "built" claim for this binary had only
ever been a successful go build, never a successful process start;
caught the first time this ran against a real docker-compose stack.
SetDataSourceClickHouseCredentials let a non-UUID id leak Postgres's
raw 22P02 (invalid_text_representation) error past the store's
ErrNotFound boundary. A malformed id can never match a row either way,
so it should be treated the same as "no such row" rather than exposing
a database-internal error past this package's boundary. Found via a
live Postgres integration test.
CreateDashboard's inline panel-creation path already called
validatePanel before insert; AddPanel and UpdatePanel relied on the
HTTP handler to validate first instead of enforcing it themselves.
Found via a live Postgres integration test: calling store.AddPanel
directly (bypassing the handler) hit a viz_config NOT NULL constraint
violation instead of getting the same default-empty-JSON treatment
every other panel-creation path gets.
Verified live against clickhouse/clickhouse-server:24.8: a freshly
created tenant user was NOT default-denied from system.* the way the
original design assumed -- system.tables listed every tenant's
database/table names to any authenticated user regardless of grants.
Fixed with an explicit REVOKE SELECT ON system.* FROM <user> after the
existing GRANT SELECT, INSERT.
Verifying this live also surfaced a real ClickHouse behavioral split
the design didn't anticipate: system.query_log is genuinely access-
checked (the REVOKE makes it hard-deny, ACCESS_DENIED), but
system.tables is a filtered catalog view that ClickHouse 24.8 never
denies outright -- it just returns zero rows for a properly-revoked
user. Both outcomes close the actual leak. Corrected
TestProvisionedUserCannotReadSystemTables to assert what each table
actually does (hard error for query_log, verified-empty-and-no-foreign-
database-names for tables) instead of demanding a hard error from both.
jcoffey-dev