Coffey Labs
252c207ccf
The status file and the README both still said Phase 4 was not shipped because the environment had lost Docker and database access partway through, and that only the audit-logging guarantees had been confirmed against a live database. That stopped being true some time ago. phase-4-runbook.md records the opposite in detail: Docker access came back, a real docker-compose stack ran with real ClickHouse and Postgres and two provisioned tenants, a local kind cluster ran the Helm chart end to end, and both SSO protocols were verified against a real Auth0 tenant with full browser round trips. Eight real bugs came out of that, six from compose and two from the chart's first real install -- none of them findable without the infrastructure. PROJECT-SPEC.md sends readers to status.md and tells them to read it before assuming a capability works end to end, so the one file that is meant to be authoritative was the one understating the project by the widest margin. Correcting it matters more now than it would have last week, because the evidence cannot be regenerated: proto.cairnobs.org and the VPS under it were retired on 2026-09-04, taking the mTLS CA, the server certificate and six enrolled agents with them. The runbooks are what is left. Three gaps are now stated rather than implied: The prototype is gone, so none of this can be re-run today without building one. The DNS was kept for that; the certificates deliberately were not. demo.cairnobs.org is live and is not evidence for Phase 4. It runs COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy demo proving multi-tenancy is exactly the wrong inference to leave available. SSO has been tried against one IdP and one local kind cluster, not two IdPs and not a production-grade cluster. The Terraform entry now names its cause instead of pointing at another file: alerting exposes no PUT for rules or targets, and neither rulestore.Store nor notifystore.Store has an Update method to wire one to, so Terraform destroys and recreates -- which resets alert_state and delivery-log continuity.
178 lines
7.2 KiB
Markdown
178 lines
7.2 KiB
Markdown
<p align="center">
|
|
<picture>
|
|
<source media="(prefers-color-scheme: dark)"
|
|
srcset="web/src/lib/assets/logo-horizontal-dark.svg">
|
|
<img src="web/src/lib/assets/logo-horizontal-light.svg"
|
|
alt="Cairn OBS" width="420">
|
|
</picture>
|
|
</p>
|
|
|
|
<p align="center">
|
|
Open-core, Kubernetes-native log aggregation and observability.<br>
|
|
Built to match Splunk on capability while winning on cost-per-GB,<br>
|
|
with honest multi-tenant RBAC and a modern language stack.
|
|
</p>
|
|
|
|
<p align="center">
|
|
Licensed <strong>AGPLv3 in its entirety</strong> — including
|
|
<code>enterprise/</code>. See <a href="#licensing">Licensing</a>.
|
|
</p>
|
|
|
|
## What it does
|
|
|
|
Logs flow from a statically-linked Rust edge agent through Redpanda into a Go
|
|
ingest pipeline, landing in ClickHouse for analytics and Tantivy for full-text
|
|
search. One query language spans both stores, compiling to a single execution
|
|
plan:
|
|
|
|
```
|
|
service=api | where status>=500 | stats count by host | sort -count
|
|
message:"connection refused" | stats count by host
|
|
```
|
|
|
|
Raw ClickHouse SQL stays available as an escape hatch and compiles to the same
|
|
IR, so performance doesn't depend on which syntax you write.
|
|
|
|
On top of that sit dashboards, an alerting evaluator with threshold and
|
|
absence rules, a CLI (`cairnobsctl`), a Terraform provider, and AI-assisted
|
|
query authoring that runs against a self-hosted Ollama model by default — no
|
|
cloud dependency.
|
|
|
|
## Architecture
|
|
|
|
| Component | Stack |
|
|
|---|---|
|
|
| Edge agent | Rust, musl static target |
|
|
| Transport | Redpanda (Kafka API) |
|
|
| Ingest / parse | Go |
|
|
| Analytical store | ClickHouse |
|
|
| Full-text index | Tantivy (Rust) |
|
|
| Control plane / API | Go, gRPC + REST gateway |
|
|
| Control-plane metadata | PostgreSQL |
|
|
| Frontend | SvelteKit + TypeScript |
|
|
| Deployment | Kubernetes Operator (kubebuilder), Helm, docker-compose |
|
|
|
|
PostgreSQL is scoped strictly to control-plane config — dashboards, panels,
|
|
alert rules and state, notification targets, delivery log — because those need
|
|
row-level locking and transactional read-modify-write that ClickHouse's
|
|
MergeTree family doesn't provide. Log data itself never touches it.
|
|
|
|
Full spec: [`docs/architecture.md`](docs/architecture.md). Read it before
|
|
changing any component; the storage/query split in particular is deliberate.
|
|
|
|
## Repository layout
|
|
|
|
Monorepo, one top-level directory per component, each with its own `README.md`,
|
|
unit tests, and Dockerfile.
|
|
|
|
```
|
|
agent/ Rust edge agent (Linux + Windows)
|
|
transport/ Redpanda topics and schemas
|
|
ingest/ Go ingest and parse pipeline
|
|
storage/ ClickHouse schema and migrations
|
|
search/ Tantivy full-text index service
|
|
api/ Control plane, query compiler, RBAC
|
|
alerting/ Rule evaluator and notification delivery
|
|
metadata/ PostgreSQL schema and migrations
|
|
web/ SvelteKit frontend
|
|
cli/ cairnobsctl
|
|
proto/ gRPC service definitions
|
|
terraform/ Terraform provider
|
|
enterprise/ SSO, multi-tenancy, per-tenant provisioning
|
|
deploy/ Helm charts and Kubernetes operator
|
|
docs/ Architecture, design docs, per-phase runbooks
|
|
hack/ Development scripts
|
|
```
|
|
|
|
`enterprise/` stays a separate module that core never imports from. Since the
|
|
Phase 6 relicensing that boundary is architectural rather than legal — it keeps
|
|
core buildable and deployable standalone, and keeps tenant resolution
|
|
server-side.
|
|
|
|
## Running locally
|
|
|
|
```sh
|
|
docker compose up
|
|
```
|
|
|
|
The web UI comes up on <http://localhost:3000> and the API on `:8080`;
|
|
alerting is on `:8081` and enterprise auth on `:8082`. The agent connects to
|
|
ingest over mTLS gRPC on `:4317`. `search` is reachable only on the compose
|
|
network — it publishes no host port.
|
|
|
|
`COMPOSE_PROFILES` in `.env` selects the query-serving binary — `single-tenant`
|
|
(default) or `enterprise` for the multi-tenant path. They're mutually
|
|
exclusive, the same choice Helm's `enterprise.enabled` flag makes for a real
|
|
cluster. Override per invocation:
|
|
|
|
```sh
|
|
COMPOSE_PROFILES=enterprise docker compose up
|
|
```
|
|
|
|
Kubernetes deployment via the Helm chart in [`deploy/`](deploy/README.md).
|
|
|
|
## Status
|
|
|
|
Built in phases; each has a runbook in `docs/` recording how it was verified.
|
|
Full per-phase detail is in [`docs/status.md`](docs/status.md).
|
|
|
|
| Phase | Scope | Status |
|
|
|---|---|---|
|
|
| 0 | Agent → Redpanda → ingest → ClickHouse, queryable end-to-end | Shipped |
|
|
| 1 | Windows Event Log + journald, SQL and full-text paths | Shipped |
|
|
| 2 | Unified query language across both stores | Shipped |
|
|
| 3 | Dashboards, alert rules, notification delivery | Shipped |
|
|
| 4 | RBAC, tenant isolation, audit logging, per-tenant ClickHouse | Shipped |
|
|
| 5 | Frontend redesign and design system | Shipped |
|
|
| 6 | License compliance audit and remediation | Shipped |
|
|
| 7 | AI-assisted query authoring | Shipped |
|
|
|
|
**Phase 4 is shipped, and the environment that proved it is gone.** Every
|
|
control the phase defines was verified against real infrastructure at least
|
|
once — a docker-compose stack with real ClickHouse and Postgres, a local `kind`
|
|
cluster, and both SSO protocols against a real Auth0 tenant — finding eight
|
|
bugs that no amount of Docker-free testing could have caught. The prototype
|
|
VPS was retired on 2026-09-04, so that verification is a record rather than
|
|
something you can re-run: see
|
|
[`docs/phase-4-runbook.md`](docs/phase-4-runbook.md).
|
|
|
|
Two limits worth stating plainly. SSO has been tried against one IdP, not two,
|
|
and no production-grade cluster has run this. And `demo.cairnobs.org` is **not**
|
|
evidence for any of it — the demo runs the single-tenant profile, so it
|
|
exercises the OSS path and says nothing about RBAC or tenant isolation.
|
|
|
|
The Windows agent code (`EvtSubscribe`, ETW, service registration) has never
|
|
run on real Windows — no Windows toolchain existed in the build environment.
|
|
ETW additionally sits behind a feature flag, since it needs elevated
|
|
privileges. Details in [`agent/README.md`](agent/README.md).
|
|
|
|
Terraform provider coverage is partial by necessity: dashboards and panels have
|
|
full CRUD, while alert rules and notification targets are create/destroy only,
|
|
because `alerting` exposes no `PUT /rules/{id}` or `PUT /targets/{id}` to
|
|
update against. Tenant and RBAC resources are disclosed future work —
|
|
[`terraform/README.md`](terraform/README.md) accounts for exactly what exists.
|
|
|
|
## Contributing
|
|
|
|
- Conventional commits. Every change should be a logically complete,
|
|
independently revertible unit.
|
|
- Rust: `cargo clippy --all-targets -- -D warnings` must pass.
|
|
- Go: `go vet` and `golangci-lint`, no globals for shared state.
|
|
- Every UI action must map to a documented REST/gRPC call — no UI-only logic.
|
|
The CLI and Terraform provider are first-class, not afterthoughts.
|
|
- Prefer boring, well-understood dependencies. This is infrastructure software;
|
|
operators need to trust it.
|
|
|
|
## Licensing
|
|
|
|
Copyright (C) 2026 Coffey Labs.
|
|
|
|
AGPLv3, no exceptions — see [`LICENSE`](LICENSE). `enterprise/` was under a
|
|
commercial-license stub from Phase 4 through Phase 5; Phase 6 relicensed it to
|
|
match core. The full record and its business-model consequences are in
|
|
[`docs/compliance/license-audit-report.md`](docs/compliance/license-audit-report.md).
|
|
|
|
The default AI deployment uses `qwen2.5-coder` (Apache-2.0) via Ollama,
|
|
chosen specifically to keep that license purity intact. The cloud adapter is
|
|
pluggable, opt-in, and off by default.
|