coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

Browse Source 
End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
This commit is contained in:
jcoffey-dev
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+105
View File
@@ -0,0 +1,105 @@
# Sentry Architecture
> **Status:** Draft, Phase 0 scope. Written from the project constraints and
> task list at kickoff, not transcribed from a pre-existing spec. Treat as a
> starting point to correct, not a settled design — flag anything that
> doesn't match your intent before implementation leans on it further.
## Mission
Open-core, Kubernetes-native centralized logging platform. Compete with
Splunk on features; win on cost-per-GB, a modern language stack, and
multi-tenant RBAC that's actually honest about its guarantees.
## Component map
```
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
└──────┬─────┘
│ batch INSERT
▼
┌────────────┐
│ ClickHouse │
└─────┬──────┘
│ SQL
┌───────────▼───────────┐
│ api (Go: gRPC+REST) │
└───────────┬───────────┘
│ REST
┌───────────▼───────────┐
│ web (SvelteKit) │
└────────────────────────┘
```
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
`ingest` service's gRPC front end produces to Redpanda rather than writing
ClickHouse directly; a separate consumer path reads from Redpanda and batches
inserts into ClickHouse. This exercises the real transport layer from day
one instead of deferring it, and keeps Kafka credentials off the edge agent.
## Storage / query split
- **ClickHouse** is the analytical store of record for structured log data:
timestamp, host, service, severity, message, plus a `Map(String,String)`
for arbitrary structured fields. Partitioned by day, ordered by
`(service, timestamp)`.
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
field and unstructured payloads, queried out-of-band from ClickHouse and
joined by a log identifier. Not built in Phase 0.
- **Schema-on-write** using OTel semantic conventions as the default log
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
the structured columns (captured via the `Map` column and/or a raw
passthrough field).
This split is not to be changed without discussion — see CLAUDE.md.
## Component responsibilities (Phase 0)
| Component | Responsibility |
|---|---|
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. |
| `proto` | Shared `.proto` contracts for the agent↔ingest gRPC service, versioned independently of either component. |
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. |
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
| `api` (Go) | gRPC + REST gateway. Phase 0: one crude `POST /query` endpoint, SELECT-only, proxying to ClickHouse. Real SPL-like query layer is Phase 2. |
| `web` (SvelteKit) | Single page: SQL text box, submit, results table. No auth, no styling polish. |
| `cli` (`sentryctl`) | Stub. Single `ping` command for now. |
| `deploy` | Helm charts, k8s manifests. Stubbed in Phase 0; docker-compose is the real local/dev path. |
## Licensing boundary
AGPLv3 for core + agents. Enterprise features (SSO, multi-tenancy,
compliance) live under `enterprise/` (not yet created — out of scope for
Phase 0) under a commercial license stub. AGPL code must never import from
`enterprise/`. No enterprise-gated code exists yet in this repo; this
section documents the boundary so nothing added later crosses it by
accident.
## Non-negotiables carried from CLAUDE.md
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
`aarch64-unknown-linux-musl`, no glibc runtime deps.
- Windows support (Phase 1+) via native ETW/Event Log API, not WSL.
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
## Explicitly out of scope for Phase 0
Windows agent, alerting, dashboards, multi-tenancy, Tantivy full-text
search, the real SPL-like query language, enterprise module code.
## Open questions for you to resolve
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
deferred until storage sizing is a real concern.
- Exact OTel log schema field mapping (which OTel resource/log attributes
map to which ClickHouse columns) — Phase 0 uses a minimal subset
(timestamp, host, service, severity, message, attributes map); full
mapping deferred.
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
a static dev CA and manually issued certs; production PKI design is
out of scope here.
docs/phase-0-runbook.md
+207
View File
@@ -0,0 +1,207 @@
# Phase 0 runbook
Walks one log line from a Linux host, through the Rust agent, Redpanda,
ingest, and ClickHouse, to a browser table. This is the actual
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
regardless of what any individual component's tests say.
**This sequence has not been run end-to-end** in the environment that
built it (no Docker available there — see the caveats each component's
summary already flagged). Individual pieces are unit-tested and built
successfully in isolation; this document is the logical sequence to run
for real, not a report that it's been run. Expect to debug something on
first attempt, and treat the "Troubleshooting" section at the bottom as a
starting point, not an exhaustive list.
## Prerequisites
- Docker with **Compose v2** (`docker compose`, not the legacy
`docker-compose` v1 binary) — the compose file uses
`service_completed_successfully` conditions that v1 doesn't support.
- Rust toolchain (`cargo`) and `protoc` — to build the agent.
- `openssl` — to generate dev mTLS certs.
- A systemd-based Linux host to run the agent on (journald is the default
source). If you're not on such a host, see `/agent/README.md`'s
`file-tail` feature as an alternative source.
You do **not** need the musl cross-compilation target for this runbook —
that's for producing the distro-agnostic release binary. A native
`cargo build --release` is enough to run the agent on the same machine
you're testing on.
## 1. Generate dev mTLS certs
```sh
./hack/dev-certs/generate.sh
```
Writes a throwaway CA plus a server cert (for `ingest`) and a client cert
(for the agent) to `hack/dev-certs/out/`. Dev-only — see the script's
header comment for why.
## 2. Bring up the backend stack
```sh
docker compose up -d --build
```
This builds and starts, in dependency order: `redpanda` → `redpanda-provision`
(creates the `sentry.logs.raw` topic, then exits) → `clickhouse` →
`clickhouse-migrate` (applies `/storage/migrations`, then exits) →
`ingest` and `api` → `web`.
Check everything came up:
```sh
docker compose ps
```
`redpanda-provision` and `clickhouse-migrate` should show `Exited (0)`
(one-shot jobs, not long-running). Everything else should show `Up` /
`healthy`.
If `ingest` or `api` crash-looped, they likely started before their
`depends_on` conditions were actually satisfied, or the dev certs from
step 1 don't exist yet — check `docker compose logs ingest`.
## 3. Sanity-check the backend before involving the agent
```sh
curl http://localhost:8080/healthz
# -> 200, empty body
curl -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"sql": "SELECT 1"}'
# -> {"columns":["1"],"rows":[[1]]} (exact column name may vary by ClickHouse version)
```
This confirms `api` can reach `clickhouse` before you go looking for bugs
anywhere else. It doesn't touch the `logs` table, so it works even before
any agent has sent data.
## 4. Install the agent's mTLS material
The agent's default config expects certs at `/etc/sentry-agent/` (see
`/agent/config/agent.example.toml`), which requires root:
```sh
sudo mkdir -p /etc/sentry-agent
sudo cp hack/dev-certs/out/ca.pem \
hack/dev-certs/out/client.pem \
hack/dev-certs/out/client-key.pem \
/etc/sentry-agent/
```
## 5. Build and run the agent
```sh
cd agent
cargo build --release
```
The agent's built-in defaults already match this setup with **zero
config file**: journald source (whole journal), service name `default`,
ingest endpoint `https://127.0.0.1:4317` (matches the port `ingest`
publishes in `docker-compose.yml`), and the cert paths from step 4. This
is the "no required flags for the common case" design goal from
`/agent/README.md` — if it doesn't just run, that design assumption is
wrong somewhere and worth reporting as a bug, not working around.
Reading the system journal generally needs root (or membership in the
`systemd-journal` group with a distro that grants it read access — varies
by distro, root is the reliable path for this runbook):
```sh
sudo ./target/release/sentry-agent
```
Leave it running in this terminal — you should see a `connected to ingest
service` log line. If you see a TLS or connection error instead, stop
here and check the Troubleshooting section before continuing.
## 6. Generate a test log line
In another terminal, **after** the agent is running and connected
(journald tailing starts from "now" — anything logged before the agent
started won't be picked up):
```sh
logger "hello from sentry phase 0"
```
`logger` (part of util-linux, present on virtually every Linux distro)
writes this to the system log, which journald captures immediately.
Give it a couple of seconds — the agent batches with a 2-second flush
interval by default, so the line won't hit ingest instantly.
## 7. Confirm it's queryable
**Via the web UI:**
`web` is already running from step 2 (`docker compose up -d --build`
starts every service in the file). Open `http://localhost:3000`, run the
default query (`SELECT * FROM logs
ORDER BY timestamp DESC LIMIT 100`), and look for a row with
`message = "hello from sentry phase 0"`.
**Or via curl, if you want to skip the browser:**
```sh
curl -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"sql": "SELECT * FROM logs ORDER BY timestamp DESC LIMIT 10"}'
```
**Or via sentryctl, just to confirm api is up (doesn't check the data
itself):**
```sh
cd cli && go run ./cmd/sentryctl ping
```
If you see the row: that's Phase 0 done, end to end. If you don't, see
Troubleshooting below.
## Tearing down
```sh
docker compose down # stops and removes containers, keeps volumes
docker compose down -v # also wipes Redpanda/ClickHouse data — start clean next time
```
## Troubleshooting
**Agent logs a TLS/certificate error on startup.**
Check the server cert's SAN actually covers how the agent is connecting
(`openssl x509 -in hack/dev-certs/out/server.pem -noout -ext
subjectAltName` — should list `DNS:ingest, DNS:localhost,
IP:127.0.0.1`). If you changed the agent's `ingest.endpoint` to something
not in that list, regenerate certs with an updated SAN in
`hack/dev-certs/generate.sh`, don't disable TLS verification.
**Agent connects but no data ever shows up in ClickHouse.**
Check each hop in order rather than guessing:
1. `docker compose logs ingest` — look for "batch produced to redpanda"
(gRPC front end got the batch) vs. errors.
2. `docker compose logs ingest` again — look for "batch flushed to
clickhouse" from the consumer half. If you see repeated "clickhouse
batch write failed... will redeliver" messages, `clickhouse-migrate`
likely hasn't finished (check `docker compose ps`) — the consumer will
keep retrying and self-heal once the table exists, per its
at-least-once design (see `/ingest/README.md`), so this may just need
more time rather than intervention.
3. `docker compose exec redpanda rpk topic list` — confirm
`sentry.logs.raw` exists (if `redpanda-provision` failed, it won't).
**`docker compose up` fails on `service_completed_successfully`.**
You're likely on Compose v1 (`docker-compose`, hyphenated) rather than v2
(`docker compose`, space) — see Prerequisites.
**Web UI query returns an error instead of rows.**
Open the browser's network tab — if the request never leaves the page
(CORS error in the console), confirm `api`'s `CORS_ALLOWED_ORIGIN`
(defaults to `*`, should not be the issue) and that `VITE_API_BASE_URL`
was set correctly at `web`'s build time (it's baked in, not read at
container start — see `/web/README.md`).