Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# Sentry Architecture
|
||||
|
||||
> **Status:** Draft, Phase 0 scope. Written from the project constraints and
|
||||
> task list at kickoff, not transcribed from a pre-existing spec. Treat as a
|
||||
> starting point to correct, not a settled design — flag anything that
|
||||
> doesn't match your intent before implementation leans on it further.
|
||||
|
||||
## Mission
|
||||
|
||||
Open-core, Kubernetes-native centralized logging platform. Compete with
|
||||
Splunk on features; win on cost-per-GB, a modern language stack, and
|
||||
multi-tenant RBAC that's actually honest about its guarantees.
|
||||
|
||||
## Component map
|
||||
|
||||
```
|
||||
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
|
||||
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
|
||||
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
|
||||
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
|
||||
└──────┬─────┘
|
||||
│ batch INSERT
|
||||
▼
|
||||
┌────────────┐
|
||||
│ ClickHouse │
|
||||
└─────┬──────┘
|
||||
│ SQL
|
||||
┌───────────▼───────────┐
|
||||
│ api (Go: gRPC+REST) │
|
||||
└───────────┬───────────┘
|
||||
│ REST
|
||||
┌───────────▼───────────┐
|
||||
│ web (SvelteKit) │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
|
||||
`ingest` service's gRPC front end produces to Redpanda rather than writing
|
||||
ClickHouse directly; a separate consumer path reads from Redpanda and batches
|
||||
inserts into ClickHouse. This exercises the real transport layer from day
|
||||
one instead of deferring it, and keeps Kafka credentials off the edge agent.
|
||||
|
||||
## Storage / query split
|
||||
|
||||
- **ClickHouse** is the analytical store of record for structured log data:
|
||||
timestamp, host, service, severity, message, plus a `Map(String,String)`
|
||||
for arbitrary structured fields. Partitioned by day, ordered by
|
||||
`(service, timestamp)`.
|
||||
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
|
||||
field and unstructured payloads, queried out-of-band from ClickHouse and
|
||||
joined by a log identifier. Not built in Phase 0.
|
||||
- **Schema-on-write** using OTel semantic conventions as the default log
|
||||
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
|
||||
the structured columns (captured via the `Map` column and/or a raw
|
||||
passthrough field).
|
||||
|
||||
This split is not to be changed without discussion — see CLAUDE.md.
|
||||
|
||||
## Component responsibilities (Phase 0)
|
||||
|
||||
| Component | Responsibility |
|
||||
|---|---|
|
||||
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. |
|
||||
| `proto` | Shared `.proto` contracts for the agent↔ingest gRPC service, versioned independently of either component. |
|
||||
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
|
||||
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. |
|
||||
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
|
||||
| `api` (Go) | gRPC + REST gateway. Phase 0: one crude `POST /query` endpoint, SELECT-only, proxying to ClickHouse. Real SPL-like query layer is Phase 2. |
|
||||
| `web` (SvelteKit) | Single page: SQL text box, submit, results table. No auth, no styling polish. |
|
||||
| `cli` (`sentryctl`) | Stub. Single `ping` command for now. |
|
||||
| `deploy` | Helm charts, k8s manifests. Stubbed in Phase 0; docker-compose is the real local/dev path. |
|
||||
|
||||
## Licensing boundary
|
||||
|
||||
AGPLv3 for core + agents. Enterprise features (SSO, multi-tenancy,
|
||||
compliance) live under `enterprise/` (not yet created — out of scope for
|
||||
Phase 0) under a commercial license stub. AGPL code must never import from
|
||||
`enterprise/`. No enterprise-gated code exists yet in this repo; this
|
||||
section documents the boundary so nothing added later crosses it by
|
||||
accident.
|
||||
|
||||
## Non-negotiables carried from CLAUDE.md
|
||||
|
||||
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
|
||||
`aarch64-unknown-linux-musl`, no glibc runtime deps.
|
||||
- Windows support (Phase 1+) via native ETW/Event Log API, not WSL.
|
||||
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
|
||||
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
|
||||
|
||||
## Explicitly out of scope for Phase 0
|
||||
|
||||
Windows agent, alerting, dashboards, multi-tenancy, Tantivy full-text
|
||||
search, the real SPL-like query language, enterprise module code.
|
||||
|
||||
## Open questions for you to resolve
|
||||
|
||||
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
|
||||
deferred until storage sizing is a real concern.
|
||||
- Exact OTel log schema field mapping (which OTel resource/log attributes
|
||||
map to which ClickHouse columns) — Phase 0 uses a minimal subset
|
||||
(timestamp, host, service, severity, message, attributes map); full
|
||||
mapping deferred.
|
||||
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
|
||||
a static dev CA and manually issued certs; production PKI design is
|
||||
out of scope here.
|
||||
Reference in New Issue
Block a user