Files

T

jcoffey-dev b6b092c912 Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.

2026-08-13 08:25:19 -07:00

3.4 KiB

Raw Blame History

sentry-agent

Distro-agnostic Linux log collector. Statically linked against musl, no glibc runtime dependency. Tails journald (default) or a file, batches lines, and ships them over mTLS gRPC to the ingest service.

Workspace layout

sentry-parser — pure-std RFC 5424 syslog parser with raw-passthrough fallback. No I/O, easy to unit test in isolation.
sentry-agent — the binary: config loading, sourcing (journald/file), batching, mTLS gRPC client.

Why journalctl, not libsystemd

The journald source shells out to journalctl -f -o json rather than linking libsystemd via FFI. Statically linking libsystemd into a musl binary is fragile — it pulls in dbus/libcap transitively and isn't designed for static linking — and would undermine the no-glibc-runtime-deps goal even where technically possible. journalctl ships on every systemd distro this agent targets, so shelling out sidesteps the problem entirely. See /docs/architecture.md.

Building

Native build (whatever target your machine is):

cargo build --release

musl targets (what actually ships):

rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl

# x86_64: works with musl-gcc installed locally (musl-tools on Debian,
# musl on Arch, etc.) — the musl target is fully static by default.
cargo build --release --target x86_64-unknown-linux-musl

# aarch64 cross-compilation needs a cross toolchain; the boring, reliable
# option is `cross` (https://github.com/cross-rs/cross), which builds
# inside a Docker container with the right linker preinstalled:
cross build --release --target aarch64-unknown-linux-musl

Building requires protoc on PATH (used by tonic-build/prost-build at compile time to generate the gRPC client from /proto/sentry/logs/v1/logs.proto).

Container build (see caveat below):

# from the repo root, not agent/
docker build -f agent/Dockerfile -t sentry-agent .

Caveat: the container image is provided for CI/completeness, but journald sourcing needs journalctl and access to the host journal — neither of which exist in the scratch image or are available to a container without deliberately bind-mounting /var/log/journal (or /run/log/journal) and the journalctl binary in. The intended Phase 0 deployment for journald sourcing is as a native binary managed by systemd on the host, not containerized.

Running

No CLI flags are required for the common case:

./sentry-agent

This uses /etc/sentry-agent/agent.toml if present, otherwise built-in defaults: journald source (whole journal, no unit filter), service name default, and mTLS material expected at /etc/sentry-agent/{ca,client,client-key}.pem. mTLS is mandatory per the project's transport requirements, so a from-scratch run with no certs in place will fail fast with a clear error rather than connecting insecurely.

See config/agent.example.toml for all fields.

./sentry-agent --config /path/to/agent.toml

Testing

cargo test --workspace

Feature flags

journald (default) — journalctl-based journald source.
file-tail — polling-based file tailer (no inotify dependency; doesn't follow rename-based log rotation yet).

Both can be enabled together; [source].kind in config picks which one runs. Building without a feature and configuring that source at runtime fails at startup with a clear error rather than silently doing nothing.

3.4 KiB Raw Blame History