Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,207 @@
|
||||
# Phase 0 runbook
|
||||
|
||||
Walks one log line from a Linux host, through the Rust agent, Redpanda,
|
||||
ingest, and ClickHouse, to a browser table. This is the actual
|
||||
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
|
||||
regardless of what any individual component's tests say.
|
||||
|
||||
**This sequence has not been run end-to-end** in the environment that
|
||||
built it (no Docker available there — see the caveats each component's
|
||||
summary already flagged). Individual pieces are unit-tested and built
|
||||
successfully in isolation; this document is the logical sequence to run
|
||||
for real, not a report that it's been run. Expect to debug something on
|
||||
first attempt, and treat the "Troubleshooting" section at the bottom as a
|
||||
starting point, not an exhaustive list.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker with **Compose v2** (`docker compose`, not the legacy
|
||||
`docker-compose` v1 binary) — the compose file uses
|
||||
`service_completed_successfully` conditions that v1 doesn't support.
|
||||
- Rust toolchain (`cargo`) and `protoc` — to build the agent.
|
||||
- `openssl` — to generate dev mTLS certs.
|
||||
- A systemd-based Linux host to run the agent on (journald is the default
|
||||
source). If you're not on such a host, see `/agent/README.md`'s
|
||||
`file-tail` feature as an alternative source.
|
||||
|
||||
You do **not** need the musl cross-compilation target for this runbook —
|
||||
that's for producing the distro-agnostic release binary. A native
|
||||
`cargo build --release` is enough to run the agent on the same machine
|
||||
you're testing on.
|
||||
|
||||
## 1. Generate dev mTLS certs
|
||||
|
||||
```sh
|
||||
./hack/dev-certs/generate.sh
|
||||
```
|
||||
|
||||
Writes a throwaway CA plus a server cert (for `ingest`) and a client cert
|
||||
(for the agent) to `hack/dev-certs/out/`. Dev-only — see the script's
|
||||
header comment for why.
|
||||
|
||||
## 2. Bring up the backend stack
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
This builds and starts, in dependency order: `redpanda` → `redpanda-provision`
|
||||
(creates the `sentry.logs.raw` topic, then exits) → `clickhouse` →
|
||||
`clickhouse-migrate` (applies `/storage/migrations`, then exits) →
|
||||
`ingest` and `api` → `web`.
|
||||
|
||||
Check everything came up:
|
||||
|
||||
```sh
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
`redpanda-provision` and `clickhouse-migrate` should show `Exited (0)`
|
||||
(one-shot jobs, not long-running). Everything else should show `Up` /
|
||||
`healthy`.
|
||||
|
||||
If `ingest` or `api` crash-looped, they likely started before their
|
||||
`depends_on` conditions were actually satisfied, or the dev certs from
|
||||
step 1 don't exist yet — check `docker compose logs ingest`.
|
||||
|
||||
## 3. Sanity-check the backend before involving the agent
|
||||
|
||||
```sh
|
||||
curl http://localhost:8080/healthz
|
||||
# -> 200, empty body
|
||||
|
||||
curl -X POST http://localhost:8080/query \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"sql": "SELECT 1"}'
|
||||
# -> {"columns":["1"],"rows":[[1]]} (exact column name may vary by ClickHouse version)
|
||||
```
|
||||
|
||||
This confirms `api` can reach `clickhouse` before you go looking for bugs
|
||||
anywhere else. It doesn't touch the `logs` table, so it works even before
|
||||
any agent has sent data.
|
||||
|
||||
## 4. Install the agent's mTLS material
|
||||
|
||||
The agent's default config expects certs at `/etc/sentry-agent/` (see
|
||||
`/agent/config/agent.example.toml`), which requires root:
|
||||
|
||||
```sh
|
||||
sudo mkdir -p /etc/sentry-agent
|
||||
sudo cp hack/dev-certs/out/ca.pem \
|
||||
hack/dev-certs/out/client.pem \
|
||||
hack/dev-certs/out/client-key.pem \
|
||||
/etc/sentry-agent/
|
||||
```
|
||||
|
||||
## 5. Build and run the agent
|
||||
|
||||
```sh
|
||||
cd agent
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
The agent's built-in defaults already match this setup with **zero
|
||||
config file**: journald source (whole journal), service name `default`,
|
||||
ingest endpoint `https://127.0.0.1:4317` (matches the port `ingest`
|
||||
publishes in `docker-compose.yml`), and the cert paths from step 4. This
|
||||
is the "no required flags for the common case" design goal from
|
||||
`/agent/README.md` — if it doesn't just run, that design assumption is
|
||||
wrong somewhere and worth reporting as a bug, not working around.
|
||||
|
||||
Reading the system journal generally needs root (or membership in the
|
||||
`systemd-journal` group with a distro that grants it read access — varies
|
||||
by distro, root is the reliable path for this runbook):
|
||||
|
||||
```sh
|
||||
sudo ./target/release/sentry-agent
|
||||
```
|
||||
|
||||
Leave it running in this terminal — you should see a `connected to ingest
|
||||
service` log line. If you see a TLS or connection error instead, stop
|
||||
here and check the Troubleshooting section before continuing.
|
||||
|
||||
## 6. Generate a test log line
|
||||
|
||||
In another terminal, **after** the agent is running and connected
|
||||
(journald tailing starts from "now" — anything logged before the agent
|
||||
started won't be picked up):
|
||||
|
||||
```sh
|
||||
logger "hello from sentry phase 0"
|
||||
```
|
||||
|
||||
`logger` (part of util-linux, present on virtually every Linux distro)
|
||||
writes this to the system log, which journald captures immediately.
|
||||
|
||||
Give it a couple of seconds — the agent batches with a 2-second flush
|
||||
interval by default, so the line won't hit ingest instantly.
|
||||
|
||||
## 7. Confirm it's queryable
|
||||
|
||||
**Via the web UI:**
|
||||
|
||||
`web` is already running from step 2 (`docker compose up -d --build`
|
||||
starts every service in the file). Open `http://localhost:3000`, run the
|
||||
default query (`SELECT * FROM logs
|
||||
ORDER BY timestamp DESC LIMIT 100`), and look for a row with
|
||||
`message = "hello from sentry phase 0"`.
|
||||
|
||||
**Or via curl, if you want to skip the browser:**
|
||||
|
||||
```sh
|
||||
curl -X POST http://localhost:8080/query \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"sql": "SELECT * FROM logs ORDER BY timestamp DESC LIMIT 10"}'
|
||||
```
|
||||
|
||||
**Or via sentryctl, just to confirm api is up (doesn't check the data
|
||||
itself):**
|
||||
|
||||
```sh
|
||||
cd cli && go run ./cmd/sentryctl ping
|
||||
```
|
||||
|
||||
If you see the row: that's Phase 0 done, end to end. If you don't, see
|
||||
Troubleshooting below.
|
||||
|
||||
## Tearing down
|
||||
|
||||
```sh
|
||||
docker compose down # stops and removes containers, keeps volumes
|
||||
docker compose down -v # also wipes Redpanda/ClickHouse data — start clean next time
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Agent logs a TLS/certificate error on startup.**
|
||||
Check the server cert's SAN actually covers how the agent is connecting
|
||||
(`openssl x509 -in hack/dev-certs/out/server.pem -noout -ext
|
||||
subjectAltName` — should list `DNS:ingest, DNS:localhost,
|
||||
IP:127.0.0.1`). If you changed the agent's `ingest.endpoint` to something
|
||||
not in that list, regenerate certs with an updated SAN in
|
||||
`hack/dev-certs/generate.sh`, don't disable TLS verification.
|
||||
|
||||
**Agent connects but no data ever shows up in ClickHouse.**
|
||||
Check each hop in order rather than guessing:
|
||||
1. `docker compose logs ingest` — look for "batch produced to redpanda"
|
||||
(gRPC front end got the batch) vs. errors.
|
||||
2. `docker compose logs ingest` again — look for "batch flushed to
|
||||
clickhouse" from the consumer half. If you see repeated "clickhouse
|
||||
batch write failed... will redeliver" messages, `clickhouse-migrate`
|
||||
likely hasn't finished (check `docker compose ps`) — the consumer will
|
||||
keep retrying and self-heal once the table exists, per its
|
||||
at-least-once design (see `/ingest/README.md`), so this may just need
|
||||
more time rather than intervention.
|
||||
3. `docker compose exec redpanda rpk topic list` — confirm
|
||||
`sentry.logs.raw` exists (if `redpanda-provision` failed, it won't).
|
||||
|
||||
**`docker compose up` fails on `service_completed_successfully`.**
|
||||
You're likely on Compose v1 (`docker-compose`, hyphenated) rather than v2
|
||||
(`docker compose`, space) — see Prerequisites.
|
||||
|
||||
**Web UI query returns an error instead of rows.**
|
||||
Open the browser's network tab — if the request never leaves the page
|
||||
(CORS error in the console), confirm `api`'s `CORS_ALLOWED_ORIGIN`
|
||||
(defaults to `*`, should not be the issue) and that `VITE_API_BASE_URL`
|
||||
was set correctly at `web`'s build time (it's baked in, not read at
|
||||
container start — see `/web/README.md`).
|
||||
Reference in New Issue
Block a user