Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
This commit is contained in:
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
+1
View File
@@ -0,0 +1 @@
out/
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Generates a throwaway CA plus a server cert (for ingest) and a client
# cert (for the agent) for local mTLS. Dev/homelab only — never use this
# CA or its certs for anything resembling production; there's no rotation,
# no revocation, and the CA key sits unencrypted on disk right next to
# everything it signed.
#
# Re-run to regenerate from scratch; existing output is overwritten.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUT_DIR="${SCRIPT_DIR}/out"
DAYS="${DEV_CERT_DAYS:-365}"
mkdir -p "${OUT_DIR}"
cd "${OUT_DIR}"
echo "Generating dev CA..."
openssl req -x509 -newkey rsa:4096 -sha256 -days "${DAYS}" -nodes \
-keyout ca-key.pem -out ca.pem \
-subj "/O=Sentry Dev/CN=Sentry Dev CA"
gen_leaf() {
local name="$1" cn="$2" san="$3"
openssl req -newkey rsa:2048 -nodes -keyout "${name}-key.pem" -out "${name}.csr" \
-subj "/O=Sentry Dev/CN=${cn}"
openssl x509 -req -in "${name}.csr" -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out "${name}.pem" -days "${DAYS}" -sha256 \
-extfile <(printf "subjectAltName=%s" "${san}")
rm -f "${name}.csr"
}
# SANs cover both "reached by another container on the compose network"
# (ingest) and "reached from the host" (localhost/127.0.0.1, for an
# agent running natively per /agent/README.md's journald caveat).
echo "Generating server (ingest) cert..."
gen_leaf server ingest "DNS:ingest,DNS:localhost,IP:127.0.0.1"
echo "Generating client (agent) cert..."
gen_leaf client sentry-agent "DNS:sentry-agent"
rm -f ca.srl
echo
echo "Done. Certs written to ${OUT_DIR}/:"
ls "${OUT_DIR}"