Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Idempotently creates the topic ingest produces/consumes. Talks to
|
||||
# Redpanda over the network via `rpk`, not `docker exec` into the broker
|
||||
# container -- this way the same script works whether it's run from the
|
||||
# host (against the standalone compose in this directory), from inside a
|
||||
# sibling container on the root compose's network, or in CI.
|
||||
set -euo pipefail
|
||||
|
||||
BROKERS="${REDPANDA_BROKERS:-localhost:9092}"
|
||||
TOPIC="${REDPANDA_TOPIC:-sentry.logs.raw}"
|
||||
PARTITIONS="${REDPANDA_TOPIC_PARTITIONS:-6}"
|
||||
|
||||
echo "Waiting for Redpanda at ${BROKERS}..."
|
||||
until rpk cluster health --brokers "${BROKERS}" --exit-when-healthy > /dev/null 2>&1; do
|
||||
sleep 1
|
||||
done
|
||||
|
||||
if rpk topic list --brokers "${BROKERS}" | awk 'NR>1{print $1}' | grep -qx "${TOPIC}"; then
|
||||
echo "Topic '${TOPIC}' already exists, skipping."
|
||||
else
|
||||
echo "Creating topic '${TOPIC}' (${PARTITIONS} partitions)..."
|
||||
rpk topic create "${TOPIC}" --brokers "${BROKERS}" --partitions "${PARTITIONS}" --replicas 1
|
||||
fi
|
||||
Reference in New Issue
Block a user