Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
This commit is contained in:
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
+8
View File
@@ -0,0 +1,8 @@
# ingest/ and api/ build with context "." (repo root) so their Dockerfiles
# can also COPY proto/. Keep that context lean.
.git/
agent/target/
web/node_modules/
web/build/
web/.svelte-kit/
hack/dev-certs/out/
+21
View File
@@ -0,0 +1,21 @@
# Rust
agent/target/
# Go build cache (go build ./... without -o doesn't normally leave
# binaries in-tree, but be defensive)
/ingest/ingest
/api/api
/cli/sentryctl
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
web/node_modules/
web/build/
web/.svelte-kit/
# Dev-only generated secrets
hack/dev-certs/out/
# OS / editor
.DS_Store
Thumbs.db
*.swp
+1510
View File
File diff suppressed because it is too large Load Diff
+13
View File
@@ -0,0 +1,13 @@
[workspace]
resolver = "2"
members = ["sentry-parser", "sentry-agent"]
[workspace.package]
version = "0.1.0"
edition = "2021"
license = "AGPL-3.0-only"
[profile.release]
lto = true
strip = true
codegen-units = 1
+16
View File
@@ -0,0 +1,16 @@
# Build context must be the repo root (sentry/), not agent/, since this
# needs both agent/ and proto/:
# docker build -f agent/Dockerfile -t sentry-agent .
FROM rust:1-alpine AS builder
RUN apk add --no-cache musl-dev protobuf-dev protobuf
WORKDIR /src
COPY proto ./proto
COPY agent ./agent
WORKDIR /src/agent
RUN rustup target add x86_64-unknown-linux-musl \
&& cargo build --release --target x86_64-unknown-linux-musl -p sentry-agent
FROM scratch
COPY --from=builder /src/agent/target/x86_64-unknown-linux-musl/release/sentry-agent /sentry-agent
ENTRYPOINT ["/sentry-agent"]
+100
View File
@@ -0,0 +1,100 @@
# sentry-agent
Distro-agnostic Linux log collector. Statically linked against musl, no
glibc runtime dependency. Tails journald (default) or a file, batches
lines, and ships them over mTLS gRPC to the ingest service.
## Workspace layout
- `sentry-parser` — pure-`std` RFC 5424 syslog parser with raw-passthrough
fallback. No I/O, easy to unit test in isolation.
- `sentry-agent` — the binary: config loading, sourcing (journald/file),
batching, mTLS gRPC client.
## Why journalctl, not libsystemd
The journald source shells out to `journalctl -f -o json` rather than
linking `libsystemd` via FFI. Statically linking libsystemd into a musl
binary is fragile — it pulls in dbus/libcap transitively and isn't designed
for static linking — and would undermine the no-glibc-runtime-deps goal
even where technically possible. `journalctl` ships on every systemd distro
this agent targets, so shelling out sidesteps the problem entirely. See
`/docs/architecture.md`.
## Building
Native build (whatever target your machine is):
```sh
cargo build --release
```
musl targets (what actually ships):
```sh
rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl
# x86_64: works with musl-gcc installed locally (musl-tools on Debian,
# musl on Arch, etc.) — the musl target is fully static by default.
cargo build --release --target x86_64-unknown-linux-musl
# aarch64 cross-compilation needs a cross toolchain; the boring, reliable
# option is `cross` (https://github.com/cross-rs/cross), which builds
# inside a Docker container with the right linker preinstalled:
cross build --release --target aarch64-unknown-linux-musl
```
Building requires `protoc` on PATH (used by `tonic-build`/`prost-build` at
compile time to generate the gRPC client from `/proto/sentry/logs/v1/logs.proto`).
Container build (see caveat below):
```sh
# from the repo root, not agent/
docker build -f agent/Dockerfile -t sentry-agent .
```
**Caveat:** the container image is provided for CI/completeness, but
journald sourcing needs `journalctl` and access to the host journal —
neither of which exist in the `scratch` image or are available to a
container without deliberately bind-mounting `/var/log/journal` (or
`/run/log/journal`) and the `journalctl` binary in. The intended Phase 0
deployment for journald sourcing is as a native binary managed by systemd
on the host, not containerized.
## Running
No CLI flags are required for the common case:
```sh
./sentry-agent
```
This uses `/etc/sentry-agent/agent.toml` if present, otherwise built-in
defaults: journald source (whole journal, no unit filter), service name
`default`, and mTLS material expected at
`/etc/sentry-agent/{ca,client,client-key}.pem`. mTLS is mandatory per the
project's transport requirements, so a from-scratch run with no certs in
place will fail fast with a clear error rather than connecting insecurely.
See `config/agent.example.toml` for all fields.
```sh
./sentry-agent --config /path/to/agent.toml
```
## Testing
```sh
cargo test --workspace
```
## Feature flags
- `journald` (default) — journalctl-based journald source.
- `file-tail` — polling-based file tailer (no inotify dependency; doesn't
follow rename-based log rotation yet).
Both can be enabled together; `[source].kind` in config picks which one
runs. Building without a feature and configuring that source at runtime
fails at startup with a clear error rather than silently doing nothing.
+3
View File
@@ -0,0 +1,3 @@
[toolchain]
channel = "stable"
targets = ["x86_64-unknown-linux-musl", "aarch64-unknown-linux-musl"]
+34
View File
@@ -0,0 +1,34 @@
[package]
name = "sentry-agent"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "Sentry distro-agnostic Linux log collector"
[[bin]]
name = "sentry-agent"
path = "src/main.rs"
[features]
default = ["journald"]
journald = []
file-tail = []
[dependencies]
sentry-parser = { path = "../sentry-parser" }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "process", "io-util", "io-std", "time", "fs", "sync", "signal"] }
tonic = { version = "0.12", features = ["tls"] }
prost = "0.13"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "0.8"
clap = { version = "4", features = ["derive"] }
anyhow = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
[build-dependencies]
tonic-build = "0.12"
+9
View File
@@ -0,0 +1,9 @@
fn main() -> Result<(), Box<dyn std::error::Error>> {
tonic_build::configure()
.build_server(false)
.compile_protos(
&["../../proto/sentry/logs/v1/logs.proto"],
&["../../proto"],
)?;
Ok(())
}
@@ -0,0 +1,33 @@
# Example sentry-agent config. Copy to /etc/sentry-agent/agent.toml, or
# pass --config /path/to/this/file.
#
# Every field has a built-in default (see src/config.rs), so this file only
# needs to contain what you're overriding. An agent with NO config file at
# all still runs: it defaults to journald, service = "default", and expects
# mTLS material at /etc/sentry-agent/{ca,client,client-key}.pem.
[agent]
# host = "explicit-hostname-override" # defaults to /etc/hostname
service = "my-service"
[source]
kind = "journald"
# unit = "nginx.service" # omit to tail the whole journal
# To tail a file instead:
# [source]
# kind = "file"
# path = "/var/log/nginx/access.log"
# from_beginning = false
[batch]
max_size = 500
flush_interval_ms = 2000
[ingest]
endpoint = "https://ingest.internal:4317"
[tls]
ca_cert = "/etc/sentry-agent/ca.pem"
client_cert = "/etc/sentry-agent/client.pem"
client_key = "/etc/sentry-agent/client-key.pem"
+108
View File
@@ -0,0 +1,108 @@
use crate::pb::LogRecord;
use std::time::{Duration, Instant};
/// Buffers `LogRecord`s and signals when to flush, either because the
/// buffer hit `max_size` (checked on every push) or because
/// `flush_interval` elapsed since the last flush (checked by the caller via
/// `poll_timeout` on a timer tick). Not thread-safe by design — one
/// batcher per agent, driven from a single async task's select loop.
pub struct Batcher {
max_size: usize,
flush_interval: Duration,
buf: Vec<LogRecord>,
last_flush: Instant,
}
impl Batcher {
pub fn new(max_size: usize, flush_interval: Duration) -> Self {
Self {
max_size,
flush_interval,
buf: Vec::with_capacity(max_size),
last_flush: Instant::now(),
}
}
/// Push a record. Returns the drained batch if this push filled the
/// buffer to `max_size`.
pub fn push(&mut self, record: LogRecord) -> Option<Vec<LogRecord>> {
self.buf.push(record);
if self.buf.len() >= self.max_size {
Some(self.drain())
} else {
None
}
}
/// Call periodically (e.g. from a timer tick). Returns the drained
/// batch if the flush interval has elapsed and there's anything
/// buffered.
pub fn poll_timeout(&mut self) -> Option<Vec<LogRecord>> {
if !self.buf.is_empty() && self.last_flush.elapsed() >= self.flush_interval {
Some(self.drain())
} else {
None
}
}
fn drain(&mut self) -> Vec<LogRecord> {
self.last_flush = Instant::now();
std::mem::replace(&mut self.buf, Vec::with_capacity(self.max_size))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn rec(msg: &str) -> LogRecord {
LogRecord {
timestamp_unix_nano: 0,
host: "h".into(),
service: "s".into(),
severity: 0,
message: msg.into(),
attributes: Default::default(),
}
}
#[test]
fn flushes_on_size() {
let mut b = Batcher::new(2, Duration::from_secs(999));
assert!(b.push(rec("a")).is_none());
let batch = b.push(rec("b")).expect("should flush at max_size");
assert_eq!(batch.len(), 2);
assert_eq!(batch[0].message, "a");
assert_eq!(batch[1].message, "b");
}
#[test]
fn buffer_empty_after_size_flush() {
let mut b = Batcher::new(1, Duration::from_secs(999));
b.push(rec("a")).expect("flush at max_size 1");
assert!(b.poll_timeout().is_none(), "buffer should be empty post-flush");
}
#[test]
fn flushes_on_timeout() {
let mut b = Batcher::new(100, Duration::from_millis(10));
assert!(b.push(rec("a")).is_none());
std::thread::sleep(Duration::from_millis(30));
let batch = b.poll_timeout().expect("should flush after timeout");
assert_eq!(batch.len(), 1);
}
#[test]
fn no_flush_when_buffer_empty() {
let mut b = Batcher::new(10, Duration::from_millis(1));
std::thread::sleep(Duration::from_millis(5));
assert!(b.poll_timeout().is_none());
}
#[test]
fn no_flush_before_timeout_elapsed() {
let mut b = Batcher::new(10, Duration::from_secs(999));
b.push(rec("a"));
assert!(b.poll_timeout().is_none());
}
}
+127
View File
@@ -0,0 +1,127 @@
use anyhow::{Context, Result};
use serde::Deserialize;
use std::path::{Path, PathBuf};
const DEFAULT_CONFIG_PATH: &str = "/etc/sentry-agent/agent.toml";
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct Config {
pub agent: AgentConfig,
pub source: SourceConfig,
pub batch: BatchConfig,
pub ingest: IngestConfig,
pub tls: TlsConfig,
}
impl Config {
/// Loads config from `explicit_path` if given, else from
/// `/etc/sentry-agent/agent.toml` if it exists, else falls back to
/// built-in defaults (journald source, default TLS cert paths). Only an
/// explicitly-passed `--config` path that doesn't exist is an error;
/// the conventional default path is optional.
pub fn load(explicit_path: Option<&Path>) -> Result<Config> {
let path = match explicit_path {
Some(p) => Some(p.to_path_buf()),
None => {
let default = PathBuf::from(DEFAULT_CONFIG_PATH);
default.exists().then_some(default)
}
};
match path {
Some(p) => {
let raw = std::fs::read_to_string(&p)
.with_context(|| format!("reading config file {}", p.display()))?;
toml::from_str(&raw).with_context(|| format!("parsing config file {}", p.display()))
}
None => Ok(Config::default()),
}
}
}
#[derive(Debug, Clone, Deserialize)]
#[serde(default)]
pub struct AgentConfig {
/// Overrides the auto-detected system hostname. Defaults to reading
/// /etc/hostname at startup when unset.
pub host: Option<String>,
pub service: String,
}
impl Default for AgentConfig {
fn default() -> Self {
Self {
host: None,
service: "default".to_string(),
}
}
}
#[derive(Debug, Clone, Deserialize)]
#[serde(rename_all = "lowercase", tag = "kind")]
pub enum SourceConfig {
Journald {
#[serde(default)]
unit: Option<String>,
},
File {
path: PathBuf,
#[serde(default)]
from_beginning: bool,
},
}
impl Default for SourceConfig {
fn default() -> Self {
SourceConfig::Journald { unit: None }
}
}
#[derive(Debug, Clone, Deserialize)]
#[serde(default)]
pub struct BatchConfig {
pub max_size: usize,
pub flush_interval_ms: u64,
}
impl Default for BatchConfig {
fn default() -> Self {
Self {
max_size: 500,
flush_interval_ms: 2000,
}
}
}
#[derive(Debug, Clone, Deserialize)]
#[serde(default)]
pub struct IngestConfig {
pub endpoint: String,
}
impl Default for IngestConfig {
fn default() -> Self {
Self {
endpoint: "https://127.0.0.1:4317".to_string(),
}
}
}
#[derive(Debug, Clone, Deserialize)]
#[serde(default)]
pub struct TlsConfig {
pub ca_cert: PathBuf,
pub client_cert: PathBuf,
pub client_key: PathBuf,
}
impl Default for TlsConfig {
fn default() -> Self {
Self {
ca_cert: PathBuf::from("/etc/sentry-agent/ca.pem"),
client_cert: PathBuf::from("/etc/sentry-agent/client.pem"),
client_key: PathBuf::from("/etc/sentry-agent/client-key.pem"),
}
}
}
+45
View File
@@ -0,0 +1,45 @@
use crate::config::{IngestConfig, TlsConfig};
use crate::pb::{log_ingest_client::LogIngestClient, LogRecord, PushBatchRequest};
use anyhow::{Context, Result};
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity};
/// Establishes an mTLS gRPC channel to the ingest service. Agents never
/// talk to Redpanda directly — this is the only network egress the agent
/// has, by design (see /docs/architecture.md).
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngestClient<Channel>> {
let ca = tokio::fs::read(&tls.ca_cert)
.await
.with_context(|| format!("reading CA cert at {}", tls.ca_cert.display()))?;
let cert = tokio::fs::read(&tls.client_cert)
.await
.with_context(|| format!("reading client cert at {}", tls.client_cert.display()))?;
let key = tokio::fs::read(&tls.client_key)
.await
.with_context(|| format!("reading client key at {}", tls.client_key.display()))?;
let tls_config = ClientTlsConfig::new()
.ca_certificate(Certificate::from_pem(ca))
.identity(Identity::from_pem(cert, key));
let channel = Channel::from_shared(ingest.endpoint.clone())
.context("invalid ingest endpoint URL")?
.tls_config(tls_config)
.context("configuring mTLS")?
.connect()
.await
.context("connecting to ingest service")?;
Ok(LogIngestClient::new(channel))
}
pub async fn send_batch(
client: &mut LogIngestClient<Channel>,
batch_id: String,
records: Vec<LogRecord>,
) -> Result<u32> {
let resp = client
.push_batch(PushBatchRequest { batch_id, records })
.await
.context("PushBatch RPC failed")?;
Ok(resp.into_inner().accepted)
}
+153
View File
@@ -0,0 +1,153 @@
mod batch;
mod config;
mod grpc;
mod source;
pub mod pb {
tonic::include_proto!("sentry.logs.v1");
}
use anyhow::{Context, Result};
use batch::Batcher;
use clap::Parser;
use config::Config;
use pb::{log_ingest_client::LogIngestClient, LogRecord, Severity};
use std::path::PathBuf;
use std::time::Duration;
use tokio::sync::mpsc;
use tonic::transport::Channel;
#[derive(Parser)]
#[command(name = "sentry-agent", about = "Sentry Linux log collector")]
struct Cli {
/// Path to a TOML config file. Defaults to /etc/sentry-agent/agent.toml
/// if present, otherwise built-in defaults (journald source, default
/// TLS cert paths under /etc/sentry-agent/).
#[arg(long)]
config: Option<PathBuf>,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
let cli = Cli::parse();
let cfg = Config::load(cli.config.as_deref()).context("loading config")?;
let host = cfg.agent.host.clone().unwrap_or_else(default_hostname);
let service = cfg.agent.service.clone();
let (tx, mut rx) = mpsc::channel(1024);
let source_handle = tokio::spawn(spawn_source(cfg.source.clone(), tx));
let mut client = grpc::connect(&cfg.ingest, &cfg.tls)
.await
.context("connecting to ingest service")?;
tracing::info!(endpoint = %cfg.ingest.endpoint, "connected to ingest service");
let flush_interval = Duration::from_millis(cfg.batch.flush_interval_ms);
let mut batcher = Batcher::new(cfg.batch.max_size, flush_interval);
let mut ticker = tokio::time::interval(flush_interval.max(Duration::from_millis(50)));
loop {
tokio::select! {
maybe_line = rx.recv() => {
let Some(raw) = maybe_line else {
tracing::warn!("source exited, flushing remaining batch and shutting down");
break;
};
let parsed = sentry_parser::parse(&raw.line);
let severity = to_pb_severity(raw.severity_hint.or(parsed.severity));
let record = LogRecord {
timestamp_unix_nano: raw.timestamp_unix_nano,
host: host.clone(),
service: service.clone(),
severity: severity as i32,
message: parsed.message,
attributes: parsed.attributes.into_iter().collect(),
};
if let Some(batch) = batcher.push(record) {
flush(&mut client, batch).await;
}
}
_ = ticker.tick() => {
if let Some(batch) = batcher.poll_timeout() {
flush(&mut client, batch).await;
}
}
}
}
if let Some(batch) = batcher.poll_timeout() {
flush(&mut client, batch).await;
}
source_handle.abort();
Ok(())
}
async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
let result = match source {
#[cfg(feature = "journald")]
config::SourceConfig::Journald { unit } => source::journald::run(unit.as_deref(), tx).await,
#[cfg(not(feature = "journald"))]
config::SourceConfig::Journald { .. } => {
Err(anyhow::anyhow!("this build was compiled without the `journald` feature"))
}
#[cfg(feature = "file-tail")]
config::SourceConfig::File { path, from_beginning } => {
source::file_tail::run(&path, from_beginning, tx).await
}
#[cfg(not(feature = "file-tail"))]
config::SourceConfig::File { .. } => {
Err(anyhow::anyhow!("this build was compiled without the `file-tail` feature"))
}
};
if let Err(e) = result {
tracing::error!(error = %e, "log source exited with error");
}
}
async fn flush(client: &mut LogIngestClient<Channel>, batch: Vec<LogRecord>) {
let n = batch.len();
let batch_id = batch_id();
match grpc::send_batch(client, batch_id, batch).await {
Ok(accepted) => tracing::debug!(accepted, sent = n, "batch flushed"),
Err(e) => tracing::error!(error = %e, sent = n, "batch flush failed"),
}
}
/// Best-effort batch identifier for ingest-side dedup on retry. Not
/// globally unique (host + nanosecond timestamp), which is good enough for
/// Phase 0's single-agent-per-host reality; revisit if agents ever share
/// an identity or clock resolution becomes a problem.
fn batch_id() -> String {
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
format!("{nanos:x}")
}
fn to_pb_severity(sev: Option<u8>) -> Severity {
match sev {
Some(0..=2) => Severity::Fatal, // emerg / alert / crit
Some(3) => Severity::Error, // err
Some(4) => Severity::Warn, // warning
Some(5) | Some(6) => Severity::Info, // notice / info
Some(7) => Severity::Debug, // debug
_ => Severity::Unspecified,
}
}
fn default_hostname() -> String {
if let Ok(s) = std::fs::read_to_string("/etc/hostname") {
let s = s.trim().to_string();
if !s.is_empty() {
return s;
}
}
std::env::var("HOSTNAME").unwrap_or_else(|_| "unknown-host".to_string())
}
@@ -0,0 +1,73 @@
use super::{LineSender, RawLine};
use anyhow::{Context, Result};
use std::io::SeekFrom;
use std::path::Path;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use tokio::fs::File;
use tokio::io::{AsyncBufReadExt, AsyncSeekExt, BufReader};
const POLL_INTERVAL: Duration = Duration::from_millis(500);
/// Polling-based file tailer: no inotify/`notify` crate dependency. Good
/// enough for Phase 0 (journald is the primary source). Handles basic
/// truncation (e.g. logrotate `copytruncate`) by detecting the file shrank
/// and reopening from the start. Does not follow rename-based rotation
/// (logrotate `create`) — that's deferred until file-tail is more than a
/// fallback path.
pub async fn run(path: &Path, from_beginning: bool, tx: LineSender) -> Result<()> {
let file = File::open(path)
.await
.with_context(|| format!("opening {}", path.display()))?;
let mut pos = if from_beginning { 0 } else { file.metadata().await?.len() };
let mut reader = BufReader::new(file);
reader.seek(SeekFrom::Start(pos)).await?;
let mut buf = String::new();
loop {
buf.clear();
let n = reader
.read_line(&mut buf)
.await
.context("reading line from file")?;
if n == 0 {
let metadata = tokio::fs::metadata(path).await.context("stat-ing file")?;
if metadata.len() < pos {
tracing::warn!(path = %path.display(), "file shrank, assuming truncation and reopening from start");
let f = File::open(path)
.await
.context("reopening file after truncation")?;
reader = BufReader::new(f);
pos = 0;
}
tokio::time::sleep(POLL_INTERVAL).await;
continue;
}
pos += n as u64;
let line = buf.trim_end_matches(['\n', '\r']).to_string();
if line.is_empty() {
continue;
}
let timestamp_unix_nano = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos() as i64)
.unwrap_or(0);
if tx
.send(RawLine {
line,
timestamp_unix_nano,
severity_hint: None,
})
.await
.is_err()
{
break;
}
}
Ok(())
}
+75
View File
@@ -0,0 +1,75 @@
use super::{LineSender, RawLine};
use anyhow::{Context, Result};
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::process::Command;
/// Reads journald entries by shelling out to `journalctl -f -o json`
/// rather than linking libsystemd via FFI. Linking libsystemd into a
/// statically-linked musl binary is fragile (it pulls in dbus/libcap
/// transitively and isn't designed for static linking) and would work
/// against the no-glibc-runtime-deps constraint in spirit even where it's
/// technically possible. `journalctl` ships on every systemd distro this
/// agent targets, so shelling out avoids the problem entirely. See
/// /docs/architecture.md.
pub async fn run(unit: Option<&str>, tx: LineSender) -> Result<()> {
let mut cmd = Command::new("journalctl");
cmd.arg("-f")
.arg("-o")
.arg("json")
.arg("--since=now")
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::null());
if let Some(unit) = unit {
cmd.arg("-u").arg(unit);
}
let mut child = cmd
.spawn()
.context("spawning journalctl -f -o json (is systemd-journal installed?)")?;
let stdout = child.stdout.take().context("journalctl child had no stdout")?;
let mut lines = BufReader::new(stdout).lines();
while let Some(line) = lines.next_line().await.context("reading journalctl output")? {
let Ok(entry) = serde_json::from_str::<serde_json::Value>(&line) else {
tracing::warn!(%line, "skipping unparseable journalctl JSON line");
continue;
};
let message = entry
.get("MESSAGE")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
if message.is_empty() {
continue;
}
let severity_hint = entry
.get("PRIORITY")
.and_then(|v| v.as_str().map(str::to_string).or_else(|| v.as_u64().map(|n| n.to_string())))
.and_then(|s| s.parse::<u8>().ok())
.filter(|&p| p <= 7);
let timestamp_unix_nano = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos() as i64)
.unwrap_or(0);
if tx
.send(RawLine {
line: message,
timestamp_unix_nano,
severity_hint,
})
.await
.is_err()
{
break; // receiver dropped, agent is shutting down
}
}
let status = child.wait().await.context("waiting for journalctl to exit")?;
tracing::warn!(?status, "journalctl exited");
Ok(())
}
+23
View File
@@ -0,0 +1,23 @@
use tokio::sync::mpsc;
/// A raw line read from a source, plus whatever metadata the source itself
/// already knows before the RFC 5424 parser ever sees it.
#[derive(Debug, Clone)]
pub struct RawLine {
pub line: String,
/// Unix epoch nanoseconds at time of read.
pub timestamp_unix_nano: i64,
/// Syslog severity (0-7) if the source already knows it independent of
/// the line's own content — e.g. journald's PRIORITY field. When set,
/// this takes precedence over whatever the RFC 5424 parser infers from
/// the message text, since it comes from a more authoritative place.
pub severity_hint: Option<u8>,
}
pub type LineSender = mpsc::Sender<RawLine>;
#[cfg(feature = "journald")]
pub mod journald;
#[cfg(feature = "file-tail")]
pub mod file_tail;
+6
View File
@@ -0,0 +1,6 @@
[package]
name = "sentry-parser"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "Minimal RFC 5424 syslog parser with raw-passthrough fallback"
+285
View File
@@ -0,0 +1,285 @@
//! Minimal RFC 5424 syslog parser with raw-passthrough fallback.
//!
//! This is intentionally not a complete RFC 5424 implementation (no BOM
//! handling on MSG, "-" nil markers are kept as literal strings rather than
//! mapped to `None`). It's the Phase 0 minimum: parse what's clearly
//! structured syslog, and never fail a log line outright — anything that
//! doesn't match the grammar becomes a raw passthrough record instead of
//! being dropped.
use std::collections::BTreeMap;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ParsedLine {
/// Syslog facility (0-23), present only when RFC 5424 framing parsed.
pub facility: Option<u8>,
/// Syslog severity (0=emergency .. 7=debug), present only when RFC 5424
/// framing parsed.
pub severity: Option<u8>,
/// Structured fields extracted from the PRI/HEADER/STRUCTURED-DATA
/// portions. Empty when the raw-passthrough fallback fires.
pub attributes: BTreeMap<String, String>,
/// The MSG portion when RFC 5424 parsing succeeded, otherwise the
/// original, unmodified line.
pub message: String,
}
/// Parse a single log line. Never fails: falls back to a raw passthrough
/// `ParsedLine` (no facility/severity, empty attributes, message = input)
/// when the line doesn't match RFC 5424 framing.
pub fn parse(line: &str) -> ParsedLine {
parse_rfc5424(line).unwrap_or_else(|| ParsedLine {
facility: None,
severity: None,
attributes: BTreeMap::new(),
message: line.to_string(),
})
}
struct Scanner<'a> {
chars: std::iter::Peekable<std::str::Chars<'a>>,
}
impl<'a> Scanner<'a> {
fn new(s: &'a str) -> Self {
Scanner {
chars: s.chars().peekable(),
}
}
fn peek(&mut self) -> Option<char> {
self.chars.peek().copied()
}
fn next(&mut self) -> Option<char> {
self.chars.next()
}
fn expect(&mut self, c: char) -> Option<()> {
if self.next()? == c {
Some(())
} else {
None
}
}
fn take_while<F: Fn(char) -> bool>(&mut self, f: F) -> String {
let mut out = String::new();
while let Some(c) = self.peek() {
if f(c) {
out.push(c);
self.next();
} else {
break;
}
}
out
}
fn skip_one_space(&mut self) -> Option<()> {
self.expect(' ')
}
}
fn parse_rfc5424(line: &str) -> Option<ParsedLine> {
let mut sc = Scanner::new(line);
sc.expect('<')?;
let pri_str = sc.take_while(|c| c.is_ascii_digit());
if pri_str.is_empty() || pri_str.len() > 3 {
return None;
}
sc.expect('>')?;
let pri: u16 = pri_str.parse().ok()?;
if pri > 191 {
return None;
}
let facility = (pri / 8) as u8;
let severity = (pri % 8) as u8;
let version = sc.take_while(|c| c.is_ascii_digit());
if version.is_empty() {
return None;
}
sc.skip_one_space()?;
let timestamp = sc.take_while(|c| c != ' ');
if timestamp.is_empty() {
return None;
}
sc.skip_one_space()?;
let hostname = sc.take_while(|c| c != ' ');
if hostname.is_empty() {
return None;
}
sc.skip_one_space()?;
let app_name = sc.take_while(|c| c != ' ');
if app_name.is_empty() {
return None;
}
sc.skip_one_space()?;
let procid = sc.take_while(|c| c != ' ');
if procid.is_empty() {
return None;
}
sc.skip_one_space()?;
let msgid = sc.take_while(|c| c != ' ');
if msgid.is_empty() {
return None;
}
sc.skip_one_space()?;
let mut sd_pairs: Vec<(String, String, String)> = Vec::new();
match sc.peek() {
Some('-') => {
sc.next();
}
Some('[') => loop {
if sc.peek() != Some('[') {
break;
}
sc.next();
let sd_id = sc.take_while(|c| c != ' ' && c != ']');
if sd_id.is_empty() {
return None;
}
loop {
match sc.peek() {
Some(' ') => {
sc.next();
let name = sc.take_while(|c| c != '=');
sc.expect('=')?;
sc.expect('"')?;
let mut val = String::new();
loop {
match sc.next() {
Some('\\') => val.push(sc.next()?),
Some('"') => break,
Some(c) => val.push(c),
None => return None,
}
}
sd_pairs.push((sd_id.clone(), name, val));
}
Some(']') => {
sc.next();
break;
}
_ => return None,
}
}
},
_ => return None,
}
let message = if sc.peek() == Some(' ') {
sc.next();
sc.take_while(|_| true)
} else {
String::new()
};
let mut attributes = BTreeMap::new();
attributes.insert("syslog.version".to_string(), version);
attributes.insert("syslog.timestamp".to_string(), timestamp);
attributes.insert("syslog.hostname".to_string(), hostname);
attributes.insert("syslog.app_name".to_string(), app_name);
attributes.insert("syslog.procid".to_string(), procid);
attributes.insert("syslog.msgid".to_string(), msgid);
for (sd_id, name, val) in sd_pairs {
attributes.insert(format!("{sd_id}.{name}"), val);
}
Some(ParsedLine {
facility: Some(facility),
severity: Some(severity),
attributes,
message,
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_full_rfc5424_with_structured_data() {
let line = r#"<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslp - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry"#;
let p = parse(line);
assert_eq!(p.facility, Some(20));
assert_eq!(p.severity, Some(5));
assert_eq!(p.message, "An application event log entry");
assert_eq!(
p.attributes.get("[email protected]"),
Some(&"3".to_string())
);
assert_eq!(
p.attributes.get("[email protected]"),
Some(&"Application".to_string())
);
assert_eq!(
p.attributes.get("syslog.hostname"),
Some(&"mymachine.example.com".to_string())
);
}
#[test]
fn parses_nil_structured_data_and_fields() {
let line = "<34>1 2003-10-11T22:14:15.003Z mymachine su - ID47 - 'su root' failed";
let p = parse(line);
assert_eq!(p.facility, Some(4));
assert_eq!(p.severity, Some(2));
assert_eq!(p.attributes.get("syslog.procid"), Some(&"-".to_string()));
assert_eq!(p.message, "'su root' failed");
}
#[test]
fn parses_multiple_structured_data_elements() {
let line = r#"<165>1 2003-10-11T22:14:15.003Z host app - ID47 [a@1 k="v"][b@1 k2="v2"] msg"#;
let p = parse(line);
assert_eq!(p.attributes.get("[email protected]"), Some(&"v".to_string()));
assert_eq!(p.attributes.get("[email protected]"), Some(&"v2".to_string()));
assert_eq!(p.message, "msg");
}
#[test]
fn handles_escaped_quote_in_param_value() {
let line = r#"<165>1 2003-10-11T22:14:15.003Z host app - ID47 [a@1 k="has \"quote\" inside"] msg"#;
let p = parse(line);
assert_eq!(
p.attributes.get("[email protected]"),
Some(&"has \"quote\" inside".to_string())
);
}
#[test]
fn falls_back_to_raw_passthrough_for_non_syslog_line() {
let line = "this is just a plain log line, not syslog at all";
let p = parse(line);
assert_eq!(p.facility, None);
assert_eq!(p.severity, None);
assert!(p.attributes.is_empty());
assert_eq!(p.message, line);
}
#[test]
fn falls_back_to_raw_passthrough_for_malformed_pri() {
let line = "<abc>1 2003-10-11T22:14:15.003Z host app - ID47 - msg";
let p = parse(line);
assert_eq!(p.facility, None);
assert_eq!(p.message, line);
}
#[test]
fn falls_back_when_structured_data_missing() {
// Missing the required "-" or "[...]" for STRUCTURED-DATA.
let line = "<34>1 2003-10-11T22:14:15.003Z host app 123 ID47";
let p = parse(line);
assert_eq!(p.facility, None);
assert_eq!(p.message, line);
}
}
+13
View File
@@ -0,0 +1,13 @@
# Build context must be the repo root (sentry/):
# docker build -f api/Dockerfile -t sentry-api .
FROM golang:1.25-alpine AS builder
WORKDIR /src
COPY api ./api
WORKDIR /src/api
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/api ./cmd/api
FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/api /api
ENTRYPOINT ["/api"]
+63
View File
@@ -0,0 +1,63 @@
# api
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
## Why plain REST, not gRPC + REST gateway
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
service is plain `net/http` instead — a deliberate Phase 0 simplification,
not a change to the pinned stack. Wiring up a `.proto` service,
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
single endpoint that Phase 2 replaces outright with a real SPL-like query
layer would be exactly the kind of premature machinery this project's
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
a second real, durable endpoint.
## Endpoints
- `POST /query` — body `{"sql": "SELECT ..."}`, response
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
SELECT-only, single-statement, basic keyword-based injection guarding
(see `internal/queryapi/validate.go` for exactly what that does and
doesn't catch — it's not a SQL parser).
- `GET /healthz` — for docker-compose/k8s liveness checks.
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
dev/homelab network.
## Configuration
Environment variables (see `internal/config/config.go`):
| Var | Default | Purpose |
|---|---|---|
| `HTTP_LISTEN_ADDR` | `:8080` | |
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
## Building & testing
```sh
go build ./...
go vet ./...
go test ./...
```
```sh
# from the repo root, not api/
docker build -f api/Dockerfile -t sentry-api .
```
## Testing notes
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
one-method `queryExecutor` interface, so routing, validation, JSON
encoding, and error-status mapping are all unit-tested against a fake —
no live ClickHouse needed. `Executor` itself (the reflection-based row
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
`driver.Rows` interface fully would be significant test-only scaffolding
for a Phase 0 placeholder, and the driver package's own docs note it isn't
meant to be implemented by adopters. It's exercised end-to-end via the
docker-compose flow in `/docs/phase-0-runbook.md` instead.
+80
View File
@@ -0,0 +1,80 @@
// Command api is the Sentry Phase 0 query API: a single crude POST /query
// endpoint proxying allowlisted SELECT statements to ClickHouse. See
// internal/queryapi for why this is plain REST rather than the pinned
// gRPC+gateway pattern for Phase 0.
package main
import (
"context"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/sentry/sentry/api/internal/config"
"github.com/sentry/sentry/api/internal/queryapi"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("loading config", "error", err)
os.Exit(1)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{cfg.ClickHouse.Addr},
Auth: clickhouse.Auth{
Database: cfg.ClickHouse.Database,
Username: cfg.ClickHouse.Username,
Password: cfg.ClickHouse.Password,
},
})
if err != nil {
logger.Error("opening clickhouse connection", "error", err)
os.Exit(1)
}
defer conn.Close()
if err := conn.Ping(ctx); err != nil {
logger.Error("pinging clickhouse", "error", err)
os.Exit(1)
}
exec := queryapi.NewExecutor(conn)
handler := queryapi.NewHandler(logger, exec, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
srv := &http.Server{
Addr: cfg.HTTPListenAddr,
Handler: handler.Routes(),
}
errCh := make(chan error, 1)
go func() {
logger.Info("api listening", "addr", cfg.HTTPListenAddr)
errCh <- srv.ListenAndServe()
}()
select {
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
logger.Error("graceful shutdown failed", "error", err)
}
case err := <-errCh:
if err != nil && err != http.ErrServerClosed {
logger.Error("server exited with error", "error", err)
os.Exit(1)
}
}
}
+22
View File
@@ -0,0 +1,22 @@
module github.com/sentry/sentry/api
go 1.25.0
require github.com/ClickHouse/clickhouse-go/v2 v2.48.0
require (
github.com/ClickHouse/ch-go v0.74.0 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.19.1 // indirect
github.com/paulmach/orb v0.13.0 // indirect
github.com/pierrec/lz4/v4 v4.1.27 // indirect
github.com/segmentio/asm v1.2.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/sys v0.47.0 // indirect
)
+42
View File
@@ -0,0 +1,42 @@
github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE=
github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M=
github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4=
github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM=
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+56
View File
@@ -0,0 +1,56 @@
// Package config loads api's configuration from environment variables,
// same convention as /ingest: no config file format for Phase 0.
package config
import (
"fmt"
"os"
"strconv"
"time"
)
type Config struct {
HTTPListenAddr string
ClickHouse ClickHouseConfig
QueryTimeout time.Duration
CORSAllowedOrigin string
}
type ClickHouseConfig struct {
Addr string
Database string
Username string
Password string
}
func Load() (Config, error) {
cfg := Config{
HTTPListenAddr: getenv("HTTP_LISTEN_ADDR", ":8080"),
ClickHouse: ClickHouseConfig{
Addr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
Database: getenv("CLICKHOUSE_DATABASE", "sentry"),
Username: getenv("CLICKHOUSE_USERNAME", "default"),
Password: getenv("CLICKHOUSE_PASSWORD", ""),
},
// Phase 0 has no auth, so this is wide open by default to keep
// the local SvelteKit dev server (a different origin/port)
// working out of the box. Tighten before this is ever reachable
// from outside a trusted dev/homelab network.
CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"),
}
timeoutSec, err := strconv.Atoi(getenv("QUERY_TIMEOUT_SECONDS", "30"))
if err != nil {
return Config{}, fmt.Errorf("QUERY_TIMEOUT_SECONDS: %w", err)
}
cfg.QueryTimeout = time.Duration(timeoutSec) * time.Second
return cfg, nil
}
func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
+29
View File
@@ -0,0 +1,29 @@
package config
import (
"testing"
"time"
)
func TestLoadDefaults(t *testing.T) {
cfg, err := Load()
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.HTTPListenAddr != ":8080" {
t.Errorf("HTTPListenAddr = %q, want :8080", cfg.HTTPListenAddr)
}
if cfg.QueryTimeout != 30*time.Second {
t.Errorf("QueryTimeout = %v, want 30s", cfg.QueryTimeout)
}
if cfg.CORSAllowedOrigin != "*" {
t.Errorf("CORSAllowedOrigin = %q, want *", cfg.CORSAllowedOrigin)
}
}
func TestLoadInvalidTimeoutErrors(t *testing.T) {
t.Setenv("QUERY_TIMEOUT_SECONDS", "not-a-number")
if _, err := Load(); err == nil {
t.Fatal("expected error for non-numeric QUERY_TIMEOUT_SECONDS, got nil")
}
}
+60
View File
@@ -0,0 +1,60 @@
package queryapi
import (
"context"
"fmt"
"reflect"
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
)
type QueryResult struct {
Columns []string `json:"columns"`
Rows [][]any `json:"rows"`
}
// Executor runs arbitrary (pre-validated) SELECT statements against
// ClickHouse and shapes the result into JSON-friendly columns/rows,
// discovering the result's column set at query time via reflection since
// the query itself is arbitrary.
type Executor struct {
conn driver.Conn
}
func NewExecutor(conn driver.Conn) *Executor {
return &Executor{conn: conn}
}
func (e *Executor) Execute(ctx context.Context, sql string) (*QueryResult, error) {
rows, err := e.conn.Query(ctx, sql)
if err != nil {
return nil, fmt.Errorf("executing query: %w", err)
}
defer rows.Close()
columnTypes := rows.ColumnTypes()
result := &QueryResult{
Columns: rows.Columns(),
Rows: [][]any{},
}
for rows.Next() {
dest := make([]any, len(columnTypes))
for i, ct := range columnTypes {
dest[i] = reflect.New(ct.ScanType()).Interface()
}
if err := rows.Scan(dest...); err != nil {
return nil, fmt.Errorf("scanning row: %w", err)
}
row := make([]any, len(dest))
for i, d := range dest {
row[i] = reflect.ValueOf(d).Elem().Interface()
}
result.Rows = append(result.Rows, row)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating rows: %w", err)
}
return result, nil
}
+112
View File
@@ -0,0 +1,112 @@
// Package queryapi is the Phase 0 query API: a single crude POST /query
// endpoint that takes a raw SQL string, allowlists it to a single SELECT
// statement, and proxies it to ClickHouse. This is a deliberate
// simplification of the pinned "gRPC + REST gateway" control-plane
// pattern (see CLAUDE.md's tech stack table): a plain net/http REST
// handler, not a gRPC service transcoded through grpc-gateway. That
// machinery (proto definitions, googleapis annotations, gateway codegen)
// buys nothing for one crude placeholder endpoint that Phase 2 replaces
// outright with the real SPL-like query layer. Revisit gRPC+gateway when
// /api grows a second real endpoint.
package queryapi
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"time"
)
// queryExecutor is the narrow interface handleQuery depends on, so tests
// can substitute a fake without a real ClickHouse connection. *Executor
// satisfies it.
type queryExecutor interface {
Execute(ctx context.Context, sql string) (*QueryResult, error)
}
type Handler struct {
logger *slog.Logger
exec queryExecutor
queryTimeout time.Duration
allowedOrigin string
}
func NewHandler(logger *slog.Logger, exec queryExecutor, queryTimeout time.Duration, allowedOrigin string) *Handler {
return &Handler{logger: logger, exec: exec, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
}
func (h *Handler) Routes() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("POST /query", h.handleQuery)
mux.HandleFunc("GET /healthz", h.handleHealthz)
return h.withCORS(mux)
}
// withCORS is deliberately permissive by default (see CORSAllowedOrigin in
// internal/config) since Phase 0 has no auth and the SvelteKit dev server
// runs on a different origin. Tighten alongside adding real auth.
func (h *Handler) withCORS(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", h.allowedOrigin)
w.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
if r.Method == http.MethodOptions {
w.WriteHeader(http.StatusNoContent)
return
}
next.ServeHTTP(w, r)
})
}
func (h *Handler) handleHealthz(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}
type queryRequest struct {
SQL string `json:"sql"`
}
type errorResponse struct {
Error string `json:"error"`
}
// maxBodyBytes caps the request body: a raw SQL string has no legitimate
// reason to be larger than this.
const maxBodyBytes = 1 << 20 // 1 MiB
func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
var req queryRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
return
}
if err := validateSelectOnly(req.SQL); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
defer cancel()
result, err := h.exec.Execute(ctx, req.SQL)
if err != nil {
h.logger.Error("query execution failed", "error", err)
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
return
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(result); err != nil {
h.logger.Error("encoding response", "error", err)
}
}
func writeError(w http.ResponseWriter, status int, msg string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(errorResponse{Error: msg})
}
+134
View File
@@ -0,0 +1,134 @@
package queryapi
import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
type fakeExecutor struct {
result *QueryResult
err error
gotSQL string
}
func (f *fakeExecutor) Execute(_ context.Context, sql string) (*QueryResult, error) {
f.gotSQL = sql
if f.err != nil {
return nil, f.err
}
return f.result, nil
}
func newTestHandler(exec queryExecutor) *Handler {
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, time.Second, "*")
}
func TestHandleQuerySuccess(t *testing.T) {
fe := &fakeExecutor{result: &QueryResult{
Columns: []string{"host", "count"},
Rows: [][]any{{"h1", 3}},
}}
h := newTestHandler(fe)
body := strings.NewReader(`{"sql": "SELECT host, count(*) FROM logs GROUP BY host"}`)
req := httptest.NewRequest(http.MethodPost, "/query", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
var got QueryResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decoding response: %v", err)
}
if len(got.Columns) != 2 || len(got.Rows) != 1 {
t.Fatalf("unexpected result: %+v", got)
}
if fe.gotSQL != "SELECT host, count(*) FROM logs GROUP BY host" {
t.Fatalf("executor received unexpected SQL: %q", fe.gotSQL)
}
}
func TestHandleQueryRejectsNonSelect(t *testing.T) {
fe := &fakeExecutor{}
h := newTestHandler(fe)
body := strings.NewReader(`{"sql": "DELETE FROM logs"}`)
req := httptest.NewRequest(http.MethodPost, "/query", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rec.Code)
}
if fe.gotSQL != "" {
t.Fatal("executor should not have been called for a rejected query")
}
}
func TestHandleQueryRejectsInvalidJSON(t *testing.T) {
h := newTestHandler(&fakeExecutor{})
body := strings.NewReader(`not json`)
req := httptest.NewRequest(http.MethodPost, "/query", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rec.Code)
}
}
func TestHandleQueryExecutorErrorReturnsBadGateway(t *testing.T) {
fe := &fakeExecutor{err: errors.New("boom")}
h := newTestHandler(fe)
body := strings.NewReader(`{"sql": "SELECT 1"}`)
req := httptest.NewRequest(http.MethodPost, "/query", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502", rec.Code)
}
}
func TestHandleHealthz(t *testing.T) {
h := newTestHandler(&fakeExecutor{})
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
}
func TestCORSPreflight(t *testing.T) {
h := newTestHandler(&fakeExecutor{})
req := httptest.NewRequest(http.MethodOptions, "/query", nil)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusNoContent {
t.Fatalf("status = %d, want 204", rec.Code)
}
if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {
t.Fatalf("Access-Control-Allow-Origin = %q, want *", got)
}
}
+47
View File
@@ -0,0 +1,47 @@
package queryapi
import (
"errors"
"regexp"
"strings"
)
// disallowedKeyword is defense-in-depth on top of the SELECT-only gate: it
// catches mutating/administrative statements appearing anywhere in the
// query (e.g. smuggled into a subquery), not just at the start. This is
// word-boundary matching, not a real SQL parser.
var disallowedKeyword = regexp.MustCompile(`(?i)\b(insert|update|delete|alter|drop|truncate|create|grant|revoke|attach|detach|rename|kill|optimize|system|set|exchange|watch)\b`)
// validateSelectOnly enforces the Phase 0 query API contract: exactly one
// SELECT statement and nothing else. This is "basic injection guarding" as
// specced, not a SQL parser: it will reject some unusual-but-valid SELECTs
// (e.g. one that references a column literally named "delete") and will
// not catch every possible abuse (e.g. a syntactically pure SELECT that's
// simply expensive to run). Both are acceptable for a Phase 0 placeholder
// that's explicitly superseded by a real query layer in Phase 2 — see
// /docs/architecture.md.
func validateSelectOnly(sql string) error {
trimmed := strings.TrimSpace(sql)
if trimmed == "" {
return errors.New("query must not be empty")
}
trimmed = strings.TrimSpace(strings.TrimSuffix(trimmed, ";"))
if trimmed == "" {
return errors.New("query must not be empty")
}
if strings.Contains(trimmed, ";") {
return errors.New("only a single statement is allowed")
}
firstWord := strings.ToUpper(strings.Fields(trimmed)[0])
if firstWord != "SELECT" {
return errors.New("only SELECT queries are allowed")
}
if disallowedKeyword.MatchString(trimmed) {
return errors.New("query contains a disallowed keyword")
}
return nil
}
+38
View File
@@ -0,0 +1,38 @@
package queryapi
import "testing"
func TestValidateSelectOnly(t *testing.T) {
cases := []struct {
name string
sql string
wantErr bool
}{
{"plain select", "SELECT * FROM logs LIMIT 10", false},
{"lowercase select", "select service, count(*) from logs group by service", false},
{"trailing semicolon allowed", "SELECT 1;", false},
{"trailing semicolon and whitespace allowed", "SELECT 1; ", false},
{"empty", "", true},
{"whitespace only", " ", true},
{"only a semicolon", ";", true},
{"multiple statements", "SELECT 1; SELECT 2", true},
{"insert", "INSERT INTO logs VALUES (1)", true},
{"delete", "DELETE FROM logs", true},
{"drop", "DROP TABLE logs", true},
{"select with drop keyword smuggled in", "SELECT * FROM logs WHERE message = 'DROP TABLE logs'", true},
{"non-select start", "WITH x AS (SELECT 1) SELECT * FROM x", true},
{"trailing garbage after semicolon", "SELECT 1; DROP TABLE logs", true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := validateSelectOnly(tc.sql)
if tc.wantErr && err == nil {
t.Errorf("validateSelectOnly(%q) = nil, want error", tc.sql)
}
if !tc.wantErr && err != nil {
t.Errorf("validateSelectOnly(%q) = %v, want nil", tc.sql, err)
}
})
}
}
+9
View File
@@ -0,0 +1,9 @@
# docker build -f cli/Dockerfile -t sentryctl cli/
FROM golang:1.25-alpine AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/sentryctl ./cmd/sentryctl
FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/sentryctl /sentryctl
ENTRYPOINT ["/sentryctl"]
+28
View File
@@ -0,0 +1,28 @@
# sentryctl
Sentry's control CLI. Phase 0: a single command.
```sh
sentryctl ping # checks http://localhost:8080/healthz
sentryctl ping --api http://api.internal:8080
SENTRYCTL_API_URL=http://api.internal:8080 sentryctl ping
```
Exits 0 and prints `ok` if `/api`'s `/healthz` responds 200; exits 1 with an
error on `stderr` otherwise.
No CLI framework (cobra/urfave-cli/etc.) — a single command doesn't need
one, and stdlib `os.Args` handling is boring enough not to need a
dependency. Revisit once there's a real command tree to justify one.
## Building & testing
```sh
go build ./...
go vet ./...
go test ./...
```
```sh
docker build -f Dockerfile -t sentryctl . # context is cli/, not the repo root
```
+87
View File
@@ -0,0 +1,87 @@
// Command sentryctl is Sentry's control CLI. Phase 0: a single "ping"
// command that checks the api service is reachable. More commands land as
// the control plane grows real operations to expose.
package main
import (
"fmt"
"io"
"net/http"
"os"
"time"
)
const defaultAPIURL = "http://localhost:8080"
func main() {
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
}
func run(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
usage(stderr)
return 1
}
switch args[0] {
case "ping":
return cmdPing(args[1:], stdout, stderr)
case "-h", "--help", "help":
usage(stdout)
return 0
default:
fmt.Fprintf(stderr, "sentryctl: unknown command %q\n", args[0])
usage(stderr)
return 1
}
}
func usage(w io.Writer) {
fmt.Fprintln(w, `sentryctl: Sentry control CLI (Phase 0: ping only)
Usage:
sentryctl ping [--api <url>]
Commands:
ping Checks that the api service is reachable via GET /healthz.
--api defaults to $SENTRYCTL_API_URL, or `+defaultAPIURL+` if unset.`)
}
// parsePingArgs resolves the api base URL for ping: --api flag wins, then
// $SENTRYCTL_API_URL, then the hardcoded default. Kept pure (env passed in
// as a function) and separate from the HTTP call so it's unit-testable
// without a real environment or server.
func parsePingArgs(args []string, env func(string) string) string {
apiURL := env("SENTRYCTL_API_URL")
if apiURL == "" {
apiURL = defaultAPIURL
}
for i := 0; i < len(args); i++ {
if args[i] == "--api" && i+1 < len(args) {
apiURL = args[i+1]
i++
}
}
return apiURL
}
func cmdPing(args []string, stdout, stderr io.Writer) int {
apiURL := parsePingArgs(args, os.Getenv)
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Get(apiURL + "/healthz")
if err != nil {
fmt.Fprintf(stderr, "ping failed: %v\n", err)
return 1
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
fmt.Fprintf(stderr, "ping failed: api returned status %d\n", resp.StatusCode)
return 1
}
fmt.Fprintln(stdout, "ok")
return 0
}
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"bytes"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestParsePingArgsDefault(t *testing.T) {
env := func(string) string { return "" }
if got := parsePingArgs(nil, env); got != defaultAPIURL {
t.Errorf("got %q, want %q", got, defaultAPIURL)
}
}
func TestParsePingArgsFromEnv(t *testing.T) {
env := func(k string) string {
if k == "SENTRYCTL_API_URL" {
return "http://env-host:1234"
}
return ""
}
if got := parsePingArgs(nil, env); got != "http://env-host:1234" {
t.Errorf("got %q, want env value", got)
}
}
func TestParsePingArgsFlagOverridesEnv(t *testing.T) {
env := func(k string) string {
if k == "SENTRYCTL_API_URL" {
return "http://env-host:1234"
}
return ""
}
got := parsePingArgs([]string{"--api", "http://flag-host:5678"}, env)
if got != "http://flag-host:5678" {
t.Errorf("got %q, want flag value", got)
}
}
func TestCmdPingSuccess(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/healthz" {
t.Errorf("unexpected path %q", r.URL.Path)
}
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
var stdout, stderr bytes.Buffer
code := cmdPing([]string{"--api", srv.URL}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%s", code, stderr.String())
}
if strings.TrimSpace(stdout.String()) != "ok" {
t.Fatalf("stdout = %q, want ok", stdout.String())
}
}
func TestCmdPingNonOKStatus(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
defer srv.Close()
var stdout, stderr bytes.Buffer
code := cmdPing([]string{"--api", srv.URL}, &stdout, &stderr)
if code != 1 {
t.Fatalf("exit code = %d, want 1", code)
}
if !strings.Contains(stderr.String(), "503") {
t.Fatalf("stderr = %q, want it to mention the status code", stderr.String())
}
}
func TestCmdPingUnreachable(t *testing.T) {
var stdout, stderr bytes.Buffer
code := cmdPing([]string{"--api", "http://127.0.0.1:1"}, &stdout, &stderr)
if code != 1 {
t.Fatalf("exit code = %d, want 1", code)
}
}
func TestRunNoArgsPrintsUsageAndFails(t *testing.T) {
var stdout, stderr bytes.Buffer
code := run(nil, &stdout, &stderr)
if code != 1 {
t.Fatalf("exit code = %d, want 1", code)
}
if !strings.Contains(stderr.String(), "Usage:") {
t.Fatalf("stderr should contain usage text, got %q", stderr.String())
}
}
func TestRunUnknownCommand(t *testing.T) {
var stdout, stderr bytes.Buffer
code := run([]string{"bogus"}, &stdout, &stderr)
if code != 1 {
t.Fatalf("exit code = %d, want 1", code)
}
if !strings.Contains(stderr.String(), "bogus") {
t.Fatalf("stderr should mention the unknown command, got %q", stderr.String())
}
}
func TestRunHelp(t *testing.T) {
var stdout, stderr bytes.Buffer
code := run([]string{"help"}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0", code)
}
if !strings.Contains(stdout.String(), "Usage:") {
t.Fatalf("stdout should contain usage text, got %q", stdout.String())
}
}
+3
View File
@@ -0,0 +1,3 @@
module github.com/sentry/sentry/cli
go 1.25
+128
View File
@@ -0,0 +1,128 @@
# Phase 0 stack: Redpanda -> ingest -> ClickHouse -> api -> web.
#
# Does NOT include the Rust agent — see /agent/README.md: journald
# sourcing needs the host's journal, which isn't something a container
# gets for free. Run the agent natively on the host per
# /docs/phase-0-runbook.md, pointed at ingest's mapped port (localhost:4317).
#
# Before first run: generate dev mTLS certs (hack/dev-certs/generate.sh).
# See /docs/phase-0-runbook.md for the full sequence.
services:
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
container_name: sentry-redpanda
command:
- redpanda
- start
- --smp=1
- --memory=1G
- --reserve-memory=0M
- --overprovisioned
- --node-id=0
- --check=false
- --kafka-addr=PLAINTEXT://0.0.0.0:9092
- --advertise-kafka-addr=PLAINTEXT://redpanda:9092
ports:
- "9092:9092"
volumes:
- redpanda-data:/var/lib/redpanda/data
healthcheck:
test: ["CMD", "rpk", "cluster", "health", "--exit-when-healthy"]
interval: 5s
timeout: 5s
retries: 30
# One-shot: creates the sentry.logs.raw topic, then exits 0. ingest
# waits on this completing successfully before it starts.
redpanda-provision:
build:
context: ./transport
container_name: sentry-redpanda-provision
depends_on:
redpanda:
condition: service_healthy
environment:
REDPANDA_BROKERS: "redpanda:9092"
clickhouse:
image: clickhouse/clickhouse-server:24.8
container_name: sentry-clickhouse
ports:
- "8123:8123" # HTTP interface, used by the migrate step
- "9000:9000" # native protocol, used by ingest and api
volumes:
- clickhouse-data:/var/lib/clickhouse
ulimits:
nofile:
soft: 262144
hard: 262144
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"]
interval: 5s
timeout: 5s
retries: 30
# One-shot: applies /storage/migrations/*.sql, then exits 0. ingest and
# api both wait on this completing successfully.
clickhouse-migrate:
build:
context: ./storage
container_name: sentry-clickhouse-migrate
depends_on:
clickhouse:
condition: service_healthy
environment:
CLICKHOUSE_HTTP: "http://clickhouse:8123"
ingest:
build:
context: . # needs both ingest/ and proto/
dockerfile: ingest/Dockerfile
container_name: sentry-ingest
depends_on:
redpanda-provision:
condition: service_completed_successfully
clickhouse-migrate:
condition: service_completed_successfully
ports:
- "4317:4317" # gRPC, mTLS — this is what the host-run agent connects to
environment:
REDPANDA_BROKERS: "redpanda:9092"
CLICKHOUSE_ADDR: "clickhouse:9000"
# TLS_*_FILE env vars are left at their defaults
# (/etc/sentry-ingest/{server,server-key,ca}.pem) — matches where
# the volume below mounts the generated dev certs.
volumes:
- ./hack/dev-certs/out:/etc/sentry-ingest:ro
api:
build:
context: .
dockerfile: api/Dockerfile
container_name: sentry-api
depends_on:
clickhouse-migrate:
condition: service_completed_successfully
ports:
- "8080:8080"
environment:
CLICKHOUSE_ADDR: "clickhouse:9000"
web:
build:
context: web
args:
# Baked in at build time (static site, not a server) as
# localhost:8080 -- this is fetched from the *browser*, which
# resolves against the host's mapped port, not the compose
# network's service DNS name.
VITE_API_BASE_URL: "http://localhost:8080"
container_name: sentry-web
depends_on:
- api
ports:
- "3000:3000"
volumes:
redpanda-data:
clickhouse-data:
+105
View File
@@ -0,0 +1,105 @@
# Sentry Architecture
> **Status:** Draft, Phase 0 scope. Written from the project constraints and
> task list at kickoff, not transcribed from a pre-existing spec. Treat as a
> starting point to correct, not a settled design — flag anything that
> doesn't match your intent before implementation leans on it further.
## Mission
Open-core, Kubernetes-native centralized logging platform. Compete with
Splunk on features; win on cost-per-GB, a modern language stack, and
multi-tenant RBAC that's actually honest about its guarantees.
## Component map
```
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
└──────┬─────┘
│ batch INSERT
┌────────────┐
│ ClickHouse │
└─────┬──────┘
│ SQL
┌───────────▼───────────┐
│ api (Go: gRPC+REST) │
└───────────┬───────────┘
│ REST
┌───────────▼───────────┐
│ web (SvelteKit) │
└────────────────────────┘
```
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
`ingest` service's gRPC front end produces to Redpanda rather than writing
ClickHouse directly; a separate consumer path reads from Redpanda and batches
inserts into ClickHouse. This exercises the real transport layer from day
one instead of deferring it, and keeps Kafka credentials off the edge agent.
## Storage / query split
- **ClickHouse** is the analytical store of record for structured log data:
timestamp, host, service, severity, message, plus a `Map(String,String)`
for arbitrary structured fields. Partitioned by day, ordered by
`(service, timestamp)`.
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
field and unstructured payloads, queried out-of-band from ClickHouse and
joined by a log identifier. Not built in Phase 0.
- **Schema-on-write** using OTel semantic conventions as the default log
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
the structured columns (captured via the `Map` column and/or a raw
passthrough field).
This split is not to be changed without discussion — see CLAUDE.md.
## Component responsibilities (Phase 0)
| Component | Responsibility |
|---|---|
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. |
| `proto` | Shared `.proto` contracts for the agent↔ingest gRPC service, versioned independently of either component. |
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. |
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
| `api` (Go) | gRPC + REST gateway. Phase 0: one crude `POST /query` endpoint, SELECT-only, proxying to ClickHouse. Real SPL-like query layer is Phase 2. |
| `web` (SvelteKit) | Single page: SQL text box, submit, results table. No auth, no styling polish. |
| `cli` (`sentryctl`) | Stub. Single `ping` command for now. |
| `deploy` | Helm charts, k8s manifests. Stubbed in Phase 0; docker-compose is the real local/dev path. |
## Licensing boundary
AGPLv3 for core + agents. Enterprise features (SSO, multi-tenancy,
compliance) live under `enterprise/` (not yet created — out of scope for
Phase 0) under a commercial license stub. AGPL code must never import from
`enterprise/`. No enterprise-gated code exists yet in this repo; this
section documents the boundary so nothing added later crosses it by
accident.
## Non-negotiables carried from CLAUDE.md
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
`aarch64-unknown-linux-musl`, no glibc runtime deps.
- Windows support (Phase 1+) via native ETW/Event Log API, not WSL.
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
## Explicitly out of scope for Phase 0
Windows agent, alerting, dashboards, multi-tenancy, Tantivy full-text
search, the real SPL-like query language, enterprise module code.
## Open questions for you to resolve
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
deferred until storage sizing is a real concern.
- Exact OTel log schema field mapping (which OTel resource/log attributes
map to which ClickHouse columns) — Phase 0 uses a minimal subset
(timestamp, host, service, severity, message, attributes map); full
mapping deferred.
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
a static dev CA and manually issued certs; production PKI design is
out of scope here.
+207
View File
@@ -0,0 +1,207 @@
# Phase 0 runbook
Walks one log line from a Linux host, through the Rust agent, Redpanda,
ingest, and ClickHouse, to a browser table. This is the actual
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
regardless of what any individual component's tests say.
**This sequence has not been run end-to-end** in the environment that
built it (no Docker available there — see the caveats each component's
summary already flagged). Individual pieces are unit-tested and built
successfully in isolation; this document is the logical sequence to run
for real, not a report that it's been run. Expect to debug something on
first attempt, and treat the "Troubleshooting" section at the bottom as a
starting point, not an exhaustive list.
## Prerequisites
- Docker with **Compose v2** (`docker compose`, not the legacy
`docker-compose` v1 binary) — the compose file uses
`service_completed_successfully` conditions that v1 doesn't support.
- Rust toolchain (`cargo`) and `protoc` — to build the agent.
- `openssl` — to generate dev mTLS certs.
- A systemd-based Linux host to run the agent on (journald is the default
source). If you're not on such a host, see `/agent/README.md`'s
`file-tail` feature as an alternative source.
You do **not** need the musl cross-compilation target for this runbook —
that's for producing the distro-agnostic release binary. A native
`cargo build --release` is enough to run the agent on the same machine
you're testing on.
## 1. Generate dev mTLS certs
```sh
./hack/dev-certs/generate.sh
```
Writes a throwaway CA plus a server cert (for `ingest`) and a client cert
(for the agent) to `hack/dev-certs/out/`. Dev-only — see the script's
header comment for why.
## 2. Bring up the backend stack
```sh
docker compose up -d --build
```
This builds and starts, in dependency order: `redpanda``redpanda-provision`
(creates the `sentry.logs.raw` topic, then exits) → `clickhouse`
`clickhouse-migrate` (applies `/storage/migrations`, then exits) →
`ingest` and `api``web`.
Check everything came up:
```sh
docker compose ps
```
`redpanda-provision` and `clickhouse-migrate` should show `Exited (0)`
(one-shot jobs, not long-running). Everything else should show `Up` /
`healthy`.
If `ingest` or `api` crash-looped, they likely started before their
`depends_on` conditions were actually satisfied, or the dev certs from
step 1 don't exist yet — check `docker compose logs ingest`.
## 3. Sanity-check the backend before involving the agent
```sh
curl http://localhost:8080/healthz
# -> 200, empty body
curl -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"sql": "SELECT 1"}'
# -> {"columns":["1"],"rows":[[1]]} (exact column name may vary by ClickHouse version)
```
This confirms `api` can reach `clickhouse` before you go looking for bugs
anywhere else. It doesn't touch the `logs` table, so it works even before
any agent has sent data.
## 4. Install the agent's mTLS material
The agent's default config expects certs at `/etc/sentry-agent/` (see
`/agent/config/agent.example.toml`), which requires root:
```sh
sudo mkdir -p /etc/sentry-agent
sudo cp hack/dev-certs/out/ca.pem \
hack/dev-certs/out/client.pem \
hack/dev-certs/out/client-key.pem \
/etc/sentry-agent/
```
## 5. Build and run the agent
```sh
cd agent
cargo build --release
```
The agent's built-in defaults already match this setup with **zero
config file**: journald source (whole journal), service name `default`,
ingest endpoint `https://127.0.0.1:4317` (matches the port `ingest`
publishes in `docker-compose.yml`), and the cert paths from step 4. This
is the "no required flags for the common case" design goal from
`/agent/README.md` — if it doesn't just run, that design assumption is
wrong somewhere and worth reporting as a bug, not working around.
Reading the system journal generally needs root (or membership in the
`systemd-journal` group with a distro that grants it read access — varies
by distro, root is the reliable path for this runbook):
```sh
sudo ./target/release/sentry-agent
```
Leave it running in this terminal — you should see a `connected to ingest
service` log line. If you see a TLS or connection error instead, stop
here and check the Troubleshooting section before continuing.
## 6. Generate a test log line
In another terminal, **after** the agent is running and connected
(journald tailing starts from "now" — anything logged before the agent
started won't be picked up):
```sh
logger "hello from sentry phase 0"
```
`logger` (part of util-linux, present on virtually every Linux distro)
writes this to the system log, which journald captures immediately.
Give it a couple of seconds — the agent batches with a 2-second flush
interval by default, so the line won't hit ingest instantly.
## 7. Confirm it's queryable
**Via the web UI:**
`web` is already running from step 2 (`docker compose up -d --build`
starts every service in the file). Open `http://localhost:3000`, run the
default query (`SELECT * FROM logs
ORDER BY timestamp DESC LIMIT 100`), and look for a row with
`message = "hello from sentry phase 0"`.
**Or via curl, if you want to skip the browser:**
```sh
curl -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"sql": "SELECT * FROM logs ORDER BY timestamp DESC LIMIT 10"}'
```
**Or via sentryctl, just to confirm api is up (doesn't check the data
itself):**
```sh
cd cli && go run ./cmd/sentryctl ping
```
If you see the row: that's Phase 0 done, end to end. If you don't, see
Troubleshooting below.
## Tearing down
```sh
docker compose down # stops and removes containers, keeps volumes
docker compose down -v # also wipes Redpanda/ClickHouse data — start clean next time
```
## Troubleshooting
**Agent logs a TLS/certificate error on startup.**
Check the server cert's SAN actually covers how the agent is connecting
(`openssl x509 -in hack/dev-certs/out/server.pem -noout -ext
subjectAltName` — should list `DNS:ingest, DNS:localhost,
IP:127.0.0.1`). If you changed the agent's `ingest.endpoint` to something
not in that list, regenerate certs with an updated SAN in
`hack/dev-certs/generate.sh`, don't disable TLS verification.
**Agent connects but no data ever shows up in ClickHouse.**
Check each hop in order rather than guessing:
1. `docker compose logs ingest` — look for "batch produced to redpanda"
(gRPC front end got the batch) vs. errors.
2. `docker compose logs ingest` again — look for "batch flushed to
clickhouse" from the consumer half. If you see repeated "clickhouse
batch write failed... will redeliver" messages, `clickhouse-migrate`
likely hasn't finished (check `docker compose ps`) — the consumer will
keep retrying and self-heal once the table exists, per its
at-least-once design (see `/ingest/README.md`), so this may just need
more time rather than intervention.
3. `docker compose exec redpanda rpk topic list` — confirm
`sentry.logs.raw` exists (if `redpanda-provision` failed, it won't).
**`docker compose up` fails on `service_completed_successfully`.**
You're likely on Compose v1 (`docker-compose`, hyphenated) rather than v2
(`docker compose`, space) — see Prerequisites.
**Web UI query returns an error instead of rows.**
Open the browser's network tab — if the request never leaves the page
(CORS error in the console), confirm `api`'s `CORS_ALLOWED_ORIGIN`
(defaults to `*`, should not be the issue) and that `VITE_API_BASE_URL`
was set correctly at `web`'s build time (it's baked in, not read at
container start — see `/web/README.md`).
+16
View File
@@ -0,0 +1,16 @@
# hack
Local developer tooling that isn't part of any shipped component — scripts
you run against your own machine/dev stack, not code that ends up in a
container image (except `dev-certs`' *output*, which mounts into the
ingest container).
Not one of the top-level directories in the original monorepo scaffold —
added because dev-only mTLS cert generation didn't have a natural home in
`/deploy` (real deployment manifests), `/transport`, or any other existing
component. `/hack` is the conventional name for this in a lot of larger Go
monorepos (Kubernetes among them).
- `dev-certs/` — generates a throwaway CA + server/client cert pair for
local mTLS between the agent and ingest. See `/docs/phase-0-runbook.md`
for when to run it.
+1
View File
@@ -0,0 +1 @@
out/
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Generates a throwaway CA plus a server cert (for ingest) and a client
# cert (for the agent) for local mTLS. Dev/homelab only — never use this
# CA or its certs for anything resembling production; there's no rotation,
# no revocation, and the CA key sits unencrypted on disk right next to
# everything it signed.
#
# Re-run to regenerate from scratch; existing output is overwritten.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUT_DIR="${SCRIPT_DIR}/out"
DAYS="${DEV_CERT_DAYS:-365}"
mkdir -p "${OUT_DIR}"
cd "${OUT_DIR}"
echo "Generating dev CA..."
openssl req -x509 -newkey rsa:4096 -sha256 -days "${DAYS}" -nodes \
-keyout ca-key.pem -out ca.pem \
-subj "/O=Sentry Dev/CN=Sentry Dev CA"
gen_leaf() {
local name="$1" cn="$2" san="$3"
openssl req -newkey rsa:2048 -nodes -keyout "${name}-key.pem" -out "${name}.csr" \
-subj "/O=Sentry Dev/CN=${cn}"
openssl x509 -req -in "${name}.csr" -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out "${name}.pem" -days "${DAYS}" -sha256 \
-extfile <(printf "subjectAltName=%s" "${san}")
rm -f "${name}.csr"
}
# SANs cover both "reached by another container on the compose network"
# (ingest) and "reached from the host" (localhost/127.0.0.1, for an
# agent running natively per /agent/README.md's journald caveat).
echo "Generating server (ingest) cert..."
gen_leaf server ingest "DNS:ingest,DNS:localhost,IP:127.0.0.1"
echo "Generating client (agent) cert..."
gen_leaf client sentry-agent "DNS:sentry-agent"
rm -f ca.srl
echo
echo "Done. Certs written to ${OUT_DIR}/:"
ls "${OUT_DIR}"
+15
View File
@@ -0,0 +1,15 @@
# Build context must be the repo root (sentry/), not ingest/, since this
# needs both ingest/ and proto/:
# docker build -f ingest/Dockerfile -t sentry-ingest .
FROM golang:1.25-alpine AS builder
WORKDIR /src
COPY proto ./proto
COPY ingest ./ingest
WORKDIR /src/ingest
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/ingest ./cmd/ingest
FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/ingest /ingest
ENTRYPOINT ["/ingest"]
+85
View File
@@ -0,0 +1,85 @@
# ingest
Go service sitting between the Rust agent and ClickHouse. Two halves in one
binary, selected with `--mode`:
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
connect to. Forwards each record, proto-encoded and unchanged, onto
Redpanda. Does no normalization — kept thin so agent-facing latency isn't
coupled to ClickHouse write performance.
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
shape (`internal/normalize`), and batch-writes via the native protocol
driver. Commits Redpanda offsets only after a successful ClickHouse
write, so a ClickHouse outage causes redelivery on restart rather than
data loss.
- **all** (default) — both, in one process. This is what docker-compose
runs. Splitting into two deployments later (e.g. to scale them
independently in k8s) is a manifest change, not a code change — see
`--mode`.
## Why Redpanda stays in the path
Confirmed with the project owner during Phase 0 planning: the gRPC front
end produces to Redpanda rather than writing ClickHouse directly. This
exercises the pinned transport layer from day one and keeps agents from
ever needing Kafka credentials — mTLS to `ingest` is the only network
egress an agent has. See `/docs/architecture.md`.
## Dependencies worth knowing about
- **github.com/segmentio/kafka-go** — pure Go, no cgo, chosen over
franz-go/confluent-kafka-go specifically to keep the distroless build
simple (confirmed with the project owner; see git history / PR
discussion for the tradeoffs considered).
- **github.com/ClickHouse/clickhouse-go/v2** — official client, native
protocol, pure Go (no cgo).
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
server and consumer halves concurrently and propagate the first error.
## Configuration
All via environment variables (see `internal/config/config.go` for the
full list and defaults) — no config file format for Phase 0:
| Var | Default | Purpose |
|---|---|---|
| `GRPC_LISTEN_ADDR` | `:4317` | Agent-facing gRPC listen address |
| `TLS_CERT_FILE` / `TLS_KEY_FILE` | `/etc/sentry-ingest/server{,-key}.pem` | ingest's own mTLS identity |
| `TLS_CLIENT_CA_FILE` | `/etc/sentry-ingest/ca.pem` | CA used to verify agent client certs |
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match the topic provisioned in `/transport` |
| `REDPANDA_CONSUMER_GROUP` | `sentry-ingest` | Consumer group id |
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port, not HTTP |
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
## Building & testing
```sh
go build ./...
go vet ./...
go test ./...
```
Requires `google.golang.org/protobuf/cmd/protoc-gen-go` and
`google.golang.org/grpc/cmd/protoc-gen-go-grpc` only if you're
regenerating `/proto`'s Go bindings — ingest itself just imports the
already-generated `github.com/sentry/sentry/proto` module (see the
`replace` directive in `go.mod`, pointing at `../proto`).
```sh
# from the repo root, not ingest/
docker build -f ingest/Dockerfile -t sentry-ingest .
```
## Testing notes
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
unit-tested against fakes — no embedded broker or database needed. What's
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
wiring and the ClickHouse native-protocol driver itself. Those are only
exercised by the docker-compose end-to-end flow described in
`/docs/phase-0-runbook.md`.
+77
View File
@@ -0,0 +1,77 @@
// Command ingest is the Sentry ingest service. It has two halves that can
// run in one process or be split across deployments via --mode:
//
// - server: mTLS gRPC front end that agents push batches to; forwards
// them onto Redpanda unchanged.
// - consumer: reads back off Redpanda, normalizes, batch-writes to
// ClickHouse.
// - all (default): both, in one process — the Phase 0 / docker-compose
// shape. Splitting into separate deployments later is a k8s manifest
// change, not a code change.
package main
import (
"context"
"flag"
"fmt"
"log/slog"
"os"
"os/signal"
"syscall"
"golang.org/x/sync/errgroup"
"github.com/sentry/sentry/ingest/internal/clickhousewriter"
"github.com/sentry/sentry/ingest/internal/config"
"github.com/sentry/sentry/ingest/internal/consumer"
"github.com/sentry/sentry/ingest/internal/grpcserver"
"github.com/sentry/sentry/ingest/internal/producer"
)
func main() {
mode := flag.String("mode", "all", "which half of ingest to run: server | consumer | all")
flag.Parse()
if *mode != "server" && *mode != "consumer" && *mode != "all" {
fmt.Fprintf(os.Stderr, "unknown --mode %q, must be server|consumer|all\n", *mode)
os.Exit(1)
}
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("loading config", "error", err)
os.Exit(1)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
g, ctx := errgroup.WithContext(ctx)
if *mode == "server" || *mode == "all" {
p := producer.New(cfg.Redpanda)
defer p.Close()
srv := grpcserver.New(logger, cfg.GRPC, cfg.TLS, p)
g.Go(func() error { return srv.Run(ctx) })
}
if *mode == "consumer" || *mode == "all" {
chw, err := clickhousewriter.New(ctx, cfg.ClickHouse)
if err != nil {
logger.Error("connecting to clickhouse", "error", err)
os.Exit(1)
}
defer chw.Close()
c := consumer.New(logger, cfg.Redpanda, cfg.Batch, chw)
g.Go(func() error { return c.Run(ctx) })
}
logger.Info("ingest started", "mode", *mode)
if err := g.Wait(); err != nil {
logger.Error("ingest exited with error", "error", err)
os.Exit(1)
}
}
+34
View File
@@ -0,0 +1,34 @@
module github.com/sentry/sentry/ingest
go 1.25.0
replace github.com/sentry/sentry/proto => ../proto
require (
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
github.com/segmentio/kafka-go v0.4.51
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
golang.org/x/sync v0.22.0
google.golang.org/grpc v1.83.0
google.golang.org/protobuf v1.36.12
)
require (
github.com/ClickHouse/ch-go v0.74.0 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.19.1 // indirect
github.com/paulmach/orb v0.13.0 // indirect
github.com/pierrec/lz4/v4 v4.1.27 // indirect
github.com/segmentio/asm v1.2.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
)
+78
View File
@@ -0,0 +1,78 @@
github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE=
github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M=
github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4=
github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM=
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/segmentio/kafka-go v0.4.51 h1:JgDPPG75tC1rWIS2Me6MwcvXJ6f49UQ4HjAOef71Hno=
github.com/segmentio/kafka-go v0.4.51/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -0,0 +1,60 @@
// Package clickhousewriter batch-inserts normalized log rows into
// ClickHouse using the native protocol driver's batch API.
package clickhousewriter
import (
"context"
"fmt"
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
"github.com/sentry/sentry/ingest/internal/config"
"github.com/sentry/sentry/ingest/internal/normalize"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
type Writer struct {
conn driver.Conn
}
func New(ctx context.Context, cfg config.ClickHouseConfig) (*Writer, error) {
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{cfg.Addr},
Auth: clickhouse.Auth{
Database: cfg.Database,
Username: cfg.Username,
Password: cfg.Password,
},
})
if err != nil {
return nil, fmt.Errorf("opening clickhouse connection: %w", err)
}
if err := conn.Ping(ctx); err != nil {
return nil, fmt.Errorf("pinging clickhouse: %w", err)
}
return &Writer{conn: conn}, nil
}
func (w *Writer) Close() error {
return w.conn.Close()
}
func (w *Writer) WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error {
batch, err := w.conn.PrepareBatch(ctx, "INSERT INTO logs (timestamp, host, service, severity, message, attributes)")
if err != nil {
return fmt.Errorf("preparing batch: %w", err)
}
for _, rec := range records {
row := normalize.ToRow(rec)
if err := batch.Append(row.Timestamp, row.Host, row.Service, row.Severity, row.Message, row.Attributes); err != nil {
return fmt.Errorf("appending row to batch: %w", err)
}
}
if err := batch.Send(); err != nil {
return fmt.Errorf("sending batch: %w", err)
}
return nil
}
+95
View File
@@ -0,0 +1,95 @@
// Package config loads ingest's configuration from environment variables.
// Phase 0 deliberately has no config file format of its own — env vars are
// enough for a docker-compose/k8s deployment and avoid pulling in a config
// library.
package config
import (
"fmt"
"os"
"strconv"
"strings"
)
type Config struct {
GRPC GRPCConfig
TLS TLSConfig
Redpanda RedpandaConfig
ClickHouse ClickHouseConfig
Batch BatchConfig
}
type GRPCConfig struct {
ListenAddr string
}
// TLSConfig is the server-side mTLS material: the ingest service's own
// cert/key, and the CA used to verify agent client certs.
type TLSConfig struct {
CertFile string
KeyFile string
ClientCAFile string
}
type RedpandaConfig struct {
Brokers []string
Topic string
ConsumerGroup string
}
type ClickHouseConfig struct {
Addr string
Database string
Username string
Password string
}
type BatchConfig struct {
MaxSize int
FlushIntervalMS int
}
func Load() (Config, error) {
cfg := Config{
GRPC: GRPCConfig{
ListenAddr: getenv("GRPC_LISTEN_ADDR", ":4317"),
},
TLS: TLSConfig{
CertFile: getenv("TLS_CERT_FILE", "/etc/sentry-ingest/server.pem"),
KeyFile: getenv("TLS_KEY_FILE", "/etc/sentry-ingest/server-key.pem"),
ClientCAFile: getenv("TLS_CLIENT_CA_FILE", "/etc/sentry-ingest/ca.pem"),
},
Redpanda: RedpandaConfig{
Brokers: strings.Split(getenv("REDPANDA_BROKERS", "localhost:9092"), ","),
Topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
ConsumerGroup: getenv("REDPANDA_CONSUMER_GROUP", "sentry-ingest"),
},
ClickHouse: ClickHouseConfig{
Addr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
Database: getenv("CLICKHOUSE_DATABASE", "sentry"),
Username: getenv("CLICKHOUSE_USERNAME", "default"),
Password: getenv("CLICKHOUSE_PASSWORD", ""),
},
}
maxSize, err := strconv.Atoi(getenv("CONSUMER_BATCH_MAX_SIZE", "500"))
if err != nil {
return Config{}, fmt.Errorf("CONSUMER_BATCH_MAX_SIZE: %w", err)
}
cfg.Batch.MaxSize = maxSize
flushMS, err := strconv.Atoi(getenv("CONSUMER_BATCH_FLUSH_INTERVAL_MS", "2000"))
if err != nil {
return Config{}, fmt.Errorf("CONSUMER_BATCH_FLUSH_INTERVAL_MS: %w", err)
}
cfg.Batch.FlushIntervalMS = flushMS
return cfg, nil
}
func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
+49
View File
@@ -0,0 +1,49 @@
package config
import "testing"
func TestLoadDefaults(t *testing.T) {
cfg, err := Load()
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.GRPC.ListenAddr != ":4317" {
t.Errorf("GRPC.ListenAddr = %q, want :4317", cfg.GRPC.ListenAddr)
}
if cfg.Redpanda.Topic != "sentry.logs.raw" {
t.Errorf("Redpanda.Topic = %q, want sentry.logs.raw", cfg.Redpanda.Topic)
}
if cfg.Batch.MaxSize != 500 {
t.Errorf("Batch.MaxSize = %d, want 500", cfg.Batch.MaxSize)
}
if cfg.Batch.FlushIntervalMS != 2000 {
t.Errorf("Batch.FlushIntervalMS = %d, want 2000", cfg.Batch.FlushIntervalMS)
}
}
func TestLoadOverridesFromEnv(t *testing.T) {
t.Setenv("GRPC_LISTEN_ADDR", ":9999")
t.Setenv("REDPANDA_BROKERS", "a:9092,b:9092")
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "10")
cfg, err := Load()
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.GRPC.ListenAddr != ":9999" {
t.Errorf("GRPC.ListenAddr = %q, want :9999", cfg.GRPC.ListenAddr)
}
if len(cfg.Redpanda.Brokers) != 2 || cfg.Redpanda.Brokers[0] != "a:9092" || cfg.Redpanda.Brokers[1] != "b:9092" {
t.Errorf("Redpanda.Brokers = %+v, want [a:9092 b:9092]", cfg.Redpanda.Brokers)
}
if cfg.Batch.MaxSize != 10 {
t.Errorf("Batch.MaxSize = %d, want 10", cfg.Batch.MaxSize)
}
}
func TestLoadInvalidBatchSizeErrors(t *testing.T) {
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "not-a-number")
if _, err := Load(); err == nil {
t.Fatal("expected error for non-numeric CONSUMER_BATCH_MAX_SIZE, got nil")
}
}
+124
View File
@@ -0,0 +1,124 @@
// Package consumer reads normalized-on-write LogRecords back off Redpanda
// and batch-writes them into ClickHouse. Offsets are committed only after
// a successful ClickHouse write, so a ClickHouse outage causes redelivery
// on restart rather than silent data loss (at-least-once, not exactly-once
// — Phase 0 doesn't dedupe on the consumer side).
package consumer
import (
"context"
"log/slog"
"time"
"github.com/segmentio/kafka-go"
"google.golang.org/protobuf/proto"
"github.com/sentry/sentry/ingest/internal/config"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
// chWriter is the subset of *clickhousewriter.Writer this package depends
// on, kept as an interface so the flush loop is unit-testable without a
// real ClickHouse connection.
type chWriter interface {
WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error
}
// reader is the subset of *kafka.Reader used here, as an interface so the
// flush/commit logic can be tested against a fake without a real broker.
type reader interface {
FetchMessage(ctx context.Context) (kafka.Message, error)
CommitMessages(ctx context.Context, msgs ...kafka.Message) error
Close() error
}
type Consumer struct {
logger *slog.Logger
reader reader
writer chWriter
batchCfg config.BatchConfig
}
func New(logger *slog.Logger, redpandaCfg config.RedpandaConfig, batchCfg config.BatchConfig, w chWriter) *Consumer {
r := kafka.NewReader(kafka.ReaderConfig{
Brokers: redpandaCfg.Brokers,
Topic: redpandaCfg.Topic,
GroupID: redpandaCfg.ConsumerGroup,
})
return &Consumer{logger: logger, reader: r, writer: w, batchCfg: batchCfg}
}
func (c *Consumer) Run(ctx context.Context) error {
defer c.reader.Close()
flushInterval := time.Duration(c.batchCfg.FlushIntervalMS) * time.Millisecond
ticker := time.NewTicker(flushInterval)
defer ticker.Stop()
msgCh := make(chan kafka.Message)
fetchErrCh := make(chan error, 1)
go func() {
for {
m, err := c.reader.FetchMessage(ctx)
if err != nil {
fetchErrCh <- err
return
}
select {
case msgCh <- m:
case <-ctx.Done():
return
}
}
}()
var records []*logsv1.LogRecord
var pending []kafka.Message
flush := func() {
if len(records) == 0 {
return
}
if err := c.writer.WriteBatch(ctx, records); err != nil {
c.logger.Error("clickhouse batch write failed, offsets not committed, will redeliver",
"records", len(records), "error", err)
} else if err := c.reader.CommitMessages(ctx, pending...); err != nil {
c.logger.Error("committing offsets after clickhouse write", "error", err)
} else {
c.logger.Debug("batch flushed to clickhouse", "records", len(records))
}
records = records[:0]
pending = pending[:0]
}
for {
select {
case <-ctx.Done():
flush()
return nil
case err := <-fetchErrCh:
flush()
if ctx.Err() != nil {
return nil
}
return err
case <-ticker.C:
flush()
case m := <-msgCh:
var rec logsv1.LogRecord
if err := proto.Unmarshal(m.Value, &rec); err != nil {
c.logger.Warn("skipping unparseable message", "error", err, "offset", m.Offset)
if cerr := c.reader.CommitMessages(ctx, m); cerr != nil {
c.logger.Error("committing offset for poison message", "error", cerr)
}
continue
}
records = append(records, &rec)
pending = append(pending, m)
if len(records) >= c.batchCfg.MaxSize {
flush()
}
}
}
}
+178
View File
@@ -0,0 +1,178 @@
package consumer
import (
"context"
"errors"
"io"
"log/slog"
"sync"
"testing"
"time"
"github.com/segmentio/kafka-go"
"google.golang.org/protobuf/proto"
"github.com/sentry/sentry/ingest/internal/config"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
type fakeReader struct {
msgs chan kafka.Message
mu sync.Mutex
committed [][]kafka.Message
}
func newFakeReader() *fakeReader {
return &fakeReader{msgs: make(chan kafka.Message, 16)}
}
func (f *fakeReader) push(m kafka.Message) { f.msgs <- m }
func (f *fakeReader) FetchMessage(ctx context.Context) (kafka.Message, error) {
select {
case m := <-f.msgs:
return m, nil
case <-ctx.Done():
return kafka.Message{}, ctx.Err()
}
}
func (f *fakeReader) CommitMessages(_ context.Context, msgs ...kafka.Message) error {
f.mu.Lock()
defer f.mu.Unlock()
f.committed = append(f.committed, msgs)
return nil
}
func (f *fakeReader) Close() error { return nil }
func (f *fakeReader) commitCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.committed)
}
type fakeWriter struct {
mu sync.Mutex
batches [][]*logsv1.LogRecord
failNext bool
}
func (f *fakeWriter) WriteBatch(_ context.Context, records []*logsv1.LogRecord) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.failNext {
f.failNext = false
return errors.New("simulated clickhouse failure")
}
batch := make([]*logsv1.LogRecord, len(records))
copy(batch, records)
f.batches = append(f.batches, batch)
return nil
}
func (f *fakeWriter) batchCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.batches)
}
func newTestConsumer(r reader, w chWriter, batchCfg config.BatchConfig) *Consumer {
return &Consumer{
logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
reader: r,
writer: w,
batchCfg: batchCfg,
}
}
func mustMarshal(t *testing.T, rec *logsv1.LogRecord) []byte {
t.Helper()
b, err := proto.Marshal(rec)
if err != nil {
t.Fatalf("marshal: %v", err)
}
return b
}
func waitFor(t *testing.T, timeout time.Duration, cond func() bool) {
t.Helper()
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
if cond() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatal("condition not met before timeout")
}
func TestConsumerFlushesOnBatchSize(t *testing.T) {
fr := newFakeReader()
fw := &fakeWriter{}
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 2, FlushIntervalMS: 60_000})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan error, 1)
go func() { done <- c.Run(ctx) }()
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "a"})})
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "b"})})
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
fw.mu.Lock()
if len(fw.batches[0]) != 2 {
t.Fatalf("expected batch of 2 records, got %d", len(fw.batches[0]))
}
fw.mu.Unlock()
waitFor(t, time.Second, func() bool { return fr.commitCount() == 1 })
}
func TestConsumerFlushesOnTimeout(t *testing.T) {
fr := newFakeReader()
fw := &fakeWriter{}
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1000, FlushIntervalMS: 20})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan error, 1)
go func() { done <- c.Run(ctx) }()
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "only-one"})})
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
fw.mu.Lock()
if len(fw.batches[0]) != 1 {
t.Fatalf("expected batch of 1 record, got %d", len(fw.batches[0]))
}
fw.mu.Unlock()
}
func TestConsumerDoesNotCommitOnWriteFailure(t *testing.T) {
fr := newFakeReader()
fw := &fakeWriter{failNext: true}
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1, FlushIntervalMS: 60_000})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan error, 1)
go func() { done <- c.Run(ctx) }()
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "will-fail"})})
// Give the flush a moment to run and fail.
time.Sleep(100 * time.Millisecond)
if got := fr.commitCount(); got != 0 {
t.Fatalf("expected no commits after a failed clickhouse write, got %d", got)
}
// The batch was attempted even though writer returned an error.
if fw.batchCount() != 0 {
t.Fatalf("fakeWriter should not record a failed batch, got %d recorded", fw.batchCount())
}
}
+96
View File
@@ -0,0 +1,96 @@
// Package grpcserver implements the agent-facing side of ingest: an mTLS
// gRPC server accepting LogIngest.PushBatch calls, which it forwards
// unchanged (proto-encoded) onto Redpanda. Normalization into the
// ClickHouse row shape happens later, on the consumer side.
package grpcserver
import (
"context"
"fmt"
"log/slog"
"net"
"github.com/segmentio/kafka-go"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
"github.com/sentry/sentry/ingest/internal/config"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
type Server struct {
logsv1.UnimplementedLogIngestServer
logger *slog.Logger
grpcCfg config.GRPCConfig
tlsCfg config.TLSConfig
producer batchProducer
}
// batchProducer is the subset of *producer.Producer this package depends
// on, so tests can substitute a fake without touching Redpanda.
type batchProducer interface {
WriteBatch(ctx context.Context, msgs []kafka.Message) error
}
func New(logger *slog.Logger, grpcCfg config.GRPCConfig, tlsCfg config.TLSConfig, p batchProducer) *Server {
return &Server{logger: logger, grpcCfg: grpcCfg, tlsCfg: tlsCfg, producer: p}
}
// Run blocks serving gRPC until ctx is canceled, then gracefully stops.
func (s *Server) Run(ctx context.Context) error {
tlsConf, err := loadServerTLSConfig(s.tlsCfg)
if err != nil {
return fmt.Errorf("loading TLS config: %w", err)
}
lis, err := net.Listen("tcp", s.grpcCfg.ListenAddr)
if err != nil {
return fmt.Errorf("listening on %s: %w", s.grpcCfg.ListenAddr, err)
}
grpcSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsConf)))
logsv1.RegisterLogIngestServer(grpcSrv, s)
s.logger.Info("gRPC server listening", "addr", s.grpcCfg.ListenAddr)
errCh := make(chan error, 1)
go func() { errCh <- grpcSrv.Serve(lis) }()
select {
case <-ctx.Done():
grpcSrv.GracefulStop()
return nil
case err := <-errCh:
return err
}
}
func (s *Server) PushBatch(ctx context.Context, req *logsv1.PushBatchRequest) (*logsv1.PushBatchResponse, error) {
if len(req.GetRecords()) == 0 {
return &logsv1.PushBatchResponse{Accepted: 0}, nil
}
msgs := make([]kafka.Message, 0, len(req.GetRecords()))
for _, rec := range req.GetRecords() {
val, err := proto.Marshal(rec)
if err != nil {
return nil, status.Errorf(codes.InvalidArgument, "marshaling record: %v", err)
}
msgs = append(msgs, kafka.Message{
Key: []byte(rec.GetHost()),
Value: val,
})
}
if err := s.producer.WriteBatch(ctx, msgs); err != nil {
s.logger.Error("failed to write batch to redpanda", "batch_id", req.GetBatchId(), "error", err)
return nil, status.Errorf(codes.Unavailable, "writing to transport: %v", err)
}
s.logger.Debug("batch produced to redpanda", "batch_id", req.GetBatchId(), "records", len(req.GetRecords()))
return &logsv1.PushBatchResponse{Accepted: uint32(len(req.GetRecords()))}, nil
}
+36
View File
@@ -0,0 +1,36 @@
package grpcserver
import (
"crypto/tls"
"crypto/x509"
"fmt"
"os"
"github.com/sentry/sentry/ingest/internal/config"
)
// loadServerTLSConfig builds the mTLS server config: ingest's own
// certificate, plus the CA used to verify agent client certificates.
// Agents are never accepted without a client cert signed by this CA.
func loadServerTLSConfig(cfg config.TLSConfig) (*tls.Config, error) {
cert, err := tls.LoadX509KeyPair(cfg.CertFile, cfg.KeyFile)
if err != nil {
return nil, fmt.Errorf("loading server cert/key: %w", err)
}
caPEM, err := os.ReadFile(cfg.ClientCAFile)
if err != nil {
return nil, fmt.Errorf("reading client CA file: %w", err)
}
caPool := x509.NewCertPool()
if !caPool.AppendCertsFromPEM(caPEM) {
return nil, fmt.Errorf("no valid certificates found in client CA file %s", cfg.ClientCAFile)
}
return &tls.Config{
Certificates: []tls.Certificate{cert},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: caPool,
MinVersion: tls.VersionTLS12,
}, nil
}
+58
View File
@@ -0,0 +1,58 @@
// Package normalize maps the wire-format LogRecord (as agents send it)
// into the ClickHouse row shape defined in /storage. This is the "OTel-log-
// like schema" normalization step called for in the ingest design — Phase
// 0 keeps it to the minimal column set; full OTel field mapping (separate
// SeverityNumber/SeverityText, resource attributes, etc.) is deferred, see
// the open questions in /docs/architecture.md.
package normalize
import (
"time"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
type Row struct {
Timestamp time.Time
Host string
Service string
Severity string
Message string
Attributes map[string]string
}
func ToRow(rec *logsv1.LogRecord) Row {
attrs := rec.GetAttributes()
if attrs == nil {
attrs = map[string]string{}
}
return Row{
Timestamp: time.Unix(0, rec.GetTimestampUnixNano()).UTC(),
Host: rec.GetHost(),
Service: rec.GetService(),
Severity: severityText(rec.GetSeverity()),
Message: rec.GetMessage(),
Attributes: attrs,
}
}
// severityText maps the proto Severity enum to short OTel-style severity
// names, stored as the `severity` column's value.
func severityText(sev logsv1.Severity) string {
switch sev {
case logsv1.Severity_SEVERITY_TRACE:
return "TRACE"
case logsv1.Severity_SEVERITY_DEBUG:
return "DEBUG"
case logsv1.Severity_SEVERITY_INFO:
return "INFO"
case logsv1.Severity_SEVERITY_WARN:
return "WARN"
case logsv1.Severity_SEVERITY_ERROR:
return "ERROR"
case logsv1.Severity_SEVERITY_FATAL:
return "FATAL"
default:
return "UNSPECIFIED"
}
}
@@ -0,0 +1,68 @@
package normalize
import (
"testing"
"time"
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
)
func TestToRowMapsFieldsAndSeverity(t *testing.T) {
rec := &logsv1.LogRecord{
TimestampUnixNano: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).UnixNano(),
Host: "host-1",
Service: "svc-a",
Severity: logsv1.Severity_SEVERITY_ERROR,
Message: "boom",
Attributes: map[string]string{"k": "v"},
}
row := ToRow(rec)
if row.Host != "host-1" || row.Service != "svc-a" || row.Message != "boom" {
t.Fatalf("unexpected row: %+v", row)
}
if row.Severity != "ERROR" {
t.Fatalf("expected severity ERROR, got %s", row.Severity)
}
if row.Attributes["k"] != "v" {
t.Fatalf("expected attribute k=v, got %+v", row.Attributes)
}
if !row.Timestamp.Equal(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) {
t.Fatalf("unexpected timestamp: %v", row.Timestamp)
}
}
func TestToRowNilAttributesBecomesEmptyMap(t *testing.T) {
rec := &logsv1.LogRecord{Host: "h", Service: "s", Message: "m"}
row := ToRow(rec)
if row.Attributes == nil {
t.Fatal("expected non-nil empty map, got nil")
}
if len(row.Attributes) != 0 {
t.Fatalf("expected empty map, got %+v", row.Attributes)
}
}
func TestSeverityTextCoversAllEnumValues(t *testing.T) {
cases := map[logsv1.Severity]string{
logsv1.Severity_SEVERITY_UNSPECIFIED: "UNSPECIFIED",
logsv1.Severity_SEVERITY_TRACE: "TRACE",
logsv1.Severity_SEVERITY_DEBUG: "DEBUG",
logsv1.Severity_SEVERITY_INFO: "INFO",
logsv1.Severity_SEVERITY_WARN: "WARN",
logsv1.Severity_SEVERITY_ERROR: "ERROR",
logsv1.Severity_SEVERITY_FATAL: "FATAL",
}
for sev, want := range cases {
if got := severityText(sev); got != want {
t.Errorf("severityText(%v) = %q, want %q", sev, got, want)
}
}
}
func TestSeverityTextUnknownValueFallsBackToUnspecified(t *testing.T) {
if got := severityText(logsv1.Severity(99)); got != "UNSPECIFIED" {
t.Fatalf("expected UNSPECIFIED for unknown severity, got %q", got)
}
}
+42
View File
@@ -0,0 +1,42 @@
// Package producer wraps the Redpanda (Kafka API) producer used by the
// gRPC front end to forward agent-submitted batches onto the transport
// layer, unchanged. OTel-log-shape normalization happens later, on the
// consumer side — see internal/normalize.
package producer
import (
"context"
"github.com/segmentio/kafka-go"
"github.com/sentry/sentry/ingest/internal/config"
)
type Producer struct {
writer *kafka.Writer
}
func New(cfg config.RedpandaConfig) *Producer {
return &Producer{
writer: &kafka.Writer{
Addr: kafka.TCP(cfg.Brokers...),
Topic: cfg.Topic,
// Partition by host so a single host's records stay in
// relative order within a partition.
Balancer: &kafka.Hash{},
RequiredAcks: kafka.RequireOne,
AllowAutoTopicCreation: false, // topics are provisioned explicitly, see /transport
},
}
}
func (p *Producer) Close() error {
return p.writer.Close()
}
// WriteBatch writes all messages in one call. kafka-go's WriteMessages
// either succeeds for the whole batch or returns an error, which matches
// the PushBatch RPC's all-or-nothing contract for Phase 0.
func (p *Producer) WriteBatch(ctx context.Context, msgs []kafka.Message) error {
return p.writer.WriteMessages(ctx, msgs...)
}
+30
View File
@@ -0,0 +1,30 @@
# proto
Shared `.proto` contracts. Source of truth for the agent↔ingest gRPC
service; each language generates its own bindings from these files rather
than sharing generated code across languages.
- `sentry/logs/v1/logs.proto``LogIngest.PushBatch`, the only RPC an
agent ever calls.
## Go bindings
Go is the one language here with pre-generated, checked-in bindings
(`sentry/logs/v1/logs.pb.go`, `logs_grpc.pb.go`), living in this directory
as its own module (`github.com/sentry/sentry/proto`) that `/ingest` and
`/api` depend on via a local `replace` directive in their `go.mod`. Rust
(`/agent`) instead generates its bindings at build time via `tonic-build`
(see `agent/sentry-agent/build.rs`) — no checked-in Rust output.
To regenerate the Go bindings after changing `logs.proto`:
```sh
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
cd proto
protoc --go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
sentry/logs/v1/logs.proto
go build ./...
```
+15
View File
@@ -0,0 +1,15 @@
module github.com/sentry/sentry/proto
go 1.25.0
require (
google.golang.org/grpc v1.83.0
google.golang.org/protobuf v1.36.12
)
require (
golang.org/x/net v0.55.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.37.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
)
+38
View File
@@ -0,0 +1,38 @@
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+376
View File
@@ -0,0 +1,376 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.36.12
// protoc v7.35.1
// source: sentry/logs/v1/logs.proto
package logsv1
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
unsafe "unsafe"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// Severity follows OTel's severity number ranges (1-24), collapsed here to
// the coarse names agents actually need to set. Numeric value stored
// downstream may be a full OTel severity_number computed by ingest.
type Severity int32
const (
Severity_SEVERITY_UNSPECIFIED Severity = 0
Severity_SEVERITY_TRACE Severity = 1
Severity_SEVERITY_DEBUG Severity = 2
Severity_SEVERITY_INFO Severity = 3
Severity_SEVERITY_WARN Severity = 4
Severity_SEVERITY_ERROR Severity = 5
Severity_SEVERITY_FATAL Severity = 6
)
// Enum value maps for Severity.
var (
Severity_name = map[int32]string{
0: "SEVERITY_UNSPECIFIED",
1: "SEVERITY_TRACE",
2: "SEVERITY_DEBUG",
3: "SEVERITY_INFO",
4: "SEVERITY_WARN",
5: "SEVERITY_ERROR",
6: "SEVERITY_FATAL",
}
Severity_value = map[string]int32{
"SEVERITY_UNSPECIFIED": 0,
"SEVERITY_TRACE": 1,
"SEVERITY_DEBUG": 2,
"SEVERITY_INFO": 3,
"SEVERITY_WARN": 4,
"SEVERITY_ERROR": 5,
"SEVERITY_FATAL": 6,
}
)
func (x Severity) Enum() *Severity {
p := new(Severity)
*p = x
return p
}
func (x Severity) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (Severity) Descriptor() protoreflect.EnumDescriptor {
return file_sentry_logs_v1_logs_proto_enumTypes[0].Descriptor()
}
func (Severity) Type() protoreflect.EnumType {
return &file_sentry_logs_v1_logs_proto_enumTypes[0]
}
func (x Severity) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use Severity.Descriptor instead.
func (Severity) EnumDescriptor() ([]byte, []int) {
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{0}
}
type LogRecord struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Unix epoch nanoseconds, set by the agent at time of read (not parse or
// send time) to preserve original ordering as closely as possible.
TimestampUnixNano int64 `protobuf:"varint,1,opt,name=timestamp_unix_nano,json=timestampUnixNano,proto3" json:"timestamp_unix_nano,omitempty"`
// Hostname the agent is running on. Agent fills this from its own config
// or system hostname; not trusted as an identity claim (mTLS client cert
// is the identity boundary).
Host string `protobuf:"bytes,2,opt,name=host,proto3" json:"host,omitempty"`
// Logical service/unit name. For journald sources, this is typically the
// systemd unit name; for file sources, it comes from agent config.
Service string `protobuf:"bytes,3,opt,name=service,proto3" json:"service,omitempty"`
Severity Severity `protobuf:"varint,4,opt,name=severity,proto3,enum=sentry.logs.v1.Severity" json:"severity,omitempty"`
// Original, unparsed log line. Always populated, even when structured
// fields below are also present, per the schema-on-read fallback
// requirement in CLAUDE.md.
Message string `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
// Structured fields extracted by the agent's parser (e.g. RFC 5424
// syslog header fields). Empty when the raw-passthrough fallback fires.
Attributes map[string]string `protobuf:"bytes,6,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *LogRecord) Reset() {
*x = LogRecord{}
mi := &file_sentry_logs_v1_logs_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *LogRecord) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*LogRecord) ProtoMessage() {}
func (x *LogRecord) ProtoReflect() protoreflect.Message {
mi := &file_sentry_logs_v1_logs_proto_msgTypes[0]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use LogRecord.ProtoReflect.Descriptor instead.
func (*LogRecord) Descriptor() ([]byte, []int) {
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{0}
}
func (x *LogRecord) GetTimestampUnixNano() int64 {
if x != nil {
return x.TimestampUnixNano
}
return 0
}
func (x *LogRecord) GetHost() string {
if x != nil {
return x.Host
}
return ""
}
func (x *LogRecord) GetService() string {
if x != nil {
return x.Service
}
return ""
}
func (x *LogRecord) GetSeverity() Severity {
if x != nil {
return x.Severity
}
return Severity_SEVERITY_UNSPECIFIED
}
func (x *LogRecord) GetMessage() string {
if x != nil {
return x.Message
}
return ""
}
func (x *LogRecord) GetAttributes() map[string]string {
if x != nil {
return x.Attributes
}
return nil
}
type PushBatchRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
// use this to avoid double-writing a batch if a retry follows a
// timeout on an actually-successful push.
BatchId string `protobuf:"bytes,1,opt,name=batch_id,json=batchId,proto3" json:"batch_id,omitempty"`
Records []*LogRecord `protobuf:"bytes,2,rep,name=records,proto3" json:"records,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *PushBatchRequest) Reset() {
*x = PushBatchRequest{}
mi := &file_sentry_logs_v1_logs_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *PushBatchRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*PushBatchRequest) ProtoMessage() {}
func (x *PushBatchRequest) ProtoReflect() protoreflect.Message {
mi := &file_sentry_logs_v1_logs_proto_msgTypes[1]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use PushBatchRequest.ProtoReflect.Descriptor instead.
func (*PushBatchRequest) Descriptor() ([]byte, []int) {
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{1}
}
func (x *PushBatchRequest) GetBatchId() string {
if x != nil {
return x.BatchId
}
return ""
}
func (x *PushBatchRequest) GetRecords() []*LogRecord {
if x != nil {
return x.Records
}
return nil
}
type PushBatchResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Number of records ingest accepted. Phase 0: batches are all-or-nothing,
// so this equals len(records) on success. Partial-acceptance semantics
// are not implemented yet.
Accepted uint32 `protobuf:"varint,1,opt,name=accepted,proto3" json:"accepted,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *PushBatchResponse) Reset() {
*x = PushBatchResponse{}
mi := &file_sentry_logs_v1_logs_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *PushBatchResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*PushBatchResponse) ProtoMessage() {}
func (x *PushBatchResponse) ProtoReflect() protoreflect.Message {
mi := &file_sentry_logs_v1_logs_proto_msgTypes[2]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use PushBatchResponse.ProtoReflect.Descriptor instead.
func (*PushBatchResponse) Descriptor() ([]byte, []int) {
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{2}
}
func (x *PushBatchResponse) GetAccepted() uint32 {
if x != nil {
return x.Accepted
}
return 0
}
var File_sentry_logs_v1_logs_proto protoreflect.FileDescriptor
const file_sentry_logs_v1_logs_proto_rawDesc = "" +
"\n" +
"\x19sentry/logs/v1/logs.proto\x12\x0esentry.logs.v1\"\xc3\x02\n" +
"\tLogRecord\x12.\n" +
"\x13timestamp_unix_nano\x18\x01 \x01(\x03R\x11timestampUnixNano\x12\x12\n" +
"\x04host\x18\x02 \x01(\tR\x04host\x12\x18\n" +
"\aservice\x18\x03 \x01(\tR\aservice\x124\n" +
"\bseverity\x18\x04 \x01(\x0e2\x18.sentry.logs.v1.SeverityR\bseverity\x12\x18\n" +
"\amessage\x18\x05 \x01(\tR\amessage\x12I\n" +
"\n" +
"attributes\x18\x06 \x03(\v2).sentry.logs.v1.LogRecord.AttributesEntryR\n" +
"attributes\x1a=\n" +
"\x0fAttributesEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"b\n" +
"\x10PushBatchRequest\x12\x19\n" +
"\bbatch_id\x18\x01 \x01(\tR\abatchId\x123\n" +
"\arecords\x18\x02 \x03(\v2\x19.sentry.logs.v1.LogRecordR\arecords\"/\n" +
"\x11PushBatchResponse\x12\x1a\n" +
"\baccepted\x18\x01 \x01(\rR\baccepted*\x9a\x01\n" +
"\bSeverity\x12\x18\n" +
"\x14SEVERITY_UNSPECIFIED\x10\x00\x12\x12\n" +
"\x0eSEVERITY_TRACE\x10\x01\x12\x12\n" +
"\x0eSEVERITY_DEBUG\x10\x02\x12\x11\n" +
"\rSEVERITY_INFO\x10\x03\x12\x11\n" +
"\rSEVERITY_WARN\x10\x04\x12\x12\n" +
"\x0eSEVERITY_ERROR\x10\x05\x12\x12\n" +
"\x0eSEVERITY_FATAL\x10\x062]\n" +
"\tLogIngest\x12P\n" +
"\tPushBatch\x12 .sentry.logs.v1.PushBatchRequest\x1a!.sentry.logs.v1.PushBatchResponseB6Z4github.com/sentry/sentry/proto/sentry/logs/v1;logsv1b\x06proto3"
var (
file_sentry_logs_v1_logs_proto_rawDescOnce sync.Once
file_sentry_logs_v1_logs_proto_rawDescData []byte
)
func file_sentry_logs_v1_logs_proto_rawDescGZIP() []byte {
file_sentry_logs_v1_logs_proto_rawDescOnce.Do(func() {
file_sentry_logs_v1_logs_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_sentry_logs_v1_logs_proto_rawDesc), len(file_sentry_logs_v1_logs_proto_rawDesc)))
})
return file_sentry_logs_v1_logs_proto_rawDescData
}
var file_sentry_logs_v1_logs_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_sentry_logs_v1_logs_proto_msgTypes = make([]protoimpl.MessageInfo, 4)
var file_sentry_logs_v1_logs_proto_goTypes = []any{
(Severity)(0), // 0: sentry.logs.v1.Severity
(*LogRecord)(nil), // 1: sentry.logs.v1.LogRecord
(*PushBatchRequest)(nil), // 2: sentry.logs.v1.PushBatchRequest
(*PushBatchResponse)(nil), // 3: sentry.logs.v1.PushBatchResponse
nil, // 4: sentry.logs.v1.LogRecord.AttributesEntry
}
var file_sentry_logs_v1_logs_proto_depIdxs = []int32{
0, // 0: sentry.logs.v1.LogRecord.severity:type_name -> sentry.logs.v1.Severity
4, // 1: sentry.logs.v1.LogRecord.attributes:type_name -> sentry.logs.v1.LogRecord.AttributesEntry
1, // 2: sentry.logs.v1.PushBatchRequest.records:type_name -> sentry.logs.v1.LogRecord
2, // 3: sentry.logs.v1.LogIngest.PushBatch:input_type -> sentry.logs.v1.PushBatchRequest
3, // 4: sentry.logs.v1.LogIngest.PushBatch:output_type -> sentry.logs.v1.PushBatchResponse
4, // [4:5] is the sub-list for method output_type
3, // [3:4] is the sub-list for method input_type
3, // [3:3] is the sub-list for extension type_name
3, // [3:3] is the sub-list for extension extendee
0, // [0:3] is the sub-list for field type_name
}
func init() { file_sentry_logs_v1_logs_proto_init() }
func file_sentry_logs_v1_logs_proto_init() {
if File_sentry_logs_v1_logs_proto != nil {
return
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_sentry_logs_v1_logs_proto_rawDesc), len(file_sentry_logs_v1_logs_proto_rawDesc)),
NumEnums: 1,
NumMessages: 4,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_sentry_logs_v1_logs_proto_goTypes,
DependencyIndexes: file_sentry_logs_v1_logs_proto_depIdxs,
EnumInfos: file_sentry_logs_v1_logs_proto_enumTypes,
MessageInfos: file_sentry_logs_v1_logs_proto_msgTypes,
}.Build()
File_sentry_logs_v1_logs_proto = out.File
file_sentry_logs_v1_logs_proto_goTypes = nil
file_sentry_logs_v1_logs_proto_depIdxs = nil
}
+68
View File
@@ -0,0 +1,68 @@
syntax = "proto3";
package sentry.logs.v1;
option go_package = "github.com/sentry/sentry/proto/sentry/logs/v1;logsv1";
// LogIngest is the service agents use to ship batched log records to the
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
// once backpressure/flow-control behavior is characterized.
service LogIngest {
rpc PushBatch(PushBatchRequest) returns (PushBatchResponse);
}
// Severity follows OTel's severity number ranges (1-24), collapsed here to
// the coarse names agents actually need to set. Numeric value stored
// downstream may be a full OTel severity_number computed by ingest.
enum Severity {
SEVERITY_UNSPECIFIED = 0;
SEVERITY_TRACE = 1;
SEVERITY_DEBUG = 2;
SEVERITY_INFO = 3;
SEVERITY_WARN = 4;
SEVERITY_ERROR = 5;
SEVERITY_FATAL = 6;
}
message LogRecord {
// Unix epoch nanoseconds, set by the agent at time of read (not parse or
// send time) to preserve original ordering as closely as possible.
int64 timestamp_unix_nano = 1;
// Hostname the agent is running on. Agent fills this from its own config
// or system hostname; not trusted as an identity claim (mTLS client cert
// is the identity boundary).
string host = 2;
// Logical service/unit name. For journald sources, this is typically the
// systemd unit name; for file sources, it comes from agent config.
string service = 3;
Severity severity = 4;
// Original, unparsed log line. Always populated, even when structured
// fields below are also present, per the schema-on-read fallback
// requirement in CLAUDE.md.
string message = 5;
// Structured fields extracted by the agent's parser (e.g. RFC 5424
// syslog header fields). Empty when the raw-passthrough fallback fires.
map<string, string> attributes = 6;
}
message PushBatchRequest {
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
// use this to avoid double-writing a batch if a retry follows a
// timeout on an actually-successful push.
string batch_id = 1;
repeated LogRecord records = 2;
}
message PushBatchResponse {
// Number of records ingest accepted. Phase 0: batches are all-or-nothing,
// so this equals len(records) on success. Partial-acceptance semantics
// are not implemented yet.
uint32 accepted = 1;
}
+131
View File
@@ -0,0 +1,131 @@
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.6.2
// - protoc v7.35.1
// source: sentry/logs/v1/logs.proto
package logsv1
import (
context "context"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
status "google.golang.org/grpc/status"
)
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
// Requires gRPC-Go v1.64.0 or later.
const _ = grpc.SupportPackageIsVersion9
const (
LogIngest_PushBatch_FullMethodName = "/sentry.logs.v1.LogIngest/PushBatch"
)
// LogIngestClient is the client API for LogIngest service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
//
// LogIngest is the service agents use to ship batched log records to the
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
// once backpressure/flow-control behavior is characterized.
type LogIngestClient interface {
PushBatch(ctx context.Context, in *PushBatchRequest, opts ...grpc.CallOption) (*PushBatchResponse, error)
}
type logIngestClient struct {
cc grpc.ClientConnInterface
}
func NewLogIngestClient(cc grpc.ClientConnInterface) LogIngestClient {
return &logIngestClient{cc}
}
func (c *logIngestClient) PushBatch(ctx context.Context, in *PushBatchRequest, opts ...grpc.CallOption) (*PushBatchResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(PushBatchResponse)
err := c.cc.Invoke(ctx, LogIngest_PushBatch_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// LogIngestServer is the server API for LogIngest service.
// All implementations must embed UnimplementedLogIngestServer
// for forward compatibility.
//
// LogIngest is the service agents use to ship batched log records to the
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
// once backpressure/flow-control behavior is characterized.
type LogIngestServer interface {
PushBatch(context.Context, *PushBatchRequest) (*PushBatchResponse, error)
mustEmbedUnimplementedLogIngestServer()
}
// UnimplementedLogIngestServer must be embedded to have
// forward compatible implementations.
//
// NOTE: this should be embedded by value instead of pointer to avoid a nil
// pointer dereference when methods are called.
type UnimplementedLogIngestServer struct{}
func (UnimplementedLogIngestServer) PushBatch(context.Context, *PushBatchRequest) (*PushBatchResponse, error) {
return nil, status.Error(codes.Unimplemented, "method PushBatch not implemented")
}
func (UnimplementedLogIngestServer) mustEmbedUnimplementedLogIngestServer() {}
func (UnimplementedLogIngestServer) testEmbeddedByValue() {}
// UnsafeLogIngestServer may be embedded to opt out of forward compatibility for this service.
// Use of this interface is not recommended, as added methods to LogIngestServer will
// result in compilation errors.
type UnsafeLogIngestServer interface {
mustEmbedUnimplementedLogIngestServer()
}
func RegisterLogIngestServer(s grpc.ServiceRegistrar, srv LogIngestServer) {
// If the following call panics, it indicates UnimplementedLogIngestServer was
// embedded by pointer and is nil. This will cause panics if an
// unimplemented method is ever invoked, so we test this at initialization
// time to prevent it from happening at runtime later due to I/O.
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
t.testEmbeddedByValue()
}
s.RegisterService(&LogIngest_ServiceDesc, srv)
}
func _LogIngest_PushBatch_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(PushBatchRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(LogIngestServer).PushBatch(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: LogIngest_PushBatch_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(LogIngestServer).PushBatch(ctx, req.(*PushBatchRequest))
}
return interceptor(ctx, in, info, handler)
}
// LogIngest_ServiceDesc is the grpc.ServiceDesc for LogIngest service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
var LogIngest_ServiceDesc = grpc.ServiceDesc{
ServiceName: "sentry.logs.v1.LogIngest",
HandlerType: (*LogIngestServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "PushBatch",
Handler: _LogIngest_PushBatch_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "sentry/logs/v1/logs.proto",
}
+10
View File
@@ -0,0 +1,10 @@
# One-shot migration runner: bash + curl baked in, migrations/*.sql copied
# in at build time. No runtime package install and no host volume mount
# needed — works offline once built.
# docker build -f storage/Dockerfile -t sentry-clickhouse-migrate storage/
FROM alpine:3.20
RUN apk add --no-cache bash curl
WORKDIR /storage
COPY migrate.sh ./
COPY migrations ./migrations
ENTRYPOINT ["bash", "migrate.sh"]
+84
View File
@@ -0,0 +1,84 @@
# storage
ClickHouse schema and migration tooling for Sentry's analytical store.
## Schema
One table for Phase 0, `logs`:
```sql
CREATE TABLE logs
(
`timestamp` DateTime64(9, 'UTC'),
`host` String,
`service` String,
`severity` LowCardinality(String),
`message` String,
`attributes` Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, timestamp)
```
Notes on choices that weren't fully specified by the task description:
- **`DateTime64(9, 'UTC')`** (nanosecond precision) rather than second or
millisecond precision, to match the agent's `timestamp_unix_nano` field
end to end without truncation.
- **`severity` as `LowCardinality(String)`**, not a numeric OTel
`SeverityNumber`. `/ingest`'s `normalize` package writes short text
values (`TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR`/`FATAL`/`UNSPECIFIED`).
`LowCardinality` gets you most of the storage/query efficiency of an enum
without committing to one at the schema level. Splitting into a proper
`SeverityNumber` + `SeverityText` pair (full OTel shape) is one of the
open questions already flagged in `/docs/architecture.md`.
- **`PARTITION BY toDate(timestamp)`** (daily partitions) and
**`ORDER BY (service, timestamp)`** are exactly what the task asked for
— service-scoped queries over a time range are the dominant access
pattern this is optimized for.
- No TTL/retention clause yet — also an open question in architecture.md,
deferred until storage sizing is a real concern.
## Migration tooling: a plain SQL-file runner, not golang-migrate
`migrate.sh` applies `migrations/*.sql` in filename order over
ClickHouse's HTTP interface, tracking what's applied in a
`schema_migrations` table. Chosen over `golang-migrate` for Phase 0
because there's exactly one migration to run — pulling in a migration
framework (another dependency, another thing to configure/vendor) for a
single `CREATE TABLE` is exactly the kind of premature machinery this
project's conventions say to avoid. Revisit `golang-migrate` once there's
real schema churn across environments (rollback support, checksums,
concurrent-apply safety become worth their cost at that point, not before).
**Convention:** one DDL statement per migration file. The ClickHouse HTTP
interface isn't reliably multi-statement, so `migrate.sh` doesn't try to
split multi-statement files — keep each migration to a single statement.
## Running
```sh
docker compose up -d # starts a standalone ClickHouse for local work
./migrate.sh # applies migrations/*.sql
```
Environment variables `migrate.sh` reads (all optional, matching
`/ingest`'s ClickHouse defaults so the two stay in sync out of the box):
| Var | Default |
|---|---|
| `CLICKHOUSE_HTTP` | `http://localhost:8123` |
| `CLICKHOUSE_USER` | `default` |
| `CLICKHOUSE_PASSWORD` | (empty) |
| `CLICKHOUSE_DATABASE` | `sentry` |
There's also a `Dockerfile` (bash + curl baked in, `migrations/` copied in
at build time) used by the root-level `docker-compose.yml` as a one-shot
init service — no runtime package install, no host volume mount needed.
## Adding a migration
Add `migrations/000N_description.sql` with the next sequential number and
a single DDL statement. `migrate.sh` picks it up automatically — no
registration step.
+20
View File
@@ -0,0 +1,20 @@
# Standalone ClickHouse for local development against /storage in
# isolation (e.g. iterating on migrations). The root-level docker-compose.yml
# runs the full Phase 0 stack and defines its own clickhouse service
# separately — this file is not included by it.
services:
clickhouse:
image: clickhouse/clickhouse-server:24.8
container_name: sentry-clickhouse
ports:
- "8123:8123" # HTTP interface, used by migrate.sh
- "9000:9000" # native protocol, used by ingest
volumes:
- clickhouse-data:/var/lib/clickhouse
ulimits:
nofile:
soft: 262144
hard: 262144
volumes:
clickhouse-data:
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# Applies migrations/*.sql to ClickHouse in filename order, tracking what's
# already been applied in a schema_migrations table. Talks to ClickHouse's
# HTTP interface via curl rather than requiring the clickhouse-client
# binary — nothing to install beyond curl, works identically on a dev
# laptop or in CI.
#
# Convention: exactly one DDL statement per migration file. The ClickHouse
# HTTP interface isn't reliably multi-statement, so keeping migrations to
# one statement each avoids relying on that.
set -euo pipefail
CLICKHOUSE_HTTP="${CLICKHOUSE_HTTP:-http://localhost:8123}"
CLICKHOUSE_USER="${CLICKHOUSE_USER:-default}"
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}"
DATABASE="${CLICKHOUSE_DATABASE:-sentry}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MIGRATIONS_DIR="${SCRIPT_DIR}/migrations"
ch_exec() {
# $1 = SQL statement, $2 = optional database to scope the query to.
local sql="$1"
local db="${2:-}"
local url="${CLICKHOUSE_HTTP}/"
if [[ -n "$db" ]]; then
url="${CLICKHOUSE_HTTP}/?database=${db}"
fi
curl -sS -f -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" "$url" --data-binary "$sql"
}
echo "Ensuring database '${DATABASE}' exists..."
ch_exec "CREATE DATABASE IF NOT EXISTS ${DATABASE}"
echo "Ensuring schema_migrations table exists..."
ch_exec "CREATE TABLE IF NOT EXISTS schema_migrations (version String, applied_at DateTime DEFAULT now()) ENGINE = MergeTree ORDER BY version" "$DATABASE"
applied="$(ch_exec "SELECT version FROM schema_migrations FORMAT TabSeparated" "$DATABASE")"
shopt -s nullglob
for file in "${MIGRATIONS_DIR}"/*.sql; do
version="$(basename "$file")"
if grep -qx "$version" <<< "$applied"; then
echo "skip ${version} (already applied)"
continue
fi
echo "apply ${version}"
ch_exec "$(cat "$file")" "$DATABASE" > /dev/null
ch_exec "INSERT INTO schema_migrations (version) VALUES ('${version}')" "$DATABASE" > /dev/null
done
echo "Migrations complete."
@@ -0,0 +1,12 @@
CREATE TABLE IF NOT EXISTS logs
(
`timestamp` DateTime64(9, 'UTC'),
`host` String,
`service` String,
`severity` LowCardinality(String),
`message` String,
`attributes` Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, timestamp)
+7
View File
@@ -0,0 +1,7 @@
# Built FROM the Redpanda image so `rpk` is already present -- no need for
# a separate client install, and no Docker socket access needed since
# provisioning happens over the network, not via `docker exec`.
# docker build -f transport/Dockerfile -t sentry-transport-provision transport/
FROM docker.redpanda.com/redpandadata/redpanda:v24.2.7
COPY provision-topics.sh /provision-topics.sh
ENTRYPOINT ["/provision-topics.sh"]
+26
View File
@@ -0,0 +1,26 @@
# transport
Redpanda for local development, plus the script that provisions the topic
`/ingest` depends on.
## Topic naming contract
`ingest` defaults to `REDPANDA_TOPIC=sentry.logs.raw` (see
`/ingest/internal/config`). `provision-topics.sh` defaults to the same
name. These aren't wired together automatically — if you change one,
change the other, or override `REDPANDA_TOPIC` consistently wherever
both are invoked.
## Running standalone
```sh
docker compose up -d
REDPANDA_BROKERS=localhost:9092 ./provision-topics.sh
```
## In the full stack
The root-level `docker-compose.yml` builds this directory's `Dockerfile`
(FROM the Redpanda image itself, so `rpk` is already present) as a
one-shot init service that runs after Redpanda reports healthy. See
`/docs/phase-0-runbook.md`.
+34
View File
@@ -0,0 +1,34 @@
# Standalone Redpanda for local development against /transport in
# isolation. The root-level docker-compose.yml runs the full Phase 0 stack
# and defines its own redpanda service separately — this file is not
# included by it.
#
# Note advertise-kafka-addr is "localhost" here (host tools connect via the
# mapped port), vs "redpanda" in the root compose (other containers connect
# via the compose network's service DNS name). Getting this wrong is the
# classic Redpanda/Kafka docker-compose footgun — clients can connect
# initially but then fail on the broker's advertised address once they try
# to actually produce/consume.
services:
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
container_name: sentry-redpanda
command:
- redpanda
- start
- --smp=1
- --memory=1G
- --reserve-memory=0M
- --overprovisioned
- --node-id=0
- --check=false
- --kafka-addr=PLAINTEXT://0.0.0.0:9092
- --advertise-kafka-addr=PLAINTEXT://localhost:9092
ports:
- "9092:9092"
- "9644:9644" # admin API, used by rpk/healthchecks
volumes:
- redpanda-data:/var/lib/redpanda/data
volumes:
redpanda-data:
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Idempotently creates the topic ingest produces/consumes. Talks to
# Redpanda over the network via `rpk`, not `docker exec` into the broker
# container -- this way the same script works whether it's run from the
# host (against the standalone compose in this directory), from inside a
# sibling container on the root compose's network, or in CI.
set -euo pipefail
BROKERS="${REDPANDA_BROKERS:-localhost:9092}"
TOPIC="${REDPANDA_TOPIC:-sentry.logs.raw}"
PARTITIONS="${REDPANDA_TOPIC_PARTITIONS:-6}"
echo "Waiting for Redpanda at ${BROKERS}..."
until rpk cluster health --brokers "${BROKERS}" --exit-when-healthy > /dev/null 2>&1; do
sleep 1
done
if rpk topic list --brokers "${BROKERS}" | awk 'NR>1{print $1}' | grep -qx "${TOPIC}"; then
echo "Topic '${TOPIC}' already exists, skipping."
else
echo "Creating topic '${TOPIC}' (${PARTITIONS} partitions)..."
rpk topic create "${TOPIC}" --brokers "${BROKERS}" --partitions "${PARTITIONS}" --replicas 1
fi
+4
View File
@@ -0,0 +1,4 @@
# Base URL of the /api service. Baked into the static build at build time
# (this is a prerendered SPA, not a server) — set this before `npm run
# build` / `docker build`, not at container start.
VITE_API_BASE_URL=http://localhost:8080
+23
View File
@@ -0,0 +1,23 @@
node_modules
# Output
.output
.vercel
.netlify
.wrangler
/.svelte-kit
/build
# OS
.DS_Store
Thumbs.db
# Env
.env
.env.*
!.env.example
!.env.test
# Vite
vite.config.js.timestamp-*
vite.config.ts.timestamp-*
+1
View File
@@ -0,0 +1 @@
engine-strict=true
+3
View File
@@ -0,0 +1,3 @@
{
"recommendations": ["svelte.svelte-vscode"]
}
+24
View File
@@ -0,0 +1,24 @@
# Build context can be just web/ (unlike agent/ingest/api, this doesn't
# need /proto):
# docker build -f web/Dockerfile -t sentry-web web/
FROM node:22-alpine AS builder
WORKDIR /src
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
# VITE_API_BASE_URL is baked in at build time — this is a prerendered
# static site, not a server. Override with --build-arg for non-default
# deployments.
ARG VITE_API_BASE_URL=http://localhost:8080
ENV VITE_API_BASE_URL=${VITE_API_BASE_URL}
RUN npm run build
# Not distroless: serving a static SPA needs *some* HTTP server, and
# nginx:alpine is the boring, well-understood choice for that job — a
# custom static-file-serving binary would be more engineering than a
# Phase 0 placeholder page warrants. See /web/README.md.
FROM nginx:alpine
COPY --from=builder /src/build /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 3000
+49
View File
@@ -0,0 +1,49 @@
# web
SvelteKit frontend. Phase 0: one page, one query box, one table. No auth,
no styling polish, no routing beyond `/`.
## What it does
Textarea for a raw SQL string → `POST {VITE_API_BASE_URL}/query` on `/api`
→ renders `{columns, rows}` as an HTML table, or shows `{error}` from a
rejected/failed query. That's the whole app — see `src/routes/+page.svelte`.
## Why a static build, not a Node server
Scaffolded with `@sveltejs/adapter-static`: this page has no server-side
data loading (all data comes from a client-side `fetch` triggered by the
submit button), so there's nothing here that needs a running SvelteKit
server. A prerendered static site is simpler to build, deploy, and reason
about than running Node in production for a page that's this thin.
Because it's static, `VITE_API_BASE_URL` is baked in at **build time**, not
read at container start. Set it before `npm run build` (or pass
`--build-arg VITE_API_BASE_URL=...` to `docker build`) — changing it later
means rebuilding, not just restarting the container.
## Building & running
```sh
npm install
cp .env.example .env # adjust VITE_API_BASE_URL if /api isn't on localhost:8080
npm run dev # local dev server with hot reload
npm run check # svelte-check, type errors
npm run build # static output to build/
npm run preview # serve the static build locally to sanity-check it
```
```sh
docker build -f Dockerfile -t sentry-web . # context is web/, not the repo root
docker run -p 3000:3000 sentry-web
```
## Why nginx, not distroless
The repo convention prefers distroless/scratch base images. Serving a
static SPA still needs *some* HTTP server, though, and `nginx:alpine` is
the boring, standard choice for that job — writing a custom static-file
binary just to stay distroless would be more engineering than a Phase 0
placeholder page justifies. `nginx.conf` here is minimal: serve `build/`,
fall back to `index.html` for client-side routing (only one route exists
today, but this is what you want the moment a second one is added).
+9
View File
@@ -0,0 +1,9 @@
server {
listen 3000;
root /usr/share/nginx/html;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
}
+1338
View File
File diff suppressed because it is too large Load Diff
+23
View File
@@ -0,0 +1,23 @@
{
"name": "web",
"private": true,
"version": "0.0.1",
"type": "module",
"scripts": {
"dev": "vite dev",
"build": "vite build",
"preview": "vite preview",
"prepare": "svelte-kit sync || echo ''",
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
"check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch"
},
"devDependencies": {
"@sveltejs/adapter-static": "^3.0.10",
"@sveltejs/kit": "^2.63.0",
"@sveltejs/vite-plugin-svelte": "^7.1.2",
"svelte": "^5.56.1",
"svelte-check": "^4.6.0",
"typescript": "^6.0.3",
"vite": "^8.0.16"
}
}
+13
View File
@@ -0,0 +1,13 @@
// See https://svelte.dev/docs/kit/types#app.d.ts
// for information about these interfaces
declare global {
namespace App {
// interface Error {}
// interface Locals {}
// interface PageData {}
// interface PageState {}
// interface Platform {}
}
}
export {};
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="text-scale" content="scale" />
%sveltekit.head%
</head>
<body data-sveltekit-preload-data="hover">
<div style="display: contents">%sveltekit.body%</div>
</body>
</html>
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="107" height="128" viewBox="0 0 107 128"><title>svelte-logo</title><path d="M94.157 22.819c-10.4-14.885-30.94-19.297-45.792-9.835L22.282 29.608A29.92 29.92 0 0 0 8.764 49.65a31.5 31.5 0 0 0 3.108 20.231 30 30 0 0 0-4.477 11.183 31.9 31.9 0 0 0 5.448 24.116c10.402 14.887 30.942 19.297 45.791 9.835l26.083-16.624A29.92 29.92 0 0 0 98.235 78.35a31.53 31.53 0 0 0-3.105-20.232 30 30 0 0 0 4.474-11.182 31.88 31.88 0 0 0-5.447-24.116" style="fill:#ff3e00"/><path d="M45.817 106.582a20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.503 18 18 0 0 1 .624-2.435l.49-1.498 1.337.981a33.6 33.6 0 0 0 10.203 5.098l.97.294-.09.968a5.85 5.85 0 0 0 1.052 3.878 6.24 6.24 0 0 0 6.695 2.485 5.8 5.8 0 0 0 1.603-.704L69.27 76.28a5.43 5.43 0 0 0 2.45-3.631 5.8 5.8 0 0 0-.987-4.371 6.24 6.24 0 0 0-6.698-2.487 5.7 5.7 0 0 0-1.6.704l-9.953 6.345a19 19 0 0 1-5.296 2.326 20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.502 17.99 17.99 0 0 1 8.13-12.052l26.081-16.623a19 19 0 0 1 5.3-2.329 20.72 20.72 0 0 1 22.237 8.243 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-.624 2.435l-.49 1.498-1.337-.98a33.6 33.6 0 0 0-10.203-5.1l-.97-.294.09-.968a5.86 5.86 0 0 0-1.052-3.878 6.24 6.24 0 0 0-6.696-2.485 5.8 5.8 0 0 0-1.602.704L37.73 51.72a5.42 5.42 0 0 0-2.449 3.63 5.79 5.79 0 0 0 .986 4.372 6.24 6.24 0 0 0 6.698 2.486 5.8 5.8 0 0 0 1.602-.704l9.952-6.342a19 19 0 0 1 5.295-2.328 20.72 20.72 0 0 1 22.237 8.242 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-8.13 12.053l-26.081 16.622a19 19 0 0 1-5.3 2.328" style="fill:#fff"/></svg>

After

Width:  |  Height:  |  Size: 1.5 KiB

+1
View File
@@ -0,0 +1 @@
// place files you want to import through the `$lib` alias in this folder.
+12
View File
@@ -0,0 +1,12 @@
<script lang="ts">
import favicon from '$lib/assets/favicon.svg';
let { children } = $props();
</script>
<svelte:head>
<title>Sentry</title>
<link rel="icon" href={favicon} />
</svelte:head>
{@render children()}
+128
View File
@@ -0,0 +1,128 @@
<script lang="ts">
// Phase 0: functional only, no styling polish, no auth. One page: a raw
// SQL box against POST /query on the api service, rendered as a table.
// This is a placeholder for the real query UI that lands once /api grows
// a real SPL-like query layer in Phase 2.
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
let sql = $state('SELECT * FROM logs ORDER BY timestamp DESC LIMIT 100');
let columns = $state<string[]>([]);
let rows = $state<unknown[][]>([]);
let error = $state('');
let loading = $state(false);
let hasRun = $state(false);
function formatCell(value: unknown): string {
if (value === null || value === undefined) return '';
if (typeof value === 'object') return JSON.stringify(value);
return String(value);
}
async function runQuery() {
loading = true;
error = '';
try {
const res = await fetch(`${apiBase}/query`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ sql })
});
const body = await res.json();
if (!res.ok) {
error = body?.error ?? `request failed with status ${res.status}`;
columns = [];
rows = [];
return;
}
columns = body.columns ?? [];
rows = body.rows ?? [];
} catch (e) {
error = e instanceof Error ? e.message : String(e);
columns = [];
rows = [];
} finally {
loading = false;
hasRun = true;
}
}
</script>
<main>
<h1>Sentry — Log Query (Phase 0)</h1>
<p>
Raw SQL only, SELECT statements against the <code>logs</code> table. No auth, no query
builder yet — see <code>/api</code> for what's actually allowed.
</p>
<textarea bind:value={sql} rows="4" cols="100" spellcheck="false"></textarea>
<div>
<button onclick={runQuery} disabled={loading}>
{loading ? 'Running…' : 'Run query'}
</button>
</div>
{#if error}
<p class="error">Error: {error}</p>
{/if}
{#if hasRun && !error}
<p>{rows.length} row(s)</p>
{/if}
{#if columns.length > 0}
<table>
<thead>
<tr>
{#each columns as col (col)}
<th>{col}</th>
{/each}
</tr>
</thead>
<tbody>
{#each rows as row, i (i)}
<tr>
{#each row as cell, j (j)}
<td>{formatCell(cell)}</td>
{/each}
</tr>
{/each}
</tbody>
</table>
{/if}
</main>
<style>
main {
font-family: system-ui, sans-serif;
max-width: 960px;
margin: 2rem auto;
padding: 0 1rem;
}
textarea {
width: 100%;
font-family: monospace;
font-size: 0.9rem;
}
button {
margin-top: 0.5rem;
}
.error {
color: #b00020;
}
table {
border-collapse: collapse;
width: 100%;
margin-top: 1rem;
}
th,
td {
border: 1px solid #ccc;
padding: 0.25rem 0.5rem;
text-align: left;
font-size: 0.85rem;
}
th {
background: #f0f0f0;
}
</style>
+4
View File
@@ -0,0 +1,4 @@
// Static adapter needs every route prerenderable. This page has no load
// function (all data comes from a client-side fetch on submit), so a plain
// prerender is enough — no need to disable SSR.
export const prerender = true;
+1
View File
@@ -0,0 +1 @@
/// <reference types="vite/client" />
+3
View File
@@ -0,0 +1,3 @@
# allow crawling everything by default
User-agent: *
Disallow:
+20
View File
@@ -0,0 +1,20 @@
{
"extends": "./.svelte-kit/tsconfig.json",
"compilerOptions": {
"rewriteRelativeImportExtensions": true,
"allowJs": true,
"checkJs": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"moduleResolution": "bundler"
}
// Path aliases are handled by https://svelte.dev/docs/kit/configuration#alias
// except $lib which is handled by https://svelte.dev/docs/kit/configuration#files
//
// To make changes to top-level options such as include and exclude, we recommend extending
// the generated config; see https://svelte.dev/docs/kit/configuration#typescript
}
+15
View File
@@ -0,0 +1,15 @@
import adapter from '@sveltejs/adapter-static';
import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vite';
export default defineConfig({
plugins: [
sveltekit({
compilerOptions: {
// Force runes mode for the project, except for libraries. Can be removed in svelte 6.
runes: ({ filename }) => filename.split(/[/\\]/).includes('node_modules') ? undefined : true
},
adapter: adapter()
})
]
});