Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
# ingest/ and api/ build with context "." (repo root) so their Dockerfiles
|
||||
# can also COPY proto/. Keep that context lean.
|
||||
.git/
|
||||
agent/target/
|
||||
web/node_modules/
|
||||
web/build/
|
||||
web/.svelte-kit/
|
||||
hack/dev-certs/out/
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# Rust
|
||||
agent/target/
|
||||
|
||||
# Go build cache (go build ./... without -o doesn't normally leave
|
||||
# binaries in-tree, but be defensive)
|
||||
/ingest/ingest
|
||||
/api/api
|
||||
/cli/sentryctl
|
||||
|
||||
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
||||
web/node_modules/
|
||||
web/build/
|
||||
web/.svelte-kit/
|
||||
|
||||
# Dev-only generated secrets
|
||||
hack/dev-certs/out/
|
||||
|
||||
# OS / editor
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
*.swp
|
||||
Generated
+1510
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
[workspace]
|
||||
resolver = "2"
|
||||
members = ["sentry-parser", "sentry-agent"]
|
||||
|
||||
[workspace.package]
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
license = "AGPL-3.0-only"
|
||||
|
||||
[profile.release]
|
||||
lto = true
|
||||
strip = true
|
||||
codegen-units = 1
|
||||
@@ -0,0 +1,16 @@
|
||||
# Build context must be the repo root (sentry/), not agent/, since this
|
||||
# needs both agent/ and proto/:
|
||||
# docker build -f agent/Dockerfile -t sentry-agent .
|
||||
|
||||
FROM rust:1-alpine AS builder
|
||||
RUN apk add --no-cache musl-dev protobuf-dev protobuf
|
||||
WORKDIR /src
|
||||
COPY proto ./proto
|
||||
COPY agent ./agent
|
||||
WORKDIR /src/agent
|
||||
RUN rustup target add x86_64-unknown-linux-musl \
|
||||
&& cargo build --release --target x86_64-unknown-linux-musl -p sentry-agent
|
||||
|
||||
FROM scratch
|
||||
COPY --from=builder /src/agent/target/x86_64-unknown-linux-musl/release/sentry-agent /sentry-agent
|
||||
ENTRYPOINT ["/sentry-agent"]
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
# sentry-agent
|
||||
|
||||
Distro-agnostic Linux log collector. Statically linked against musl, no
|
||||
glibc runtime dependency. Tails journald (default) or a file, batches
|
||||
lines, and ships them over mTLS gRPC to the ingest service.
|
||||
|
||||
## Workspace layout
|
||||
|
||||
- `sentry-parser` — pure-`std` RFC 5424 syslog parser with raw-passthrough
|
||||
fallback. No I/O, easy to unit test in isolation.
|
||||
- `sentry-agent` — the binary: config loading, sourcing (journald/file),
|
||||
batching, mTLS gRPC client.
|
||||
|
||||
## Why journalctl, not libsystemd
|
||||
|
||||
The journald source shells out to `journalctl -f -o json` rather than
|
||||
linking `libsystemd` via FFI. Statically linking libsystemd into a musl
|
||||
binary is fragile — it pulls in dbus/libcap transitively and isn't designed
|
||||
for static linking — and would undermine the no-glibc-runtime-deps goal
|
||||
even where technically possible. `journalctl` ships on every systemd distro
|
||||
this agent targets, so shelling out sidesteps the problem entirely. See
|
||||
`/docs/architecture.md`.
|
||||
|
||||
## Building
|
||||
|
||||
Native build (whatever target your machine is):
|
||||
|
||||
```sh
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
musl targets (what actually ships):
|
||||
|
||||
```sh
|
||||
rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl
|
||||
|
||||
# x86_64: works with musl-gcc installed locally (musl-tools on Debian,
|
||||
# musl on Arch, etc.) — the musl target is fully static by default.
|
||||
cargo build --release --target x86_64-unknown-linux-musl
|
||||
|
||||
# aarch64 cross-compilation needs a cross toolchain; the boring, reliable
|
||||
# option is `cross` (https://github.com/cross-rs/cross), which builds
|
||||
# inside a Docker container with the right linker preinstalled:
|
||||
cross build --release --target aarch64-unknown-linux-musl
|
||||
```
|
||||
|
||||
Building requires `protoc` on PATH (used by `tonic-build`/`prost-build` at
|
||||
compile time to generate the gRPC client from `/proto/sentry/logs/v1/logs.proto`).
|
||||
|
||||
Container build (see caveat below):
|
||||
|
||||
```sh
|
||||
# from the repo root, not agent/
|
||||
docker build -f agent/Dockerfile -t sentry-agent .
|
||||
```
|
||||
|
||||
**Caveat:** the container image is provided for CI/completeness, but
|
||||
journald sourcing needs `journalctl` and access to the host journal —
|
||||
neither of which exist in the `scratch` image or are available to a
|
||||
container without deliberately bind-mounting `/var/log/journal` (or
|
||||
`/run/log/journal`) and the `journalctl` binary in. The intended Phase 0
|
||||
deployment for journald sourcing is as a native binary managed by systemd
|
||||
on the host, not containerized.
|
||||
|
||||
## Running
|
||||
|
||||
No CLI flags are required for the common case:
|
||||
|
||||
```sh
|
||||
./sentry-agent
|
||||
```
|
||||
|
||||
This uses `/etc/sentry-agent/agent.toml` if present, otherwise built-in
|
||||
defaults: journald source (whole journal, no unit filter), service name
|
||||
`default`, and mTLS material expected at
|
||||
`/etc/sentry-agent/{ca,client,client-key}.pem`. mTLS is mandatory per the
|
||||
project's transport requirements, so a from-scratch run with no certs in
|
||||
place will fail fast with a clear error rather than connecting insecurely.
|
||||
|
||||
See `config/agent.example.toml` for all fields.
|
||||
|
||||
```sh
|
||||
./sentry-agent --config /path/to/agent.toml
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
cargo test --workspace
|
||||
```
|
||||
|
||||
## Feature flags
|
||||
|
||||
- `journald` (default) — journalctl-based journald source.
|
||||
- `file-tail` — polling-based file tailer (no inotify dependency; doesn't
|
||||
follow rename-based log rotation yet).
|
||||
|
||||
Both can be enabled together; `[source].kind` in config picks which one
|
||||
runs. Building without a feature and configuring that source at runtime
|
||||
fails at startup with a clear error rather than silently doing nothing.
|
||||
@@ -0,0 +1,3 @@
|
||||
[toolchain]
|
||||
channel = "stable"
|
||||
targets = ["x86_64-unknown-linux-musl", "aarch64-unknown-linux-musl"]
|
||||
@@ -0,0 +1,34 @@
|
||||
[package]
|
||||
name = "sentry-agent"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
description = "Sentry distro-agnostic Linux log collector"
|
||||
|
||||
[[bin]]
|
||||
name = "sentry-agent"
|
||||
path = "src/main.rs"
|
||||
|
||||
[features]
|
||||
default = ["journald"]
|
||||
journald = []
|
||||
file-tail = []
|
||||
|
||||
[dependencies]
|
||||
sentry-parser = { path = "../sentry-parser" }
|
||||
|
||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "process", "io-util", "io-std", "time", "fs", "sync", "signal"] }
|
||||
tonic = { version = "0.12", features = ["tls"] }
|
||||
prost = "0.13"
|
||||
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
toml = "0.8"
|
||||
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
anyhow = "1"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
|
||||
[build-dependencies]
|
||||
tonic-build = "0.12"
|
||||
@@ -0,0 +1,9 @@
|
||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
tonic_build::configure()
|
||||
.build_server(false)
|
||||
.compile_protos(
|
||||
&["../../proto/sentry/logs/v1/logs.proto"],
|
||||
&["../../proto"],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
# Example sentry-agent config. Copy to /etc/sentry-agent/agent.toml, or
|
||||
# pass --config /path/to/this/file.
|
||||
#
|
||||
# Every field has a built-in default (see src/config.rs), so this file only
|
||||
# needs to contain what you're overriding. An agent with NO config file at
|
||||
# all still runs: it defaults to journald, service = "default", and expects
|
||||
# mTLS material at /etc/sentry-agent/{ca,client,client-key}.pem.
|
||||
|
||||
[agent]
|
||||
# host = "explicit-hostname-override" # defaults to /etc/hostname
|
||||
service = "my-service"
|
||||
|
||||
[source]
|
||||
kind = "journald"
|
||||
# unit = "nginx.service" # omit to tail the whole journal
|
||||
|
||||
# To tail a file instead:
|
||||
# [source]
|
||||
# kind = "file"
|
||||
# path = "/var/log/nginx/access.log"
|
||||
# from_beginning = false
|
||||
|
||||
[batch]
|
||||
max_size = 500
|
||||
flush_interval_ms = 2000
|
||||
|
||||
[ingest]
|
||||
endpoint = "https://ingest.internal:4317"
|
||||
|
||||
[tls]
|
||||
ca_cert = "/etc/sentry-agent/ca.pem"
|
||||
client_cert = "/etc/sentry-agent/client.pem"
|
||||
client_key = "/etc/sentry-agent/client-key.pem"
|
||||
@@ -0,0 +1,108 @@
|
||||
use crate::pb::LogRecord;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// Buffers `LogRecord`s and signals when to flush, either because the
|
||||
/// buffer hit `max_size` (checked on every push) or because
|
||||
/// `flush_interval` elapsed since the last flush (checked by the caller via
|
||||
/// `poll_timeout` on a timer tick). Not thread-safe by design — one
|
||||
/// batcher per agent, driven from a single async task's select loop.
|
||||
pub struct Batcher {
|
||||
max_size: usize,
|
||||
flush_interval: Duration,
|
||||
buf: Vec<LogRecord>,
|
||||
last_flush: Instant,
|
||||
}
|
||||
|
||||
impl Batcher {
|
||||
pub fn new(max_size: usize, flush_interval: Duration) -> Self {
|
||||
Self {
|
||||
max_size,
|
||||
flush_interval,
|
||||
buf: Vec::with_capacity(max_size),
|
||||
last_flush: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Push a record. Returns the drained batch if this push filled the
|
||||
/// buffer to `max_size`.
|
||||
pub fn push(&mut self, record: LogRecord) -> Option<Vec<LogRecord>> {
|
||||
self.buf.push(record);
|
||||
if self.buf.len() >= self.max_size {
|
||||
Some(self.drain())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Call periodically (e.g. from a timer tick). Returns the drained
|
||||
/// batch if the flush interval has elapsed and there's anything
|
||||
/// buffered.
|
||||
pub fn poll_timeout(&mut self) -> Option<Vec<LogRecord>> {
|
||||
if !self.buf.is_empty() && self.last_flush.elapsed() >= self.flush_interval {
|
||||
Some(self.drain())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn drain(&mut self) -> Vec<LogRecord> {
|
||||
self.last_flush = Instant::now();
|
||||
std::mem::replace(&mut self.buf, Vec::with_capacity(self.max_size))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn rec(msg: &str) -> LogRecord {
|
||||
LogRecord {
|
||||
timestamp_unix_nano: 0,
|
||||
host: "h".into(),
|
||||
service: "s".into(),
|
||||
severity: 0,
|
||||
message: msg.into(),
|
||||
attributes: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flushes_on_size() {
|
||||
let mut b = Batcher::new(2, Duration::from_secs(999));
|
||||
assert!(b.push(rec("a")).is_none());
|
||||
let batch = b.push(rec("b")).expect("should flush at max_size");
|
||||
assert_eq!(batch.len(), 2);
|
||||
assert_eq!(batch[0].message, "a");
|
||||
assert_eq!(batch[1].message, "b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn buffer_empty_after_size_flush() {
|
||||
let mut b = Batcher::new(1, Duration::from_secs(999));
|
||||
b.push(rec("a")).expect("flush at max_size 1");
|
||||
assert!(b.poll_timeout().is_none(), "buffer should be empty post-flush");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flushes_on_timeout() {
|
||||
let mut b = Batcher::new(100, Duration::from_millis(10));
|
||||
assert!(b.push(rec("a")).is_none());
|
||||
std::thread::sleep(Duration::from_millis(30));
|
||||
let batch = b.poll_timeout().expect("should flush after timeout");
|
||||
assert_eq!(batch.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_flush_when_buffer_empty() {
|
||||
let mut b = Batcher::new(10, Duration::from_millis(1));
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
assert!(b.poll_timeout().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_flush_before_timeout_elapsed() {
|
||||
let mut b = Batcher::new(10, Duration::from_secs(999));
|
||||
b.push(rec("a"));
|
||||
assert!(b.poll_timeout().is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
use anyhow::{Context, Result};
|
||||
use serde::Deserialize;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
const DEFAULT_CONFIG_PATH: &str = "/etc/sentry-agent/agent.toml";
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Default)]
|
||||
#[serde(default)]
|
||||
pub struct Config {
|
||||
pub agent: AgentConfig,
|
||||
pub source: SourceConfig,
|
||||
pub batch: BatchConfig,
|
||||
pub ingest: IngestConfig,
|
||||
pub tls: TlsConfig,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
/// Loads config from `explicit_path` if given, else from
|
||||
/// `/etc/sentry-agent/agent.toml` if it exists, else falls back to
|
||||
/// built-in defaults (journald source, default TLS cert paths). Only an
|
||||
/// explicitly-passed `--config` path that doesn't exist is an error;
|
||||
/// the conventional default path is optional.
|
||||
pub fn load(explicit_path: Option<&Path>) -> Result<Config> {
|
||||
let path = match explicit_path {
|
||||
Some(p) => Some(p.to_path_buf()),
|
||||
None => {
|
||||
let default = PathBuf::from(DEFAULT_CONFIG_PATH);
|
||||
default.exists().then_some(default)
|
||||
}
|
||||
};
|
||||
|
||||
match path {
|
||||
Some(p) => {
|
||||
let raw = std::fs::read_to_string(&p)
|
||||
.with_context(|| format!("reading config file {}", p.display()))?;
|
||||
toml::from_str(&raw).with_context(|| format!("parsing config file {}", p.display()))
|
||||
}
|
||||
None => Ok(Config::default()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct AgentConfig {
|
||||
/// Overrides the auto-detected system hostname. Defaults to reading
|
||||
/// /etc/hostname at startup when unset.
|
||||
pub host: Option<String>,
|
||||
pub service: String,
|
||||
}
|
||||
|
||||
impl Default for AgentConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
host: None,
|
||||
service: "default".to_string(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
#[serde(rename_all = "lowercase", tag = "kind")]
|
||||
pub enum SourceConfig {
|
||||
Journald {
|
||||
#[serde(default)]
|
||||
unit: Option<String>,
|
||||
},
|
||||
File {
|
||||
path: PathBuf,
|
||||
#[serde(default)]
|
||||
from_beginning: bool,
|
||||
},
|
||||
}
|
||||
|
||||
impl Default for SourceConfig {
|
||||
fn default() -> Self {
|
||||
SourceConfig::Journald { unit: None }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct BatchConfig {
|
||||
pub max_size: usize,
|
||||
pub flush_interval_ms: u64,
|
||||
}
|
||||
|
||||
impl Default for BatchConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_size: 500,
|
||||
flush_interval_ms: 2000,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct IngestConfig {
|
||||
pub endpoint: String,
|
||||
}
|
||||
|
||||
impl Default for IngestConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
endpoint: "https://127.0.0.1:4317".to_string(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct TlsConfig {
|
||||
pub ca_cert: PathBuf,
|
||||
pub client_cert: PathBuf,
|
||||
pub client_key: PathBuf,
|
||||
}
|
||||
|
||||
impl Default for TlsConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
ca_cert: PathBuf::from("/etc/sentry-agent/ca.pem"),
|
||||
client_cert: PathBuf::from("/etc/sentry-agent/client.pem"),
|
||||
client_key: PathBuf::from("/etc/sentry-agent/client-key.pem"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
use crate::config::{IngestConfig, TlsConfig};
|
||||
use crate::pb::{log_ingest_client::LogIngestClient, LogRecord, PushBatchRequest};
|
||||
use anyhow::{Context, Result};
|
||||
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity};
|
||||
|
||||
/// Establishes an mTLS gRPC channel to the ingest service. Agents never
|
||||
/// talk to Redpanda directly — this is the only network egress the agent
|
||||
/// has, by design (see /docs/architecture.md).
|
||||
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngestClient<Channel>> {
|
||||
let ca = tokio::fs::read(&tls.ca_cert)
|
||||
.await
|
||||
.with_context(|| format!("reading CA cert at {}", tls.ca_cert.display()))?;
|
||||
let cert = tokio::fs::read(&tls.client_cert)
|
||||
.await
|
||||
.with_context(|| format!("reading client cert at {}", tls.client_cert.display()))?;
|
||||
let key = tokio::fs::read(&tls.client_key)
|
||||
.await
|
||||
.with_context(|| format!("reading client key at {}", tls.client_key.display()))?;
|
||||
|
||||
let tls_config = ClientTlsConfig::new()
|
||||
.ca_certificate(Certificate::from_pem(ca))
|
||||
.identity(Identity::from_pem(cert, key));
|
||||
|
||||
let channel = Channel::from_shared(ingest.endpoint.clone())
|
||||
.context("invalid ingest endpoint URL")?
|
||||
.tls_config(tls_config)
|
||||
.context("configuring mTLS")?
|
||||
.connect()
|
||||
.await
|
||||
.context("connecting to ingest service")?;
|
||||
|
||||
Ok(LogIngestClient::new(channel))
|
||||
}
|
||||
|
||||
pub async fn send_batch(
|
||||
client: &mut LogIngestClient<Channel>,
|
||||
batch_id: String,
|
||||
records: Vec<LogRecord>,
|
||||
) -> Result<u32> {
|
||||
let resp = client
|
||||
.push_batch(PushBatchRequest { batch_id, records })
|
||||
.await
|
||||
.context("PushBatch RPC failed")?;
|
||||
Ok(resp.into_inner().accepted)
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
mod batch;
|
||||
mod config;
|
||||
mod grpc;
|
||||
mod source;
|
||||
|
||||
pub mod pb {
|
||||
tonic::include_proto!("sentry.logs.v1");
|
||||
}
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use batch::Batcher;
|
||||
use clap::Parser;
|
||||
use config::Config;
|
||||
use pb::{log_ingest_client::LogIngestClient, LogRecord, Severity};
|
||||
use std::path::PathBuf;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::mpsc;
|
||||
use tonic::transport::Channel;
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(name = "sentry-agent", about = "Sentry Linux log collector")]
|
||||
struct Cli {
|
||||
/// Path to a TOML config file. Defaults to /etc/sentry-agent/agent.toml
|
||||
/// if present, otherwise built-in defaults (journald source, default
|
||||
/// TLS cert paths under /etc/sentry-agent/).
|
||||
#[arg(long)]
|
||||
config: Option<PathBuf>,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||||
.init();
|
||||
|
||||
let cli = Cli::parse();
|
||||
let cfg = Config::load(cli.config.as_deref()).context("loading config")?;
|
||||
|
||||
let host = cfg.agent.host.clone().unwrap_or_else(default_hostname);
|
||||
let service = cfg.agent.service.clone();
|
||||
|
||||
let (tx, mut rx) = mpsc::channel(1024);
|
||||
let source_handle = tokio::spawn(spawn_source(cfg.source.clone(), tx));
|
||||
|
||||
let mut client = grpc::connect(&cfg.ingest, &cfg.tls)
|
||||
.await
|
||||
.context("connecting to ingest service")?;
|
||||
tracing::info!(endpoint = %cfg.ingest.endpoint, "connected to ingest service");
|
||||
|
||||
let flush_interval = Duration::from_millis(cfg.batch.flush_interval_ms);
|
||||
let mut batcher = Batcher::new(cfg.batch.max_size, flush_interval);
|
||||
let mut ticker = tokio::time::interval(flush_interval.max(Duration::from_millis(50)));
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
maybe_line = rx.recv() => {
|
||||
let Some(raw) = maybe_line else {
|
||||
tracing::warn!("source exited, flushing remaining batch and shutting down");
|
||||
break;
|
||||
};
|
||||
let parsed = sentry_parser::parse(&raw.line);
|
||||
let severity = to_pb_severity(raw.severity_hint.or(parsed.severity));
|
||||
let record = LogRecord {
|
||||
timestamp_unix_nano: raw.timestamp_unix_nano,
|
||||
host: host.clone(),
|
||||
service: service.clone(),
|
||||
severity: severity as i32,
|
||||
message: parsed.message,
|
||||
attributes: parsed.attributes.into_iter().collect(),
|
||||
};
|
||||
if let Some(batch) = batcher.push(record) {
|
||||
flush(&mut client, batch).await;
|
||||
}
|
||||
}
|
||||
_ = ticker.tick() => {
|
||||
if let Some(batch) = batcher.poll_timeout() {
|
||||
flush(&mut client, batch).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(batch) = batcher.poll_timeout() {
|
||||
flush(&mut client, batch).await;
|
||||
}
|
||||
source_handle.abort();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
|
||||
let result = match source {
|
||||
#[cfg(feature = "journald")]
|
||||
config::SourceConfig::Journald { unit } => source::journald::run(unit.as_deref(), tx).await,
|
||||
#[cfg(not(feature = "journald"))]
|
||||
config::SourceConfig::Journald { .. } => {
|
||||
Err(anyhow::anyhow!("this build was compiled without the `journald` feature"))
|
||||
}
|
||||
|
||||
#[cfg(feature = "file-tail")]
|
||||
config::SourceConfig::File { path, from_beginning } => {
|
||||
source::file_tail::run(&path, from_beginning, tx).await
|
||||
}
|
||||
#[cfg(not(feature = "file-tail"))]
|
||||
config::SourceConfig::File { .. } => {
|
||||
Err(anyhow::anyhow!("this build was compiled without the `file-tail` feature"))
|
||||
}
|
||||
};
|
||||
if let Err(e) = result {
|
||||
tracing::error!(error = %e, "log source exited with error");
|
||||
}
|
||||
}
|
||||
|
||||
async fn flush(client: &mut LogIngestClient<Channel>, batch: Vec<LogRecord>) {
|
||||
let n = batch.len();
|
||||
let batch_id = batch_id();
|
||||
match grpc::send_batch(client, batch_id, batch).await {
|
||||
Ok(accepted) => tracing::debug!(accepted, sent = n, "batch flushed"),
|
||||
Err(e) => tracing::error!(error = %e, sent = n, "batch flush failed"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Best-effort batch identifier for ingest-side dedup on retry. Not
|
||||
/// globally unique (host + nanosecond timestamp), which is good enough for
|
||||
/// Phase 0's single-agent-per-host reality; revisit if agents ever share
|
||||
/// an identity or clock resolution becomes a problem.
|
||||
fn batch_id() -> String {
|
||||
let nanos = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
format!("{nanos:x}")
|
||||
}
|
||||
|
||||
fn to_pb_severity(sev: Option<u8>) -> Severity {
|
||||
match sev {
|
||||
Some(0..=2) => Severity::Fatal, // emerg / alert / crit
|
||||
Some(3) => Severity::Error, // err
|
||||
Some(4) => Severity::Warn, // warning
|
||||
Some(5) | Some(6) => Severity::Info, // notice / info
|
||||
Some(7) => Severity::Debug, // debug
|
||||
_ => Severity::Unspecified,
|
||||
}
|
||||
}
|
||||
|
||||
fn default_hostname() -> String {
|
||||
if let Ok(s) = std::fs::read_to_string("/etc/hostname") {
|
||||
let s = s.trim().to_string();
|
||||
if !s.is_empty() {
|
||||
return s;
|
||||
}
|
||||
}
|
||||
std::env::var("HOSTNAME").unwrap_or_else(|_| "unknown-host".to_string())
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
use super::{LineSender, RawLine};
|
||||
use anyhow::{Context, Result};
|
||||
use std::io::SeekFrom;
|
||||
use std::path::Path;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use tokio::fs::File;
|
||||
use tokio::io::{AsyncBufReadExt, AsyncSeekExt, BufReader};
|
||||
|
||||
const POLL_INTERVAL: Duration = Duration::from_millis(500);
|
||||
|
||||
/// Polling-based file tailer: no inotify/`notify` crate dependency. Good
|
||||
/// enough for Phase 0 (journald is the primary source). Handles basic
|
||||
/// truncation (e.g. logrotate `copytruncate`) by detecting the file shrank
|
||||
/// and reopening from the start. Does not follow rename-based rotation
|
||||
/// (logrotate `create`) — that's deferred until file-tail is more than a
|
||||
/// fallback path.
|
||||
pub async fn run(path: &Path, from_beginning: bool, tx: LineSender) -> Result<()> {
|
||||
let file = File::open(path)
|
||||
.await
|
||||
.with_context(|| format!("opening {}", path.display()))?;
|
||||
|
||||
let mut pos = if from_beginning { 0 } else { file.metadata().await?.len() };
|
||||
|
||||
let mut reader = BufReader::new(file);
|
||||
reader.seek(SeekFrom::Start(pos)).await?;
|
||||
let mut buf = String::new();
|
||||
|
||||
loop {
|
||||
buf.clear();
|
||||
let n = reader
|
||||
.read_line(&mut buf)
|
||||
.await
|
||||
.context("reading line from file")?;
|
||||
if n == 0 {
|
||||
let metadata = tokio::fs::metadata(path).await.context("stat-ing file")?;
|
||||
if metadata.len() < pos {
|
||||
tracing::warn!(path = %path.display(), "file shrank, assuming truncation and reopening from start");
|
||||
let f = File::open(path)
|
||||
.await
|
||||
.context("reopening file after truncation")?;
|
||||
reader = BufReader::new(f);
|
||||
pos = 0;
|
||||
}
|
||||
tokio::time::sleep(POLL_INTERVAL).await;
|
||||
continue;
|
||||
}
|
||||
pos += n as u64;
|
||||
|
||||
let line = buf.trim_end_matches(['\n', '\r']).to_string();
|
||||
if line.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let timestamp_unix_nano = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos() as i64)
|
||||
.unwrap_or(0);
|
||||
|
||||
if tx
|
||||
.send(RawLine {
|
||||
line,
|
||||
timestamp_unix_nano,
|
||||
severity_hint: None,
|
||||
})
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
use super::{LineSender, RawLine};
|
||||
use anyhow::{Context, Result};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::process::Command;
|
||||
|
||||
/// Reads journald entries by shelling out to `journalctl -f -o json`
|
||||
/// rather than linking libsystemd via FFI. Linking libsystemd into a
|
||||
/// statically-linked musl binary is fragile (it pulls in dbus/libcap
|
||||
/// transitively and isn't designed for static linking) and would work
|
||||
/// against the no-glibc-runtime-deps constraint in spirit even where it's
|
||||
/// technically possible. `journalctl` ships on every systemd distro this
|
||||
/// agent targets, so shelling out avoids the problem entirely. See
|
||||
/// /docs/architecture.md.
|
||||
pub async fn run(unit: Option<&str>, tx: LineSender) -> Result<()> {
|
||||
let mut cmd = Command::new("journalctl");
|
||||
cmd.arg("-f")
|
||||
.arg("-o")
|
||||
.arg("json")
|
||||
.arg("--since=now")
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::null());
|
||||
if let Some(unit) = unit {
|
||||
cmd.arg("-u").arg(unit);
|
||||
}
|
||||
|
||||
let mut child = cmd
|
||||
.spawn()
|
||||
.context("spawning journalctl -f -o json (is systemd-journal installed?)")?;
|
||||
let stdout = child.stdout.take().context("journalctl child had no stdout")?;
|
||||
let mut lines = BufReader::new(stdout).lines();
|
||||
|
||||
while let Some(line) = lines.next_line().await.context("reading journalctl output")? {
|
||||
let Ok(entry) = serde_json::from_str::<serde_json::Value>(&line) else {
|
||||
tracing::warn!(%line, "skipping unparseable journalctl JSON line");
|
||||
continue;
|
||||
};
|
||||
|
||||
let message = entry
|
||||
.get("MESSAGE")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
if message.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let severity_hint = entry
|
||||
.get("PRIORITY")
|
||||
.and_then(|v| v.as_str().map(str::to_string).or_else(|| v.as_u64().map(|n| n.to_string())))
|
||||
.and_then(|s| s.parse::<u8>().ok())
|
||||
.filter(|&p| p <= 7);
|
||||
|
||||
let timestamp_unix_nano = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos() as i64)
|
||||
.unwrap_or(0);
|
||||
|
||||
if tx
|
||||
.send(RawLine {
|
||||
line: message,
|
||||
timestamp_unix_nano,
|
||||
severity_hint,
|
||||
})
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
break; // receiver dropped, agent is shutting down
|
||||
}
|
||||
}
|
||||
|
||||
let status = child.wait().await.context("waiting for journalctl to exit")?;
|
||||
tracing::warn!(?status, "journalctl exited");
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
/// A raw line read from a source, plus whatever metadata the source itself
|
||||
/// already knows before the RFC 5424 parser ever sees it.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RawLine {
|
||||
pub line: String,
|
||||
/// Unix epoch nanoseconds at time of read.
|
||||
pub timestamp_unix_nano: i64,
|
||||
/// Syslog severity (0-7) if the source already knows it independent of
|
||||
/// the line's own content — e.g. journald's PRIORITY field. When set,
|
||||
/// this takes precedence over whatever the RFC 5424 parser infers from
|
||||
/// the message text, since it comes from a more authoritative place.
|
||||
pub severity_hint: Option<u8>,
|
||||
}
|
||||
|
||||
pub type LineSender = mpsc::Sender<RawLine>;
|
||||
|
||||
#[cfg(feature = "journald")]
|
||||
pub mod journald;
|
||||
|
||||
#[cfg(feature = "file-tail")]
|
||||
pub mod file_tail;
|
||||
@@ -0,0 +1,6 @@
|
||||
[package]
|
||||
name = "sentry-parser"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
description = "Minimal RFC 5424 syslog parser with raw-passthrough fallback"
|
||||
@@ -0,0 +1,285 @@
|
||||
//! Minimal RFC 5424 syslog parser with raw-passthrough fallback.
|
||||
//!
|
||||
//! This is intentionally not a complete RFC 5424 implementation (no BOM
|
||||
//! handling on MSG, "-" nil markers are kept as literal strings rather than
|
||||
//! mapped to `None`). It's the Phase 0 minimum: parse what's clearly
|
||||
//! structured syslog, and never fail a log line outright — anything that
|
||||
//! doesn't match the grammar becomes a raw passthrough record instead of
|
||||
//! being dropped.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ParsedLine {
|
||||
/// Syslog facility (0-23), present only when RFC 5424 framing parsed.
|
||||
pub facility: Option<u8>,
|
||||
/// Syslog severity (0=emergency .. 7=debug), present only when RFC 5424
|
||||
/// framing parsed.
|
||||
pub severity: Option<u8>,
|
||||
/// Structured fields extracted from the PRI/HEADER/STRUCTURED-DATA
|
||||
/// portions. Empty when the raw-passthrough fallback fires.
|
||||
pub attributes: BTreeMap<String, String>,
|
||||
/// The MSG portion when RFC 5424 parsing succeeded, otherwise the
|
||||
/// original, unmodified line.
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
/// Parse a single log line. Never fails: falls back to a raw passthrough
|
||||
/// `ParsedLine` (no facility/severity, empty attributes, message = input)
|
||||
/// when the line doesn't match RFC 5424 framing.
|
||||
pub fn parse(line: &str) -> ParsedLine {
|
||||
parse_rfc5424(line).unwrap_or_else(|| ParsedLine {
|
||||
facility: None,
|
||||
severity: None,
|
||||
attributes: BTreeMap::new(),
|
||||
message: line.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
struct Scanner<'a> {
|
||||
chars: std::iter::Peekable<std::str::Chars<'a>>,
|
||||
}
|
||||
|
||||
impl<'a> Scanner<'a> {
|
||||
fn new(s: &'a str) -> Self {
|
||||
Scanner {
|
||||
chars: s.chars().peekable(),
|
||||
}
|
||||
}
|
||||
|
||||
fn peek(&mut self) -> Option<char> {
|
||||
self.chars.peek().copied()
|
||||
}
|
||||
|
||||
fn next(&mut self) -> Option<char> {
|
||||
self.chars.next()
|
||||
}
|
||||
|
||||
fn expect(&mut self, c: char) -> Option<()> {
|
||||
if self.next()? == c {
|
||||
Some(())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn take_while<F: Fn(char) -> bool>(&mut self, f: F) -> String {
|
||||
let mut out = String::new();
|
||||
while let Some(c) = self.peek() {
|
||||
if f(c) {
|
||||
out.push(c);
|
||||
self.next();
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn skip_one_space(&mut self) -> Option<()> {
|
||||
self.expect(' ')
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_rfc5424(line: &str) -> Option<ParsedLine> {
|
||||
let mut sc = Scanner::new(line);
|
||||
|
||||
sc.expect('<')?;
|
||||
let pri_str = sc.take_while(|c| c.is_ascii_digit());
|
||||
if pri_str.is_empty() || pri_str.len() > 3 {
|
||||
return None;
|
||||
}
|
||||
sc.expect('>')?;
|
||||
let pri: u16 = pri_str.parse().ok()?;
|
||||
if pri > 191 {
|
||||
return None;
|
||||
}
|
||||
let facility = (pri / 8) as u8;
|
||||
let severity = (pri % 8) as u8;
|
||||
|
||||
let version = sc.take_while(|c| c.is_ascii_digit());
|
||||
if version.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let timestamp = sc.take_while(|c| c != ' ');
|
||||
if timestamp.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let hostname = sc.take_while(|c| c != ' ');
|
||||
if hostname.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let app_name = sc.take_while(|c| c != ' ');
|
||||
if app_name.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let procid = sc.take_while(|c| c != ' ');
|
||||
if procid.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let msgid = sc.take_while(|c| c != ' ');
|
||||
if msgid.is_empty() {
|
||||
return None;
|
||||
}
|
||||
sc.skip_one_space()?;
|
||||
|
||||
let mut sd_pairs: Vec<(String, String, String)> = Vec::new();
|
||||
match sc.peek() {
|
||||
Some('-') => {
|
||||
sc.next();
|
||||
}
|
||||
Some('[') => loop {
|
||||
if sc.peek() != Some('[') {
|
||||
break;
|
||||
}
|
||||
sc.next();
|
||||
let sd_id = sc.take_while(|c| c != ' ' && c != ']');
|
||||
if sd_id.is_empty() {
|
||||
return None;
|
||||
}
|
||||
loop {
|
||||
match sc.peek() {
|
||||
Some(' ') => {
|
||||
sc.next();
|
||||
let name = sc.take_while(|c| c != '=');
|
||||
sc.expect('=')?;
|
||||
sc.expect('"')?;
|
||||
let mut val = String::new();
|
||||
loop {
|
||||
match sc.next() {
|
||||
Some('\\') => val.push(sc.next()?),
|
||||
Some('"') => break,
|
||||
Some(c) => val.push(c),
|
||||
None => return None,
|
||||
}
|
||||
}
|
||||
sd_pairs.push((sd_id.clone(), name, val));
|
||||
}
|
||||
Some(']') => {
|
||||
sc.next();
|
||||
break;
|
||||
}
|
||||
_ => return None,
|
||||
}
|
||||
}
|
||||
},
|
||||
_ => return None,
|
||||
}
|
||||
|
||||
let message = if sc.peek() == Some(' ') {
|
||||
sc.next();
|
||||
sc.take_while(|_| true)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
|
||||
let mut attributes = BTreeMap::new();
|
||||
attributes.insert("syslog.version".to_string(), version);
|
||||
attributes.insert("syslog.timestamp".to_string(), timestamp);
|
||||
attributes.insert("syslog.hostname".to_string(), hostname);
|
||||
attributes.insert("syslog.app_name".to_string(), app_name);
|
||||
attributes.insert("syslog.procid".to_string(), procid);
|
||||
attributes.insert("syslog.msgid".to_string(), msgid);
|
||||
for (sd_id, name, val) in sd_pairs {
|
||||
attributes.insert(format!("{sd_id}.{name}"), val);
|
||||
}
|
||||
|
||||
Some(ParsedLine {
|
||||
facility: Some(facility),
|
||||
severity: Some(severity),
|
||||
attributes,
|
||||
message,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parses_full_rfc5424_with_structured_data() {
|
||||
let line = r#"<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslp - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry"#;
|
||||
let p = parse(line);
|
||||
assert_eq!(p.facility, Some(20));
|
||||
assert_eq!(p.severity, Some(5));
|
||||
assert_eq!(p.message, "An application event log entry");
|
||||
assert_eq!(
|
||||
p.attributes.get("[email protected]"),
|
||||
Some(&"3".to_string())
|
||||
);
|
||||
assert_eq!(
|
||||
p.attributes.get("[email protected]"),
|
||||
Some(&"Application".to_string())
|
||||
);
|
||||
assert_eq!(
|
||||
p.attributes.get("syslog.hostname"),
|
||||
Some(&"mymachine.example.com".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_nil_structured_data_and_fields() {
|
||||
let line = "<34>1 2003-10-11T22:14:15.003Z mymachine su - ID47 - 'su root' failed";
|
||||
let p = parse(line);
|
||||
assert_eq!(p.facility, Some(4));
|
||||
assert_eq!(p.severity, Some(2));
|
||||
assert_eq!(p.attributes.get("syslog.procid"), Some(&"-".to_string()));
|
||||
assert_eq!(p.message, "'su root' failed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_multiple_structured_data_elements() {
|
||||
let line = r#"<165>1 2003-10-11T22:14:15.003Z host app - ID47 [a@1 k="v"][b@1 k2="v2"] msg"#;
|
||||
let p = parse(line);
|
||||
assert_eq!(p.attributes.get("[email protected]"), Some(&"v".to_string()));
|
||||
assert_eq!(p.attributes.get("[email protected]"), Some(&"v2".to_string()));
|
||||
assert_eq!(p.message, "msg");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handles_escaped_quote_in_param_value() {
|
||||
let line = r#"<165>1 2003-10-11T22:14:15.003Z host app - ID47 [a@1 k="has \"quote\" inside"] msg"#;
|
||||
let p = parse(line);
|
||||
assert_eq!(
|
||||
p.attributes.get("[email protected]"),
|
||||
Some(&"has \"quote\" inside".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn falls_back_to_raw_passthrough_for_non_syslog_line() {
|
||||
let line = "this is just a plain log line, not syslog at all";
|
||||
let p = parse(line);
|
||||
assert_eq!(p.facility, None);
|
||||
assert_eq!(p.severity, None);
|
||||
assert!(p.attributes.is_empty());
|
||||
assert_eq!(p.message, line);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn falls_back_to_raw_passthrough_for_malformed_pri() {
|
||||
let line = "<abc>1 2003-10-11T22:14:15.003Z host app - ID47 - msg";
|
||||
let p = parse(line);
|
||||
assert_eq!(p.facility, None);
|
||||
assert_eq!(p.message, line);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn falls_back_when_structured_data_missing() {
|
||||
// Missing the required "-" or "[...]" for STRUCTURED-DATA.
|
||||
let line = "<34>1 2003-10-11T22:14:15.003Z host app 123 ID47";
|
||||
let p = parse(line);
|
||||
assert_eq!(p.facility, None);
|
||||
assert_eq!(p.message, line);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# Build context must be the repo root (sentry/):
|
||||
# docker build -f api/Dockerfile -t sentry-api .
|
||||
|
||||
FROM golang:1.25-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY api ./api
|
||||
WORKDIR /src/api
|
||||
RUN go mod download
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/api ./cmd/api
|
||||
|
||||
FROM gcr.io/distroless/static-debian12
|
||||
COPY --from=builder /out/api /api
|
||||
ENTRYPOINT ["/api"]
|
||||
@@ -0,0 +1,63 @@
|
||||
# api
|
||||
|
||||
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
|
||||
|
||||
## Why plain REST, not gRPC + REST gateway
|
||||
|
||||
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
||||
service is plain `net/http` instead — a deliberate Phase 0 simplification,
|
||||
not a change to the pinned stack. Wiring up a `.proto` service,
|
||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
|
||||
single endpoint that Phase 2 replaces outright with a real SPL-like query
|
||||
layer would be exactly the kind of premature machinery this project's
|
||||
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
|
||||
a second real, durable endpoint.
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `POST /query` — body `{"sql": "SELECT ..."}`, response
|
||||
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
||||
SELECT-only, single-statement, basic keyword-based injection guarding
|
||||
(see `internal/queryapi/validate.go` for exactly what that does and
|
||||
doesn't catch — it's not a SQL parser).
|
||||
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
||||
|
||||
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
|
||||
dev/homelab network.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables (see `internal/config/config.go`):
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `HTTP_LISTEN_ADDR` | `:8080` | |
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
|
||||
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
```sh
|
||||
# from the repo root, not api/
|
||||
docker build -f api/Dockerfile -t sentry-api .
|
||||
```
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
|
||||
one-method `queryExecutor` interface, so routing, validation, JSON
|
||||
encoding, and error-status mapping are all unit-tested against a fake —
|
||||
no live ClickHouse needed. `Executor` itself (the reflection-based row
|
||||
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
|
||||
`driver.Rows` interface fully would be significant test-only scaffolding
|
||||
for a Phase 0 placeholder, and the driver package's own docs note it isn't
|
||||
meant to be implemented by adopters. It's exercised end-to-end via the
|
||||
docker-compose flow in `/docs/phase-0-runbook.md` instead.
|
||||
@@ -0,0 +1,80 @@
|
||||
// Command api is the Sentry Phase 0 query API: a single crude POST /query
|
||||
// endpoint proxying allowlisted SELECT statements to ClickHouse. See
|
||||
// internal/queryapi for why this is plain REST rather than the pinned
|
||||
// gRPC+gateway pattern for Phase 0.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
|
||||
"github.com/sentry/sentry/api/internal/config"
|
||||
"github.com/sentry/sentry/api/internal/queryapi"
|
||||
)
|
||||
|
||||
func main() {
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
logger.Error("loading config", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
conn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{cfg.ClickHouse.Addr},
|
||||
Auth: clickhouse.Auth{
|
||||
Database: cfg.ClickHouse.Database,
|
||||
Username: cfg.ClickHouse.Username,
|
||||
Password: cfg.ClickHouse.Password,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
logger.Error("opening clickhouse connection", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
if err := conn.Ping(ctx); err != nil {
|
||||
logger.Error("pinging clickhouse", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
exec := queryapi.NewExecutor(conn)
|
||||
handler := queryapi.NewHandler(logger, exec, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: cfg.HTTPListenAddr,
|
||||
Handler: handler.Routes(),
|
||||
}
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
logger.Info("api listening", "addr", cfg.HTTPListenAddr)
|
||||
errCh <- srv.ListenAndServe()
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shutdownCtx); err != nil {
|
||||
logger.Error("graceful shutdown failed", "error", err)
|
||||
}
|
||||
case err := <-errCh:
|
||||
if err != nil && err != http.ErrServerClosed {
|
||||
logger.Error("server exited with error", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
module github.com/sentry/sentry/api
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
||||
|
||||
require (
|
||||
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
||||
github.com/andybalholm/brotli v1.2.2 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/go-faster/city v1.0.1 // indirect
|
||||
github.com/go-faster/errors v0.7.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/klauspost/compress v1.19.1 // indirect
|
||||
github.com/paulmach/orb v0.13.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||
github.com/segmentio/asm v1.2.1 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
go.opentelemetry.io/otel v1.44.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
)
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE=
|
||||
github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM=
|
||||
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
|
||||
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
|
||||
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
|
||||
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
||||
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
|
||||
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
|
||||
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
|
||||
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
|
||||
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,56 @@
|
||||
// Package config loads api's configuration from environment variables,
|
||||
// same convention as /ingest: no config file format for Phase 0.
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
HTTPListenAddr string
|
||||
ClickHouse ClickHouseConfig
|
||||
QueryTimeout time.Duration
|
||||
CORSAllowedOrigin string
|
||||
}
|
||||
|
||||
type ClickHouseConfig struct {
|
||||
Addr string
|
||||
Database string
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
cfg := Config{
|
||||
HTTPListenAddr: getenv("HTTP_LISTEN_ADDR", ":8080"),
|
||||
ClickHouse: ClickHouseConfig{
|
||||
Addr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
|
||||
Database: getenv("CLICKHOUSE_DATABASE", "sentry"),
|
||||
Username: getenv("CLICKHOUSE_USERNAME", "default"),
|
||||
Password: getenv("CLICKHOUSE_PASSWORD", ""),
|
||||
},
|
||||
// Phase 0 has no auth, so this is wide open by default to keep
|
||||
// the local SvelteKit dev server (a different origin/port)
|
||||
// working out of the box. Tighten before this is ever reachable
|
||||
// from outside a trusted dev/homelab network.
|
||||
CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"),
|
||||
}
|
||||
|
||||
timeoutSec, err := strconv.Atoi(getenv("QUERY_TIMEOUT_SECONDS", "30"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("QUERY_TIMEOUT_SECONDS: %w", err)
|
||||
}
|
||||
cfg.QueryTimeout = time.Duration(timeoutSec) * time.Second
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLoadDefaults(t *testing.T) {
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.HTTPListenAddr != ":8080" {
|
||||
t.Errorf("HTTPListenAddr = %q, want :8080", cfg.HTTPListenAddr)
|
||||
}
|
||||
if cfg.QueryTimeout != 30*time.Second {
|
||||
t.Errorf("QueryTimeout = %v, want 30s", cfg.QueryTimeout)
|
||||
}
|
||||
if cfg.CORSAllowedOrigin != "*" {
|
||||
t.Errorf("CORSAllowedOrigin = %q, want *", cfg.CORSAllowedOrigin)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadInvalidTimeoutErrors(t *testing.T) {
|
||||
t.Setenv("QUERY_TIMEOUT_SECONDS", "not-a-number")
|
||||
if _, err := Load(); err == nil {
|
||||
t.Fatal("expected error for non-numeric QUERY_TIMEOUT_SECONDS, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package queryapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
)
|
||||
|
||||
type QueryResult struct {
|
||||
Columns []string `json:"columns"`
|
||||
Rows [][]any `json:"rows"`
|
||||
}
|
||||
|
||||
// Executor runs arbitrary (pre-validated) SELECT statements against
|
||||
// ClickHouse and shapes the result into JSON-friendly columns/rows,
|
||||
// discovering the result's column set at query time via reflection since
|
||||
// the query itself is arbitrary.
|
||||
type Executor struct {
|
||||
conn driver.Conn
|
||||
}
|
||||
|
||||
func NewExecutor(conn driver.Conn) *Executor {
|
||||
return &Executor{conn: conn}
|
||||
}
|
||||
|
||||
func (e *Executor) Execute(ctx context.Context, sql string) (*QueryResult, error) {
|
||||
rows, err := e.conn.Query(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("executing query: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
columnTypes := rows.ColumnTypes()
|
||||
result := &QueryResult{
|
||||
Columns: rows.Columns(),
|
||||
Rows: [][]any{},
|
||||
}
|
||||
|
||||
for rows.Next() {
|
||||
dest := make([]any, len(columnTypes))
|
||||
for i, ct := range columnTypes {
|
||||
dest[i] = reflect.New(ct.ScanType()).Interface()
|
||||
}
|
||||
if err := rows.Scan(dest...); err != nil {
|
||||
return nil, fmt.Errorf("scanning row: %w", err)
|
||||
}
|
||||
row := make([]any, len(dest))
|
||||
for i, d := range dest {
|
||||
row[i] = reflect.ValueOf(d).Elem().Interface()
|
||||
}
|
||||
result.Rows = append(result.Rows, row)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("iterating rows: %w", err)
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
// Package queryapi is the Phase 0 query API: a single crude POST /query
|
||||
// endpoint that takes a raw SQL string, allowlists it to a single SELECT
|
||||
// statement, and proxies it to ClickHouse. This is a deliberate
|
||||
// simplification of the pinned "gRPC + REST gateway" control-plane
|
||||
// pattern (see CLAUDE.md's tech stack table): a plain net/http REST
|
||||
// handler, not a gRPC service transcoded through grpc-gateway. That
|
||||
// machinery (proto definitions, googleapis annotations, gateway codegen)
|
||||
// buys nothing for one crude placeholder endpoint that Phase 2 replaces
|
||||
// outright with the real SPL-like query layer. Revisit gRPC+gateway when
|
||||
// /api grows a second real endpoint.
|
||||
package queryapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// queryExecutor is the narrow interface handleQuery depends on, so tests
|
||||
// can substitute a fake without a real ClickHouse connection. *Executor
|
||||
// satisfies it.
|
||||
type queryExecutor interface {
|
||||
Execute(ctx context.Context, sql string) (*QueryResult, error)
|
||||
}
|
||||
|
||||
type Handler struct {
|
||||
logger *slog.Logger
|
||||
exec queryExecutor
|
||||
queryTimeout time.Duration
|
||||
allowedOrigin string
|
||||
}
|
||||
|
||||
func NewHandler(logger *slog.Logger, exec queryExecutor, queryTimeout time.Duration, allowedOrigin string) *Handler {
|
||||
return &Handler{logger: logger, exec: exec, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
|
||||
}
|
||||
|
||||
func (h *Handler) Routes() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("POST /query", h.handleQuery)
|
||||
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
||||
return h.withCORS(mux)
|
||||
}
|
||||
|
||||
// withCORS is deliberately permissive by default (see CORSAllowedOrigin in
|
||||
// internal/config) since Phase 0 has no auth and the SvelteKit dev server
|
||||
// runs on a different origin. Tighten alongside adding real auth.
|
||||
func (h *Handler) withCORS(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Access-Control-Allow-Origin", h.allowedOrigin)
|
||||
w.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
|
||||
if r.Method == http.MethodOptions {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
type queryRequest struct {
|
||||
SQL string `json:"sql"`
|
||||
}
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
// maxBodyBytes caps the request body: a raw SQL string has no legitimate
|
||||
// reason to be larger than this.
|
||||
const maxBodyBytes = 1 << 20 // 1 MiB
|
||||
|
||||
func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
|
||||
|
||||
var req queryRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if err := validateSelectOnly(req.SQL); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
|
||||
defer cancel()
|
||||
|
||||
result, err := h.exec.Execute(ctx, req.SQL)
|
||||
if err != nil {
|
||||
h.logger.Error("query execution failed", "error", err)
|
||||
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(result); err != nil {
|
||||
h.logger.Error("encoding response", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(errorResponse{Error: msg})
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
package queryapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type fakeExecutor struct {
|
||||
result *QueryResult
|
||||
err error
|
||||
gotSQL string
|
||||
}
|
||||
|
||||
func (f *fakeExecutor) Execute(_ context.Context, sql string) (*QueryResult, error) {
|
||||
f.gotSQL = sql
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return f.result, nil
|
||||
}
|
||||
|
||||
func newTestHandler(exec queryExecutor) *Handler {
|
||||
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, time.Second, "*")
|
||||
}
|
||||
|
||||
func TestHandleQuerySuccess(t *testing.T) {
|
||||
fe := &fakeExecutor{result: &QueryResult{
|
||||
Columns: []string{"host", "count"},
|
||||
Rows: [][]any{{"h1", 3}},
|
||||
}}
|
||||
h := newTestHandler(fe)
|
||||
|
||||
body := strings.NewReader(`{"sql": "SELECT host, count(*) FROM logs GROUP BY host"}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var got QueryResult
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if len(got.Columns) != 2 || len(got.Rows) != 1 {
|
||||
t.Fatalf("unexpected result: %+v", got)
|
||||
}
|
||||
if fe.gotSQL != "SELECT host, count(*) FROM logs GROUP BY host" {
|
||||
t.Fatalf("executor received unexpected SQL: %q", fe.gotSQL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleQueryRejectsNonSelect(t *testing.T) {
|
||||
fe := &fakeExecutor{}
|
||||
h := newTestHandler(fe)
|
||||
|
||||
body := strings.NewReader(`{"sql": "DELETE FROM logs"}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
if fe.gotSQL != "" {
|
||||
t.Fatal("executor should not have been called for a rejected query")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleQueryRejectsInvalidJSON(t *testing.T) {
|
||||
h := newTestHandler(&fakeExecutor{})
|
||||
|
||||
body := strings.NewReader(`not json`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleQueryExecutorErrorReturnsBadGateway(t *testing.T) {
|
||||
fe := &fakeExecutor{err: errors.New("boom")}
|
||||
h := newTestHandler(fe)
|
||||
|
||||
body := strings.NewReader(`{"sql": "SELECT 1"}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status = %d, want 502", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleHealthz(t *testing.T) {
|
||||
h := newTestHandler(&fakeExecutor{})
|
||||
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCORSPreflight(t *testing.T) {
|
||||
h := newTestHandler(&fakeExecutor{})
|
||||
req := httptest.NewRequest(http.MethodOptions, "/query", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h.Routes().ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204", rec.Code)
|
||||
}
|
||||
if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" {
|
||||
t.Fatalf("Access-Control-Allow-Origin = %q, want *", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package queryapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// disallowedKeyword is defense-in-depth on top of the SELECT-only gate: it
|
||||
// catches mutating/administrative statements appearing anywhere in the
|
||||
// query (e.g. smuggled into a subquery), not just at the start. This is
|
||||
// word-boundary matching, not a real SQL parser.
|
||||
var disallowedKeyword = regexp.MustCompile(`(?i)\b(insert|update|delete|alter|drop|truncate|create|grant|revoke|attach|detach|rename|kill|optimize|system|set|exchange|watch)\b`)
|
||||
|
||||
// validateSelectOnly enforces the Phase 0 query API contract: exactly one
|
||||
// SELECT statement and nothing else. This is "basic injection guarding" as
|
||||
// specced, not a SQL parser: it will reject some unusual-but-valid SELECTs
|
||||
// (e.g. one that references a column literally named "delete") and will
|
||||
// not catch every possible abuse (e.g. a syntactically pure SELECT that's
|
||||
// simply expensive to run). Both are acceptable for a Phase 0 placeholder
|
||||
// that's explicitly superseded by a real query layer in Phase 2 — see
|
||||
// /docs/architecture.md.
|
||||
func validateSelectOnly(sql string) error {
|
||||
trimmed := strings.TrimSpace(sql)
|
||||
if trimmed == "" {
|
||||
return errors.New("query must not be empty")
|
||||
}
|
||||
|
||||
trimmed = strings.TrimSpace(strings.TrimSuffix(trimmed, ";"))
|
||||
if trimmed == "" {
|
||||
return errors.New("query must not be empty")
|
||||
}
|
||||
if strings.Contains(trimmed, ";") {
|
||||
return errors.New("only a single statement is allowed")
|
||||
}
|
||||
|
||||
firstWord := strings.ToUpper(strings.Fields(trimmed)[0])
|
||||
if firstWord != "SELECT" {
|
||||
return errors.New("only SELECT queries are allowed")
|
||||
}
|
||||
|
||||
if disallowedKeyword.MatchString(trimmed) {
|
||||
return errors.New("query contains a disallowed keyword")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package queryapi
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestValidateSelectOnly(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
sql string
|
||||
wantErr bool
|
||||
}{
|
||||
{"plain select", "SELECT * FROM logs LIMIT 10", false},
|
||||
{"lowercase select", "select service, count(*) from logs group by service", false},
|
||||
{"trailing semicolon allowed", "SELECT 1;", false},
|
||||
{"trailing semicolon and whitespace allowed", "SELECT 1; ", false},
|
||||
{"empty", "", true},
|
||||
{"whitespace only", " ", true},
|
||||
{"only a semicolon", ";", true},
|
||||
{"multiple statements", "SELECT 1; SELECT 2", true},
|
||||
{"insert", "INSERT INTO logs VALUES (1)", true},
|
||||
{"delete", "DELETE FROM logs", true},
|
||||
{"drop", "DROP TABLE logs", true},
|
||||
{"select with drop keyword smuggled in", "SELECT * FROM logs WHERE message = 'DROP TABLE logs'", true},
|
||||
{"non-select start", "WITH x AS (SELECT 1) SELECT * FROM x", true},
|
||||
{"trailing garbage after semicolon", "SELECT 1; DROP TABLE logs", true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := validateSelectOnly(tc.sql)
|
||||
if tc.wantErr && err == nil {
|
||||
t.Errorf("validateSelectOnly(%q) = nil, want error", tc.sql)
|
||||
}
|
||||
if !tc.wantErr && err != nil {
|
||||
t.Errorf("validateSelectOnly(%q) = %v, want nil", tc.sql, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
# docker build -f cli/Dockerfile -t sentryctl cli/
|
||||
FROM golang:1.25-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY . .
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/sentryctl ./cmd/sentryctl
|
||||
|
||||
FROM gcr.io/distroless/static-debian12
|
||||
COPY --from=builder /out/sentryctl /sentryctl
|
||||
ENTRYPOINT ["/sentryctl"]
|
||||
@@ -0,0 +1,28 @@
|
||||
# sentryctl
|
||||
|
||||
Sentry's control CLI. Phase 0: a single command.
|
||||
|
||||
```sh
|
||||
sentryctl ping # checks http://localhost:8080/healthz
|
||||
sentryctl ping --api http://api.internal:8080
|
||||
SENTRYCTL_API_URL=http://api.internal:8080 sentryctl ping
|
||||
```
|
||||
|
||||
Exits 0 and prints `ok` if `/api`'s `/healthz` responds 200; exits 1 with an
|
||||
error on `stderr` otherwise.
|
||||
|
||||
No CLI framework (cobra/urfave-cli/etc.) — a single command doesn't need
|
||||
one, and stdlib `os.Args` handling is boring enough not to need a
|
||||
dependency. Revisit once there's a real command tree to justify one.
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
```sh
|
||||
docker build -f Dockerfile -t sentryctl . # context is cli/, not the repo root
|
||||
```
|
||||
@@ -0,0 +1,87 @@
|
||||
// Command sentryctl is Sentry's control CLI. Phase 0: a single "ping"
|
||||
// command that checks the api service is reachable. More commands land as
|
||||
// the control plane grows real operations to expose.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
const defaultAPIURL = "http://localhost:8080"
|
||||
|
||||
func main() {
|
||||
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
|
||||
}
|
||||
|
||||
func run(args []string, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 {
|
||||
usage(stderr)
|
||||
return 1
|
||||
}
|
||||
|
||||
switch args[0] {
|
||||
case "ping":
|
||||
return cmdPing(args[1:], stdout, stderr)
|
||||
case "-h", "--help", "help":
|
||||
usage(stdout)
|
||||
return 0
|
||||
default:
|
||||
fmt.Fprintf(stderr, "sentryctl: unknown command %q\n", args[0])
|
||||
usage(stderr)
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
func usage(w io.Writer) {
|
||||
fmt.Fprintln(w, `sentryctl: Sentry control CLI (Phase 0: ping only)
|
||||
|
||||
Usage:
|
||||
sentryctl ping [--api <url>]
|
||||
|
||||
Commands:
|
||||
ping Checks that the api service is reachable via GET /healthz.
|
||||
|
||||
--api defaults to $SENTRYCTL_API_URL, or `+defaultAPIURL+` if unset.`)
|
||||
}
|
||||
|
||||
// parsePingArgs resolves the api base URL for ping: --api flag wins, then
|
||||
// $SENTRYCTL_API_URL, then the hardcoded default. Kept pure (env passed in
|
||||
// as a function) and separate from the HTTP call so it's unit-testable
|
||||
// without a real environment or server.
|
||||
func parsePingArgs(args []string, env func(string) string) string {
|
||||
apiURL := env("SENTRYCTL_API_URL")
|
||||
if apiURL == "" {
|
||||
apiURL = defaultAPIURL
|
||||
}
|
||||
for i := 0; i < len(args); i++ {
|
||||
if args[i] == "--api" && i+1 < len(args) {
|
||||
apiURL = args[i+1]
|
||||
i++
|
||||
}
|
||||
}
|
||||
return apiURL
|
||||
}
|
||||
|
||||
func cmdPing(args []string, stdout, stderr io.Writer) int {
|
||||
apiURL := parsePingArgs(args, os.Getenv)
|
||||
|
||||
client := &http.Client{Timeout: 5 * time.Second}
|
||||
resp, err := client.Get(apiURL + "/healthz")
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "ping failed: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
fmt.Fprintf(stderr, "ping failed: api returned status %d\n", resp.StatusCode)
|
||||
return 1
|
||||
}
|
||||
|
||||
fmt.Fprintln(stdout, "ok")
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParsePingArgsDefault(t *testing.T) {
|
||||
env := func(string) string { return "" }
|
||||
if got := parsePingArgs(nil, env); got != defaultAPIURL {
|
||||
t.Errorf("got %q, want %q", got, defaultAPIURL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePingArgsFromEnv(t *testing.T) {
|
||||
env := func(k string) string {
|
||||
if k == "SENTRYCTL_API_URL" {
|
||||
return "http://env-host:1234"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if got := parsePingArgs(nil, env); got != "http://env-host:1234" {
|
||||
t.Errorf("got %q, want env value", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePingArgsFlagOverridesEnv(t *testing.T) {
|
||||
env := func(k string) string {
|
||||
if k == "SENTRYCTL_API_URL" {
|
||||
return "http://env-host:1234"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
got := parsePingArgs([]string{"--api", "http://flag-host:5678"}, env)
|
||||
if got != "http://flag-host:5678" {
|
||||
t.Errorf("got %q, want flag value", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdPingSuccess(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/healthz" {
|
||||
t.Errorf("unexpected path %q", r.URL.Path)
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdPing([]string{"--api", srv.URL}, &stdout, &stderr)
|
||||
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%s", code, stderr.String())
|
||||
}
|
||||
if strings.TrimSpace(stdout.String()) != "ok" {
|
||||
t.Fatalf("stdout = %q, want ok", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdPingNonOKStatus(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdPing([]string{"--api", srv.URL}, &stdout, &stderr)
|
||||
|
||||
if code != 1 {
|
||||
t.Fatalf("exit code = %d, want 1", code)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "503") {
|
||||
t.Fatalf("stderr = %q, want it to mention the status code", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdPingUnreachable(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdPing([]string{"--api", "http://127.0.0.1:1"}, &stdout, &stderr)
|
||||
|
||||
if code != 1 {
|
||||
t.Fatalf("exit code = %d, want 1", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunNoArgsPrintsUsageAndFails(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := run(nil, &stdout, &stderr)
|
||||
|
||||
if code != 1 {
|
||||
t.Fatalf("exit code = %d, want 1", code)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "Usage:") {
|
||||
t.Fatalf("stderr should contain usage text, got %q", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunUnknownCommand(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := run([]string{"bogus"}, &stdout, &stderr)
|
||||
|
||||
if code != 1 {
|
||||
t.Fatalf("exit code = %d, want 1", code)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "bogus") {
|
||||
t.Fatalf("stderr should mention the unknown command, got %q", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHelp(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := run([]string{"help"}, &stdout, &stderr)
|
||||
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0", code)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Usage:") {
|
||||
t.Fatalf("stdout should contain usage text, got %q", stdout.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module github.com/sentry/sentry/cli
|
||||
|
||||
go 1.25
|
||||
@@ -0,0 +1,128 @@
|
||||
# Phase 0 stack: Redpanda -> ingest -> ClickHouse -> api -> web.
|
||||
#
|
||||
# Does NOT include the Rust agent — see /agent/README.md: journald
|
||||
# sourcing needs the host's journal, which isn't something a container
|
||||
# gets for free. Run the agent natively on the host per
|
||||
# /docs/phase-0-runbook.md, pointed at ingest's mapped port (localhost:4317).
|
||||
#
|
||||
# Before first run: generate dev mTLS certs (hack/dev-certs/generate.sh).
|
||||
# See /docs/phase-0-runbook.md for the full sequence.
|
||||
services:
|
||||
redpanda:
|
||||
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
|
||||
container_name: sentry-redpanda
|
||||
command:
|
||||
- redpanda
|
||||
- start
|
||||
- --smp=1
|
||||
- --memory=1G
|
||||
- --reserve-memory=0M
|
||||
- --overprovisioned
|
||||
- --node-id=0
|
||||
- --check=false
|
||||
- --kafka-addr=PLAINTEXT://0.0.0.0:9092
|
||||
- --advertise-kafka-addr=PLAINTEXT://redpanda:9092
|
||||
ports:
|
||||
- "9092:9092"
|
||||
volumes:
|
||||
- redpanda-data:/var/lib/redpanda/data
|
||||
healthcheck:
|
||||
test: ["CMD", "rpk", "cluster", "health", "--exit-when-healthy"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
|
||||
# One-shot: creates the sentry.logs.raw topic, then exits 0. ingest
|
||||
# waits on this completing successfully before it starts.
|
||||
redpanda-provision:
|
||||
build:
|
||||
context: ./transport
|
||||
container_name: sentry-redpanda-provision
|
||||
depends_on:
|
||||
redpanda:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
REDPANDA_BROKERS: "redpanda:9092"
|
||||
|
||||
clickhouse:
|
||||
image: clickhouse/clickhouse-server:24.8
|
||||
container_name: sentry-clickhouse
|
||||
ports:
|
||||
- "8123:8123" # HTTP interface, used by the migrate step
|
||||
- "9000:9000" # native protocol, used by ingest and api
|
||||
volumes:
|
||||
- clickhouse-data:/var/lib/clickhouse
|
||||
ulimits:
|
||||
nofile:
|
||||
soft: 262144
|
||||
hard: 262144
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
|
||||
# One-shot: applies /storage/migrations/*.sql, then exits 0. ingest and
|
||||
# api both wait on this completing successfully.
|
||||
clickhouse-migrate:
|
||||
build:
|
||||
context: ./storage
|
||||
container_name: sentry-clickhouse-migrate
|
||||
depends_on:
|
||||
clickhouse:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
CLICKHOUSE_HTTP: "http://clickhouse:8123"
|
||||
|
||||
ingest:
|
||||
build:
|
||||
context: . # needs both ingest/ and proto/
|
||||
dockerfile: ingest/Dockerfile
|
||||
container_name: sentry-ingest
|
||||
depends_on:
|
||||
redpanda-provision:
|
||||
condition: service_completed_successfully
|
||||
clickhouse-migrate:
|
||||
condition: service_completed_successfully
|
||||
ports:
|
||||
- "4317:4317" # gRPC, mTLS — this is what the host-run agent connects to
|
||||
environment:
|
||||
REDPANDA_BROKERS: "redpanda:9092"
|
||||
CLICKHOUSE_ADDR: "clickhouse:9000"
|
||||
# TLS_*_FILE env vars are left at their defaults
|
||||
# (/etc/sentry-ingest/{server,server-key,ca}.pem) — matches where
|
||||
# the volume below mounts the generated dev certs.
|
||||
volumes:
|
||||
- ./hack/dev-certs/out:/etc/sentry-ingest:ro
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: api/Dockerfile
|
||||
container_name: sentry-api
|
||||
depends_on:
|
||||
clickhouse-migrate:
|
||||
condition: service_completed_successfully
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
CLICKHOUSE_ADDR: "clickhouse:9000"
|
||||
|
||||
web:
|
||||
build:
|
||||
context: web
|
||||
args:
|
||||
# Baked in at build time (static site, not a server) as
|
||||
# localhost:8080 -- this is fetched from the *browser*, which
|
||||
# resolves against the host's mapped port, not the compose
|
||||
# network's service DNS name.
|
||||
VITE_API_BASE_URL: "http://localhost:8080"
|
||||
container_name: sentry-web
|
||||
depends_on:
|
||||
- api
|
||||
ports:
|
||||
- "3000:3000"
|
||||
|
||||
volumes:
|
||||
redpanda-data:
|
||||
clickhouse-data:
|
||||
@@ -0,0 +1,105 @@
|
||||
# Sentry Architecture
|
||||
|
||||
> **Status:** Draft, Phase 0 scope. Written from the project constraints and
|
||||
> task list at kickoff, not transcribed from a pre-existing spec. Treat as a
|
||||
> starting point to correct, not a settled design — flag anything that
|
||||
> doesn't match your intent before implementation leans on it further.
|
||||
|
||||
## Mission
|
||||
|
||||
Open-core, Kubernetes-native centralized logging platform. Compete with
|
||||
Splunk on features; win on cost-per-GB, a modern language stack, and
|
||||
multi-tenant RBAC that's actually honest about its guarantees.
|
||||
|
||||
## Component map
|
||||
|
||||
```
|
||||
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
|
||||
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
|
||||
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
|
||||
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
|
||||
└──────┬─────┘
|
||||
│ batch INSERT
|
||||
▼
|
||||
┌────────────┐
|
||||
│ ClickHouse │
|
||||
└─────┬──────┘
|
||||
│ SQL
|
||||
┌───────────▼───────────┐
|
||||
│ api (Go: gRPC+REST) │
|
||||
└───────────┬───────────┘
|
||||
│ REST
|
||||
┌───────────▼───────────┐
|
||||
│ web (SvelteKit) │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
|
||||
`ingest` service's gRPC front end produces to Redpanda rather than writing
|
||||
ClickHouse directly; a separate consumer path reads from Redpanda and batches
|
||||
inserts into ClickHouse. This exercises the real transport layer from day
|
||||
one instead of deferring it, and keeps Kafka credentials off the edge agent.
|
||||
|
||||
## Storage / query split
|
||||
|
||||
- **ClickHouse** is the analytical store of record for structured log data:
|
||||
timestamp, host, service, severity, message, plus a `Map(String,String)`
|
||||
for arbitrary structured fields. Partitioned by day, ordered by
|
||||
`(service, timestamp)`.
|
||||
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
|
||||
field and unstructured payloads, queried out-of-band from ClickHouse and
|
||||
joined by a log identifier. Not built in Phase 0.
|
||||
- **Schema-on-write** using OTel semantic conventions as the default log
|
||||
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
|
||||
the structured columns (captured via the `Map` column and/or a raw
|
||||
passthrough field).
|
||||
|
||||
This split is not to be changed without discussion — see CLAUDE.md.
|
||||
|
||||
## Component responsibilities (Phase 0)
|
||||
|
||||
| Component | Responsibility |
|
||||
|---|---|
|
||||
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. |
|
||||
| `proto` | Shared `.proto` contracts for the agent↔ingest gRPC service, versioned independently of either component. |
|
||||
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
|
||||
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. |
|
||||
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
|
||||
| `api` (Go) | gRPC + REST gateway. Phase 0: one crude `POST /query` endpoint, SELECT-only, proxying to ClickHouse. Real SPL-like query layer is Phase 2. |
|
||||
| `web` (SvelteKit) | Single page: SQL text box, submit, results table. No auth, no styling polish. |
|
||||
| `cli` (`sentryctl`) | Stub. Single `ping` command for now. |
|
||||
| `deploy` | Helm charts, k8s manifests. Stubbed in Phase 0; docker-compose is the real local/dev path. |
|
||||
|
||||
## Licensing boundary
|
||||
|
||||
AGPLv3 for core + agents. Enterprise features (SSO, multi-tenancy,
|
||||
compliance) live under `enterprise/` (not yet created — out of scope for
|
||||
Phase 0) under a commercial license stub. AGPL code must never import from
|
||||
`enterprise/`. No enterprise-gated code exists yet in this repo; this
|
||||
section documents the boundary so nothing added later crosses it by
|
||||
accident.
|
||||
|
||||
## Non-negotiables carried from CLAUDE.md
|
||||
|
||||
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
|
||||
`aarch64-unknown-linux-musl`, no glibc runtime deps.
|
||||
- Windows support (Phase 1+) via native ETW/Event Log API, not WSL.
|
||||
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
|
||||
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
|
||||
|
||||
## Explicitly out of scope for Phase 0
|
||||
|
||||
Windows agent, alerting, dashboards, multi-tenancy, Tantivy full-text
|
||||
search, the real SPL-like query language, enterprise module code.
|
||||
|
||||
## Open questions for you to resolve
|
||||
|
||||
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
|
||||
deferred until storage sizing is a real concern.
|
||||
- Exact OTel log schema field mapping (which OTel resource/log attributes
|
||||
map to which ClickHouse columns) — Phase 0 uses a minimal subset
|
||||
(timestamp, host, service, severity, message, attributes map); full
|
||||
mapping deferred.
|
||||
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
|
||||
a static dev CA and manually issued certs; production PKI design is
|
||||
out of scope here.
|
||||
@@ -0,0 +1,207 @@
|
||||
# Phase 0 runbook
|
||||
|
||||
Walks one log line from a Linux host, through the Rust agent, Redpanda,
|
||||
ingest, and ClickHouse, to a browser table. This is the actual
|
||||
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
|
||||
regardless of what any individual component's tests say.
|
||||
|
||||
**This sequence has not been run end-to-end** in the environment that
|
||||
built it (no Docker available there — see the caveats each component's
|
||||
summary already flagged). Individual pieces are unit-tested and built
|
||||
successfully in isolation; this document is the logical sequence to run
|
||||
for real, not a report that it's been run. Expect to debug something on
|
||||
first attempt, and treat the "Troubleshooting" section at the bottom as a
|
||||
starting point, not an exhaustive list.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker with **Compose v2** (`docker compose`, not the legacy
|
||||
`docker-compose` v1 binary) — the compose file uses
|
||||
`service_completed_successfully` conditions that v1 doesn't support.
|
||||
- Rust toolchain (`cargo`) and `protoc` — to build the agent.
|
||||
- `openssl` — to generate dev mTLS certs.
|
||||
- A systemd-based Linux host to run the agent on (journald is the default
|
||||
source). If you're not on such a host, see `/agent/README.md`'s
|
||||
`file-tail` feature as an alternative source.
|
||||
|
||||
You do **not** need the musl cross-compilation target for this runbook —
|
||||
that's for producing the distro-agnostic release binary. A native
|
||||
`cargo build --release` is enough to run the agent on the same machine
|
||||
you're testing on.
|
||||
|
||||
## 1. Generate dev mTLS certs
|
||||
|
||||
```sh
|
||||
./hack/dev-certs/generate.sh
|
||||
```
|
||||
|
||||
Writes a throwaway CA plus a server cert (for `ingest`) and a client cert
|
||||
(for the agent) to `hack/dev-certs/out/`. Dev-only — see the script's
|
||||
header comment for why.
|
||||
|
||||
## 2. Bring up the backend stack
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
This builds and starts, in dependency order: `redpanda` → `redpanda-provision`
|
||||
(creates the `sentry.logs.raw` topic, then exits) → `clickhouse` →
|
||||
`clickhouse-migrate` (applies `/storage/migrations`, then exits) →
|
||||
`ingest` and `api` → `web`.
|
||||
|
||||
Check everything came up:
|
||||
|
||||
```sh
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
`redpanda-provision` and `clickhouse-migrate` should show `Exited (0)`
|
||||
(one-shot jobs, not long-running). Everything else should show `Up` /
|
||||
`healthy`.
|
||||
|
||||
If `ingest` or `api` crash-looped, they likely started before their
|
||||
`depends_on` conditions were actually satisfied, or the dev certs from
|
||||
step 1 don't exist yet — check `docker compose logs ingest`.
|
||||
|
||||
## 3. Sanity-check the backend before involving the agent
|
||||
|
||||
```sh
|
||||
curl http://localhost:8080/healthz
|
||||
# -> 200, empty body
|
||||
|
||||
curl -X POST http://localhost:8080/query \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"sql": "SELECT 1"}'
|
||||
# -> {"columns":["1"],"rows":[[1]]} (exact column name may vary by ClickHouse version)
|
||||
```
|
||||
|
||||
This confirms `api` can reach `clickhouse` before you go looking for bugs
|
||||
anywhere else. It doesn't touch the `logs` table, so it works even before
|
||||
any agent has sent data.
|
||||
|
||||
## 4. Install the agent's mTLS material
|
||||
|
||||
The agent's default config expects certs at `/etc/sentry-agent/` (see
|
||||
`/agent/config/agent.example.toml`), which requires root:
|
||||
|
||||
```sh
|
||||
sudo mkdir -p /etc/sentry-agent
|
||||
sudo cp hack/dev-certs/out/ca.pem \
|
||||
hack/dev-certs/out/client.pem \
|
||||
hack/dev-certs/out/client-key.pem \
|
||||
/etc/sentry-agent/
|
||||
```
|
||||
|
||||
## 5. Build and run the agent
|
||||
|
||||
```sh
|
||||
cd agent
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
The agent's built-in defaults already match this setup with **zero
|
||||
config file**: journald source (whole journal), service name `default`,
|
||||
ingest endpoint `https://127.0.0.1:4317` (matches the port `ingest`
|
||||
publishes in `docker-compose.yml`), and the cert paths from step 4. This
|
||||
is the "no required flags for the common case" design goal from
|
||||
`/agent/README.md` — if it doesn't just run, that design assumption is
|
||||
wrong somewhere and worth reporting as a bug, not working around.
|
||||
|
||||
Reading the system journal generally needs root (or membership in the
|
||||
`systemd-journal` group with a distro that grants it read access — varies
|
||||
by distro, root is the reliable path for this runbook):
|
||||
|
||||
```sh
|
||||
sudo ./target/release/sentry-agent
|
||||
```
|
||||
|
||||
Leave it running in this terminal — you should see a `connected to ingest
|
||||
service` log line. If you see a TLS or connection error instead, stop
|
||||
here and check the Troubleshooting section before continuing.
|
||||
|
||||
## 6. Generate a test log line
|
||||
|
||||
In another terminal, **after** the agent is running and connected
|
||||
(journald tailing starts from "now" — anything logged before the agent
|
||||
started won't be picked up):
|
||||
|
||||
```sh
|
||||
logger "hello from sentry phase 0"
|
||||
```
|
||||
|
||||
`logger` (part of util-linux, present on virtually every Linux distro)
|
||||
writes this to the system log, which journald captures immediately.
|
||||
|
||||
Give it a couple of seconds — the agent batches with a 2-second flush
|
||||
interval by default, so the line won't hit ingest instantly.
|
||||
|
||||
## 7. Confirm it's queryable
|
||||
|
||||
**Via the web UI:**
|
||||
|
||||
`web` is already running from step 2 (`docker compose up -d --build`
|
||||
starts every service in the file). Open `http://localhost:3000`, run the
|
||||
default query (`SELECT * FROM logs
|
||||
ORDER BY timestamp DESC LIMIT 100`), and look for a row with
|
||||
`message = "hello from sentry phase 0"`.
|
||||
|
||||
**Or via curl, if you want to skip the browser:**
|
||||
|
||||
```sh
|
||||
curl -X POST http://localhost:8080/query \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"sql": "SELECT * FROM logs ORDER BY timestamp DESC LIMIT 10"}'
|
||||
```
|
||||
|
||||
**Or via sentryctl, just to confirm api is up (doesn't check the data
|
||||
itself):**
|
||||
|
||||
```sh
|
||||
cd cli && go run ./cmd/sentryctl ping
|
||||
```
|
||||
|
||||
If you see the row: that's Phase 0 done, end to end. If you don't, see
|
||||
Troubleshooting below.
|
||||
|
||||
## Tearing down
|
||||
|
||||
```sh
|
||||
docker compose down # stops and removes containers, keeps volumes
|
||||
docker compose down -v # also wipes Redpanda/ClickHouse data — start clean next time
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Agent logs a TLS/certificate error on startup.**
|
||||
Check the server cert's SAN actually covers how the agent is connecting
|
||||
(`openssl x509 -in hack/dev-certs/out/server.pem -noout -ext
|
||||
subjectAltName` — should list `DNS:ingest, DNS:localhost,
|
||||
IP:127.0.0.1`). If you changed the agent's `ingest.endpoint` to something
|
||||
not in that list, regenerate certs with an updated SAN in
|
||||
`hack/dev-certs/generate.sh`, don't disable TLS verification.
|
||||
|
||||
**Agent connects but no data ever shows up in ClickHouse.**
|
||||
Check each hop in order rather than guessing:
|
||||
1. `docker compose logs ingest` — look for "batch produced to redpanda"
|
||||
(gRPC front end got the batch) vs. errors.
|
||||
2. `docker compose logs ingest` again — look for "batch flushed to
|
||||
clickhouse" from the consumer half. If you see repeated "clickhouse
|
||||
batch write failed... will redeliver" messages, `clickhouse-migrate`
|
||||
likely hasn't finished (check `docker compose ps`) — the consumer will
|
||||
keep retrying and self-heal once the table exists, per its
|
||||
at-least-once design (see `/ingest/README.md`), so this may just need
|
||||
more time rather than intervention.
|
||||
3. `docker compose exec redpanda rpk topic list` — confirm
|
||||
`sentry.logs.raw` exists (if `redpanda-provision` failed, it won't).
|
||||
|
||||
**`docker compose up` fails on `service_completed_successfully`.**
|
||||
You're likely on Compose v1 (`docker-compose`, hyphenated) rather than v2
|
||||
(`docker compose`, space) — see Prerequisites.
|
||||
|
||||
**Web UI query returns an error instead of rows.**
|
||||
Open the browser's network tab — if the request never leaves the page
|
||||
(CORS error in the console), confirm `api`'s `CORS_ALLOWED_ORIGIN`
|
||||
(defaults to `*`, should not be the issue) and that `VITE_API_BASE_URL`
|
||||
was set correctly at `web`'s build time (it's baked in, not read at
|
||||
container start — see `/web/README.md`).
|
||||
@@ -0,0 +1,16 @@
|
||||
# hack
|
||||
|
||||
Local developer tooling that isn't part of any shipped component — scripts
|
||||
you run against your own machine/dev stack, not code that ends up in a
|
||||
container image (except `dev-certs`' *output*, which mounts into the
|
||||
ingest container).
|
||||
|
||||
Not one of the top-level directories in the original monorepo scaffold —
|
||||
added because dev-only mTLS cert generation didn't have a natural home in
|
||||
`/deploy` (real deployment manifests), `/transport`, or any other existing
|
||||
component. `/hack` is the conventional name for this in a lot of larger Go
|
||||
monorepos (Kubernetes among them).
|
||||
|
||||
- `dev-certs/` — generates a throwaway CA + server/client cert pair for
|
||||
local mTLS between the agent and ingest. See `/docs/phase-0-runbook.md`
|
||||
for when to run it.
|
||||
@@ -0,0 +1 @@
|
||||
out/
|
||||
Executable
+46
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
# Generates a throwaway CA plus a server cert (for ingest) and a client
|
||||
# cert (for the agent) for local mTLS. Dev/homelab only — never use this
|
||||
# CA or its certs for anything resembling production; there's no rotation,
|
||||
# no revocation, and the CA key sits unencrypted on disk right next to
|
||||
# everything it signed.
|
||||
#
|
||||
# Re-run to regenerate from scratch; existing output is overwritten.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
OUT_DIR="${SCRIPT_DIR}/out"
|
||||
DAYS="${DEV_CERT_DAYS:-365}"
|
||||
|
||||
mkdir -p "${OUT_DIR}"
|
||||
cd "${OUT_DIR}"
|
||||
|
||||
echo "Generating dev CA..."
|
||||
openssl req -x509 -newkey rsa:4096 -sha256 -days "${DAYS}" -nodes \
|
||||
-keyout ca-key.pem -out ca.pem \
|
||||
-subj "/O=Sentry Dev/CN=Sentry Dev CA"
|
||||
|
||||
gen_leaf() {
|
||||
local name="$1" cn="$2" san="$3"
|
||||
openssl req -newkey rsa:2048 -nodes -keyout "${name}-key.pem" -out "${name}.csr" \
|
||||
-subj "/O=Sentry Dev/CN=${cn}"
|
||||
openssl x509 -req -in "${name}.csr" -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
|
||||
-out "${name}.pem" -days "${DAYS}" -sha256 \
|
||||
-extfile <(printf "subjectAltName=%s" "${san}")
|
||||
rm -f "${name}.csr"
|
||||
}
|
||||
|
||||
# SANs cover both "reached by another container on the compose network"
|
||||
# (ingest) and "reached from the host" (localhost/127.0.0.1, for an
|
||||
# agent running natively per /agent/README.md's journald caveat).
|
||||
echo "Generating server (ingest) cert..."
|
||||
gen_leaf server ingest "DNS:ingest,DNS:localhost,IP:127.0.0.1"
|
||||
|
||||
echo "Generating client (agent) cert..."
|
||||
gen_leaf client sentry-agent "DNS:sentry-agent"
|
||||
|
||||
rm -f ca.srl
|
||||
|
||||
echo
|
||||
echo "Done. Certs written to ${OUT_DIR}/:"
|
||||
ls "${OUT_DIR}"
|
||||
@@ -0,0 +1,15 @@
|
||||
# Build context must be the repo root (sentry/), not ingest/, since this
|
||||
# needs both ingest/ and proto/:
|
||||
# docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
|
||||
FROM golang:1.25-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY proto ./proto
|
||||
COPY ingest ./ingest
|
||||
WORKDIR /src/ingest
|
||||
RUN go mod download
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/ingest ./cmd/ingest
|
||||
|
||||
FROM gcr.io/distroless/static-debian12
|
||||
COPY --from=builder /out/ingest /ingest
|
||||
ENTRYPOINT ["/ingest"]
|
||||
@@ -0,0 +1,85 @@
|
||||
# ingest
|
||||
|
||||
Go service sitting between the Rust agent and ClickHouse. Two halves in one
|
||||
binary, selected with `--mode`:
|
||||
|
||||
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
||||
connect to. Forwards each record, proto-encoded and unchanged, onto
|
||||
Redpanda. Does no normalization — kept thin so agent-facing latency isn't
|
||||
coupled to ClickHouse write performance.
|
||||
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
||||
shape (`internal/normalize`), and batch-writes via the native protocol
|
||||
driver. Commits Redpanda offsets only after a successful ClickHouse
|
||||
write, so a ClickHouse outage causes redelivery on restart rather than
|
||||
data loss.
|
||||
- **all** (default) — both, in one process. This is what docker-compose
|
||||
runs. Splitting into two deployments later (e.g. to scale them
|
||||
independently in k8s) is a manifest change, not a code change — see
|
||||
`--mode`.
|
||||
|
||||
## Why Redpanda stays in the path
|
||||
|
||||
Confirmed with the project owner during Phase 0 planning: the gRPC front
|
||||
end produces to Redpanda rather than writing ClickHouse directly. This
|
||||
exercises the pinned transport layer from day one and keeps agents from
|
||||
ever needing Kafka credentials — mTLS to `ingest` is the only network
|
||||
egress an agent has. See `/docs/architecture.md`.
|
||||
|
||||
## Dependencies worth knowing about
|
||||
|
||||
- **github.com/segmentio/kafka-go** — pure Go, no cgo, chosen over
|
||||
franz-go/confluent-kafka-go specifically to keep the distroless build
|
||||
simple (confirmed with the project owner; see git history / PR
|
||||
discussion for the tradeoffs considered).
|
||||
- **github.com/ClickHouse/clickhouse-go/v2** — official client, native
|
||||
protocol, pure Go (no cgo).
|
||||
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
||||
server and consumer halves concurrently and propagate the first error.
|
||||
|
||||
## Configuration
|
||||
|
||||
All via environment variables (see `internal/config/config.go` for the
|
||||
full list and defaults) — no config file format for Phase 0:
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `GRPC_LISTEN_ADDR` | `:4317` | Agent-facing gRPC listen address |
|
||||
| `TLS_CERT_FILE` / `TLS_KEY_FILE` | `/etc/sentry-ingest/server{,-key}.pem` | ingest's own mTLS identity |
|
||||
| `TLS_CLIENT_CA_FILE` | `/etc/sentry-ingest/ca.pem` | CA used to verify agent client certs |
|
||||
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
|
||||
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match the topic provisioned in `/transport` |
|
||||
| `REDPANDA_CONSUMER_GROUP` | `sentry-ingest` | Consumer group id |
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port, not HTTP |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
|
||||
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Requires `google.golang.org/protobuf/cmd/protoc-gen-go` and
|
||||
`google.golang.org/grpc/cmd/protoc-gen-go-grpc` only if you're
|
||||
regenerating `/proto`'s Go bindings — ingest itself just imports the
|
||||
already-generated `github.com/sentry/sentry/proto` module (see the
|
||||
`replace` directive in `go.mod`, pointing at `../proto`).
|
||||
|
||||
```sh
|
||||
# from the repo root, not ingest/
|
||||
docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
```
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
|
||||
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
|
||||
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
|
||||
unit-tested against fakes — no embedded broker or database needed. What's
|
||||
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
|
||||
wiring and the ClickHouse native-protocol driver itself. Those are only
|
||||
exercised by the docker-compose end-to-end flow described in
|
||||
`/docs/phase-0-runbook.md`.
|
||||
@@ -0,0 +1,77 @@
|
||||
// Command ingest is the Sentry ingest service. It has two halves that can
|
||||
// run in one process or be split across deployments via --mode:
|
||||
//
|
||||
// - server: mTLS gRPC front end that agents push batches to; forwards
|
||||
// them onto Redpanda unchanged.
|
||||
// - consumer: reads back off Redpanda, normalizes, batch-writes to
|
||||
// ClickHouse.
|
||||
// - all (default): both, in one process — the Phase 0 / docker-compose
|
||||
// shape. Splitting into separate deployments later is a k8s manifest
|
||||
// change, not a code change.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"golang.org/x/sync/errgroup"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/clickhousewriter"
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
"github.com/sentry/sentry/ingest/internal/consumer"
|
||||
"github.com/sentry/sentry/ingest/internal/grpcserver"
|
||||
"github.com/sentry/sentry/ingest/internal/producer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
mode := flag.String("mode", "all", "which half of ingest to run: server | consumer | all")
|
||||
flag.Parse()
|
||||
|
||||
if *mode != "server" && *mode != "consumer" && *mode != "all" {
|
||||
fmt.Fprintf(os.Stderr, "unknown --mode %q, must be server|consumer|all\n", *mode)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
logger.Error("loading config", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
|
||||
if *mode == "server" || *mode == "all" {
|
||||
p := producer.New(cfg.Redpanda)
|
||||
defer p.Close()
|
||||
srv := grpcserver.New(logger, cfg.GRPC, cfg.TLS, p)
|
||||
g.Go(func() error { return srv.Run(ctx) })
|
||||
}
|
||||
|
||||
if *mode == "consumer" || *mode == "all" {
|
||||
chw, err := clickhousewriter.New(ctx, cfg.ClickHouse)
|
||||
if err != nil {
|
||||
logger.Error("connecting to clickhouse", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer chw.Close()
|
||||
c := consumer.New(logger, cfg.Redpanda, cfg.Batch, chw)
|
||||
g.Go(func() error { return c.Run(ctx) })
|
||||
}
|
||||
|
||||
logger.Info("ingest started", "mode", *mode)
|
||||
|
||||
if err := g.Wait(); err != nil {
|
||||
logger.Error("ingest exited with error", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
module github.com/sentry/sentry/ingest
|
||||
|
||||
go 1.25.0
|
||||
|
||||
replace github.com/sentry/sentry/proto => ../proto
|
||||
|
||||
require (
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
||||
github.com/segmentio/kafka-go v0.4.51
|
||||
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||
golang.org/x/sync v0.22.0
|
||||
google.golang.org/grpc v1.83.0
|
||||
google.golang.org/protobuf v1.36.12
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
||||
github.com/andybalholm/brotli v1.2.2 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/go-faster/city v1.0.1 // indirect
|
||||
github.com/go-faster/errors v0.7.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/klauspost/compress v1.19.1 // indirect
|
||||
github.com/paulmach/orb v0.13.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||
github.com/segmentio/asm v1.2.1 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
go.opentelemetry.io/otel v1.44.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||
)
|
||||
@@ -0,0 +1,78 @@
|
||||
github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE=
|
||||
github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM=
|
||||
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
|
||||
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
|
||||
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
|
||||
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
||||
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
|
||||
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
|
||||
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
|
||||
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
|
||||
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||
github.com/segmentio/kafka-go v0.4.51 h1:JgDPPG75tC1rWIS2Me6MwcvXJ6f49UQ4HjAOef71Hno=
|
||||
github.com/segmentio/kafka-go v0.4.51/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
|
||||
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
|
||||
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
|
||||
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
|
||||
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,60 @@
|
||||
// Package clickhousewriter batch-inserts normalized log rows into
|
||||
// ClickHouse using the native protocol driver's batch API.
|
||||
package clickhousewriter
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
"github.com/sentry/sentry/ingest/internal/normalize"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Writer struct {
|
||||
conn driver.Conn
|
||||
}
|
||||
|
||||
func New(ctx context.Context, cfg config.ClickHouseConfig) (*Writer, error) {
|
||||
conn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{cfg.Addr},
|
||||
Auth: clickhouse.Auth{
|
||||
Database: cfg.Database,
|
||||
Username: cfg.Username,
|
||||
Password: cfg.Password,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening clickhouse connection: %w", err)
|
||||
}
|
||||
if err := conn.Ping(ctx); err != nil {
|
||||
return nil, fmt.Errorf("pinging clickhouse: %w", err)
|
||||
}
|
||||
return &Writer{conn: conn}, nil
|
||||
}
|
||||
|
||||
func (w *Writer) Close() error {
|
||||
return w.conn.Close()
|
||||
}
|
||||
|
||||
func (w *Writer) WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error {
|
||||
batch, err := w.conn.PrepareBatch(ctx, "INSERT INTO logs (timestamp, host, service, severity, message, attributes)")
|
||||
if err != nil {
|
||||
return fmt.Errorf("preparing batch: %w", err)
|
||||
}
|
||||
|
||||
for _, rec := range records {
|
||||
row := normalize.ToRow(rec)
|
||||
if err := batch.Append(row.Timestamp, row.Host, row.Service, row.Severity, row.Message, row.Attributes); err != nil {
|
||||
return fmt.Errorf("appending row to batch: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := batch.Send(); err != nil {
|
||||
return fmt.Errorf("sending batch: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
// Package config loads ingest's configuration from environment variables.
|
||||
// Phase 0 deliberately has no config file format of its own — env vars are
|
||||
// enough for a docker-compose/k8s deployment and avoid pulling in a config
|
||||
// library.
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
GRPC GRPCConfig
|
||||
TLS TLSConfig
|
||||
Redpanda RedpandaConfig
|
||||
ClickHouse ClickHouseConfig
|
||||
Batch BatchConfig
|
||||
}
|
||||
|
||||
type GRPCConfig struct {
|
||||
ListenAddr string
|
||||
}
|
||||
|
||||
// TLSConfig is the server-side mTLS material: the ingest service's own
|
||||
// cert/key, and the CA used to verify agent client certs.
|
||||
type TLSConfig struct {
|
||||
CertFile string
|
||||
KeyFile string
|
||||
ClientCAFile string
|
||||
}
|
||||
|
||||
type RedpandaConfig struct {
|
||||
Brokers []string
|
||||
Topic string
|
||||
ConsumerGroup string
|
||||
}
|
||||
|
||||
type ClickHouseConfig struct {
|
||||
Addr string
|
||||
Database string
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
type BatchConfig struct {
|
||||
MaxSize int
|
||||
FlushIntervalMS int
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
cfg := Config{
|
||||
GRPC: GRPCConfig{
|
||||
ListenAddr: getenv("GRPC_LISTEN_ADDR", ":4317"),
|
||||
},
|
||||
TLS: TLSConfig{
|
||||
CertFile: getenv("TLS_CERT_FILE", "/etc/sentry-ingest/server.pem"),
|
||||
KeyFile: getenv("TLS_KEY_FILE", "/etc/sentry-ingest/server-key.pem"),
|
||||
ClientCAFile: getenv("TLS_CLIENT_CA_FILE", "/etc/sentry-ingest/ca.pem"),
|
||||
},
|
||||
Redpanda: RedpandaConfig{
|
||||
Brokers: strings.Split(getenv("REDPANDA_BROKERS", "localhost:9092"), ","),
|
||||
Topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
|
||||
ConsumerGroup: getenv("REDPANDA_CONSUMER_GROUP", "sentry-ingest"),
|
||||
},
|
||||
ClickHouse: ClickHouseConfig{
|
||||
Addr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
|
||||
Database: getenv("CLICKHOUSE_DATABASE", "sentry"),
|
||||
Username: getenv("CLICKHOUSE_USERNAME", "default"),
|
||||
Password: getenv("CLICKHOUSE_PASSWORD", ""),
|
||||
},
|
||||
}
|
||||
|
||||
maxSize, err := strconv.Atoi(getenv("CONSUMER_BATCH_MAX_SIZE", "500"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("CONSUMER_BATCH_MAX_SIZE: %w", err)
|
||||
}
|
||||
cfg.Batch.MaxSize = maxSize
|
||||
|
||||
flushMS, err := strconv.Atoi(getenv("CONSUMER_BATCH_FLUSH_INTERVAL_MS", "2000"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("CONSUMER_BATCH_FLUSH_INTERVAL_MS: %w", err)
|
||||
}
|
||||
cfg.Batch.FlushIntervalMS = flushMS
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package config
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestLoadDefaults(t *testing.T) {
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.GRPC.ListenAddr != ":4317" {
|
||||
t.Errorf("GRPC.ListenAddr = %q, want :4317", cfg.GRPC.ListenAddr)
|
||||
}
|
||||
if cfg.Redpanda.Topic != "sentry.logs.raw" {
|
||||
t.Errorf("Redpanda.Topic = %q, want sentry.logs.raw", cfg.Redpanda.Topic)
|
||||
}
|
||||
if cfg.Batch.MaxSize != 500 {
|
||||
t.Errorf("Batch.MaxSize = %d, want 500", cfg.Batch.MaxSize)
|
||||
}
|
||||
if cfg.Batch.FlushIntervalMS != 2000 {
|
||||
t.Errorf("Batch.FlushIntervalMS = %d, want 2000", cfg.Batch.FlushIntervalMS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadOverridesFromEnv(t *testing.T) {
|
||||
t.Setenv("GRPC_LISTEN_ADDR", ":9999")
|
||||
t.Setenv("REDPANDA_BROKERS", "a:9092,b:9092")
|
||||
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "10")
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.GRPC.ListenAddr != ":9999" {
|
||||
t.Errorf("GRPC.ListenAddr = %q, want :9999", cfg.GRPC.ListenAddr)
|
||||
}
|
||||
if len(cfg.Redpanda.Brokers) != 2 || cfg.Redpanda.Brokers[0] != "a:9092" || cfg.Redpanda.Brokers[1] != "b:9092" {
|
||||
t.Errorf("Redpanda.Brokers = %+v, want [a:9092 b:9092]", cfg.Redpanda.Brokers)
|
||||
}
|
||||
if cfg.Batch.MaxSize != 10 {
|
||||
t.Errorf("Batch.MaxSize = %d, want 10", cfg.Batch.MaxSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadInvalidBatchSizeErrors(t *testing.T) {
|
||||
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "not-a-number")
|
||||
if _, err := Load(); err == nil {
|
||||
t.Fatal("expected error for non-numeric CONSUMER_BATCH_MAX_SIZE, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
// Package consumer reads normalized-on-write LogRecords back off Redpanda
|
||||
// and batch-writes them into ClickHouse. Offsets are committed only after
|
||||
// a successful ClickHouse write, so a ClickHouse outage causes redelivery
|
||||
// on restart rather than silent data loss (at-least-once, not exactly-once
|
||||
// — Phase 0 doesn't dedupe on the consumer side).
|
||||
package consumer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
// chWriter is the subset of *clickhousewriter.Writer this package depends
|
||||
// on, kept as an interface so the flush loop is unit-testable without a
|
||||
// real ClickHouse connection.
|
||||
type chWriter interface {
|
||||
WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error
|
||||
}
|
||||
|
||||
// reader is the subset of *kafka.Reader used here, as an interface so the
|
||||
// flush/commit logic can be tested against a fake without a real broker.
|
||||
type reader interface {
|
||||
FetchMessage(ctx context.Context) (kafka.Message, error)
|
||||
CommitMessages(ctx context.Context, msgs ...kafka.Message) error
|
||||
Close() error
|
||||
}
|
||||
|
||||
type Consumer struct {
|
||||
logger *slog.Logger
|
||||
reader reader
|
||||
writer chWriter
|
||||
batchCfg config.BatchConfig
|
||||
}
|
||||
|
||||
func New(logger *slog.Logger, redpandaCfg config.RedpandaConfig, batchCfg config.BatchConfig, w chWriter) *Consumer {
|
||||
r := kafka.NewReader(kafka.ReaderConfig{
|
||||
Brokers: redpandaCfg.Brokers,
|
||||
Topic: redpandaCfg.Topic,
|
||||
GroupID: redpandaCfg.ConsumerGroup,
|
||||
})
|
||||
return &Consumer{logger: logger, reader: r, writer: w, batchCfg: batchCfg}
|
||||
}
|
||||
|
||||
func (c *Consumer) Run(ctx context.Context) error {
|
||||
defer c.reader.Close()
|
||||
|
||||
flushInterval := time.Duration(c.batchCfg.FlushIntervalMS) * time.Millisecond
|
||||
ticker := time.NewTicker(flushInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
msgCh := make(chan kafka.Message)
|
||||
fetchErrCh := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
for {
|
||||
m, err := c.reader.FetchMessage(ctx)
|
||||
if err != nil {
|
||||
fetchErrCh <- err
|
||||
return
|
||||
}
|
||||
select {
|
||||
case msgCh <- m:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
var records []*logsv1.LogRecord
|
||||
var pending []kafka.Message
|
||||
|
||||
flush := func() {
|
||||
if len(records) == 0 {
|
||||
return
|
||||
}
|
||||
if err := c.writer.WriteBatch(ctx, records); err != nil {
|
||||
c.logger.Error("clickhouse batch write failed, offsets not committed, will redeliver",
|
||||
"records", len(records), "error", err)
|
||||
} else if err := c.reader.CommitMessages(ctx, pending...); err != nil {
|
||||
c.logger.Error("committing offsets after clickhouse write", "error", err)
|
||||
} else {
|
||||
c.logger.Debug("batch flushed to clickhouse", "records", len(records))
|
||||
}
|
||||
records = records[:0]
|
||||
pending = pending[:0]
|
||||
}
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
flush()
|
||||
return nil
|
||||
case err := <-fetchErrCh:
|
||||
flush()
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
case <-ticker.C:
|
||||
flush()
|
||||
case m := <-msgCh:
|
||||
var rec logsv1.LogRecord
|
||||
if err := proto.Unmarshal(m.Value, &rec); err != nil {
|
||||
c.logger.Warn("skipping unparseable message", "error", err, "offset", m.Offset)
|
||||
if cerr := c.reader.CommitMessages(ctx, m); cerr != nil {
|
||||
c.logger.Error("committing offset for poison message", "error", cerr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
records = append(records, &rec)
|
||||
pending = append(pending, m)
|
||||
if len(records) >= c.batchCfg.MaxSize {
|
||||
flush()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
package consumer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type fakeReader struct {
|
||||
msgs chan kafka.Message
|
||||
|
||||
mu sync.Mutex
|
||||
committed [][]kafka.Message
|
||||
}
|
||||
|
||||
func newFakeReader() *fakeReader {
|
||||
return &fakeReader{msgs: make(chan kafka.Message, 16)}
|
||||
}
|
||||
|
||||
func (f *fakeReader) push(m kafka.Message) { f.msgs <- m }
|
||||
|
||||
func (f *fakeReader) FetchMessage(ctx context.Context) (kafka.Message, error) {
|
||||
select {
|
||||
case m := <-f.msgs:
|
||||
return m, nil
|
||||
case <-ctx.Done():
|
||||
return kafka.Message{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeReader) CommitMessages(_ context.Context, msgs ...kafka.Message) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.committed = append(f.committed, msgs)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeReader) Close() error { return nil }
|
||||
|
||||
func (f *fakeReader) commitCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.committed)
|
||||
}
|
||||
|
||||
type fakeWriter struct {
|
||||
mu sync.Mutex
|
||||
batches [][]*logsv1.LogRecord
|
||||
failNext bool
|
||||
}
|
||||
|
||||
func (f *fakeWriter) WriteBatch(_ context.Context, records []*logsv1.LogRecord) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.failNext {
|
||||
f.failNext = false
|
||||
return errors.New("simulated clickhouse failure")
|
||||
}
|
||||
batch := make([]*logsv1.LogRecord, len(records))
|
||||
copy(batch, records)
|
||||
f.batches = append(f.batches, batch)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeWriter) batchCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.batches)
|
||||
}
|
||||
|
||||
func newTestConsumer(r reader, w chWriter, batchCfg config.BatchConfig) *Consumer {
|
||||
return &Consumer{
|
||||
logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
reader: r,
|
||||
writer: w,
|
||||
batchCfg: batchCfg,
|
||||
}
|
||||
}
|
||||
|
||||
func mustMarshal(t *testing.T, rec *logsv1.LogRecord) []byte {
|
||||
t.Helper()
|
||||
b, err := proto.Marshal(rec)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func waitFor(t *testing.T, timeout time.Duration, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if cond() {
|
||||
return
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("condition not met before timeout")
|
||||
}
|
||||
|
||||
func TestConsumerFlushesOnBatchSize(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 2, FlushIntervalMS: 60_000})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "a"})})
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "b"})})
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
|
||||
|
||||
fw.mu.Lock()
|
||||
if len(fw.batches[0]) != 2 {
|
||||
t.Fatalf("expected batch of 2 records, got %d", len(fw.batches[0]))
|
||||
}
|
||||
fw.mu.Unlock()
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fr.commitCount() == 1 })
|
||||
}
|
||||
|
||||
func TestConsumerFlushesOnTimeout(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1000, FlushIntervalMS: 20})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "only-one"})})
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
|
||||
|
||||
fw.mu.Lock()
|
||||
if len(fw.batches[0]) != 1 {
|
||||
t.Fatalf("expected batch of 1 record, got %d", len(fw.batches[0]))
|
||||
}
|
||||
fw.mu.Unlock()
|
||||
}
|
||||
|
||||
func TestConsumerDoesNotCommitOnWriteFailure(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{failNext: true}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1, FlushIntervalMS: 60_000})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "will-fail"})})
|
||||
|
||||
// Give the flush a moment to run and fail.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
if got := fr.commitCount(); got != 0 {
|
||||
t.Fatalf("expected no commits after a failed clickhouse write, got %d", got)
|
||||
}
|
||||
// The batch was attempted even though writer returned an error.
|
||||
if fw.batchCount() != 0 {
|
||||
t.Fatalf("fakeWriter should not record a failed batch, got %d recorded", fw.batchCount())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
// Package grpcserver implements the agent-facing side of ingest: an mTLS
|
||||
// gRPC server accepting LogIngest.PushBatch calls, which it forwards
|
||||
// unchanged (proto-encoded) onto Redpanda. Normalization into the
|
||||
// ClickHouse row shape happens later, on the consumer side.
|
||||
package grpcserver
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials"
|
||||
"google.golang.org/grpc/status"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
logsv1.UnimplementedLogIngestServer
|
||||
|
||||
logger *slog.Logger
|
||||
grpcCfg config.GRPCConfig
|
||||
tlsCfg config.TLSConfig
|
||||
producer batchProducer
|
||||
}
|
||||
|
||||
// batchProducer is the subset of *producer.Producer this package depends
|
||||
// on, so tests can substitute a fake without touching Redpanda.
|
||||
type batchProducer interface {
|
||||
WriteBatch(ctx context.Context, msgs []kafka.Message) error
|
||||
}
|
||||
|
||||
func New(logger *slog.Logger, grpcCfg config.GRPCConfig, tlsCfg config.TLSConfig, p batchProducer) *Server {
|
||||
return &Server{logger: logger, grpcCfg: grpcCfg, tlsCfg: tlsCfg, producer: p}
|
||||
}
|
||||
|
||||
// Run blocks serving gRPC until ctx is canceled, then gracefully stops.
|
||||
func (s *Server) Run(ctx context.Context) error {
|
||||
tlsConf, err := loadServerTLSConfig(s.tlsCfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading TLS config: %w", err)
|
||||
}
|
||||
|
||||
lis, err := net.Listen("tcp", s.grpcCfg.ListenAddr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", s.grpcCfg.ListenAddr, err)
|
||||
}
|
||||
|
||||
grpcSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsConf)))
|
||||
logsv1.RegisterLogIngestServer(grpcSrv, s)
|
||||
|
||||
s.logger.Info("gRPC server listening", "addr", s.grpcCfg.ListenAddr)
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() { errCh <- grpcSrv.Serve(lis) }()
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
grpcSrv.GracefulStop()
|
||||
return nil
|
||||
case err := <-errCh:
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) PushBatch(ctx context.Context, req *logsv1.PushBatchRequest) (*logsv1.PushBatchResponse, error) {
|
||||
if len(req.GetRecords()) == 0 {
|
||||
return &logsv1.PushBatchResponse{Accepted: 0}, nil
|
||||
}
|
||||
|
||||
msgs := make([]kafka.Message, 0, len(req.GetRecords()))
|
||||
for _, rec := range req.GetRecords() {
|
||||
val, err := proto.Marshal(rec)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "marshaling record: %v", err)
|
||||
}
|
||||
msgs = append(msgs, kafka.Message{
|
||||
Key: []byte(rec.GetHost()),
|
||||
Value: val,
|
||||
})
|
||||
}
|
||||
|
||||
if err := s.producer.WriteBatch(ctx, msgs); err != nil {
|
||||
s.logger.Error("failed to write batch to redpanda", "batch_id", req.GetBatchId(), "error", err)
|
||||
return nil, status.Errorf(codes.Unavailable, "writing to transport: %v", err)
|
||||
}
|
||||
|
||||
s.logger.Debug("batch produced to redpanda", "batch_id", req.GetBatchId(), "records", len(req.GetRecords()))
|
||||
return &logsv1.PushBatchResponse{Accepted: uint32(len(req.GetRecords()))}, nil
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package grpcserver
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
)
|
||||
|
||||
// loadServerTLSConfig builds the mTLS server config: ingest's own
|
||||
// certificate, plus the CA used to verify agent client certificates.
|
||||
// Agents are never accepted without a client cert signed by this CA.
|
||||
func loadServerTLSConfig(cfg config.TLSConfig) (*tls.Config, error) {
|
||||
cert, err := tls.LoadX509KeyPair(cfg.CertFile, cfg.KeyFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("loading server cert/key: %w", err)
|
||||
}
|
||||
|
||||
caPEM, err := os.ReadFile(cfg.ClientCAFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading client CA file: %w", err)
|
||||
}
|
||||
caPool := x509.NewCertPool()
|
||||
if !caPool.AppendCertsFromPEM(caPEM) {
|
||||
return nil, fmt.Errorf("no valid certificates found in client CA file %s", cfg.ClientCAFile)
|
||||
}
|
||||
|
||||
return &tls.Config{
|
||||
Certificates: []tls.Certificate{cert},
|
||||
ClientAuth: tls.RequireAndVerifyClientCert,
|
||||
ClientCAs: caPool,
|
||||
MinVersion: tls.VersionTLS12,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
// Package normalize maps the wire-format LogRecord (as agents send it)
|
||||
// into the ClickHouse row shape defined in /storage. This is the "OTel-log-
|
||||
// like schema" normalization step called for in the ingest design — Phase
|
||||
// 0 keeps it to the minimal column set; full OTel field mapping (separate
|
||||
// SeverityNumber/SeverityText, resource attributes, etc.) is deferred, see
|
||||
// the open questions in /docs/architecture.md.
|
||||
package normalize
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Row struct {
|
||||
Timestamp time.Time
|
||||
Host string
|
||||
Service string
|
||||
Severity string
|
||||
Message string
|
||||
Attributes map[string]string
|
||||
}
|
||||
|
||||
func ToRow(rec *logsv1.LogRecord) Row {
|
||||
attrs := rec.GetAttributes()
|
||||
if attrs == nil {
|
||||
attrs = map[string]string{}
|
||||
}
|
||||
return Row{
|
||||
Timestamp: time.Unix(0, rec.GetTimestampUnixNano()).UTC(),
|
||||
Host: rec.GetHost(),
|
||||
Service: rec.GetService(),
|
||||
Severity: severityText(rec.GetSeverity()),
|
||||
Message: rec.GetMessage(),
|
||||
Attributes: attrs,
|
||||
}
|
||||
}
|
||||
|
||||
// severityText maps the proto Severity enum to short OTel-style severity
|
||||
// names, stored as the `severity` column's value.
|
||||
func severityText(sev logsv1.Severity) string {
|
||||
switch sev {
|
||||
case logsv1.Severity_SEVERITY_TRACE:
|
||||
return "TRACE"
|
||||
case logsv1.Severity_SEVERITY_DEBUG:
|
||||
return "DEBUG"
|
||||
case logsv1.Severity_SEVERITY_INFO:
|
||||
return "INFO"
|
||||
case logsv1.Severity_SEVERITY_WARN:
|
||||
return "WARN"
|
||||
case logsv1.Severity_SEVERITY_ERROR:
|
||||
return "ERROR"
|
||||
case logsv1.Severity_SEVERITY_FATAL:
|
||||
return "FATAL"
|
||||
default:
|
||||
return "UNSPECIFIED"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package normalize
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
||||
rec := &logsv1.LogRecord{
|
||||
TimestampUnixNano: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).UnixNano(),
|
||||
Host: "host-1",
|
||||
Service: "svc-a",
|
||||
Severity: logsv1.Severity_SEVERITY_ERROR,
|
||||
Message: "boom",
|
||||
Attributes: map[string]string{"k": "v"},
|
||||
}
|
||||
|
||||
row := ToRow(rec)
|
||||
|
||||
if row.Host != "host-1" || row.Service != "svc-a" || row.Message != "boom" {
|
||||
t.Fatalf("unexpected row: %+v", row)
|
||||
}
|
||||
if row.Severity != "ERROR" {
|
||||
t.Fatalf("expected severity ERROR, got %s", row.Severity)
|
||||
}
|
||||
if row.Attributes["k"] != "v" {
|
||||
t.Fatalf("expected attribute k=v, got %+v", row.Attributes)
|
||||
}
|
||||
if !row.Timestamp.Equal(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) {
|
||||
t.Fatalf("unexpected timestamp: %v", row.Timestamp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestToRowNilAttributesBecomesEmptyMap(t *testing.T) {
|
||||
rec := &logsv1.LogRecord{Host: "h", Service: "s", Message: "m"}
|
||||
row := ToRow(rec)
|
||||
if row.Attributes == nil {
|
||||
t.Fatal("expected non-nil empty map, got nil")
|
||||
}
|
||||
if len(row.Attributes) != 0 {
|
||||
t.Fatalf("expected empty map, got %+v", row.Attributes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeverityTextCoversAllEnumValues(t *testing.T) {
|
||||
cases := map[logsv1.Severity]string{
|
||||
logsv1.Severity_SEVERITY_UNSPECIFIED: "UNSPECIFIED",
|
||||
logsv1.Severity_SEVERITY_TRACE: "TRACE",
|
||||
logsv1.Severity_SEVERITY_DEBUG: "DEBUG",
|
||||
logsv1.Severity_SEVERITY_INFO: "INFO",
|
||||
logsv1.Severity_SEVERITY_WARN: "WARN",
|
||||
logsv1.Severity_SEVERITY_ERROR: "ERROR",
|
||||
logsv1.Severity_SEVERITY_FATAL: "FATAL",
|
||||
}
|
||||
for sev, want := range cases {
|
||||
if got := severityText(sev); got != want {
|
||||
t.Errorf("severityText(%v) = %q, want %q", sev, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeverityTextUnknownValueFallsBackToUnspecified(t *testing.T) {
|
||||
if got := severityText(logsv1.Severity(99)); got != "UNSPECIFIED" {
|
||||
t.Fatalf("expected UNSPECIFIED for unknown severity, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
// Package producer wraps the Redpanda (Kafka API) producer used by the
|
||||
// gRPC front end to forward agent-submitted batches onto the transport
|
||||
// layer, unchanged. OTel-log-shape normalization happens later, on the
|
||||
// consumer side — see internal/normalize.
|
||||
package producer
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
)
|
||||
|
||||
type Producer struct {
|
||||
writer *kafka.Writer
|
||||
}
|
||||
|
||||
func New(cfg config.RedpandaConfig) *Producer {
|
||||
return &Producer{
|
||||
writer: &kafka.Writer{
|
||||
Addr: kafka.TCP(cfg.Brokers...),
|
||||
Topic: cfg.Topic,
|
||||
// Partition by host so a single host's records stay in
|
||||
// relative order within a partition.
|
||||
Balancer: &kafka.Hash{},
|
||||
RequiredAcks: kafka.RequireOne,
|
||||
AllowAutoTopicCreation: false, // topics are provisioned explicitly, see /transport
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Producer) Close() error {
|
||||
return p.writer.Close()
|
||||
}
|
||||
|
||||
// WriteBatch writes all messages in one call. kafka-go's WriteMessages
|
||||
// either succeeds for the whole batch or returns an error, which matches
|
||||
// the PushBatch RPC's all-or-nothing contract for Phase 0.
|
||||
func (p *Producer) WriteBatch(ctx context.Context, msgs []kafka.Message) error {
|
||||
return p.writer.WriteMessages(ctx, msgs...)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
# proto
|
||||
|
||||
Shared `.proto` contracts. Source of truth for the agent↔ingest gRPC
|
||||
service; each language generates its own bindings from these files rather
|
||||
than sharing generated code across languages.
|
||||
|
||||
- `sentry/logs/v1/logs.proto` — `LogIngest.PushBatch`, the only RPC an
|
||||
agent ever calls.
|
||||
|
||||
## Go bindings
|
||||
|
||||
Go is the one language here with pre-generated, checked-in bindings
|
||||
(`sentry/logs/v1/logs.pb.go`, `logs_grpc.pb.go`), living in this directory
|
||||
as its own module (`github.com/sentry/sentry/proto`) that `/ingest` and
|
||||
`/api` depend on via a local `replace` directive in their `go.mod`. Rust
|
||||
(`/agent`) instead generates its bindings at build time via `tonic-build`
|
||||
(see `agent/sentry-agent/build.rs`) — no checked-in Rust output.
|
||||
|
||||
To regenerate the Go bindings after changing `logs.proto`:
|
||||
|
||||
```sh
|
||||
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
|
||||
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
|
||||
|
||||
cd proto
|
||||
protoc --go_out=. --go_opt=paths=source_relative \
|
||||
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
|
||||
sentry/logs/v1/logs.proto
|
||||
go build ./...
|
||||
```
|
||||
@@ -0,0 +1,15 @@
|
||||
module github.com/sentry/sentry/proto
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
google.golang.org/grpc v1.83.0
|
||||
google.golang.org/protobuf v1.36.12
|
||||
)
|
||||
|
||||
require (
|
||||
golang.org/x/net v0.55.0 // indirect
|
||||
golang.org/x/sys v0.45.0 // indirect
|
||||
golang.org/x/text v0.37.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||
)
|
||||
@@ -0,0 +1,38 @@
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
|
||||
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
@@ -0,0 +1,376 @@
|
||||
// Code generated by protoc-gen-go. DO NOT EDIT.
|
||||
// versions:
|
||||
// protoc-gen-go v1.36.12
|
||||
// protoc v7.35.1
|
||||
// source: sentry/logs/v1/logs.proto
|
||||
|
||||
package logsv1
|
||||
|
||||
import (
|
||||
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
|
||||
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
|
||||
reflect "reflect"
|
||||
sync "sync"
|
||||
unsafe "unsafe"
|
||||
)
|
||||
|
||||
const (
|
||||
// Verify that this generated code is sufficiently up-to-date.
|
||||
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
|
||||
// Verify that runtime/protoimpl is sufficiently up-to-date.
|
||||
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
|
||||
)
|
||||
|
||||
// Severity follows OTel's severity number ranges (1-24), collapsed here to
|
||||
// the coarse names agents actually need to set. Numeric value stored
|
||||
// downstream may be a full OTel severity_number computed by ingest.
|
||||
type Severity int32
|
||||
|
||||
const (
|
||||
Severity_SEVERITY_UNSPECIFIED Severity = 0
|
||||
Severity_SEVERITY_TRACE Severity = 1
|
||||
Severity_SEVERITY_DEBUG Severity = 2
|
||||
Severity_SEVERITY_INFO Severity = 3
|
||||
Severity_SEVERITY_WARN Severity = 4
|
||||
Severity_SEVERITY_ERROR Severity = 5
|
||||
Severity_SEVERITY_FATAL Severity = 6
|
||||
)
|
||||
|
||||
// Enum value maps for Severity.
|
||||
var (
|
||||
Severity_name = map[int32]string{
|
||||
0: "SEVERITY_UNSPECIFIED",
|
||||
1: "SEVERITY_TRACE",
|
||||
2: "SEVERITY_DEBUG",
|
||||
3: "SEVERITY_INFO",
|
||||
4: "SEVERITY_WARN",
|
||||
5: "SEVERITY_ERROR",
|
||||
6: "SEVERITY_FATAL",
|
||||
}
|
||||
Severity_value = map[string]int32{
|
||||
"SEVERITY_UNSPECIFIED": 0,
|
||||
"SEVERITY_TRACE": 1,
|
||||
"SEVERITY_DEBUG": 2,
|
||||
"SEVERITY_INFO": 3,
|
||||
"SEVERITY_WARN": 4,
|
||||
"SEVERITY_ERROR": 5,
|
||||
"SEVERITY_FATAL": 6,
|
||||
}
|
||||
)
|
||||
|
||||
func (x Severity) Enum() *Severity {
|
||||
p := new(Severity)
|
||||
*p = x
|
||||
return p
|
||||
}
|
||||
|
||||
func (x Severity) String() string {
|
||||
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
|
||||
}
|
||||
|
||||
func (Severity) Descriptor() protoreflect.EnumDescriptor {
|
||||
return file_sentry_logs_v1_logs_proto_enumTypes[0].Descriptor()
|
||||
}
|
||||
|
||||
func (Severity) Type() protoreflect.EnumType {
|
||||
return &file_sentry_logs_v1_logs_proto_enumTypes[0]
|
||||
}
|
||||
|
||||
func (x Severity) Number() protoreflect.EnumNumber {
|
||||
return protoreflect.EnumNumber(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use Severity.Descriptor instead.
|
||||
func (Severity) EnumDescriptor() ([]byte, []int) {
|
||||
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{0}
|
||||
}
|
||||
|
||||
type LogRecord struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// Unix epoch nanoseconds, set by the agent at time of read (not parse or
|
||||
// send time) to preserve original ordering as closely as possible.
|
||||
TimestampUnixNano int64 `protobuf:"varint,1,opt,name=timestamp_unix_nano,json=timestampUnixNano,proto3" json:"timestamp_unix_nano,omitempty"`
|
||||
// Hostname the agent is running on. Agent fills this from its own config
|
||||
// or system hostname; not trusted as an identity claim (mTLS client cert
|
||||
// is the identity boundary).
|
||||
Host string `protobuf:"bytes,2,opt,name=host,proto3" json:"host,omitempty"`
|
||||
// Logical service/unit name. For journald sources, this is typically the
|
||||
// systemd unit name; for file sources, it comes from agent config.
|
||||
Service string `protobuf:"bytes,3,opt,name=service,proto3" json:"service,omitempty"`
|
||||
Severity Severity `protobuf:"varint,4,opt,name=severity,proto3,enum=sentry.logs.v1.Severity" json:"severity,omitempty"`
|
||||
// Original, unparsed log line. Always populated, even when structured
|
||||
// fields below are also present, per the schema-on-read fallback
|
||||
// requirement in CLAUDE.md.
|
||||
Message string `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
|
||||
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
||||
// syslog header fields). Empty when the raw-passthrough fallback fires.
|
||||
Attributes map[string]string `protobuf:"bytes,6,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *LogRecord) Reset() {
|
||||
*x = LogRecord{}
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[0]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *LogRecord) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*LogRecord) ProtoMessage() {}
|
||||
|
||||
func (x *LogRecord) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[0]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use LogRecord.ProtoReflect.Descriptor instead.
|
||||
func (*LogRecord) Descriptor() ([]byte, []int) {
|
||||
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{0}
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetTimestampUnixNano() int64 {
|
||||
if x != nil {
|
||||
return x.TimestampUnixNano
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetHost() string {
|
||||
if x != nil {
|
||||
return x.Host
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetService() string {
|
||||
if x != nil {
|
||||
return x.Service
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetSeverity() Severity {
|
||||
if x != nil {
|
||||
return x.Severity
|
||||
}
|
||||
return Severity_SEVERITY_UNSPECIFIED
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetMessage() string {
|
||||
if x != nil {
|
||||
return x.Message
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *LogRecord) GetAttributes() map[string]string {
|
||||
if x != nil {
|
||||
return x.Attributes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type PushBatchRequest struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
|
||||
// use this to avoid double-writing a batch if a retry follows a
|
||||
// timeout on an actually-successful push.
|
||||
BatchId string `protobuf:"bytes,1,opt,name=batch_id,json=batchId,proto3" json:"batch_id,omitempty"`
|
||||
Records []*LogRecord `protobuf:"bytes,2,rep,name=records,proto3" json:"records,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *PushBatchRequest) Reset() {
|
||||
*x = PushBatchRequest{}
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[1]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *PushBatchRequest) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*PushBatchRequest) ProtoMessage() {}
|
||||
|
||||
func (x *PushBatchRequest) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[1]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use PushBatchRequest.ProtoReflect.Descriptor instead.
|
||||
func (*PushBatchRequest) Descriptor() ([]byte, []int) {
|
||||
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{1}
|
||||
}
|
||||
|
||||
func (x *PushBatchRequest) GetBatchId() string {
|
||||
if x != nil {
|
||||
return x.BatchId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *PushBatchRequest) GetRecords() []*LogRecord {
|
||||
if x != nil {
|
||||
return x.Records
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type PushBatchResponse struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// Number of records ingest accepted. Phase 0: batches are all-or-nothing,
|
||||
// so this equals len(records) on success. Partial-acceptance semantics
|
||||
// are not implemented yet.
|
||||
Accepted uint32 `protobuf:"varint,1,opt,name=accepted,proto3" json:"accepted,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *PushBatchResponse) Reset() {
|
||||
*x = PushBatchResponse{}
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[2]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *PushBatchResponse) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*PushBatchResponse) ProtoMessage() {}
|
||||
|
||||
func (x *PushBatchResponse) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_sentry_logs_v1_logs_proto_msgTypes[2]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use PushBatchResponse.ProtoReflect.Descriptor instead.
|
||||
func (*PushBatchResponse) Descriptor() ([]byte, []int) {
|
||||
return file_sentry_logs_v1_logs_proto_rawDescGZIP(), []int{2}
|
||||
}
|
||||
|
||||
func (x *PushBatchResponse) GetAccepted() uint32 {
|
||||
if x != nil {
|
||||
return x.Accepted
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
var File_sentry_logs_v1_logs_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_sentry_logs_v1_logs_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x19sentry/logs/v1/logs.proto\x12\x0esentry.logs.v1\"\xc3\x02\n" +
|
||||
"\tLogRecord\x12.\n" +
|
||||
"\x13timestamp_unix_nano\x18\x01 \x01(\x03R\x11timestampUnixNano\x12\x12\n" +
|
||||
"\x04host\x18\x02 \x01(\tR\x04host\x12\x18\n" +
|
||||
"\aservice\x18\x03 \x01(\tR\aservice\x124\n" +
|
||||
"\bseverity\x18\x04 \x01(\x0e2\x18.sentry.logs.v1.SeverityR\bseverity\x12\x18\n" +
|
||||
"\amessage\x18\x05 \x01(\tR\amessage\x12I\n" +
|
||||
"\n" +
|
||||
"attributes\x18\x06 \x03(\v2).sentry.logs.v1.LogRecord.AttributesEntryR\n" +
|
||||
"attributes\x1a=\n" +
|
||||
"\x0fAttributesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"b\n" +
|
||||
"\x10PushBatchRequest\x12\x19\n" +
|
||||
"\bbatch_id\x18\x01 \x01(\tR\abatchId\x123\n" +
|
||||
"\arecords\x18\x02 \x03(\v2\x19.sentry.logs.v1.LogRecordR\arecords\"/\n" +
|
||||
"\x11PushBatchResponse\x12\x1a\n" +
|
||||
"\baccepted\x18\x01 \x01(\rR\baccepted*\x9a\x01\n" +
|
||||
"\bSeverity\x12\x18\n" +
|
||||
"\x14SEVERITY_UNSPECIFIED\x10\x00\x12\x12\n" +
|
||||
"\x0eSEVERITY_TRACE\x10\x01\x12\x12\n" +
|
||||
"\x0eSEVERITY_DEBUG\x10\x02\x12\x11\n" +
|
||||
"\rSEVERITY_INFO\x10\x03\x12\x11\n" +
|
||||
"\rSEVERITY_WARN\x10\x04\x12\x12\n" +
|
||||
"\x0eSEVERITY_ERROR\x10\x05\x12\x12\n" +
|
||||
"\x0eSEVERITY_FATAL\x10\x062]\n" +
|
||||
"\tLogIngest\x12P\n" +
|
||||
"\tPushBatch\x12 .sentry.logs.v1.PushBatchRequest\x1a!.sentry.logs.v1.PushBatchResponseB6Z4github.com/sentry/sentry/proto/sentry/logs/v1;logsv1b\x06proto3"
|
||||
|
||||
var (
|
||||
file_sentry_logs_v1_logs_proto_rawDescOnce sync.Once
|
||||
file_sentry_logs_v1_logs_proto_rawDescData []byte
|
||||
)
|
||||
|
||||
func file_sentry_logs_v1_logs_proto_rawDescGZIP() []byte {
|
||||
file_sentry_logs_v1_logs_proto_rawDescOnce.Do(func() {
|
||||
file_sentry_logs_v1_logs_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_sentry_logs_v1_logs_proto_rawDesc), len(file_sentry_logs_v1_logs_proto_rawDesc)))
|
||||
})
|
||||
return file_sentry_logs_v1_logs_proto_rawDescData
|
||||
}
|
||||
|
||||
var file_sentry_logs_v1_logs_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_sentry_logs_v1_logs_proto_msgTypes = make([]protoimpl.MessageInfo, 4)
|
||||
var file_sentry_logs_v1_logs_proto_goTypes = []any{
|
||||
(Severity)(0), // 0: sentry.logs.v1.Severity
|
||||
(*LogRecord)(nil), // 1: sentry.logs.v1.LogRecord
|
||||
(*PushBatchRequest)(nil), // 2: sentry.logs.v1.PushBatchRequest
|
||||
(*PushBatchResponse)(nil), // 3: sentry.logs.v1.PushBatchResponse
|
||||
nil, // 4: sentry.logs.v1.LogRecord.AttributesEntry
|
||||
}
|
||||
var file_sentry_logs_v1_logs_proto_depIdxs = []int32{
|
||||
0, // 0: sentry.logs.v1.LogRecord.severity:type_name -> sentry.logs.v1.Severity
|
||||
4, // 1: sentry.logs.v1.LogRecord.attributes:type_name -> sentry.logs.v1.LogRecord.AttributesEntry
|
||||
1, // 2: sentry.logs.v1.PushBatchRequest.records:type_name -> sentry.logs.v1.LogRecord
|
||||
2, // 3: sentry.logs.v1.LogIngest.PushBatch:input_type -> sentry.logs.v1.PushBatchRequest
|
||||
3, // 4: sentry.logs.v1.LogIngest.PushBatch:output_type -> sentry.logs.v1.PushBatchResponse
|
||||
4, // [4:5] is the sub-list for method output_type
|
||||
3, // [3:4] is the sub-list for method input_type
|
||||
3, // [3:3] is the sub-list for extension type_name
|
||||
3, // [3:3] is the sub-list for extension extendee
|
||||
0, // [0:3] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_sentry_logs_v1_logs_proto_init() }
|
||||
func file_sentry_logs_v1_logs_proto_init() {
|
||||
if File_sentry_logs_v1_logs_proto != nil {
|
||||
return
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
File: protoimpl.DescBuilder{
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_sentry_logs_v1_logs_proto_rawDesc), len(file_sentry_logs_v1_logs_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 4,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
GoTypes: file_sentry_logs_v1_logs_proto_goTypes,
|
||||
DependencyIndexes: file_sentry_logs_v1_logs_proto_depIdxs,
|
||||
EnumInfos: file_sentry_logs_v1_logs_proto_enumTypes,
|
||||
MessageInfos: file_sentry_logs_v1_logs_proto_msgTypes,
|
||||
}.Build()
|
||||
File_sentry_logs_v1_logs_proto = out.File
|
||||
file_sentry_logs_v1_logs_proto_goTypes = nil
|
||||
file_sentry_logs_v1_logs_proto_depIdxs = nil
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
syntax = "proto3";
|
||||
|
||||
package sentry.logs.v1;
|
||||
|
||||
option go_package = "github.com/sentry/sentry/proto/sentry/logs/v1;logsv1";
|
||||
|
||||
// LogIngest is the service agents use to ship batched log records to the
|
||||
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
|
||||
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
|
||||
// once backpressure/flow-control behavior is characterized.
|
||||
service LogIngest {
|
||||
rpc PushBatch(PushBatchRequest) returns (PushBatchResponse);
|
||||
}
|
||||
|
||||
// Severity follows OTel's severity number ranges (1-24), collapsed here to
|
||||
// the coarse names agents actually need to set. Numeric value stored
|
||||
// downstream may be a full OTel severity_number computed by ingest.
|
||||
enum Severity {
|
||||
SEVERITY_UNSPECIFIED = 0;
|
||||
SEVERITY_TRACE = 1;
|
||||
SEVERITY_DEBUG = 2;
|
||||
SEVERITY_INFO = 3;
|
||||
SEVERITY_WARN = 4;
|
||||
SEVERITY_ERROR = 5;
|
||||
SEVERITY_FATAL = 6;
|
||||
}
|
||||
|
||||
message LogRecord {
|
||||
// Unix epoch nanoseconds, set by the agent at time of read (not parse or
|
||||
// send time) to preserve original ordering as closely as possible.
|
||||
int64 timestamp_unix_nano = 1;
|
||||
|
||||
// Hostname the agent is running on. Agent fills this from its own config
|
||||
// or system hostname; not trusted as an identity claim (mTLS client cert
|
||||
// is the identity boundary).
|
||||
string host = 2;
|
||||
|
||||
// Logical service/unit name. For journald sources, this is typically the
|
||||
// systemd unit name; for file sources, it comes from agent config.
|
||||
string service = 3;
|
||||
|
||||
Severity severity = 4;
|
||||
|
||||
// Original, unparsed log line. Always populated, even when structured
|
||||
// fields below are also present, per the schema-on-read fallback
|
||||
// requirement in CLAUDE.md.
|
||||
string message = 5;
|
||||
|
||||
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
||||
// syslog header fields). Empty when the raw-passthrough fallback fires.
|
||||
map<string, string> attributes = 6;
|
||||
}
|
||||
|
||||
message PushBatchRequest {
|
||||
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
|
||||
// use this to avoid double-writing a batch if a retry follows a
|
||||
// timeout on an actually-successful push.
|
||||
string batch_id = 1;
|
||||
|
||||
repeated LogRecord records = 2;
|
||||
}
|
||||
|
||||
message PushBatchResponse {
|
||||
// Number of records ingest accepted. Phase 0: batches are all-or-nothing,
|
||||
// so this equals len(records) on success. Partial-acceptance semantics
|
||||
// are not implemented yet.
|
||||
uint32 accepted = 1;
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
|
||||
// versions:
|
||||
// - protoc-gen-go-grpc v1.6.2
|
||||
// - protoc v7.35.1
|
||||
// source: sentry/logs/v1/logs.proto
|
||||
|
||||
package logsv1
|
||||
|
||||
import (
|
||||
context "context"
|
||||
grpc "google.golang.org/grpc"
|
||||
codes "google.golang.org/grpc/codes"
|
||||
status "google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// This is a compile-time assertion to ensure that this generated file
|
||||
// is compatible with the grpc package it is being compiled against.
|
||||
// Requires gRPC-Go v1.64.0 or later.
|
||||
const _ = grpc.SupportPackageIsVersion9
|
||||
|
||||
const (
|
||||
LogIngest_PushBatch_FullMethodName = "/sentry.logs.v1.LogIngest/PushBatch"
|
||||
)
|
||||
|
||||
// LogIngestClient is the client API for LogIngest service.
|
||||
//
|
||||
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
|
||||
//
|
||||
// LogIngest is the service agents use to ship batched log records to the
|
||||
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
|
||||
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
|
||||
// once backpressure/flow-control behavior is characterized.
|
||||
type LogIngestClient interface {
|
||||
PushBatch(ctx context.Context, in *PushBatchRequest, opts ...grpc.CallOption) (*PushBatchResponse, error)
|
||||
}
|
||||
|
||||
type logIngestClient struct {
|
||||
cc grpc.ClientConnInterface
|
||||
}
|
||||
|
||||
func NewLogIngestClient(cc grpc.ClientConnInterface) LogIngestClient {
|
||||
return &logIngestClient{cc}
|
||||
}
|
||||
|
||||
func (c *logIngestClient) PushBatch(ctx context.Context, in *PushBatchRequest, opts ...grpc.CallOption) (*PushBatchResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(PushBatchResponse)
|
||||
err := c.cc.Invoke(ctx, LogIngest_PushBatch_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// LogIngestServer is the server API for LogIngest service.
|
||||
// All implementations must embed UnimplementedLogIngestServer
|
||||
// for forward compatibility.
|
||||
//
|
||||
// LogIngest is the service agents use to ship batched log records to the
|
||||
// ingest service over mTLS. Phase 0: single unary batch push. Streaming
|
||||
// (client-streaming for continuous shipping) is a likely Phase 1 upgrade
|
||||
// once backpressure/flow-control behavior is characterized.
|
||||
type LogIngestServer interface {
|
||||
PushBatch(context.Context, *PushBatchRequest) (*PushBatchResponse, error)
|
||||
mustEmbedUnimplementedLogIngestServer()
|
||||
}
|
||||
|
||||
// UnimplementedLogIngestServer must be embedded to have
|
||||
// forward compatible implementations.
|
||||
//
|
||||
// NOTE: this should be embedded by value instead of pointer to avoid a nil
|
||||
// pointer dereference when methods are called.
|
||||
type UnimplementedLogIngestServer struct{}
|
||||
|
||||
func (UnimplementedLogIngestServer) PushBatch(context.Context, *PushBatchRequest) (*PushBatchResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method PushBatch not implemented")
|
||||
}
|
||||
func (UnimplementedLogIngestServer) mustEmbedUnimplementedLogIngestServer() {}
|
||||
func (UnimplementedLogIngestServer) testEmbeddedByValue() {}
|
||||
|
||||
// UnsafeLogIngestServer may be embedded to opt out of forward compatibility for this service.
|
||||
// Use of this interface is not recommended, as added methods to LogIngestServer will
|
||||
// result in compilation errors.
|
||||
type UnsafeLogIngestServer interface {
|
||||
mustEmbedUnimplementedLogIngestServer()
|
||||
}
|
||||
|
||||
func RegisterLogIngestServer(s grpc.ServiceRegistrar, srv LogIngestServer) {
|
||||
// If the following call panics, it indicates UnimplementedLogIngestServer was
|
||||
// embedded by pointer and is nil. This will cause panics if an
|
||||
// unimplemented method is ever invoked, so we test this at initialization
|
||||
// time to prevent it from happening at runtime later due to I/O.
|
||||
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
|
||||
t.testEmbeddedByValue()
|
||||
}
|
||||
s.RegisterService(&LogIngest_ServiceDesc, srv)
|
||||
}
|
||||
|
||||
func _LogIngest_PushBatch_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(PushBatchRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(LogIngestServer).PushBatch(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: LogIngest_PushBatch_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(LogIngestServer).PushBatch(ctx, req.(*PushBatchRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
// LogIngest_ServiceDesc is the grpc.ServiceDesc for LogIngest service.
|
||||
// It's only intended for direct use with grpc.RegisterService,
|
||||
// and not to be introspected or modified (even as a copy)
|
||||
var LogIngest_ServiceDesc = grpc.ServiceDesc{
|
||||
ServiceName: "sentry.logs.v1.LogIngest",
|
||||
HandlerType: (*LogIngestServer)(nil),
|
||||
Methods: []grpc.MethodDesc{
|
||||
{
|
||||
MethodName: "PushBatch",
|
||||
Handler: _LogIngest_PushBatch_Handler,
|
||||
},
|
||||
},
|
||||
Streams: []grpc.StreamDesc{},
|
||||
Metadata: "sentry/logs/v1/logs.proto",
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
# One-shot migration runner: bash + curl baked in, migrations/*.sql copied
|
||||
# in at build time. No runtime package install and no host volume mount
|
||||
# needed — works offline once built.
|
||||
# docker build -f storage/Dockerfile -t sentry-clickhouse-migrate storage/
|
||||
FROM alpine:3.20
|
||||
RUN apk add --no-cache bash curl
|
||||
WORKDIR /storage
|
||||
COPY migrate.sh ./
|
||||
COPY migrations ./migrations
|
||||
ENTRYPOINT ["bash", "migrate.sh"]
|
||||
@@ -0,0 +1,84 @@
|
||||
# storage
|
||||
|
||||
ClickHouse schema and migration tooling for Sentry's analytical store.
|
||||
|
||||
## Schema
|
||||
|
||||
One table for Phase 0, `logs`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE logs
|
||||
(
|
||||
`timestamp` DateTime64(9, 'UTC'),
|
||||
`host` String,
|
||||
`service` String,
|
||||
`severity` LowCardinality(String),
|
||||
`message` String,
|
||||
`attributes` Map(String, String)
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toDate(timestamp)
|
||||
ORDER BY (service, timestamp)
|
||||
```
|
||||
|
||||
Notes on choices that weren't fully specified by the task description:
|
||||
|
||||
- **`DateTime64(9, 'UTC')`** (nanosecond precision) rather than second or
|
||||
millisecond precision, to match the agent's `timestamp_unix_nano` field
|
||||
end to end without truncation.
|
||||
- **`severity` as `LowCardinality(String)`**, not a numeric OTel
|
||||
`SeverityNumber`. `/ingest`'s `normalize` package writes short text
|
||||
values (`TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR`/`FATAL`/`UNSPECIFIED`).
|
||||
`LowCardinality` gets you most of the storage/query efficiency of an enum
|
||||
without committing to one at the schema level. Splitting into a proper
|
||||
`SeverityNumber` + `SeverityText` pair (full OTel shape) is one of the
|
||||
open questions already flagged in `/docs/architecture.md`.
|
||||
- **`PARTITION BY toDate(timestamp)`** (daily partitions) and
|
||||
**`ORDER BY (service, timestamp)`** are exactly what the task asked for
|
||||
— service-scoped queries over a time range are the dominant access
|
||||
pattern this is optimized for.
|
||||
- No TTL/retention clause yet — also an open question in architecture.md,
|
||||
deferred until storage sizing is a real concern.
|
||||
|
||||
## Migration tooling: a plain SQL-file runner, not golang-migrate
|
||||
|
||||
`migrate.sh` applies `migrations/*.sql` in filename order over
|
||||
ClickHouse's HTTP interface, tracking what's applied in a
|
||||
`schema_migrations` table. Chosen over `golang-migrate` for Phase 0
|
||||
because there's exactly one migration to run — pulling in a migration
|
||||
framework (another dependency, another thing to configure/vendor) for a
|
||||
single `CREATE TABLE` is exactly the kind of premature machinery this
|
||||
project's conventions say to avoid. Revisit `golang-migrate` once there's
|
||||
real schema churn across environments (rollback support, checksums,
|
||||
concurrent-apply safety become worth their cost at that point, not before).
|
||||
|
||||
**Convention:** one DDL statement per migration file. The ClickHouse HTTP
|
||||
interface isn't reliably multi-statement, so `migrate.sh` doesn't try to
|
||||
split multi-statement files — keep each migration to a single statement.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
docker compose up -d # starts a standalone ClickHouse for local work
|
||||
./migrate.sh # applies migrations/*.sql
|
||||
```
|
||||
|
||||
Environment variables `migrate.sh` reads (all optional, matching
|
||||
`/ingest`'s ClickHouse defaults so the two stay in sync out of the box):
|
||||
|
||||
| Var | Default |
|
||||
|---|---|
|
||||
| `CLICKHOUSE_HTTP` | `http://localhost:8123` |
|
||||
| `CLICKHOUSE_USER` | `default` |
|
||||
| `CLICKHOUSE_PASSWORD` | (empty) |
|
||||
| `CLICKHOUSE_DATABASE` | `sentry` |
|
||||
|
||||
There's also a `Dockerfile` (bash + curl baked in, `migrations/` copied in
|
||||
at build time) used by the root-level `docker-compose.yml` as a one-shot
|
||||
init service — no runtime package install, no host volume mount needed.
|
||||
|
||||
## Adding a migration
|
||||
|
||||
Add `migrations/000N_description.sql` with the next sequential number and
|
||||
a single DDL statement. `migrate.sh` picks it up automatically — no
|
||||
registration step.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Standalone ClickHouse for local development against /storage in
|
||||
# isolation (e.g. iterating on migrations). The root-level docker-compose.yml
|
||||
# runs the full Phase 0 stack and defines its own clickhouse service
|
||||
# separately — this file is not included by it.
|
||||
services:
|
||||
clickhouse:
|
||||
image: clickhouse/clickhouse-server:24.8
|
||||
container_name: sentry-clickhouse
|
||||
ports:
|
||||
- "8123:8123" # HTTP interface, used by migrate.sh
|
||||
- "9000:9000" # native protocol, used by ingest
|
||||
volumes:
|
||||
- clickhouse-data:/var/lib/clickhouse
|
||||
ulimits:
|
||||
nofile:
|
||||
soft: 262144
|
||||
hard: 262144
|
||||
|
||||
volumes:
|
||||
clickhouse-data:
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Applies migrations/*.sql to ClickHouse in filename order, tracking what's
|
||||
# already been applied in a schema_migrations table. Talks to ClickHouse's
|
||||
# HTTP interface via curl rather than requiring the clickhouse-client
|
||||
# binary — nothing to install beyond curl, works identically on a dev
|
||||
# laptop or in CI.
|
||||
#
|
||||
# Convention: exactly one DDL statement per migration file. The ClickHouse
|
||||
# HTTP interface isn't reliably multi-statement, so keeping migrations to
|
||||
# one statement each avoids relying on that.
|
||||
set -euo pipefail
|
||||
|
||||
CLICKHOUSE_HTTP="${CLICKHOUSE_HTTP:-http://localhost:8123}"
|
||||
CLICKHOUSE_USER="${CLICKHOUSE_USER:-default}"
|
||||
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}"
|
||||
DATABASE="${CLICKHOUSE_DATABASE:-sentry}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
MIGRATIONS_DIR="${SCRIPT_DIR}/migrations"
|
||||
|
||||
ch_exec() {
|
||||
# $1 = SQL statement, $2 = optional database to scope the query to.
|
||||
local sql="$1"
|
||||
local db="${2:-}"
|
||||
local url="${CLICKHOUSE_HTTP}/"
|
||||
if [[ -n "$db" ]]; then
|
||||
url="${CLICKHOUSE_HTTP}/?database=${db}"
|
||||
fi
|
||||
curl -sS -f -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" "$url" --data-binary "$sql"
|
||||
}
|
||||
|
||||
echo "Ensuring database '${DATABASE}' exists..."
|
||||
ch_exec "CREATE DATABASE IF NOT EXISTS ${DATABASE}"
|
||||
|
||||
echo "Ensuring schema_migrations table exists..."
|
||||
ch_exec "CREATE TABLE IF NOT EXISTS schema_migrations (version String, applied_at DateTime DEFAULT now()) ENGINE = MergeTree ORDER BY version" "$DATABASE"
|
||||
|
||||
applied="$(ch_exec "SELECT version FROM schema_migrations FORMAT TabSeparated" "$DATABASE")"
|
||||
|
||||
shopt -s nullglob
|
||||
for file in "${MIGRATIONS_DIR}"/*.sql; do
|
||||
version="$(basename "$file")"
|
||||
if grep -qx "$version" <<< "$applied"; then
|
||||
echo "skip ${version} (already applied)"
|
||||
continue
|
||||
fi
|
||||
echo "apply ${version}"
|
||||
ch_exec "$(cat "$file")" "$DATABASE" > /dev/null
|
||||
ch_exec "INSERT INTO schema_migrations (version) VALUES ('${version}')" "$DATABASE" > /dev/null
|
||||
done
|
||||
|
||||
echo "Migrations complete."
|
||||
@@ -0,0 +1,12 @@
|
||||
CREATE TABLE IF NOT EXISTS logs
|
||||
(
|
||||
`timestamp` DateTime64(9, 'UTC'),
|
||||
`host` String,
|
||||
`service` String,
|
||||
`severity` LowCardinality(String),
|
||||
`message` String,
|
||||
`attributes` Map(String, String)
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toDate(timestamp)
|
||||
ORDER BY (service, timestamp)
|
||||
@@ -0,0 +1,7 @@
|
||||
# Built FROM the Redpanda image so `rpk` is already present -- no need for
|
||||
# a separate client install, and no Docker socket access needed since
|
||||
# provisioning happens over the network, not via `docker exec`.
|
||||
# docker build -f transport/Dockerfile -t sentry-transport-provision transport/
|
||||
FROM docker.redpanda.com/redpandadata/redpanda:v24.2.7
|
||||
COPY provision-topics.sh /provision-topics.sh
|
||||
ENTRYPOINT ["/provision-topics.sh"]
|
||||
@@ -0,0 +1,26 @@
|
||||
# transport
|
||||
|
||||
Redpanda for local development, plus the script that provisions the topic
|
||||
`/ingest` depends on.
|
||||
|
||||
## Topic naming contract
|
||||
|
||||
`ingest` defaults to `REDPANDA_TOPIC=sentry.logs.raw` (see
|
||||
`/ingest/internal/config`). `provision-topics.sh` defaults to the same
|
||||
name. These aren't wired together automatically — if you change one,
|
||||
change the other, or override `REDPANDA_TOPIC` consistently wherever
|
||||
both are invoked.
|
||||
|
||||
## Running standalone
|
||||
|
||||
```sh
|
||||
docker compose up -d
|
||||
REDPANDA_BROKERS=localhost:9092 ./provision-topics.sh
|
||||
```
|
||||
|
||||
## In the full stack
|
||||
|
||||
The root-level `docker-compose.yml` builds this directory's `Dockerfile`
|
||||
(FROM the Redpanda image itself, so `rpk` is already present) as a
|
||||
one-shot init service that runs after Redpanda reports healthy. See
|
||||
`/docs/phase-0-runbook.md`.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Standalone Redpanda for local development against /transport in
|
||||
# isolation. The root-level docker-compose.yml runs the full Phase 0 stack
|
||||
# and defines its own redpanda service separately — this file is not
|
||||
# included by it.
|
||||
#
|
||||
# Note advertise-kafka-addr is "localhost" here (host tools connect via the
|
||||
# mapped port), vs "redpanda" in the root compose (other containers connect
|
||||
# via the compose network's service DNS name). Getting this wrong is the
|
||||
# classic Redpanda/Kafka docker-compose footgun — clients can connect
|
||||
# initially but then fail on the broker's advertised address once they try
|
||||
# to actually produce/consume.
|
||||
services:
|
||||
redpanda:
|
||||
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
|
||||
container_name: sentry-redpanda
|
||||
command:
|
||||
- redpanda
|
||||
- start
|
||||
- --smp=1
|
||||
- --memory=1G
|
||||
- --reserve-memory=0M
|
||||
- --overprovisioned
|
||||
- --node-id=0
|
||||
- --check=false
|
||||
- --kafka-addr=PLAINTEXT://0.0.0.0:9092
|
||||
- --advertise-kafka-addr=PLAINTEXT://localhost:9092
|
||||
ports:
|
||||
- "9092:9092"
|
||||
- "9644:9644" # admin API, used by rpk/healthchecks
|
||||
volumes:
|
||||
- redpanda-data:/var/lib/redpanda/data
|
||||
|
||||
volumes:
|
||||
redpanda-data:
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Idempotently creates the topic ingest produces/consumes. Talks to
|
||||
# Redpanda over the network via `rpk`, not `docker exec` into the broker
|
||||
# container -- this way the same script works whether it's run from the
|
||||
# host (against the standalone compose in this directory), from inside a
|
||||
# sibling container on the root compose's network, or in CI.
|
||||
set -euo pipefail
|
||||
|
||||
BROKERS="${REDPANDA_BROKERS:-localhost:9092}"
|
||||
TOPIC="${REDPANDA_TOPIC:-sentry.logs.raw}"
|
||||
PARTITIONS="${REDPANDA_TOPIC_PARTITIONS:-6}"
|
||||
|
||||
echo "Waiting for Redpanda at ${BROKERS}..."
|
||||
until rpk cluster health --brokers "${BROKERS}" --exit-when-healthy > /dev/null 2>&1; do
|
||||
sleep 1
|
||||
done
|
||||
|
||||
if rpk topic list --brokers "${BROKERS}" | awk 'NR>1{print $1}' | grep -qx "${TOPIC}"; then
|
||||
echo "Topic '${TOPIC}' already exists, skipping."
|
||||
else
|
||||
echo "Creating topic '${TOPIC}' (${PARTITIONS} partitions)..."
|
||||
rpk topic create "${TOPIC}" --brokers "${BROKERS}" --partitions "${PARTITIONS}" --replicas 1
|
||||
fi
|
||||
@@ -0,0 +1,4 @@
|
||||
# Base URL of the /api service. Baked into the static build at build time
|
||||
# (this is a prerendered SPA, not a server) — set this before `npm run
|
||||
# build` / `docker build`, not at container start.
|
||||
VITE_API_BASE_URL=http://localhost:8080
|
||||
@@ -0,0 +1,23 @@
|
||||
node_modules
|
||||
|
||||
# Output
|
||||
.output
|
||||
.vercel
|
||||
.netlify
|
||||
.wrangler
|
||||
/.svelte-kit
|
||||
/build
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Env
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
!.env.test
|
||||
|
||||
# Vite
|
||||
vite.config.js.timestamp-*
|
||||
vite.config.ts.timestamp-*
|
||||
@@ -0,0 +1 @@
|
||||
engine-strict=true
|
||||
Vendored
+3
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"recommendations": ["svelte.svelte-vscode"]
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
# Build context can be just web/ (unlike agent/ingest/api, this doesn't
|
||||
# need /proto):
|
||||
# docker build -f web/Dockerfile -t sentry-web web/
|
||||
|
||||
FROM node:22-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY package.json package-lock.json ./
|
||||
RUN npm ci
|
||||
COPY . .
|
||||
# VITE_API_BASE_URL is baked in at build time — this is a prerendered
|
||||
# static site, not a server. Override with --build-arg for non-default
|
||||
# deployments.
|
||||
ARG VITE_API_BASE_URL=http://localhost:8080
|
||||
ENV VITE_API_BASE_URL=${VITE_API_BASE_URL}
|
||||
RUN npm run build
|
||||
|
||||
# Not distroless: serving a static SPA needs *some* HTTP server, and
|
||||
# nginx:alpine is the boring, well-understood choice for that job — a
|
||||
# custom static-file-serving binary would be more engineering than a
|
||||
# Phase 0 placeholder page warrants. See /web/README.md.
|
||||
FROM nginx:alpine
|
||||
COPY --from=builder /src/build /usr/share/nginx/html
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
EXPOSE 3000
|
||||
@@ -0,0 +1,49 @@
|
||||
# web
|
||||
|
||||
SvelteKit frontend. Phase 0: one page, one query box, one table. No auth,
|
||||
no styling polish, no routing beyond `/`.
|
||||
|
||||
## What it does
|
||||
|
||||
Textarea for a raw SQL string → `POST {VITE_API_BASE_URL}/query` on `/api`
|
||||
→ renders `{columns, rows}` as an HTML table, or shows `{error}` from a
|
||||
rejected/failed query. That's the whole app — see `src/routes/+page.svelte`.
|
||||
|
||||
## Why a static build, not a Node server
|
||||
|
||||
Scaffolded with `@sveltejs/adapter-static`: this page has no server-side
|
||||
data loading (all data comes from a client-side `fetch` triggered by the
|
||||
submit button), so there's nothing here that needs a running SvelteKit
|
||||
server. A prerendered static site is simpler to build, deploy, and reason
|
||||
about than running Node in production for a page that's this thin.
|
||||
|
||||
Because it's static, `VITE_API_BASE_URL` is baked in at **build time**, not
|
||||
read at container start. Set it before `npm run build` (or pass
|
||||
`--build-arg VITE_API_BASE_URL=...` to `docker build`) — changing it later
|
||||
means rebuilding, not just restarting the container.
|
||||
|
||||
## Building & running
|
||||
|
||||
```sh
|
||||
npm install
|
||||
cp .env.example .env # adjust VITE_API_BASE_URL if /api isn't on localhost:8080
|
||||
npm run dev # local dev server with hot reload
|
||||
npm run check # svelte-check, type errors
|
||||
npm run build # static output to build/
|
||||
npm run preview # serve the static build locally to sanity-check it
|
||||
```
|
||||
|
||||
```sh
|
||||
docker build -f Dockerfile -t sentry-web . # context is web/, not the repo root
|
||||
docker run -p 3000:3000 sentry-web
|
||||
```
|
||||
|
||||
## Why nginx, not distroless
|
||||
|
||||
The repo convention prefers distroless/scratch base images. Serving a
|
||||
static SPA still needs *some* HTTP server, though, and `nginx:alpine` is
|
||||
the boring, standard choice for that job — writing a custom static-file
|
||||
binary just to stay distroless would be more engineering than a Phase 0
|
||||
placeholder page justifies. `nginx.conf` here is minimal: serve `build/`,
|
||||
fall back to `index.html` for client-side routing (only one route exists
|
||||
today, but this is what you want the moment a second one is added).
|
||||
@@ -0,0 +1,9 @@
|
||||
server {
|
||||
listen 3000;
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
}
|
||||
Generated
+1338
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "web",
|
||||
"private": true,
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview",
|
||||
"prepare": "svelte-kit sync || echo ''",
|
||||
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
|
||||
"check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@sveltejs/adapter-static": "^3.0.10",
|
||||
"@sveltejs/kit": "^2.63.0",
|
||||
"@sveltejs/vite-plugin-svelte": "^7.1.2",
|
||||
"svelte": "^5.56.1",
|
||||
"svelte-check": "^4.6.0",
|
||||
"typescript": "^6.0.3",
|
||||
"vite": "^8.0.16"
|
||||
}
|
||||
}
|
||||
Vendored
+13
@@ -0,0 +1,13 @@
|
||||
// See https://svelte.dev/docs/kit/types#app.d.ts
|
||||
// for information about these interfaces
|
||||
declare global {
|
||||
namespace App {
|
||||
// interface Error {}
|
||||
// interface Locals {}
|
||||
// interface PageData {}
|
||||
// interface PageState {}
|
||||
// interface Platform {}
|
||||
}
|
||||
}
|
||||
|
||||
export {};
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="text-scale" content="scale" />
|
||||
%sveltekit.head%
|
||||
</head>
|
||||
<body data-sveltekit-preload-data="hover">
|
||||
<div style="display: contents">%sveltekit.body%</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="107" height="128" viewBox="0 0 107 128"><title>svelte-logo</title><path d="M94.157 22.819c-10.4-14.885-30.94-19.297-45.792-9.835L22.282 29.608A29.92 29.92 0 0 0 8.764 49.65a31.5 31.5 0 0 0 3.108 20.231 30 30 0 0 0-4.477 11.183 31.9 31.9 0 0 0 5.448 24.116c10.402 14.887 30.942 19.297 45.791 9.835l26.083-16.624A29.92 29.92 0 0 0 98.235 78.35a31.53 31.53 0 0 0-3.105-20.232 30 30 0 0 0 4.474-11.182 31.88 31.88 0 0 0-5.447-24.116" style="fill:#ff3e00"/><path d="M45.817 106.582a20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.503 18 18 0 0 1 .624-2.435l.49-1.498 1.337.981a33.6 33.6 0 0 0 10.203 5.098l.97.294-.09.968a5.85 5.85 0 0 0 1.052 3.878 6.24 6.24 0 0 0 6.695 2.485 5.8 5.8 0 0 0 1.603-.704L69.27 76.28a5.43 5.43 0 0 0 2.45-3.631 5.8 5.8 0 0 0-.987-4.371 6.24 6.24 0 0 0-6.698-2.487 5.7 5.7 0 0 0-1.6.704l-9.953 6.345a19 19 0 0 1-5.296 2.326 20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.502 17.99 17.99 0 0 1 8.13-12.052l26.081-16.623a19 19 0 0 1 5.3-2.329 20.72 20.72 0 0 1 22.237 8.243 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-.624 2.435l-.49 1.498-1.337-.98a33.6 33.6 0 0 0-10.203-5.1l-.97-.294.09-.968a5.86 5.86 0 0 0-1.052-3.878 6.24 6.24 0 0 0-6.696-2.485 5.8 5.8 0 0 0-1.602.704L37.73 51.72a5.42 5.42 0 0 0-2.449 3.63 5.79 5.79 0 0 0 .986 4.372 6.24 6.24 0 0 0 6.698 2.486 5.8 5.8 0 0 0 1.602-.704l9.952-6.342a19 19 0 0 1 5.295-2.328 20.72 20.72 0 0 1 22.237 8.242 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-8.13 12.053l-26.081 16.622a19 19 0 0 1-5.3 2.328" style="fill:#fff"/></svg>
|
||||
|
After Width: | Height: | Size: 1.5 KiB |
@@ -0,0 +1 @@
|
||||
// place files you want to import through the `$lib` alias in this folder.
|
||||
@@ -0,0 +1,12 @@
|
||||
<script lang="ts">
|
||||
import favicon from '$lib/assets/favicon.svg';
|
||||
|
||||
let { children } = $props();
|
||||
</script>
|
||||
|
||||
<svelte:head>
|
||||
<title>Sentry</title>
|
||||
<link rel="icon" href={favicon} />
|
||||
</svelte:head>
|
||||
|
||||
{@render children()}
|
||||
@@ -0,0 +1,128 @@
|
||||
<script lang="ts">
|
||||
// Phase 0: functional only, no styling polish, no auth. One page: a raw
|
||||
// SQL box against POST /query on the api service, rendered as a table.
|
||||
// This is a placeholder for the real query UI that lands once /api grows
|
||||
// a real SPL-like query layer in Phase 2.
|
||||
|
||||
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
||||
|
||||
let sql = $state('SELECT * FROM logs ORDER BY timestamp DESC LIMIT 100');
|
||||
let columns = $state<string[]>([]);
|
||||
let rows = $state<unknown[][]>([]);
|
||||
let error = $state('');
|
||||
let loading = $state(false);
|
||||
let hasRun = $state(false);
|
||||
|
||||
function formatCell(value: unknown): string {
|
||||
if (value === null || value === undefined) return '';
|
||||
if (typeof value === 'object') return JSON.stringify(value);
|
||||
return String(value);
|
||||
}
|
||||
|
||||
async function runQuery() {
|
||||
loading = true;
|
||||
error = '';
|
||||
try {
|
||||
const res = await fetch(`${apiBase}/query`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ sql })
|
||||
});
|
||||
const body = await res.json();
|
||||
if (!res.ok) {
|
||||
error = body?.error ?? `request failed with status ${res.status}`;
|
||||
columns = [];
|
||||
rows = [];
|
||||
return;
|
||||
}
|
||||
columns = body.columns ?? [];
|
||||
rows = body.rows ?? [];
|
||||
} catch (e) {
|
||||
error = e instanceof Error ? e.message : String(e);
|
||||
columns = [];
|
||||
rows = [];
|
||||
} finally {
|
||||
loading = false;
|
||||
hasRun = true;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<main>
|
||||
<h1>Sentry — Log Query (Phase 0)</h1>
|
||||
<p>
|
||||
Raw SQL only, SELECT statements against the <code>logs</code> table. No auth, no query
|
||||
builder yet — see <code>/api</code> for what's actually allowed.
|
||||
</p>
|
||||
|
||||
<textarea bind:value={sql} rows="4" cols="100" spellcheck="false"></textarea>
|
||||
<div>
|
||||
<button onclick={runQuery} disabled={loading}>
|
||||
{loading ? 'Running…' : 'Run query'}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{#if error}
|
||||
<p class="error">Error: {error}</p>
|
||||
{/if}
|
||||
|
||||
{#if hasRun && !error}
|
||||
<p>{rows.length} row(s)</p>
|
||||
{/if}
|
||||
|
||||
{#if columns.length > 0}
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
{#each columns as col (col)}
|
||||
<th>{col}</th>
|
||||
{/each}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{#each rows as row, i (i)}
|
||||
<tr>
|
||||
{#each row as cell, j (j)}
|
||||
<td>{formatCell(cell)}</td>
|
||||
{/each}
|
||||
</tr>
|
||||
{/each}
|
||||
</tbody>
|
||||
</table>
|
||||
{/if}
|
||||
</main>
|
||||
|
||||
<style>
|
||||
main {
|
||||
font-family: system-ui, sans-serif;
|
||||
max-width: 960px;
|
||||
margin: 2rem auto;
|
||||
padding: 0 1rem;
|
||||
}
|
||||
textarea {
|
||||
width: 100%;
|
||||
font-family: monospace;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
button {
|
||||
margin-top: 0.5rem;
|
||||
}
|
||||
.error {
|
||||
color: #b00020;
|
||||
}
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
margin-top: 1rem;
|
||||
}
|
||||
th,
|
||||
td {
|
||||
border: 1px solid #ccc;
|
||||
padding: 0.25rem 0.5rem;
|
||||
text-align: left;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
th {
|
||||
background: #f0f0f0;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,4 @@
|
||||
// Static adapter needs every route prerenderable. This page has no load
|
||||
// function (all data comes from a client-side fetch on submit), so a plain
|
||||
// prerender is enough — no need to disable SSR.
|
||||
export const prerender = true;
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
/// <reference types="vite/client" />
|
||||
@@ -0,0 +1,3 @@
|
||||
# allow crawling everything by default
|
||||
User-agent: *
|
||||
Disallow:
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"extends": "./.svelte-kit/tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"rewriteRelativeImportExtensions": true,
|
||||
"allowJs": true,
|
||||
"checkJs": true,
|
||||
"esModuleInterop": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"skipLibCheck": true,
|
||||
"sourceMap": true,
|
||||
"strict": true,
|
||||
"moduleResolution": "bundler"
|
||||
}
|
||||
// Path aliases are handled by https://svelte.dev/docs/kit/configuration#alias
|
||||
// except $lib which is handled by https://svelte.dev/docs/kit/configuration#files
|
||||
//
|
||||
// To make changes to top-level options such as include and exclude, we recommend extending
|
||||
// the generated config; see https://svelte.dev/docs/kit/configuration#typescript
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import adapter from '@sveltejs/adapter-static';
|
||||
import { sveltekit } from '@sveltejs/kit/vite';
|
||||
import { defineConfig } from 'vite';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [
|
||||
sveltekit({
|
||||
compilerOptions: {
|
||||
// Force runes mode for the project, except for libraries. Can be removed in svelte 6.
|
||||
runes: ({ filename }) => filename.split(/[/\\]/).includes('node_modules') ? undefined : true
|
||||
},
|
||||
adapter: adapter()
|
||||
})
|
||||
]
|
||||
});
|
||||
Reference in New Issue
Block a user