Phase 1: Windows log collection + full-text search
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
This commit is contained in:
+3
-2
@@ -1,7 +1,8 @@
|
|||||||
# ingest/ and api/ build with context "." (repo root) so their Dockerfiles
|
# ingest/, api/, and search/ build with context "." (repo root) so their
|
||||||
# can also COPY proto/. Keep that context lean.
|
# Dockerfiles can also COPY proto/. Keep that context lean.
|
||||||
.git/
|
.git/
|
||||||
agent/target/
|
agent/target/
|
||||||
|
search/target/
|
||||||
web/node_modules/
|
web/node_modules/
|
||||||
web/build/
|
web/build/
|
||||||
web/.svelte-kit/
|
web/.svelte-kit/
|
||||||
|
|||||||
@@ -1,11 +1,13 @@
|
|||||||
# Rust
|
# Rust
|
||||||
agent/target/
|
agent/target/
|
||||||
|
search/target/
|
||||||
|
|
||||||
# Go build cache (go build ./... without -o doesn't normally leave
|
# Go build cache (go build ./... without -o doesn't normally leave
|
||||||
# binaries in-tree, but be defensive)
|
# binaries in-tree, but be defensive)
|
||||||
/ingest/ingest
|
/ingest/ingest
|
||||||
/api/api
|
/api/api
|
||||||
/cli/sentryctl
|
/cli/sentryctl
|
||||||
|
/hack/windows-fixture/windows-fixture
|
||||||
|
|
||||||
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
||||||
web/node_modules/
|
web/node_modules/
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# Project: Sentry — Distributed Log Aggregation & Observability Platform
|
||||||
|
|
||||||
|
## Mission
|
||||||
|
Build an open-core, Kubernetes-native centralized logging platform that rivals
|
||||||
|
Splunk on features but wins on cost-per-GB, modern language stack, and honest
|
||||||
|
multi-tenant RBAC. Full architecture spec is in `/docs/architecture.md` — read
|
||||||
|
it before touching any component. Do not deviate from the storage/query split
|
||||||
|
described there without flagging it to me first.
|
||||||
|
|
||||||
|
## Non-negotiable constraints
|
||||||
|
- Distro-agnostic Linux agent: must run identically on RHEL/Debian/Arch/SUSE
|
||||||
|
derivatives via a statically-linked musl binary. No glibc runtime deps.
|
||||||
|
- Windows support via native ETW/Event Log API, not a WSL shim.
|
||||||
|
- AGPLv3 for core + agents. Enterprise module (SSO/multi-tenancy/compliance)
|
||||||
|
lives in a separate `enterprise/` directory under a commercial license stub
|
||||||
|
— keep the boundary clean from day one, don't let AGPL code import from it.
|
||||||
|
- Schema-on-write with OTel semantic conventions as the default schema, with
|
||||||
|
schema-on-read fallback for unstructured text.
|
||||||
|
- Every UI action must correspond to a documented REST/gRPC call. No
|
||||||
|
UI-only logic. CLI (`sentryctl`) and Terraform provider are first-class,
|
||||||
|
not afterthoughts.
|
||||||
|
|
||||||
|
## Tech stack (pinned — do not substitute without discussion)
|
||||||
|
| Component | Language/Tool |
|
||||||
|
|-------------------|------------------------|
|
||||||
|
| Edge agent | Rust, musl target |
|
||||||
|
| Transport | Redpanda (Kafka API) |
|
||||||
|
| Ingest/parse | Go |
|
||||||
|
| Analytical store | ClickHouse |
|
||||||
|
| Full-text index | Tantivy (Rust) |
|
||||||
|
| Control plane/API | Go, gRPC + REST gateway |
|
||||||
|
| Frontend | SvelteKit + TypeScript |
|
||||||
|
| Deployment | Kubernetes Operator (Go, kubebuilder), Helm, docker-compose for local/homelab |
|
||||||
|
|
||||||
|
## Repo conventions
|
||||||
|
- Monorepo, one top-level dir per component (see structure below).
|
||||||
|
- Rust: workspace-based, `cargo clippy --all-targets -- -D warnings` must pass.
|
||||||
|
- Go: standard `go vet` + `golangci-lint`, no globals for shared state.
|
||||||
|
- Every component ships with: unit tests, a `README.md`, and a Dockerfile
|
||||||
|
using distroless or scratch base images where feasible.
|
||||||
|
- Conventional commits. Every PR-sized change should be a logically complete,
|
||||||
|
independently revertible unit.
|
||||||
|
- Prefer boring, well-understood dependencies over novel ones. This is
|
||||||
|
infrastructure software; operators need to trust it.
|
||||||
|
|
||||||
|
## What "done" looks like for Phase 0 (MVP)
|
||||||
|
|
||||||
|
**Status: shipped.** A single log line, generated on a Linux host by the
|
||||||
|
Rust agent, flows: agent → Redpanda → Go ingest service → ClickHouse, and
|
||||||
|
is queryable via a minimal SQL endpoint and visible in a bare-bones
|
||||||
|
SvelteKit table view. Verified end-to-end on real hardware, not just in
|
||||||
|
CI — see `/docs/phase-0-runbook.md`. No alerting, no multi-tenancy, no
|
||||||
|
dashboards — that discipline held for the whole phase.
|
||||||
|
|
||||||
|
## What "done" looks like for Phase 1
|
||||||
|
|
||||||
|
A Windows Event Log entry and a Linux journald entry should both be
|
||||||
|
queryable via SQL (the ClickHouse path) and via free-text search (the
|
||||||
|
Tantivy path), from the same UI, within a few seconds of being generated.
|
||||||
|
|
||||||
|
Non-goals for this phase (same "resist scope creep" discipline as Phase 0):
|
||||||
|
no alerting, no dashboards, no SPL-like query layer, no multi-tenancy, and
|
||||||
|
no unified query experience — two separate boxes on two separate pages is
|
||||||
|
correct for Phase 1; unifying them is Phase 2's job.
|
||||||
|
|
||||||
|
ETW and WEF (Windows Event Forwarding) are *designed* in this phase but not
|
||||||
|
required to be running for "done": ETW ships behind a feature flag most
|
||||||
|
environments won't enable (it needs elevated privileges), and WEF's
|
||||||
|
receiver-side is explicitly deferred rather than built now — see
|
||||||
|
`/docs/phase-1-runbook.md` for both. Only the Event Log source needs to
|
||||||
|
actually be running end-to-end for this phase to count as done.
|
||||||
|
|
||||||
|
## When in doubt
|
||||||
|
Ask before: changing the pinned stack, adding a new external dependency
|
||||||
|
that pulls in a large transitive tree, or making an architectural decision
|
||||||
|
that isn't already specified in `/docs/architecture.md`.
|
||||||
Generated
+93
@@ -735,6 +735,15 @@ dependencies = [
|
|||||||
"prost",
|
"prost",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quick-xml"
|
||||||
|
version = "0.36.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f7649a7b4df05aed9ea7ec6f628c67c9953a43869b8bc50929569b2999d443fe"
|
||||||
|
dependencies = [
|
||||||
|
"memchr",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "quote"
|
name = "quote"
|
||||||
version = "1.0.47"
|
version = "1.0.47"
|
||||||
@@ -893,6 +902,7 @@ dependencies = [
|
|||||||
"anyhow",
|
"anyhow",
|
||||||
"clap",
|
"clap",
|
||||||
"prost",
|
"prost",
|
||||||
|
"quick-xml",
|
||||||
"sentry-parser",
|
"sentry-parser",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
@@ -902,6 +912,8 @@ dependencies = [
|
|||||||
"tonic-build",
|
"tonic-build",
|
||||||
"tracing",
|
"tracing",
|
||||||
"tracing-subscriber",
|
"tracing-subscriber",
|
||||||
|
"windows",
|
||||||
|
"windows-service",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1380,12 +1392,93 @@ version = "0.11.1+wasi-snapshot-preview1"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
|
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "widestring"
|
||||||
|
version = "1.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "72069c3113ab32ab29e5584db3c6ec55d416895e60715417b5b883a357c3e471"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows"
|
||||||
|
version = "0.58.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "dd04d41d93c4992d421894c18c8b43496aa748dd4c081bac0dc93eb0489272b6"
|
||||||
|
dependencies = [
|
||||||
|
"windows-core",
|
||||||
|
"windows-targets",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-core"
|
||||||
|
version = "0.58.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "6ba6d44ec8c2591c134257ce647b7ea6b20335bf6379a27dac5f1641fcf59f99"
|
||||||
|
dependencies = [
|
||||||
|
"windows-implement",
|
||||||
|
"windows-interface",
|
||||||
|
"windows-result",
|
||||||
|
"windows-strings",
|
||||||
|
"windows-targets",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-implement"
|
||||||
|
version = "0.58.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "2bbd5b46c938e506ecbce286b6628a02171d56153ba733b6c741fc627ec9579b"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn 2.0.119",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-interface"
|
||||||
|
version = "0.58.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "053c4c462dc91d3b1504c6fe5a726dd15e216ba718e84a0e46a88fbe5ded3515"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn 2.0.119",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-link"
|
name = "windows-link"
|
||||||
version = "0.2.1"
|
version = "0.2.1"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-result"
|
||||||
|
version = "0.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e"
|
||||||
|
dependencies = [
|
||||||
|
"windows-targets",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-service"
|
||||||
|
version = "0.7.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d24d6bcc7f734a4091ecf8d7a64c5f7d7066f45585c1861eba06449909609c8a"
|
||||||
|
dependencies = [
|
||||||
|
"bitflags",
|
||||||
|
"widestring",
|
||||||
|
"windows-sys 0.52.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "windows-strings"
|
||||||
|
version = "0.1.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10"
|
||||||
|
dependencies = [
|
||||||
|
"windows-result",
|
||||||
|
"windows-targets",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "windows-sys"
|
name = "windows-sys"
|
||||||
version = "0.52.0"
|
version = "0.52.0"
|
||||||
|
|||||||
+146
-16
@@ -1,15 +1,39 @@
|
|||||||
# sentry-agent
|
# sentry-agent
|
||||||
|
|
||||||
Distro-agnostic Linux log collector. Statically linked against musl, no
|
Distro-agnostic Linux/Windows log collector. On Linux, statically linked
|
||||||
glibc runtime dependency. Tails journald (default) or a file, batches
|
against musl, no glibc runtime dependency. Tails journald (Linux default),
|
||||||
lines, and ships them over mTLS gRPC to the ingest service.
|
a file, Windows Event Log, or ETW, batches lines, and ships them over mTLS
|
||||||
|
gRPC to the ingest service.
|
||||||
|
|
||||||
|
**Windows support status:** the Windows-specific code
|
||||||
|
(`source/windows_eventlog.rs`, `source/etw.rs`, `service.rs`) was written
|
||||||
|
against documented Win32/ETW API shapes but has **not been compiled or run
|
||||||
|
on Windows** — no Windows toolchain was available in the environment this
|
||||||
|
was built in (confirmed: only the Linux target's std library was
|
||||||
|
installed, no way to even `cargo check --target x86_64-pc-windows-*`).
|
||||||
|
Linux builds/tests/clippy are verified clean across every feature
|
||||||
|
combination; Windows code is a first draft to compile-check and test for
|
||||||
|
real before trusting it. See `/docs/phase-1-runbook.md`.
|
||||||
|
|
||||||
## Workspace layout
|
## Workspace layout
|
||||||
|
|
||||||
- `sentry-parser` — pure-`std` RFC 5424 syslog parser with raw-passthrough
|
- `sentry-parser` — pure-`std` RFC 5424 syslog parser with raw-passthrough
|
||||||
fallback. No I/O, easy to unit test in isolation.
|
fallback. No I/O, easy to unit test in isolation.
|
||||||
- `sentry-agent` — the binary: config loading, sourcing (journald/file),
|
- `sentry-agent` — the binary: config loading, sourcing (journald/file/
|
||||||
batching, mTLS gRPC client.
|
Windows Event Log/ETW), batching, mTLS gRPC client, Windows service
|
||||||
|
wrapper.
|
||||||
|
|
||||||
|
## Why one crate for both platforms, not a platform split
|
||||||
|
|
||||||
|
`config.rs`, `batch.rs`, `grpc.rs`, and `main.rs`'s event loop are already
|
||||||
|
100% cross-platform Rust — nothing in them is Linux- or Windows-specific.
|
||||||
|
Only the `source/` modules differ per platform, and that boundary already
|
||||||
|
existed before Windows support was added (it's exactly what made adding
|
||||||
|
Windows sources a matter of adding two files, not restructuring anything).
|
||||||
|
Windows-only dependencies (`windows`, `windows-service`, `quick-xml`) live
|
||||||
|
in a `[target.'cfg(windows)'.dependencies]` section in `Cargo.toml`, so
|
||||||
|
they're not in the Linux build's dependency graph at all — no crate split
|
||||||
|
needed to keep the two platforms from stepping on each other.
|
||||||
|
|
||||||
## Why journalctl, not libsystemd
|
## Why journalctl, not libsystemd
|
||||||
|
|
||||||
@@ -62,6 +86,32 @@ container without deliberately bind-mounting `/var/log/journal` (or
|
|||||||
deployment for journald sourcing is as a native binary managed by systemd
|
deployment for journald sourcing is as a native binary managed by systemd
|
||||||
on the host, not containerized.
|
on the host, not containerized.
|
||||||
|
|
||||||
|
### Building for Windows
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Cross-compiling FROM Linux, for the build step only:
|
||||||
|
rustup target add x86_64-pc-windows-gnu
|
||||||
|
cargo build --release --target x86_64-pc-windows-gnu \
|
||||||
|
--no-default-features --features windows-eventlog,etw
|
||||||
|
|
||||||
|
# Natively on Windows (MSVC toolchain):
|
||||||
|
cargo build --release --target x86_64-pc-windows-msvc \
|
||||||
|
--no-default-features --features windows-eventlog,etw
|
||||||
|
```
|
||||||
|
|
||||||
|
`--no-default-features` matters: the default feature set is `journald`,
|
||||||
|
which is Linux-only (the module is `target_os = "linux"`-gated and simply
|
||||||
|
won't compile in on Windows, but there's no reason to carry the dead
|
||||||
|
feature flag). Drop `,etw` from `--features` if you only want Event Log —
|
||||||
|
see the privilege note below for why most environments will want to.
|
||||||
|
|
||||||
|
**Cross-compilation only covers the *build* step.** Running/testing the
|
||||||
|
Windows sources — actually calling `EvtSubscribe`, starting an ETW
|
||||||
|
session, registering a Windows service — needs a real or virtualized
|
||||||
|
Windows host. There is no way around that, and nothing in this repo
|
||||||
|
pretends otherwise; see `/docs/phase-1-runbook.md` for exactly what's
|
||||||
|
automatable vs. manual-only.
|
||||||
|
|
||||||
## Running
|
## Running
|
||||||
|
|
||||||
No CLI flags are required for the common case:
|
No CLI flags are required for the common case:
|
||||||
@@ -70,12 +120,14 @@ No CLI flags are required for the common case:
|
|||||||
./sentry-agent
|
./sentry-agent
|
||||||
```
|
```
|
||||||
|
|
||||||
This uses `/etc/sentry-agent/agent.toml` if present, otherwise built-in
|
This uses the platform's conventional config path if present
|
||||||
defaults: journald source (whole journal, no unit filter), service name
|
(`/etc/sentry-agent/agent.toml` on Linux, `C:\ProgramData\SentryAgent\agent.toml`
|
||||||
`default`, and mTLS material expected at
|
on Windows), otherwise built-in defaults: journald source on Linux (whole
|
||||||
`/etc/sentry-agent/{ca,client,client-key}.pem`. mTLS is mandatory per the
|
journal, no unit filter), service name `default`, and mTLS material
|
||||||
project's transport requirements, so a from-scratch run with no certs in
|
expected under the same conventional directory
|
||||||
place will fail fast with a clear error rather than connecting insecurely.
|
(`{ca,client,client-key}.pem`). mTLS is mandatory per the project's
|
||||||
|
transport requirements, so a from-scratch run with no certs in place will
|
||||||
|
fail fast with a clear error rather than connecting insecurely.
|
||||||
|
|
||||||
See `config/agent.example.toml` for all fields.
|
See `config/agent.example.toml` for all fields.
|
||||||
|
|
||||||
@@ -83,6 +135,70 @@ See `config/agent.example.toml` for all fields.
|
|||||||
./sentry-agent --config /path/to/agent.toml
|
./sentry-agent --config /path/to/agent.toml
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Running as a Windows service
|
||||||
|
|
||||||
|
"A native Windows service, not a WSL wrapper" means implementing the Win32
|
||||||
|
Service Control Manager protocol, not just running the binary in a
|
||||||
|
console — that's what `service.rs` (via the `windows-service` crate)
|
||||||
|
does. From an administrator shell:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
sentry-agent.exe install # registers the service, Automatic start, LocalSystem account
|
||||||
|
sc.exe start SentryAgent
|
||||||
|
sc.exe stop SentryAgent
|
||||||
|
sentry-agent.exe uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
`install`/`uninstall`/`run-service` are subcommands only present in
|
||||||
|
Windows builds (`sentry-agent` with no subcommand is still the normal
|
||||||
|
foreground/console run, same as on Linux) — `run-service` specifically is
|
||||||
|
what the SCM itself invokes at service start; don't run it directly.
|
||||||
|
|
||||||
|
**Known limitation:** when running as a service, there's no console
|
||||||
|
attached, so `tracing_subscriber::fmt()`'s stdout writer has nowhere to
|
||||||
|
go — logs won't be visible anywhere useful until this is redirected to a
|
||||||
|
file or a proper Windows Event Log tracing sink is written. Not addressed
|
||||||
|
in Phase 1; flagging it here rather than shipping it silently broken.
|
||||||
|
|
||||||
|
## ETW: read this before enabling it
|
||||||
|
|
||||||
|
ETW needs elevated privileges to subscribe to most providers — running
|
||||||
|
the agent under an administrator token or a service account with
|
||||||
|
`SeSystemProfilePrivilege`/ETW-specific rights. This is a real privilege
|
||||||
|
escalation, not a footnote: think about whether your environment wants
|
||||||
|
the log-shipping agent running with that level of access before turning
|
||||||
|
on the `etw` feature and an `[source] kind = "etw"` config. Event Log
|
||||||
|
alone (no elevated privileges needed) covers the common case and is what
|
||||||
|
Phase 1's exit criteria in `/CLAUDE.md` actually requires to be running.
|
||||||
|
|
||||||
|
Providers are configured by **GUID**, not friendly name — ETW's own API
|
||||||
|
requires it. Look one up with `logman query providers "<Friendly Name>"`.
|
||||||
|
|
||||||
|
## Windows Event Forwarding (WEF)
|
||||||
|
|
||||||
|
Two different things people mean by "WEF support," worth being explicit
|
||||||
|
about since they're very different amounts of work:
|
||||||
|
|
||||||
|
1. **What this repo supports today, with zero extra code:** WEF is a
|
||||||
|
native Windows-to-Windows mechanism (`wecsvc`, the built-in Windows
|
||||||
|
Event Collector role) — endpoints forward to a Windows Server acting
|
||||||
|
as collector using Windows' own mechanism, no Sentry code involved in
|
||||||
|
the forwarding itself. Run this agent *on the collector box*,
|
||||||
|
subscribed to the `ForwardedEvents` channel instead of the usual three:
|
||||||
|
```toml
|
||||||
|
[source]
|
||||||
|
kind = "eventlog"
|
||||||
|
channels = ["ForwardedEvents"]
|
||||||
|
```
|
||||||
|
2. **What this repo does *not* implement:** a true agentless receiver —
|
||||||
|
Sentry itself speaking the WS-Management/WinRM event-subscription
|
||||||
|
protocol so endpoints can forward directly to `ingest` without any
|
||||||
|
Windows Event Collector role or Sentry agent anywhere. That's a
|
||||||
|
standalone protocol implementation (SOAP-ish subscription/heartbeat/
|
||||||
|
delivery over WinRM), not an agent or ingest-side tweak, and it's out
|
||||||
|
of scope for Phase 1. If you need this, it's a real project of its
|
||||||
|
own — say so before assuming it's a small addition.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -91,10 +207,24 @@ cargo test --workspace
|
|||||||
|
|
||||||
## Feature flags
|
## Feature flags
|
||||||
|
|
||||||
- `journald` (default) — journalctl-based journald source.
|
- `journald` (default) — journalctl-based journald source. `target_os =
|
||||||
|
"linux"`-gated: enabling this on a Windows build is a no-op, not a
|
||||||
|
build failure.
|
||||||
- `file-tail` — polling-based file tailer (no inotify dependency; doesn't
|
- `file-tail` — polling-based file tailer (no inotify dependency; doesn't
|
||||||
follow rename-based log rotation yet).
|
follow rename-based log rotation yet). Cross-platform, works on Windows
|
||||||
|
too.
|
||||||
|
- `windows-eventlog` — Windows Event Log via `EvtSubscribe`.
|
||||||
|
`target_os = "windows"`-gated the same way; a no-op on Linux.
|
||||||
|
- `etw` — ETW real-time session. Same gating. See the privilege section
|
||||||
|
above before enabling.
|
||||||
|
|
||||||
Both can be enabled together; `[source].kind` in config picks which one
|
Any combination can be enabled together; `[source].kind` in config picks
|
||||||
runs. Building without a feature and configuring that source at runtime
|
which one actually runs. Building without a feature and configuring that
|
||||||
fails at startup with a clear error rather than silently doing nothing.
|
source at runtime fails at startup with a clear error rather than
|
||||||
|
silently doing nothing.
|
||||||
|
|
||||||
|
Dependencies added for Windows support, worth knowing about:
|
||||||
|
`windows` (Microsoft's official Win32/ETW bindings), `windows-service`
|
||||||
|
(Windows Service Control Manager wrapper), `quick-xml` (parses
|
||||||
|
EvtSubscribe's rendered event XML). All three are `[target.'cfg(windows)'.dependencies]`
|
||||||
|
— not in the Linux build's dependency graph at all.
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ name = "sentry-agent"
|
|||||||
version.workspace = true
|
version.workspace = true
|
||||||
edition.workspace = true
|
edition.workspace = true
|
||||||
license.workspace = true
|
license.workspace = true
|
||||||
description = "Sentry distro-agnostic Linux log collector"
|
description = "Sentry distro-agnostic Linux/Windows log collector"
|
||||||
|
|
||||||
[[bin]]
|
[[bin]]
|
||||||
name = "sentry-agent"
|
name = "sentry-agent"
|
||||||
@@ -13,6 +13,12 @@ path = "src/main.rs"
|
|||||||
default = ["journald"]
|
default = ["journald"]
|
||||||
journald = []
|
journald = []
|
||||||
file-tail = []
|
file-tail = []
|
||||||
|
# Windows-only sources. Feature-enabled AND target_os="windows"-gated at
|
||||||
|
# the module level (see src/source/mod.rs), so enabling these on a
|
||||||
|
# non-Windows build is a harmless no-op, not a build failure -- keeps
|
||||||
|
# `cargo test --workspace --all-features` green on Linux CI.
|
||||||
|
windows-eventlog = []
|
||||||
|
etw = []
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
sentry-parser = { path = "../sentry-parser" }
|
sentry-parser = { path = "../sentry-parser" }
|
||||||
@@ -30,5 +36,18 @@ anyhow = "1"
|
|||||||
tracing = "0.1"
|
tracing = "0.1"
|
||||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
|
||||||
|
# Windows-only: not in the dependency graph at all on other targets, so
|
||||||
|
# they don't affect Linux build times or the musl release binary.
|
||||||
|
[target.'cfg(windows)'.dependencies]
|
||||||
|
windows = { version = "0.58", features = [
|
||||||
|
"Win32_Foundation",
|
||||||
|
"Win32_System_EventLog",
|
||||||
|
"Win32_System_Threading",
|
||||||
|
"Win32_System_Diagnostics_Etw",
|
||||||
|
"Win32_Security",
|
||||||
|
] }
|
||||||
|
windows-service = "0.7"
|
||||||
|
quick-xml = "0.36"
|
||||||
|
|
||||||
[build-dependencies]
|
[build-dependencies]
|
||||||
tonic-build = "0.12"
|
tonic-build = "0.12"
|
||||||
|
|||||||
@@ -1,25 +1,39 @@
|
|||||||
# Example sentry-agent config. Copy to /etc/sentry-agent/agent.toml, or
|
# Example sentry-agent config. Copy to the platform's conventional path
|
||||||
# pass --config /path/to/this/file.
|
# (/etc/sentry-agent/agent.toml on Linux, C:\ProgramData\SentryAgent\agent.toml
|
||||||
|
# on Windows), or pass --config /path/to/this/file.
|
||||||
#
|
#
|
||||||
# Every field has a built-in default (see src/config.rs), so this file only
|
# Every field has a built-in default (see src/config.rs), so this file only
|
||||||
# needs to contain what you're overriding. An agent with NO config file at
|
# needs to contain what you're overriding. An agent with NO config file at
|
||||||
# all still runs: it defaults to journald, service = "default", and expects
|
# all still runs: on Linux it defaults to journald, service = "default",
|
||||||
# mTLS material at /etc/sentry-agent/{ca,client,client-key}.pem.
|
# and expects mTLS material at /etc/sentry-agent/{ca,client,client-key}.pem
|
||||||
|
# (Windows equivalents under C:\ProgramData\SentryAgent\).
|
||||||
|
|
||||||
[agent]
|
[agent]
|
||||||
# host = "explicit-hostname-override" # defaults to /etc/hostname
|
# host = "explicit-hostname-override" # defaults to /etc/hostname (Linux) or %COMPUTERNAME% (Windows)
|
||||||
service = "my-service"
|
service = "my-service"
|
||||||
|
|
||||||
[source]
|
[source]
|
||||||
kind = "journald"
|
kind = "journald"
|
||||||
# unit = "nginx.service" # omit to tail the whole journal
|
# unit = "nginx.service" # omit to tail the whole journal
|
||||||
|
|
||||||
# To tail a file instead:
|
# To tail a file instead (works on both Linux and Windows):
|
||||||
# [source]
|
# [source]
|
||||||
# kind = "file"
|
# kind = "file"
|
||||||
# path = "/var/log/nginx/access.log"
|
# path = "/var/log/nginx/access.log"
|
||||||
# from_beginning = false
|
# from_beginning = false
|
||||||
|
|
||||||
|
# Windows Event Log (requires the agent to be built with the
|
||||||
|
# `windows-eventlog` feature — see /agent/README.md):
|
||||||
|
# [source]
|
||||||
|
# kind = "eventlog"
|
||||||
|
# channels = ["Application", "System", "Security"] # this is the default if omitted
|
||||||
|
|
||||||
|
# ETW (requires the `etw` feature, and usually elevated privileges — read
|
||||||
|
# /agent/README.md's privilege section before enabling this):
|
||||||
|
# [source]
|
||||||
|
# kind = "etw"
|
||||||
|
# providers = ["{22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}"] # GUIDs, not friendly names
|
||||||
|
|
||||||
[batch]
|
[batch]
|
||||||
max_size = 500
|
max_size = 500
|
||||||
flush_interval_ms = 2000
|
flush_interval_ms = 2000
|
||||||
|
|||||||
@@ -63,6 +63,7 @@ mod tests {
|
|||||||
severity: 0,
|
severity: 0,
|
||||||
message: msg.into(),
|
message: msg.into(),
|
||||||
attributes: Default::default(),
|
attributes: Default::default(),
|
||||||
|
record_id: String::new(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,10 @@ use anyhow::{Context, Result};
|
|||||||
use serde::Deserialize;
|
use serde::Deserialize;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
const DEFAULT_CONFIG_PATH: &str = "/etc/sentry-agent/agent.toml";
|
const DEFAULT_CONFIG_PATH: &str = "/etc/sentry-agent/agent.toml";
|
||||||
|
#[cfg(windows)]
|
||||||
|
const DEFAULT_CONFIG_PATH: &str = r"C:\ProgramData\SentryAgent\agent.toml";
|
||||||
|
|
||||||
#[derive(Debug, Clone, Deserialize, Default)]
|
#[derive(Debug, Clone, Deserialize, Default)]
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
@@ -15,11 +18,13 @@ pub struct Config {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl Config {
|
impl Config {
|
||||||
/// Loads config from `explicit_path` if given, else from
|
/// Loads config from `explicit_path` if given, else from the
|
||||||
/// `/etc/sentry-agent/agent.toml` if it exists, else falls back to
|
/// platform's conventional config path if it exists
|
||||||
/// built-in defaults (journald source, default TLS cert paths). Only an
|
/// (`/etc/sentry-agent/agent.toml` on Linux,
|
||||||
/// explicitly-passed `--config` path that doesn't exist is an error;
|
/// `C:\ProgramData\SentryAgent\agent.toml` on Windows), else falls
|
||||||
/// the conventional default path is optional.
|
/// back to built-in defaults (journald source on Linux, default TLS
|
||||||
|
/// cert paths). Only an explicitly-passed `--config` path that doesn't
|
||||||
|
/// exist is an error; the conventional default path is optional.
|
||||||
pub fn load(explicit_path: Option<&Path>) -> Result<Config> {
|
pub fn load(explicit_path: Option<&Path>) -> Result<Config> {
|
||||||
let path = match explicit_path {
|
let path = match explicit_path {
|
||||||
Some(p) => Some(p.to_path_buf()),
|
Some(p) => Some(p.to_path_buf()),
|
||||||
@@ -63,13 +68,55 @@ impl Default for AgentConfig {
|
|||||||
pub enum SourceConfig {
|
pub enum SourceConfig {
|
||||||
Journald {
|
Journald {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
|
#[cfg_attr(not(all(feature = "journald", target_os = "linux")), allow(dead_code))]
|
||||||
unit: Option<String>,
|
unit: Option<String>,
|
||||||
},
|
},
|
||||||
|
// Dead-code-on-default-build, same reasoning as EventLog/Etw below:
|
||||||
|
// these fields are only read by the `file-tail`-gated arm in
|
||||||
|
// spawn_source (main.rs), which doesn't exist in the default build
|
||||||
|
// (`default = ["journald"]`). Pre-existing gap from Phase 0 — CLAUDE.md
|
||||||
|
// mandates plain `cargo clippy --all-targets -- -D warnings` (no
|
||||||
|
// --all-features), which this broke silently since only
|
||||||
|
// --all-features clippy was ever actually run.
|
||||||
File {
|
File {
|
||||||
|
#[cfg_attr(not(feature = "file-tail"), allow(dead_code))]
|
||||||
path: PathBuf,
|
path: PathBuf,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
|
#[cfg_attr(not(feature = "file-tail"), allow(dead_code))]
|
||||||
from_beginning: bool,
|
from_beginning: bool,
|
||||||
},
|
},
|
||||||
|
/// Windows Event Log via EvtSubscribe. Requires the agent to be built
|
||||||
|
/// with the `windows-eventlog` feature; see /agent/README.md.
|
||||||
|
///
|
||||||
|
/// `channels`/`providers` below are read only by the Windows-only
|
||||||
|
/// consumers in `spawn_source` (main.rs), which don't exist at all on
|
||||||
|
/// non-Windows builds — unlike `File`'s fields (dead only when the
|
||||||
|
/// `file-tail` feature happens to be off), these are dead on *every*
|
||||||
|
/// non-Windows build regardless of feature flags, since their sole
|
||||||
|
/// consumer is `target_os = "windows"`-gated. `cfg_attr` here keeps
|
||||||
|
/// clippy honest: still flags genuine dead code on an actual Windows
|
||||||
|
/// build, just not on the platform where these fields can never be
|
||||||
|
/// read no matter what.
|
||||||
|
EventLog {
|
||||||
|
#[serde(default = "default_eventlog_channels")]
|
||||||
|
#[cfg_attr(not(windows), allow(dead_code))]
|
||||||
|
channels: Vec<String>,
|
||||||
|
},
|
||||||
|
/// ETW (Event Tracing for Windows). Requires the `etw` feature and
|
||||||
|
/// (usually) elevated privileges — see /agent/README.md before
|
||||||
|
/// enabling this in any environment that isn't Windows-first.
|
||||||
|
Etw {
|
||||||
|
#[cfg_attr(not(windows), allow(dead_code))]
|
||||||
|
providers: Vec<String>,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_eventlog_channels() -> Vec<String> {
|
||||||
|
vec![
|
||||||
|
"Application".to_string(),
|
||||||
|
"System".to_string(),
|
||||||
|
"Security".to_string(),
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for SourceConfig {
|
impl Default for SourceConfig {
|
||||||
@@ -119,9 +166,19 @@ pub struct TlsConfig {
|
|||||||
impl Default for TlsConfig {
|
impl Default for TlsConfig {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
Self {
|
Self {
|
||||||
ca_cert: PathBuf::from("/etc/sentry-agent/ca.pem"),
|
ca_cert: default_cert_path("ca.pem"),
|
||||||
client_cert: PathBuf::from("/etc/sentry-agent/client.pem"),
|
client_cert: default_cert_path("client.pem"),
|
||||||
client_key: PathBuf::from("/etc/sentry-agent/client-key.pem"),
|
client_key: default_cert_path("client-key.pem"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
fn default_cert_path(name: &str) -> PathBuf {
|
||||||
|
PathBuf::from(format!("/etc/sentry-agent/{name}"))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn default_cert_path(name: &str) -> PathBuf {
|
||||||
|
PathBuf::from(format!(r"C:\ProgramData\SentryAgent\{name}"))
|
||||||
|
}
|
||||||
|
|||||||
+103
-13
@@ -3,6 +3,9 @@ mod config;
|
|||||||
mod grpc;
|
mod grpc;
|
||||||
mod source;
|
mod source;
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
mod service;
|
||||||
|
|
||||||
pub mod pb {
|
pub mod pb {
|
||||||
tonic::include_proto!("sentry.logs.v1");
|
tonic::include_proto!("sentry.logs.v1");
|
||||||
}
|
}
|
||||||
@@ -18,23 +21,66 @@ use tokio::sync::mpsc;
|
|||||||
use tonic::transport::Channel;
|
use tonic::transport::Channel;
|
||||||
|
|
||||||
#[derive(Parser)]
|
#[derive(Parser)]
|
||||||
#[command(name = "sentry-agent", about = "Sentry Linux log collector")]
|
#[command(name = "sentry-agent", about = "Sentry Linux/Windows log collector")]
|
||||||
struct Cli {
|
struct Cli {
|
||||||
/// Path to a TOML config file. Defaults to /etc/sentry-agent/agent.toml
|
/// Path to a TOML config file. Defaults to the platform's conventional
|
||||||
/// if present, otherwise built-in defaults (journald source, default
|
/// path if present, otherwise built-in defaults — see config::Config::load.
|
||||||
/// TLS cert paths under /etc/sentry-agent/).
|
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
config: Option<PathBuf>,
|
config: Option<PathBuf>,
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
#[command(subcommand)]
|
||||||
|
command: Option<WindowsCommand>,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
#[cfg(windows)]
|
||||||
async fn main() -> Result<()> {
|
#[derive(clap::Subcommand)]
|
||||||
|
enum WindowsCommand {
|
||||||
|
/// Registers this binary as a Windows service (Automatic start,
|
||||||
|
/// LocalSystem account). Requires an administrator shell.
|
||||||
|
Install,
|
||||||
|
/// Removes the Windows service registration.
|
||||||
|
Uninstall,
|
||||||
|
/// Entry point the Service Control Manager invokes when starting the
|
||||||
|
/// registered service. Not meant to be run directly by a user — use
|
||||||
|
/// `sentry-agent` with no subcommand for a normal foreground/console
|
||||||
|
/// run, same as on Linux.
|
||||||
|
RunService,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Not `#[tokio::main]`: the Windows service dispatcher
|
||||||
|
/// (`service_dispatcher::start`, see service.rs) is a blocking, synchronous
|
||||||
|
/// FFI call into the Service Control Manager and needs to be invoked
|
||||||
|
/// directly from a plain thread, not from inside an already-running tokio
|
||||||
|
/// runtime. Every other path builds its own runtime explicitly instead.
|
||||||
|
fn main() -> Result<()> {
|
||||||
|
let cli = Cli::parse();
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
match cli.command {
|
||||||
|
Some(WindowsCommand::Install) => return service::install().context("installing Windows service"),
|
||||||
|
Some(WindowsCommand::Uninstall) => return service::uninstall().context("removing Windows service"),
|
||||||
|
Some(WindowsCommand::RunService) => return service::run_as_service().context("running as a Windows service"),
|
||||||
|
None => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let rt = tokio::runtime::Runtime::new().context("building tokio runtime")?;
|
||||||
|
rt.block_on(run_agent(cli.config))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The actual agent: load config, connect to ingest, run the source ->
|
||||||
|
/// parse -> batch -> ship loop until the source exits or the process is
|
||||||
|
/// signaled to stop. Called from `main()` directly for a normal run, and
|
||||||
|
/// from within the Windows service's own thread when running as a
|
||||||
|
/// service (see service.rs) — same logic either way.
|
||||||
|
pub async fn run_agent(config_path: Option<PathBuf>) -> Result<()> {
|
||||||
tracing_subscriber::fmt()
|
tracing_subscriber::fmt()
|
||||||
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||||||
.init();
|
.init();
|
||||||
|
|
||||||
let cli = Cli::parse();
|
let cfg = Config::load(config_path.as_deref()).context("loading config")?;
|
||||||
let cfg = Config::load(cli.config.as_deref()).context("loading config")?;
|
|
||||||
|
|
||||||
let host = cfg.agent.host.clone().unwrap_or_else(default_hostname);
|
let host = cfg.agent.host.clone().unwrap_or_else(default_hostname);
|
||||||
let service = cfg.agent.service.clone();
|
let service = cfg.agent.service.clone();
|
||||||
@@ -60,13 +106,23 @@ async fn main() -> Result<()> {
|
|||||||
};
|
};
|
||||||
let parsed = sentry_parser::parse(&raw.line);
|
let parsed = sentry_parser::parse(&raw.line);
|
||||||
let severity = to_pb_severity(raw.severity_hint.or(parsed.severity));
|
let severity = to_pb_severity(raw.severity_hint.or(parsed.severity));
|
||||||
|
let mut attributes: std::collections::HashMap<String, String> =
|
||||||
|
parsed.attributes.into_iter().collect();
|
||||||
|
// Source-provided structured fields (e.g. Windows Event
|
||||||
|
// Log's EventID/Provider/Channel) win over anything the
|
||||||
|
// RFC 5424 parser inferred from the raw text, since they
|
||||||
|
// come from a more authoritative place.
|
||||||
|
attributes.extend(raw.extra_attributes);
|
||||||
let record = LogRecord {
|
let record = LogRecord {
|
||||||
timestamp_unix_nano: raw.timestamp_unix_nano,
|
timestamp_unix_nano: raw.timestamp_unix_nano,
|
||||||
host: host.clone(),
|
host: host.clone(),
|
||||||
service: service.clone(),
|
service: service.clone(),
|
||||||
severity: severity as i32,
|
severity: severity as i32,
|
||||||
message: parsed.message,
|
message: parsed.message,
|
||||||
attributes: parsed.attributes.into_iter().collect(),
|
attributes,
|
||||||
|
// Always empty as sent by the agent -- ingest assigns
|
||||||
|
// this server-side. See the proto field comment.
|
||||||
|
record_id: String::new(),
|
||||||
};
|
};
|
||||||
if let Some(batch) = batcher.push(record) {
|
if let Some(batch) = batcher.push(record) {
|
||||||
flush(&mut client, batch).await;
|
flush(&mut client, batch).await;
|
||||||
@@ -87,13 +143,24 @@ async fn main() -> Result<()> {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// `tx` genuinely goes unused in one rare-but-valid combination: Windows
|
||||||
|
// features enabled while targeting a non-Windows platform (e.g. sanity-
|
||||||
|
// checking the Windows source arms compile shape from Linux, which is
|
||||||
|
// exactly how these were checked before a real Windows toolchain was
|
||||||
|
// available) collapses every arm to the tx-free `Err(...)` fallback.
|
||||||
|
#[allow(unused_variables)]
|
||||||
async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
|
async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
|
||||||
let result = match source {
|
// Explicit type: with an unusual feature combination (e.g.
|
||||||
#[cfg(feature = "journald")]
|
// windows-eventlog enabled while targeting Linux), every arm below can
|
||||||
|
// collapse to the same untyped `Err(...)` fallback, and Rust can't
|
||||||
|
// infer the Ok type without at least one real `.await` call anywhere
|
||||||
|
// in the compiled match to anchor it.
|
||||||
|
let result: Result<(), anyhow::Error> = match source {
|
||||||
|
#[cfg(all(feature = "journald", target_os = "linux"))]
|
||||||
config::SourceConfig::Journald { unit } => source::journald::run(unit.as_deref(), tx).await,
|
config::SourceConfig::Journald { unit } => source::journald::run(unit.as_deref(), tx).await,
|
||||||
#[cfg(not(feature = "journald"))]
|
#[cfg(not(all(feature = "journald", target_os = "linux")))]
|
||||||
config::SourceConfig::Journald { .. } => {
|
config::SourceConfig::Journald { .. } => {
|
||||||
Err(anyhow::anyhow!("this build was compiled without the `journald` feature"))
|
Err(anyhow::anyhow!("this build was compiled without the `journald` feature (or isn't targeting Linux)"))
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(feature = "file-tail")]
|
#[cfg(feature = "file-tail")]
|
||||||
@@ -104,6 +171,20 @@ async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
|
|||||||
config::SourceConfig::File { .. } => {
|
config::SourceConfig::File { .. } => {
|
||||||
Err(anyhow::anyhow!("this build was compiled without the `file-tail` feature"))
|
Err(anyhow::anyhow!("this build was compiled without the `file-tail` feature"))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(all(feature = "windows-eventlog", target_os = "windows"))]
|
||||||
|
config::SourceConfig::EventLog { channels } => source::windows_eventlog::run(&channels, tx).await,
|
||||||
|
#[cfg(not(all(feature = "windows-eventlog", target_os = "windows")))]
|
||||||
|
config::SourceConfig::EventLog { .. } => {
|
||||||
|
Err(anyhow::anyhow!("this build was compiled without the `windows-eventlog` feature (or isn't targeting Windows)"))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(all(feature = "etw", target_os = "windows"))]
|
||||||
|
config::SourceConfig::Etw { providers } => source::etw::run(&providers, tx).await,
|
||||||
|
#[cfg(not(all(feature = "etw", target_os = "windows")))]
|
||||||
|
config::SourceConfig::Etw { .. } => {
|
||||||
|
Err(anyhow::anyhow!("this build was compiled without the `etw` feature (or isn't targeting Windows)"))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
if let Err(e) = result {
|
if let Err(e) = result {
|
||||||
tracing::error!(error = %e, "log source exited with error");
|
tracing::error!(error = %e, "log source exited with error");
|
||||||
@@ -142,6 +223,7 @@ fn to_pb_severity(sev: Option<u8>) -> Severity {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
fn default_hostname() -> String {
|
fn default_hostname() -> String {
|
||||||
if let Ok(s) = std::fs::read_to_string("/etc/hostname") {
|
if let Ok(s) = std::fs::read_to_string("/etc/hostname") {
|
||||||
let s = s.trim().to_string();
|
let s = s.trim().to_string();
|
||||||
@@ -151,3 +233,11 @@ fn default_hostname() -> String {
|
|||||||
}
|
}
|
||||||
std::env::var("HOSTNAME").unwrap_or_else(|_| "unknown-host".to_string())
|
std::env::var("HOSTNAME").unwrap_or_else(|_| "unknown-host".to_string())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn default_hostname() -> String {
|
||||||
|
// Windows sets this in every process's environment; no Win32 API call
|
||||||
|
// needed (GetComputerNameW would be the "proper" way, but this is the
|
||||||
|
// same value and far simpler).
|
||||||
|
std::env::var("COMPUTERNAME").unwrap_or_else(|_| "unknown-host".to_string())
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,170 @@
|
|||||||
|
//! Windows Service Control Manager integration: install/uninstall the
|
||||||
|
//! agent as a native Windows service, and the SCM-invoked entry point
|
||||||
|
//! that actually runs it as one.
|
||||||
|
//!
|
||||||
|
//! UNVERIFIED, same caveat as source/windows_eventlog.rs and
|
||||||
|
//! source/etw.rs -- written against the `windows-service` crate's
|
||||||
|
//! documented usage pattern, not compiled or run (no Windows toolchain
|
||||||
|
//! available). This one is lower-risk than etw.rs (no raw FFI struct
|
||||||
|
//! layout to get right; `windows-service` wraps that), but the
|
||||||
|
//! stop-signal plumbing between the SCM callback and the tokio-running
|
||||||
|
//! agent thread is new code worth testing carefully.
|
||||||
|
//!
|
||||||
|
//! Known limitation, not addressed here: when running as a service (no
|
||||||
|
//! console attached), `tracing_subscriber::fmt()`'s stdout writer has
|
||||||
|
//! nowhere to go. Logs won't be visible anywhere useful until this is
|
||||||
|
//! redirected to a file or an actual Windows Event Log tracing sink is
|
||||||
|
//! written -- flagging this now rather than shipping it silently broken.
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::ffi::OsString;
|
||||||
|
use std::time::Duration;
|
||||||
|
use windows_service::service::{
|
||||||
|
ServiceAccess, ServiceControl, ServiceControlAccept, ServiceErrorControl, ServiceExitCode,
|
||||||
|
ServiceInfo, ServiceStartType, ServiceState, ServiceStatus, ServiceType,
|
||||||
|
};
|
||||||
|
use windows_service::service_control_handler::{self, ServiceControlHandlerResult};
|
||||||
|
use windows_service::service_manager::{ServiceManager, ServiceManagerAccess};
|
||||||
|
use windows_service::{define_windows_service, service_dispatcher};
|
||||||
|
|
||||||
|
pub const SERVICE_NAME: &str = "SentryAgent";
|
||||||
|
const SERVICE_TYPE: ServiceType = ServiceType::OWN_PROCESS;
|
||||||
|
|
||||||
|
/// Registers this binary as a Windows service: Automatic start,
|
||||||
|
/// LocalSystem account, invoked with the `run-service` subcommand (which
|
||||||
|
/// is what the SCM actually launches — not a bare `sentry-agent` with no
|
||||||
|
/// arguments). Requires an administrator shell.
|
||||||
|
pub fn install() -> Result<()> {
|
||||||
|
let manager = ServiceManager::local_computer(None::<&str>, ServiceManagerAccess::CREATE_SERVICE)
|
||||||
|
.context("opening Service Control Manager")?;
|
||||||
|
|
||||||
|
let exe_path = std::env::current_exe().context("resolving current executable path")?;
|
||||||
|
|
||||||
|
let service_info = ServiceInfo {
|
||||||
|
name: OsString::from(SERVICE_NAME),
|
||||||
|
display_name: OsString::from("Sentry Log Agent"),
|
||||||
|
service_type: SERVICE_TYPE,
|
||||||
|
start_type: ServiceStartType::AutoStart,
|
||||||
|
error_control: ServiceErrorControl::Normal,
|
||||||
|
executable_path: exe_path,
|
||||||
|
launch_arguments: vec![OsString::from("run-service")],
|
||||||
|
dependencies: vec![],
|
||||||
|
account_name: None, // LocalSystem
|
||||||
|
account_password: None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let service = manager
|
||||||
|
.create_service(&service_info, ServiceAccess::CHANGE_CONFIG)
|
||||||
|
.context("creating service")?;
|
||||||
|
service
|
||||||
|
.set_description("Ships local logs to Sentry ingest over mTLS.")
|
||||||
|
.context("setting service description")?;
|
||||||
|
|
||||||
|
tracing::info!(service = SERVICE_NAME, "installed Windows service");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Removes the service registration. Does not stop a currently-running
|
||||||
|
/// instance first — stop it via `services.msc`/`sc.exe stop` before
|
||||||
|
/// uninstalling if it's running.
|
||||||
|
pub fn uninstall() -> Result<()> {
|
||||||
|
let manager = ServiceManager::local_computer(None::<&str>, ServiceManagerAccess::CONNECT)
|
||||||
|
.context("opening Service Control Manager")?;
|
||||||
|
let service = manager
|
||||||
|
.open_service(SERVICE_NAME, ServiceAccess::DELETE)
|
||||||
|
.context("opening service for deletion")?;
|
||||||
|
service.delete().context("deleting service")?;
|
||||||
|
|
||||||
|
tracing::info!(service = SERVICE_NAME, "removed Windows service");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
define_windows_service!(ffi_service_main, service_main);
|
||||||
|
|
||||||
|
/// Blocks, handing control to the SCM dispatch loop -- this is what
|
||||||
|
/// `main()` calls for the `run-service` subcommand, which is what the SCM
|
||||||
|
/// itself launches when the service starts. Must not be called from
|
||||||
|
/// inside a tokio runtime (see the doc comment on `main()` in main.rs).
|
||||||
|
pub fn run_as_service() -> Result<()> {
|
||||||
|
service_dispatcher::start(SERVICE_NAME, ffi_service_main)
|
||||||
|
.context("starting Windows service dispatcher")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn service_main(_arguments: Vec<OsString>) {
|
||||||
|
if let Err(e) = run_service() {
|
||||||
|
// Nowhere better to put this yet -- see the module-level caveat
|
||||||
|
// about tracing having no attached console under the SCM.
|
||||||
|
tracing::error!(error = ?e, "windows service run failed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_service() -> Result<()> {
|
||||||
|
let (shutdown_tx, shutdown_rx) = std::sync::mpsc::channel::<()>();
|
||||||
|
|
||||||
|
let event_handler = move |control_event| -> ServiceControlHandlerResult {
|
||||||
|
match control_event {
|
||||||
|
ServiceControl::Interrogate => ServiceControlHandlerResult::NoError,
|
||||||
|
ServiceControl::Stop => {
|
||||||
|
let _ = shutdown_tx.send(());
|
||||||
|
ServiceControlHandlerResult::NoError
|
||||||
|
}
|
||||||
|
_ => ServiceControlHandlerResult::NotImplemented,
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let status_handle = service_control_handler::register(SERVICE_NAME, event_handler)
|
||||||
|
.context("registering service control handler")?;
|
||||||
|
|
||||||
|
status_handle
|
||||||
|
.set_service_status(ServiceStatus {
|
||||||
|
service_type: SERVICE_TYPE,
|
||||||
|
current_state: ServiceState::Running,
|
||||||
|
controls_accepted: ServiceControlAccept::STOP,
|
||||||
|
exit_code: ServiceExitCode::Win32(0),
|
||||||
|
checkpoint: 0,
|
||||||
|
wait_hint: Duration::default(),
|
||||||
|
process_id: None,
|
||||||
|
})
|
||||||
|
.context("reporting Running status to the SCM")?;
|
||||||
|
|
||||||
|
// service_main is invoked by the SCM on a plain thread, not an async
|
||||||
|
// context -- build a dedicated tokio runtime here and run the actual
|
||||||
|
// agent on it, same `run_agent` entry point a normal foreground run
|
||||||
|
// uses. Block this thread until either the agent exits on its own or
|
||||||
|
// the SCM asks us to stop.
|
||||||
|
let agent_thread = std::thread::spawn(|| {
|
||||||
|
let rt = match tokio::runtime::Runtime::new() {
|
||||||
|
Ok(rt) => rt,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::error!(error = %e, "building tokio runtime for service run");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if let Err(e) = rt.block_on(crate::run_agent(None)) {
|
||||||
|
tracing::error!(error = %e, "agent exited with error while running as a service");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if shutdown_rx.recv_timeout(Duration::from_millis(500)).is_ok() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if agent_thread.is_finished() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
status_handle
|
||||||
|
.set_service_status(ServiceStatus {
|
||||||
|
service_type: SERVICE_TYPE,
|
||||||
|
current_state: ServiceState::Stopped,
|
||||||
|
controls_accepted: ServiceControlAccept::empty(),
|
||||||
|
exit_code: ServiceExitCode::Win32(0),
|
||||||
|
checkpoint: 0,
|
||||||
|
wait_hint: Duration::default(),
|
||||||
|
process_id: None,
|
||||||
|
})
|
||||||
|
.context("reporting Stopped status to the SCM")?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
//! ETW (Event Tracing for Windows) source: a real-time trace session
|
||||||
|
//! subscribed to specific provider GUIDs.
|
||||||
|
//!
|
||||||
|
//! UNVERIFIED, and the highest-risk file in this whole Windows integration
|
||||||
|
//! -- more so than windows_eventlog.rs. `EVENT_TRACE_PROPERTIES` requires
|
||||||
|
//! a variable-length buffer appended after the fixed struct (a classic C
|
||||||
|
//! "flexible array member" pattern for LoggerName), which is exactly the
|
||||||
|
//! kind of FFI layout detail most likely to be subtly wrong without a
|
||||||
|
//! Windows toolchain to actually compile and run this against. No Windows
|
||||||
|
//! target was available in the environment this was written in -- see the
|
||||||
|
//! module-level note in windows_eventlog.rs for what that means. Compile-
|
||||||
|
//! check and test this file specifically, first, before trusting any of
|
||||||
|
//! it.
|
||||||
|
//!
|
||||||
|
//! Providers are configured by **GUID**, not friendly name (e.g.
|
||||||
|
//! `"{22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}"`) -- ETW's own
|
||||||
|
//! `EnableTraceEx2` API takes a GUID, not a name, and there's no simple
|
||||||
|
//! name-to-GUID resolution in the raw ETW API (that needs the separate TDH
|
||||||
|
//! provider-enumeration API, not implemented here). Look up a provider's
|
||||||
|
//! GUID with `logman query providers "<Friendly Name>"`.
|
||||||
|
//!
|
||||||
|
//! Message extraction here is deliberately limited to what's available
|
||||||
|
//! directly on `EVENT_RECORD`'s header (ProviderId, EventID, Level,
|
||||||
|
//! Keywords, timestamp, process/thread ID) -- no TDH-based property
|
||||||
|
//! decoding or message-template rendering (`TdhGetEventInformation`),
|
||||||
|
//! which is a meaningfully larger undertaking left for a follow-up. This
|
||||||
|
//! gives real session/provider/callback plumbing with a coarse message,
|
||||||
|
//! not full structured event decoding.
|
||||||
|
|
||||||
|
use super::{LineSender, RawLine};
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::ffi::c_void;
|
||||||
|
use std::sync::mpsc as std_mpsc;
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
use tokio::sync::mpsc as tokio_mpsc;
|
||||||
|
|
||||||
|
use windows::core::{GUID, PCWSTR};
|
||||||
|
use windows::Win32::System::Diagnostics::Etw::{
|
||||||
|
CloseTrace, ControlTraceW, EnableTraceEx2, OpenTraceW, ProcessTrace, StartTraceW,
|
||||||
|
EVENT_CONTROL_CODE_ENABLE_PROVIDER, EVENT_RECORD, EVENT_TRACE_CONTROL_STOP,
|
||||||
|
EVENT_TRACE_LOGFILEW, EVENT_TRACE_LOGFILEW_0, EVENT_TRACE_LOGFILEW_1,
|
||||||
|
EVENT_TRACE_PROPERTIES, EVENT_TRACE_REAL_TIME_MODE, PROCESS_TRACE_MODE_EVENT_RECORD,
|
||||||
|
PROCESS_TRACE_MODE_REAL_TIME, TRACE_LEVEL_VERBOSE,
|
||||||
|
};
|
||||||
|
|
||||||
|
const SESSION_NAME: &str = "SentryAgentEtw";
|
||||||
|
|
||||||
|
pub async fn run(providers: &[String], tx: LineSender) -> Result<()> {
|
||||||
|
let providers = providers.to_vec();
|
||||||
|
let (blocking_tx, mut blocking_rx) = tokio_mpsc::channel::<RawLine>(256);
|
||||||
|
|
||||||
|
let handle = tokio::task::spawn_blocking(move || run_session(&providers, blocking_tx));
|
||||||
|
|
||||||
|
while let Some(line) = blocking_rx.recv().await {
|
||||||
|
if tx.send(line).await.is_err() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
handle.await.context("ETW session task panicked")??;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn to_wide(s: &str) -> Vec<u16> {
|
||||||
|
s.encode_utf16().chain(std::iter::once(0)).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Thread-local-ish channel used to get the async sender into the
|
||||||
|
/// C-callable `event_record_callback`, which has a fixed extern "system"
|
||||||
|
/// signature and can't capture a closure. Set once per `run_session` call
|
||||||
|
/// before `ProcessTrace` starts invoking the callback.
|
||||||
|
thread_local! {
|
||||||
|
static CALLBACK_TX: std::cell::RefCell<Option<tokio_mpsc::Sender<RawLine>>> =
|
||||||
|
const { std::cell::RefCell::new(None) };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_session(providers: &[String], tx: tokio_mpsc::Sender<RawLine>) -> Result<()> {
|
||||||
|
let guids: Vec<GUID> = providers
|
||||||
|
.iter()
|
||||||
|
.map(|p| GUID::try_from(p.as_str()).with_context(|| format!("invalid provider GUID: {p}")))
|
||||||
|
.collect::<Result<_>>()?;
|
||||||
|
|
||||||
|
unsafe {
|
||||||
|
let session_handle = start_session()?;
|
||||||
|
for guid in &guids {
|
||||||
|
enable_provider(session_handle, guid)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProcessTrace runs the consumer loop on *this* thread until
|
||||||
|
// CloseTrace is called (from the callback, or from another
|
||||||
|
// thread against the same handle) -- there's no separate
|
||||||
|
// "shutdown channel" here because the agent's top-level shutdown
|
||||||
|
// path currently aborts the whole spawn_blocking task rather
|
||||||
|
// than signaling sources to stop gracefully (same as the other
|
||||||
|
// sources today).
|
||||||
|
CALLBACK_TX.with(|cell| *cell.borrow_mut() = Some(tx));
|
||||||
|
|
||||||
|
let mut logfile = EVENT_TRACE_LOGFILEW::default();
|
||||||
|
let mut session_name_wide = to_wide(SESSION_NAME);
|
||||||
|
logfile.LoggerName = PCWSTR(session_name_wide.as_mut_ptr());
|
||||||
|
logfile.Anonymous1 = EVENT_TRACE_LOGFILEW_0 {
|
||||||
|
ProcessTraceMode: PROCESS_TRACE_MODE_REAL_TIME.0 | PROCESS_TRACE_MODE_EVENT_RECORD.0,
|
||||||
|
};
|
||||||
|
logfile.Anonymous2 = EVENT_TRACE_LOGFILEW_1 {
|
||||||
|
EventRecordCallback: Some(event_record_callback),
|
||||||
|
};
|
||||||
|
|
||||||
|
let trace_handle = OpenTraceW(&mut logfile);
|
||||||
|
if trace_handle.0 == u64::MAX as usize {
|
||||||
|
anyhow::bail!("OpenTraceW failed");
|
||||||
|
}
|
||||||
|
|
||||||
|
let result = ProcessTrace(&[trace_handle], None, None);
|
||||||
|
|
||||||
|
let _ = CloseTrace(trace_handle);
|
||||||
|
stop_session(session_handle);
|
||||||
|
|
||||||
|
result.ok().context("ProcessTrace failed")?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
unsafe fn start_session() -> Result<windows::Win32::System::Diagnostics::Etw::CONTROLTRACE_HANDLE> {
|
||||||
|
// EVENT_TRACE_PROPERTIES needs a trailing buffer (appended after the
|
||||||
|
// fixed struct) for the session's LoggerName -- this is the flexible-
|
||||||
|
// array-member pattern flagged in the module doc comment as the
|
||||||
|
// highest-risk detail in this file. LogFileNameOffset is left 0 (no
|
||||||
|
// log file; real-time only).
|
||||||
|
const LOGGER_NAME_CAPACITY: usize = 256;
|
||||||
|
let total_size = std::mem::size_of::<EVENT_TRACE_PROPERTIES>() + LOGGER_NAME_CAPACITY * 2;
|
||||||
|
let mut buffer = vec![0u8; total_size];
|
||||||
|
|
||||||
|
let props = buffer.as_mut_ptr() as *mut EVENT_TRACE_PROPERTIES;
|
||||||
|
(*props).Wnode.BufferSize = total_size as u32;
|
||||||
|
(*props).Wnode.Flags = windows::Win32::System::Diagnostics::Etw::WNODE_FLAG_TRACED_GUID;
|
||||||
|
(*props).LogFileMode = EVENT_TRACE_REAL_TIME_MODE;
|
||||||
|
(*props).LoggerNameOffset = std::mem::size_of::<EVENT_TRACE_PROPERTIES>() as u32;
|
||||||
|
|
||||||
|
let session_name_wide = to_wide(SESSION_NAME);
|
||||||
|
let mut session_handle = Default::default();
|
||||||
|
StartTraceW(
|
||||||
|
&mut session_handle,
|
||||||
|
PCWSTR(session_name_wide.as_ptr()),
|
||||||
|
props,
|
||||||
|
)
|
||||||
|
.ok()
|
||||||
|
.context("StartTraceW failed")?;
|
||||||
|
|
||||||
|
Ok(session_handle)
|
||||||
|
}
|
||||||
|
|
||||||
|
unsafe fn enable_provider(
|
||||||
|
session_handle: windows::Win32::System::Diagnostics::Etw::CONTROLTRACE_HANDLE,
|
||||||
|
guid: &GUID,
|
||||||
|
) -> Result<()> {
|
||||||
|
EnableTraceEx2(
|
||||||
|
session_handle,
|
||||||
|
guid,
|
||||||
|
EVENT_CONTROL_CODE_ENABLE_PROVIDER.0,
|
||||||
|
TRACE_LEVEL_VERBOSE as u8,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
.ok()
|
||||||
|
.with_context(|| format!("EnableTraceEx2 failed for provider {guid:?}"))
|
||||||
|
}
|
||||||
|
|
||||||
|
unsafe fn stop_session(session_handle: windows::Win32::System::Diagnostics::Etw::CONTROLTRACE_HANDLE) {
|
||||||
|
let mut buffer = vec![0u8; std::mem::size_of::<EVENT_TRACE_PROPERTIES>() + 512];
|
||||||
|
let props = buffer.as_mut_ptr() as *mut EVENT_TRACE_PROPERTIES;
|
||||||
|
(*props).Wnode.BufferSize = buffer.len() as u32;
|
||||||
|
let _ = ControlTraceW(session_handle, PCWSTR::null(), props, EVENT_TRACE_CONTROL_STOP);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `extern "system"` callback ETW invokes per event during `ProcessTrace`.
|
||||||
|
/// Deliberately minimal: header fields only, no TDH property decoding
|
||||||
|
/// (see module doc comment).
|
||||||
|
unsafe extern "system" fn event_record_callback(record: *mut EVENT_RECORD) {
|
||||||
|
if record.is_null() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let record = &*record;
|
||||||
|
let header = &record.EventHeader;
|
||||||
|
|
||||||
|
let mut attributes = HashMap::new();
|
||||||
|
attributes.insert(
|
||||||
|
"etw.provider_guid".to_string(),
|
||||||
|
format!("{:?}", header.ProviderId),
|
||||||
|
);
|
||||||
|
attributes.insert("etw.event_id".to_string(), header.EventDescriptor.Id.to_string());
|
||||||
|
attributes.insert(
|
||||||
|
"etw.opcode".to_string(),
|
||||||
|
header.EventDescriptor.Opcode.to_string(),
|
||||||
|
);
|
||||||
|
attributes.insert(
|
||||||
|
"etw.task".to_string(),
|
||||||
|
header.EventDescriptor.Task.to_string(),
|
||||||
|
);
|
||||||
|
attributes.insert("etw.process_id".to_string(), header.ProcessId.to_string());
|
||||||
|
attributes.insert("etw.thread_id".to_string(), header.ThreadId.to_string());
|
||||||
|
|
||||||
|
let severity_hint = etw_level_to_syslog_severity(header.EventDescriptor.Level);
|
||||||
|
|
||||||
|
let timestamp_unix_nano = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_nanos() as i64)
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
|
// No TDH-based message rendering (see module doc comment) -- this is
|
||||||
|
// a coarse, structured summary rather than a human-authored message.
|
||||||
|
// Downstream (sentry_parser's raw-passthrough fallback) handles a
|
||||||
|
// non-RFC5424 line like this the same as any other raw line.
|
||||||
|
let message = format!(
|
||||||
|
"ETW event: provider={:?} id={} level={}",
|
||||||
|
header.ProviderId, header.EventDescriptor.Id, header.EventDescriptor.Level
|
||||||
|
);
|
||||||
|
|
||||||
|
let raw = RawLine {
|
||||||
|
line: message,
|
||||||
|
timestamp_unix_nano,
|
||||||
|
severity_hint,
|
||||||
|
extra_attributes: attributes,
|
||||||
|
};
|
||||||
|
|
||||||
|
CALLBACK_TX.with(|cell| {
|
||||||
|
if let Some(tx) = cell.borrow().as_ref() {
|
||||||
|
let _ = tx.blocking_send(raw);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maps ETW `Level` (0=LogAlways/Verbose-ish through 5=Verbose, following
|
||||||
|
/// the same TRACE_LEVEL_* scale as windows_eventlog's Level values) onto
|
||||||
|
/// the syslog 0-7 scale, same reasoning as windows_eventlog.rs.
|
||||||
|
fn etw_level_to_syslog_severity(level: u8) -> Option<u8> {
|
||||||
|
match level {
|
||||||
|
1 => Some(2), // Critical -> crit
|
||||||
|
2 => Some(3), // Error -> err
|
||||||
|
3 => Some(4), // Warning -> warning
|
||||||
|
4 => Some(6), // Informational -> info
|
||||||
|
5 => Some(7), // Verbose -> debug
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -61,6 +61,7 @@ pub async fn run(path: &Path, from_beginning: bool, tx: LineSender) -> Result<()
|
|||||||
line,
|
line,
|
||||||
timestamp_unix_nano,
|
timestamp_unix_nano,
|
||||||
severity_hint: None,
|
severity_hint: None,
|
||||||
|
extra_attributes: Default::default(),
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
.is_err()
|
.is_err()
|
||||||
|
|||||||
@@ -61,6 +61,7 @@ pub async fn run(unit: Option<&str>, tx: LineSender) -> Result<()> {
|
|||||||
line: message,
|
line: message,
|
||||||
timestamp_unix_nano,
|
timestamp_unix_nano,
|
||||||
severity_hint,
|
severity_hint,
|
||||||
|
extra_attributes: Default::default(),
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
.is_err()
|
.is_err()
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
use tokio::sync::mpsc;
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
/// A raw line read from a source, plus whatever metadata the source itself
|
/// A raw line read from a source, plus whatever metadata the source itself
|
||||||
@@ -8,16 +9,31 @@ pub struct RawLine {
|
|||||||
/// Unix epoch nanoseconds at time of read.
|
/// Unix epoch nanoseconds at time of read.
|
||||||
pub timestamp_unix_nano: i64,
|
pub timestamp_unix_nano: i64,
|
||||||
/// Syslog severity (0-7) if the source already knows it independent of
|
/// Syslog severity (0-7) if the source already knows it independent of
|
||||||
/// the line's own content — e.g. journald's PRIORITY field. When set,
|
/// the line's own content — e.g. journald's PRIORITY field, or a
|
||||||
/// this takes precedence over whatever the RFC 5424 parser infers from
|
/// Windows Event Log Level mapped onto the same scale. When set, this
|
||||||
/// the message text, since it comes from a more authoritative place.
|
/// takes precedence over whatever the RFC 5424 parser infers from the
|
||||||
|
/// message text, since it comes from a more authoritative place.
|
||||||
pub severity_hint: Option<u8>,
|
pub severity_hint: Option<u8>,
|
||||||
|
/// Structured fields the source already knows, independent of the raw
|
||||||
|
/// message text — e.g. Windows Event Log's EventID/Provider/Channel.
|
||||||
|
/// Merged into the record's attributes alongside whatever the RFC 5424
|
||||||
|
/// parser extracts from `line`; on key collision, these win, since
|
||||||
|
/// they also come from a more authoritative place than text parsing.
|
||||||
|
/// Sources that have nothing to add (journald, file-tail) just leave
|
||||||
|
/// this empty.
|
||||||
|
pub extra_attributes: HashMap<String, String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub type LineSender = mpsc::Sender<RawLine>;
|
pub type LineSender = mpsc::Sender<RawLine>;
|
||||||
|
|
||||||
#[cfg(feature = "journald")]
|
#[cfg(all(feature = "journald", target_os = "linux"))]
|
||||||
pub mod journald;
|
pub mod journald;
|
||||||
|
|
||||||
#[cfg(feature = "file-tail")]
|
#[cfg(feature = "file-tail")]
|
||||||
pub mod file_tail;
|
pub mod file_tail;
|
||||||
|
|
||||||
|
#[cfg(all(feature = "windows-eventlog", target_os = "windows"))]
|
||||||
|
pub mod windows_eventlog;
|
||||||
|
|
||||||
|
#[cfg(all(feature = "etw", target_os = "windows"))]
|
||||||
|
pub mod etw;
|
||||||
|
|||||||
@@ -0,0 +1,283 @@
|
|||||||
|
//! Windows Event Log source via `EvtSubscribe`.
|
||||||
|
//!
|
||||||
|
//! UNVERIFIED: this module was written against the documented
|
||||||
|
//! EvtSubscribe/EvtNext/EvtRender API shape (the same pull-model pattern
|
||||||
|
//! Microsoft's own C++ samples use for subscriptions), but has not been
|
||||||
|
//! compiled or run on a real Windows host — no Windows target toolchain
|
||||||
|
//! was available in the environment this was written in (confirmed: only
|
||||||
|
//! x86_64-unknown-linux-gnu std was installed, no rustup, no way to even
|
||||||
|
//! `cargo check --target x86_64-pc-windows-*`). Treat this as a first
|
||||||
|
//! draft to compile-check and test for real before trusting it. See
|
||||||
|
//! /docs/phase-1-runbook.md for what's actually been verified vs. not.
|
||||||
|
|
||||||
|
use super::{LineSender, RawLine};
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
use tokio::sync::mpsc as tokio_mpsc;
|
||||||
|
|
||||||
|
use windows::core::PCWSTR;
|
||||||
|
use windows::Win32::Foundation::{ERROR_NO_MORE_ITEMS, WAIT_OBJECT_0};
|
||||||
|
use windows::Win32::System::EventLog::{
|
||||||
|
EvtClose, EvtNext, EvtRender, EvtRenderEventXml, EvtSubscribe, EVT_HANDLE,
|
||||||
|
EVT_SUBSCRIBE_TO_FUTURE_EVENTS,
|
||||||
|
};
|
||||||
|
use windows::Win32::System::Threading::{CreateEventW, WaitForSingleObject};
|
||||||
|
|
||||||
|
/// Tails one or more Windows Event Log channels. Runs the blocking
|
||||||
|
/// EvtSubscribe/EvtNext calls on a dedicated OS thread per channel (via
|
||||||
|
/// `spawn_blocking`) and forwards parsed lines back over `tx`, same shape
|
||||||
|
/// as the journald source's subprocess-reading loop.
|
||||||
|
pub async fn run(channels: &[String], tx: LineSender) -> Result<()> {
|
||||||
|
let channels = channels.to_vec();
|
||||||
|
let (blocking_tx, mut blocking_rx) = tokio_mpsc::channel::<RawLine>(256);
|
||||||
|
|
||||||
|
let handle = tokio::task::spawn_blocking(move || subscribe_all(&channels, blocking_tx));
|
||||||
|
|
||||||
|
while let Some(line) = blocking_rx.recv().await {
|
||||||
|
if tx.send(line).await.is_err() {
|
||||||
|
break; // receiver dropped, agent is shutting down
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
handle
|
||||||
|
.await
|
||||||
|
.context("windows event log subscription task panicked")??;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One `std::thread` per channel, each blocked in its own
|
||||||
|
/// wait-then-drain loop. Simpler and still correct for the common case of
|
||||||
|
/// 1-3 channels; a single `WaitForMultipleObjects`-based dispatcher would
|
||||||
|
/// scale better to many channels but isn't needed for Phase 1's default
|
||||||
|
/// three (Application/System/Security).
|
||||||
|
fn subscribe_all(channels: &[String], tx: tokio_mpsc::Sender<RawLine>) -> Result<()> {
|
||||||
|
let mut threads = Vec::with_capacity(channels.len());
|
||||||
|
for channel in channels {
|
||||||
|
let channel = channel.clone();
|
||||||
|
let tx = tx.clone();
|
||||||
|
threads.push(std::thread::spawn(move || subscribe_one(&channel, tx)));
|
||||||
|
}
|
||||||
|
for t in threads {
|
||||||
|
t.join()
|
||||||
|
.map_err(|_| anyhow::anyhow!("event log subscriber thread panicked"))??;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn to_wide(s: &str) -> Vec<u16> {
|
||||||
|
s.encode_utf16().chain(std::iter::once(0)).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn subscribe_one(channel: &str, tx: tokio_mpsc::Sender<RawLine>) -> Result<()> {
|
||||||
|
unsafe {
|
||||||
|
let signal_event = CreateEventW(None, true, false, None)
|
||||||
|
.context("CreateEventW for subscription signal failed")?;
|
||||||
|
|
||||||
|
let channel_wide = to_wide(channel);
|
||||||
|
// NULL query (PCWSTR::null()) means "all events on this channel".
|
||||||
|
// No callback (None) -- pull model via the signal event instead,
|
||||||
|
// so this stays a plain loop rather than a Win32 callback that
|
||||||
|
// would need to cross back into the tokio runtime.
|
||||||
|
let subscription = EvtSubscribe(
|
||||||
|
None,
|
||||||
|
Some(signal_event),
|
||||||
|
PCWSTR(channel_wide.as_ptr()),
|
||||||
|
PCWSTR::null(),
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
EVT_SUBSCRIBE_TO_FUTURE_EVENTS.0,
|
||||||
|
)
|
||||||
|
.context("EvtSubscribe failed")?;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let wait = WaitForSingleObject(signal_event, u32::MAX);
|
||||||
|
if wait != WAIT_OBJECT_0 {
|
||||||
|
anyhow::bail!("WaitForSingleObject on event log subscription failed");
|
||||||
|
}
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let mut events: [EVT_HANDLE; 16] = [EVT_HANDLE::default(); 16];
|
||||||
|
let mut returned = 0u32;
|
||||||
|
let next = EvtNext(subscription, &mut events, 0, 0, &mut returned);
|
||||||
|
if let Err(err) = next {
|
||||||
|
if err.code() == ERROR_NO_MORE_ITEMS.into() {
|
||||||
|
break; // drained this batch; go back to waiting on the signal
|
||||||
|
}
|
||||||
|
return Err(err).context("EvtNext failed");
|
||||||
|
}
|
||||||
|
|
||||||
|
for &event in &events[..returned as usize] {
|
||||||
|
if let Some(raw) = render_event(event, channel) {
|
||||||
|
if tx.blocking_send(raw).is_err() {
|
||||||
|
let _ = EvtClose(event);
|
||||||
|
let _ = EvtClose(subscription);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let _ = EvtClose(event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn render_event(event: EVT_HANDLE, channel: &str) -> Option<RawLine> {
|
||||||
|
unsafe {
|
||||||
|
let mut buffer_used = 0u32;
|
||||||
|
let mut property_count = 0u32;
|
||||||
|
// First call with a zero-length buffer to learn the required size.
|
||||||
|
let _ = EvtRender(
|
||||||
|
None,
|
||||||
|
event,
|
||||||
|
EvtRenderEventXml.0,
|
||||||
|
0,
|
||||||
|
None,
|
||||||
|
&mut buffer_used,
|
||||||
|
&mut property_count,
|
||||||
|
);
|
||||||
|
if buffer_used == 0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut buffer = vec![0u16; (buffer_used as usize).div_ceil(2)];
|
||||||
|
let rendered = EvtRender(
|
||||||
|
None,
|
||||||
|
event,
|
||||||
|
EvtRenderEventXml.0,
|
||||||
|
buffer_used,
|
||||||
|
Some(buffer.as_mut_ptr() as *mut _),
|
||||||
|
&mut buffer_used,
|
||||||
|
&mut property_count,
|
||||||
|
);
|
||||||
|
if rendered.is_err() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
let xml = String::from_utf16_lossy(&buffer);
|
||||||
|
let xml = xml.trim_end_matches('\0');
|
||||||
|
parse_event_xml(xml, channel)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Minimal, deliberately non-validating extraction of the fields Phase 1
|
||||||
|
/// needs from the rendered event XML: EventID, Provider, Level, Computer,
|
||||||
|
/// Windows' own EventRecordID, and a best-effort message. Not a full XML
|
||||||
|
/// parser in the schema-aware sense -- uses `quick-xml`'s streaming
|
||||||
|
/// reader to pull specific elements/attributes rather than hand-rolled
|
||||||
|
/// string search, but doesn't attempt full EventData/UserData schema
|
||||||
|
/// awareness across every provider's custom shape. Worth revisiting once
|
||||||
|
/// this is running against real events from real providers.
|
||||||
|
fn parse_event_xml(xml: &str, channel: &str) -> Option<RawLine> {
|
||||||
|
use quick_xml::events::Event as XmlEvent;
|
||||||
|
use quick_xml::reader::Reader;
|
||||||
|
|
||||||
|
let mut reader = Reader::from_str(xml);
|
||||||
|
reader.config_mut().trim_text(true);
|
||||||
|
|
||||||
|
let mut event_id = None;
|
||||||
|
let mut provider = None;
|
||||||
|
let mut level = None;
|
||||||
|
let mut computer = None;
|
||||||
|
let mut record_id = None;
|
||||||
|
let mut event_data_values: Vec<String> = Vec::new();
|
||||||
|
|
||||||
|
let mut current_tag: Option<String> = None;
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
|
||||||
|
loop {
|
||||||
|
match reader.read_event_into(&mut buf) {
|
||||||
|
Ok(XmlEvent::Start(e)) | Ok(XmlEvent::Empty(e)) => {
|
||||||
|
let name = local_name(&e);
|
||||||
|
if name == "Provider" {
|
||||||
|
for attr in e.attributes().flatten() {
|
||||||
|
if attr.key.as_ref() == b"Name" {
|
||||||
|
provider = attr
|
||||||
|
.decode_and_unescape_value(reader.decoder())
|
||||||
|
.ok()
|
||||||
|
.map(|v| v.into_owned());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
current_tag = Some(name);
|
||||||
|
}
|
||||||
|
Ok(XmlEvent::Text(t)) => {
|
||||||
|
let text = t.unescape().unwrap_or_default().into_owned();
|
||||||
|
match current_tag.as_deref() {
|
||||||
|
Some("EventID") => event_id = Some(text),
|
||||||
|
Some("Level") => level = text.parse::<u8>().ok(),
|
||||||
|
Some("Computer") => computer = Some(text),
|
||||||
|
Some("EventRecordID") => record_id = Some(text),
|
||||||
|
Some("Data") => event_data_values.push(text),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(XmlEvent::Eof) => break,
|
||||||
|
Err(_) => return None,
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
// <EventData> commonly holds one or more <Data Name="...">value</Data>
|
||||||
|
// elements rather than a single free-text message; joining them is a
|
||||||
|
// reasonable Phase 1 default until per-provider message templates are
|
||||||
|
// rendered properly. Real message-template rendering needs
|
||||||
|
// EvtFormatMessage against the provider's message-table resource --
|
||||||
|
// worth a follow-up, not required for a raw-passthrough-shaped record
|
||||||
|
// (sentry_parser's raw fallback handles this fine either way).
|
||||||
|
let message = if event_data_values.is_empty() {
|
||||||
|
xml.to_string()
|
||||||
|
} else {
|
||||||
|
event_data_values.join(" | ")
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut attributes = HashMap::new();
|
||||||
|
if let Some(id) = &event_id {
|
||||||
|
attributes.insert("winevt.event_id".to_string(), id.clone());
|
||||||
|
}
|
||||||
|
if let Some(p) = provider {
|
||||||
|
attributes.insert("winevt.provider".to_string(), p);
|
||||||
|
}
|
||||||
|
attributes.insert("winevt.channel".to_string(), channel.to_string());
|
||||||
|
if let Some(c) = computer {
|
||||||
|
attributes.insert("winevt.computer".to_string(), c);
|
||||||
|
}
|
||||||
|
if let Some(r) = record_id {
|
||||||
|
attributes.insert("winevt.record_number".to_string(), r);
|
||||||
|
}
|
||||||
|
|
||||||
|
let severity_hint = level.and_then(windows_level_to_syslog_severity);
|
||||||
|
|
||||||
|
let timestamp_unix_nano = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_nanos() as i64)
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
|
Some(RawLine {
|
||||||
|
line: message,
|
||||||
|
timestamp_unix_nano,
|
||||||
|
severity_hint,
|
||||||
|
extra_attributes: attributes,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_name(e: &quick_xml::events::BytesStart) -> String {
|
||||||
|
String::from_utf8_lossy(e.local_name().as_ref()).into_owned()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maps Windows Event Log `Level` values (0=LogAlways, 1=Critical,
|
||||||
|
/// 2=Error, 3=Warning, 4=Informational, 5=Verbose) onto the same syslog
|
||||||
|
/// 0-7 severity scale `severity_hint` uses everywhere else in the agent,
|
||||||
|
/// so `main.rs`'s `to_pb_severity` needs no Windows-specific knowledge.
|
||||||
|
fn windows_level_to_syslog_severity(level: u8) -> Option<u8> {
|
||||||
|
match level {
|
||||||
|
1 => Some(2), // Critical -> crit
|
||||||
|
2 => Some(3), // Error -> err
|
||||||
|
3 => Some(4), // Warning -> warning
|
||||||
|
4 => Some(6), // Informational -> info
|
||||||
|
5 => Some(7), // Verbose -> debug
|
||||||
|
_ => None, // 0 (LogAlways) or unrecognized -- let the parser decide
|
||||||
|
}
|
||||||
|
}
|
||||||
+4
-1
@@ -1,8 +1,11 @@
|
|||||||
# Build context must be the repo root (sentry/):
|
# Build context must be the repo root (sentry/), since this needs both
|
||||||
|
# api/ and proto/ (api now speaks gRPC to /search, using proto's checked-in
|
||||||
|
# Go bindings via the `replace` directive in api/go.mod):
|
||||||
# docker build -f api/Dockerfile -t sentry-api .
|
# docker build -f api/Dockerfile -t sentry-api .
|
||||||
|
|
||||||
FROM golang:1.25-alpine AS builder
|
FROM golang:1.25-alpine AS builder
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
COPY proto ./proto
|
||||||
COPY api ./api
|
COPY api ./api
|
||||||
WORKDIR /src/api
|
WORKDIR /src/api
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
|
|||||||
+41
-19
@@ -1,17 +1,19 @@
|
|||||||
# api
|
# api
|
||||||
|
|
||||||
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
|
Sentry's query API: two intentionally crude endpoints — raw SQL and
|
||||||
|
free-text search — that Phase 2's real query layer replaces outright.
|
||||||
|
|
||||||
## Why plain REST, not gRPC + REST gateway
|
## Why plain REST, not gRPC + REST gateway
|
||||||
|
|
||||||
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
||||||
service is plain `net/http` instead — a deliberate Phase 0 simplification,
|
service is plain `net/http` instead — a deliberate simplification, not a
|
||||||
not a change to the pinned stack. Wiring up a `.proto` service,
|
change to the pinned stack. Wiring up `.proto` services,
|
||||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
|
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
||||||
single endpoint that Phase 2 replaces outright with a real SPL-like query
|
two endpoints that Phase 2 replaces outright with a real SPL-like query
|
||||||
layer would be exactly the kind of premature machinery this project's
|
layer would be exactly the kind of premature machinery this project's
|
||||||
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
|
conventions warn against. `api` *does* speak gRPC internally though — to
|
||||||
a second real, durable endpoint.
|
`/search` (see below) — this simplification is specifically about the
|
||||||
|
public-facing surface, not a blanket avoidance of gRPC.
|
||||||
|
|
||||||
## Endpoints
|
## Endpoints
|
||||||
|
|
||||||
@@ -20,10 +22,21 @@ a second real, durable endpoint.
|
|||||||
SELECT-only, single-statement, basic keyword-based injection guarding
|
SELECT-only, single-statement, basic keyword-based injection guarding
|
||||||
(see `internal/queryapi/validate.go` for exactly what that does and
|
(see `internal/queryapi/validate.go` for exactly what that does and
|
||||||
doesn't catch — it's not a SQL parser).
|
doesn't catch — it's not a SQL parser).
|
||||||
|
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
|
||||||
|
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
|
||||||
|
resolve the free-text query into matching `record_id`s, then joins
|
||||||
|
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
|
||||||
|
(...)`) to return full rows — so both endpoints return the same
|
||||||
|
`{columns, rows}` shape and `/web` can reuse one table component for
|
||||||
|
both. Every `record_id` is validated as a real UUID before being
|
||||||
|
embedded in the generated SQL (defense in depth: `record_id`s come from
|
||||||
|
an internal, trusted service, not raw user input, but a value that
|
||||||
|
fails to parse as a UUID can't contain SQL-breaking characters either
|
||||||
|
way).
|
||||||
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
||||||
|
|
||||||
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
|
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
||||||
dev/homelab network.
|
network.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
@@ -34,9 +47,15 @@ Environment variables (see `internal/config/config.go`):
|
|||||||
| `HTTP_LISTEN_ADDR` | `:8080` | |
|
| `HTTP_LISTEN_ADDR` | `:8080` | |
|
||||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
||||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
|
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
||||||
|
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
|
||||||
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
||||||
|
|
||||||
|
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
||||||
|
trust boundary as `api`'s existing plain-TCP connection to ClickHouse.
|
||||||
|
mTLS in this project is specifically the agent↔ingest edge boundary, not
|
||||||
|
every internal hop.
|
||||||
|
|
||||||
## Building & testing
|
## Building & testing
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -52,12 +71,15 @@ docker build -f api/Dockerfile -t sentry-api .
|
|||||||
|
|
||||||
## Testing notes
|
## Testing notes
|
||||||
|
|
||||||
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
|
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
|
||||||
one-method `queryExecutor` interface, so routing, validation, JSON
|
only through narrow interfaces (`queryExecutor`, `searchClient`), so
|
||||||
encoding, and error-status mapping are all unit-tested against a fake —
|
routing, validation, JSON encoding, error-status mapping, and the
|
||||||
no live ClickHouse needed. `Executor` itself (the reflection-based row
|
record_id-to-SQL query building are all unit-tested against fakes — no
|
||||||
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
|
live ClickHouse or `/search` instance needed. `Executor` itself (the
|
||||||
`driver.Rows` interface fully would be significant test-only scaffolding
|
reflection-based row scanning against `driver.Rows`) and
|
||||||
for a Phase 0 placeholder, and the driver package's own docs note it isn't
|
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
||||||
meant to be implemented by adopters. It's exercised end-to-end via the
|
former because faking ClickHouse's `driver.Rows` interface fully would be
|
||||||
docker-compose flow in `/docs/phase-0-runbook.md` instead.
|
significant test-only scaffolding the driver's own docs say isn't meant
|
||||||
|
to be implemented by adopters; the latter because it's a thin wrapper
|
||||||
|
with nothing but wiring to test. Both are exercised end-to-end via the
|
||||||
|
docker-compose flow in `/docs/phase-1-runbook.md` instead.
|
||||||
|
|||||||
+13
-5
@@ -1,7 +1,7 @@
|
|||||||
// Command api is the Sentry Phase 0 query API: a single crude POST /query
|
// Command api is Sentry's query API: POST /query (raw SQL, SELECT-only)
|
||||||
// endpoint proxying allowlisted SELECT statements to ClickHouse. See
|
// and POST /search (free-text, via the search service). See
|
||||||
// internal/queryapi for why this is plain REST rather than the pinned
|
// internal/queryapi for why these are plain REST rather than the pinned
|
||||||
// gRPC+gateway pattern for Phase 0.
|
// gRPC+gateway pattern.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -17,6 +17,7 @@ import (
|
|||||||
|
|
||||||
"github.com/sentry/sentry/api/internal/config"
|
"github.com/sentry/sentry/api/internal/config"
|
||||||
"github.com/sentry/sentry/api/internal/queryapi"
|
"github.com/sentry/sentry/api/internal/queryapi"
|
||||||
|
"github.com/sentry/sentry/api/internal/searchclient"
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
@@ -50,8 +51,15 @@ func main() {
|
|||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
search, err := searchclient.Dial(cfg.SearchGRPCAddr)
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("dialing search service", "error", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer search.Close()
|
||||||
|
|
||||||
exec := queryapi.NewExecutor(conn)
|
exec := queryapi.NewExecutor(conn)
|
||||||
handler := queryapi.NewHandler(logger, exec, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
|
handler := queryapi.NewHandler(logger, exec, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
|
||||||
|
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
Addr: cfg.HTTPListenAddr,
|
Addr: cfg.HTTPListenAddr,
|
||||||
|
|||||||
+12
-2
@@ -2,7 +2,14 @@ module github.com/sentry/sentry/api
|
|||||||
|
|
||||||
go 1.25.0
|
go 1.25.0
|
||||||
|
|
||||||
require github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
require (
|
||||||
|
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
||||||
|
github.com/google/uuid v1.6.0
|
||||||
|
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||||
|
google.golang.org/grpc v1.83.0
|
||||||
|
)
|
||||||
|
|
||||||
|
replace github.com/sentry/sentry/proto => ../proto
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
||||||
@@ -10,7 +17,6 @@ require (
|
|||||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||||
github.com/go-faster/city v1.0.1 // indirect
|
github.com/go-faster/city v1.0.1 // indirect
|
||||||
github.com/go-faster/errors v0.7.1 // indirect
|
github.com/go-faster/errors v0.7.1 // indirect
|
||||||
github.com/google/uuid v1.6.0 // indirect
|
|
||||||
github.com/klauspost/compress v1.19.1 // indirect
|
github.com/klauspost/compress v1.19.1 // indirect
|
||||||
github.com/paulmach/orb v0.13.0 // indirect
|
github.com/paulmach/orb v0.13.0 // indirect
|
||||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||||
@@ -18,5 +24,9 @@ require (
|
|||||||
github.com/shopspring/decimal v1.4.0 // indirect
|
github.com/shopspring/decimal v1.4.0 // indirect
|
||||||
go.opentelemetry.io/otel v1.44.0 // indirect
|
go.opentelemetry.io/otel v1.44.0 // indirect
|
||||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||||
|
golang.org/x/net v0.57.0 // indirect
|
||||||
golang.org/x/sys v0.47.0 // indirect
|
golang.org/x/sys v0.47.0 // indirect
|
||||||
|
golang.org/x/text v0.40.0 // indirect
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||||
|
google.golang.org/protobuf v1.36.12 // indirect
|
||||||
)
|
)
|
||||||
|
|||||||
+26
@@ -12,6 +12,12 @@ github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
|
|||||||
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
|
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
|
||||||
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
||||||
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||||
|
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||||
|
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||||
|
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||||
|
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||||
|
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||||
|
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
@@ -32,11 +38,31 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
|
|||||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||||
|
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||||
|
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
|
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||||
|
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||||
|
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||||
|
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||||
|
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||||
|
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||||
|
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||||
|
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
type Config struct {
|
type Config struct {
|
||||||
HTTPListenAddr string
|
HTTPListenAddr string
|
||||||
ClickHouse ClickHouseConfig
|
ClickHouse ClickHouseConfig
|
||||||
|
SearchGRPCAddr string
|
||||||
QueryTimeout time.Duration
|
QueryTimeout time.Duration
|
||||||
CORSAllowedOrigin string
|
CORSAllowedOrigin string
|
||||||
}
|
}
|
||||||
@@ -32,6 +33,9 @@ func Load() (Config, error) {
|
|||||||
Username: getenv("CLICKHOUSE_USERNAME", "default"),
|
Username: getenv("CLICKHOUSE_USERNAME", "default"),
|
||||||
Password: getenv("CLICKHOUSE_PASSWORD", ""),
|
Password: getenv("CLICKHOUSE_PASSWORD", ""),
|
||||||
},
|
},
|
||||||
|
// Search service's gRPC address (see /search) -- default matches
|
||||||
|
// /search's own default GRPC_LISTEN_ADDR.
|
||||||
|
SearchGRPCAddr: getenv("SEARCH_GRPC_ADDR", "localhost:50052"),
|
||||||
// Phase 0 has no auth, so this is wide open by default to keep
|
// Phase 0 has no auth, so this is wide open by default to keep
|
||||||
// the local SvelteKit dev server (a different origin/port)
|
// the local SvelteKit dev server (a different origin/port)
|
||||||
// working out of the box. Tighten before this is ever reachable
|
// working out of the box. Tighten before this is ever reachable
|
||||||
|
|||||||
@@ -19,6 +19,9 @@ func TestLoadDefaults(t *testing.T) {
|
|||||||
if cfg.CORSAllowedOrigin != "*" {
|
if cfg.CORSAllowedOrigin != "*" {
|
||||||
t.Errorf("CORSAllowedOrigin = %q, want *", cfg.CORSAllowedOrigin)
|
t.Errorf("CORSAllowedOrigin = %q, want *", cfg.CORSAllowedOrigin)
|
||||||
}
|
}
|
||||||
|
if cfg.SearchGRPCAddr != "localhost:50052" {
|
||||||
|
t.Errorf("SearchGRPCAddr = %q, want localhost:50052", cfg.SearchGRPCAddr)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoadInvalidTimeoutErrors(t *testing.T) {
|
func TestLoadInvalidTimeoutErrors(t *testing.T) {
|
||||||
|
|||||||
@@ -1,13 +1,13 @@
|
|||||||
// Package queryapi is the Phase 0 query API: a single crude POST /query
|
// Package queryapi is Sentry's query API: POST /query (Phase 0, a crude
|
||||||
// endpoint that takes a raw SQL string, allowlists it to a single SELECT
|
// raw-SQL passthrough allowlisted to SELECT) and POST /search (Phase 1,
|
||||||
// statement, and proxies it to ClickHouse. This is a deliberate
|
// free-text search via the search service, joined back against
|
||||||
// simplification of the pinned "gRPC + REST gateway" control-plane
|
// ClickHouse). This is a deliberate simplification of the pinned "gRPC +
|
||||||
// pattern (see CLAUDE.md's tech stack table): a plain net/http REST
|
// REST gateway" control-plane pattern (see CLAUDE.md's tech stack table):
|
||||||
// handler, not a gRPC service transcoded through grpc-gateway. That
|
// plain net/http REST handlers, not a gRPC service transcoded through
|
||||||
// machinery (proto definitions, googleapis annotations, gateway codegen)
|
// grpc-gateway. That machinery (proto definitions, googleapis
|
||||||
// buys nothing for one crude placeholder endpoint that Phase 2 replaces
|
// annotations, gateway codegen) doesn't buy much for two crude endpoints
|
||||||
// outright with the real SPL-like query layer. Revisit gRPC+gateway when
|
// that Phase 2's real SPL-like query layer replaces outright. Revisit
|
||||||
// /api grows a second real endpoint.
|
// gRPC+gateway once /api's endpoint count and lifespan justify it.
|
||||||
package queryapi
|
package queryapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -28,17 +28,19 @@ type queryExecutor interface {
|
|||||||
type Handler struct {
|
type Handler struct {
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
exec queryExecutor
|
exec queryExecutor
|
||||||
|
search searchClient
|
||||||
queryTimeout time.Duration
|
queryTimeout time.Duration
|
||||||
allowedOrigin string
|
allowedOrigin string
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewHandler(logger *slog.Logger, exec queryExecutor, queryTimeout time.Duration, allowedOrigin string) *Handler {
|
func NewHandler(logger *slog.Logger, exec queryExecutor, search searchClient, queryTimeout time.Duration, allowedOrigin string) *Handler {
|
||||||
return &Handler{logger: logger, exec: exec, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
|
return &Handler{logger: logger, exec: exec, search: search, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *Handler) Routes() http.Handler {
|
func (h *Handler) Routes() http.Handler {
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.HandleFunc("POST /query", h.handleQuery)
|
mux.HandleFunc("POST /query", h.handleQuery)
|
||||||
|
mux.HandleFunc("POST /search", h.handleSearch)
|
||||||
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
||||||
return h.withCORS(mux)
|
return h.withCORS(mux)
|
||||||
}
|
}
|
||||||
@@ -99,10 +101,7 @@ func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
writeJSON(w, result)
|
||||||
if err := json.NewEncoder(w).Encode(result); err != nil {
|
|
||||||
h.logger.Error("encoding response", "error", err)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||||
|
|||||||
@@ -27,8 +27,24 @@ func (f *fakeExecutor) Execute(_ context.Context, sql string) (*QueryResult, err
|
|||||||
return f.result, nil
|
return f.result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fakeSearchClient struct {
|
||||||
|
recordIDs []string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSearchClient) Search(_ context.Context, _ string, _ uint32) ([]string, error) {
|
||||||
|
if f.err != nil {
|
||||||
|
return nil, f.err
|
||||||
|
}
|
||||||
|
return f.recordIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
func newTestHandler(exec queryExecutor) *Handler {
|
func newTestHandler(exec queryExecutor) *Handler {
|
||||||
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, time.Second, "*")
|
return newTestHandlerWithSearch(exec, &fakeSearchClient{})
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestHandlerWithSearch(exec queryExecutor, search searchClient) *Handler {
|
||||||
|
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, search, time.Second, "*")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandleQuerySuccess(t *testing.T) {
|
func TestHandleQuerySuccess(t *testing.T) {
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
package queryapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// searchClient is the narrow interface handleSearch depends on, so tests
|
||||||
|
// can substitute a fake without a real search service. A small gRPC
|
||||||
|
// adapter in cmd/api satisfies this.
|
||||||
|
type searchClient interface {
|
||||||
|
Search(ctx context.Context, query string, limit uint32) ([]string, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type searchRequest struct {
|
||||||
|
Query string `json:"query"`
|
||||||
|
Limit uint32 `json:"limit"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) handleSearch(w http.ResponseWriter, r *http.Request) {
|
||||||
|
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
|
||||||
|
|
||||||
|
var req searchRequest
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||||
|
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Query) == "" {
|
||||||
|
writeError(w, http.StatusBadRequest, "query must not be empty")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
recordIDs, err := h.search.Search(ctx, req.Query, req.Limit)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("search failed", "error", err)
|
||||||
|
writeError(w, http.StatusBadGateway, "search failed: "+err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(recordIDs) == 0 {
|
||||||
|
writeJSON(w, &QueryResult{Columns: []string{}, Rows: [][]any{}})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
sql, err := recordIDsQuery(recordIDs)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("building record_id query", "error", err)
|
||||||
|
writeError(w, http.StatusBadGateway, "search returned unusable results")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := h.exec.Execute(ctx, sql)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("joining search results against clickhouse failed", "error", err)
|
||||||
|
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, result)
|
||||||
|
}
|
||||||
|
|
||||||
|
// recordIDsQuery builds a SELECT ... WHERE record_id IN (...) against the
|
||||||
|
// IDs the search service returned. Every ID is validated as a real UUID
|
||||||
|
// before being embedded in the query string -- record_ids come from an
|
||||||
|
// internal, trusted service (not raw user input), but a UUID that fails
|
||||||
|
// to parse can't contain SQL-breaking characters either way, so this is
|
||||||
|
// defense in depth, not a response to a specific threat.
|
||||||
|
func recordIDsQuery(recordIDs []string) (string, error) {
|
||||||
|
quoted := make([]string, 0, len(recordIDs))
|
||||||
|
for _, id := range recordIDs {
|
||||||
|
if _, err := uuid.Parse(id); err != nil {
|
||||||
|
continue // skip anything not a valid UUID rather than failing the whole query
|
||||||
|
}
|
||||||
|
quoted = append(quoted, "'"+id+"'")
|
||||||
|
}
|
||||||
|
if len(quoted) == 0 {
|
||||||
|
return "", fmt.Errorf("no valid record_ids in search response")
|
||||||
|
}
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"SELECT * FROM logs WHERE record_id IN (%s) ORDER BY timestamp DESC",
|
||||||
|
strings.Join(quoted, ","),
|
||||||
|
), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(w http.ResponseWriter, v any) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(v)
|
||||||
|
}
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
package queryapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestHandleSearchSuccess(t *testing.T) {
|
||||||
|
id := "5754b062-ec8b-45b1-b1b8-a50f263adcd3"
|
||||||
|
fe := &fakeExecutor{result: &QueryResult{
|
||||||
|
Columns: []string{"message"},
|
||||||
|
Rows: [][]any{{"hello world"}},
|
||||||
|
}}
|
||||||
|
fs := &fakeSearchClient{recordIDs: []string{id}}
|
||||||
|
h := newTestHandlerWithSearch(fe, fs)
|
||||||
|
|
||||||
|
body := strings.NewReader(`{"query": "hello"}`)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.Routes().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(fe.gotSQL, id) {
|
||||||
|
t.Fatalf("expected the record_id in the generated SQL, got %q", fe.gotSQL)
|
||||||
|
}
|
||||||
|
if !strings.Contains(fe.gotSQL, "WHERE record_id IN") {
|
||||||
|
t.Fatalf("expected an IN clause, got %q", fe.gotSQL)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got QueryResult
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||||
|
t.Fatalf("decoding response: %v", err)
|
||||||
|
}
|
||||||
|
if len(got.Rows) != 1 {
|
||||||
|
t.Fatalf("unexpected result: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleSearchRejectsEmptyQuery(t *testing.T) {
|
||||||
|
fe := &fakeExecutor{}
|
||||||
|
fs := &fakeSearchClient{}
|
||||||
|
h := newTestHandlerWithSearch(fe, fs)
|
||||||
|
|
||||||
|
body := strings.NewReader(`{"query": " "}`)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.Routes().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("status = %d, want 400", rec.Code)
|
||||||
|
}
|
||||||
|
if fe.gotSQL != "" {
|
||||||
|
t.Fatal("executor should not have been called for an empty query")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleSearchNoResultsReturnsEmptyNotError(t *testing.T) {
|
||||||
|
fe := &fakeExecutor{}
|
||||||
|
fs := &fakeSearchClient{recordIDs: nil}
|
||||||
|
h := newTestHandlerWithSearch(fe, fs)
|
||||||
|
|
||||||
|
body := strings.NewReader(`{"query": "nothing matches this"}`)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.Routes().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if fe.gotSQL != "" {
|
||||||
|
t.Fatal("executor should not have been called when search returns no IDs")
|
||||||
|
}
|
||||||
|
|
||||||
|
var got QueryResult
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||||
|
t.Fatalf("decoding response: %v", err)
|
||||||
|
}
|
||||||
|
if len(got.Rows) != 0 {
|
||||||
|
t.Fatalf("expected empty rows, got %+v", got.Rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleSearchServiceErrorReturnsBadGateway(t *testing.T) {
|
||||||
|
fe := &fakeExecutor{}
|
||||||
|
fs := &fakeSearchClient{err: errors.New("search service unreachable")}
|
||||||
|
h := newTestHandlerWithSearch(fe, fs)
|
||||||
|
|
||||||
|
body := strings.NewReader(`{"query": "hello"}`)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.Routes().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusBadGateway {
|
||||||
|
t.Fatalf("status = %d, want 502", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRecordIDsQuerySkipsInvalidUUIDs(t *testing.T) {
|
||||||
|
sql, err := recordIDsQuery([]string{"not-a-uuid", "5754b062-ec8b-45b1-b1b8-a50f263adcd3"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("recordIDsQuery() error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(sql, "not-a-uuid") {
|
||||||
|
t.Fatalf("expected the invalid UUID to be skipped, got %q", sql)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sql, "5754b062-ec8b-45b1-b1b8-a50f263adcd3") {
|
||||||
|
t.Fatalf("expected the valid UUID to be included, got %q", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRecordIDsQueryAllInvalidReturnsError(t *testing.T) {
|
||||||
|
if _, err := recordIDsQuery([]string{"not-a-uuid", "also-not-one"}); err == nil {
|
||||||
|
t.Fatal("expected an error when no IDs are valid UUIDs")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
// Package searchclient adapts the generated gRPC SearchServiceClient to
|
||||||
|
// the narrow queryapi.searchClient interface, so queryapi doesn't need to
|
||||||
|
// know anything about gRPC/protobuf directly.
|
||||||
|
package searchclient
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"google.golang.org/grpc"
|
||||||
|
"google.golang.org/grpc/credentials/insecure"
|
||||||
|
|
||||||
|
searchv1 "github.com/sentry/sentry/proto/sentry/search/v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Client struct {
|
||||||
|
grpc searchv1.SearchServiceClient
|
||||||
|
conn *grpc.ClientConn
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dial connects to the search service. Plain TCP, no TLS: internal
|
||||||
|
// service-to-service traffic (api <-> search), same trust boundary as
|
||||||
|
// api's existing plain-TCP connection to ClickHouse -- mTLS in this
|
||||||
|
// project is specifically the agent<->ingest edge boundary, not every
|
||||||
|
// internal hop.
|
||||||
|
func Dial(addr string) (*Client, error) {
|
||||||
|
conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("dialing search service at %s: %w", addr, err)
|
||||||
|
}
|
||||||
|
return &Client{grpc: searchv1.NewSearchServiceClient(conn), conn: conn}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) Close() error {
|
||||||
|
return c.conn.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) Search(ctx context.Context, query string, limit uint32) ([]string, error) {
|
||||||
|
resp, err := c.grpc.Search(ctx, &searchv1.SearchRequest{Query: query, Limit: limit})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return resp.GetRecordIds(), nil
|
||||||
|
}
|
||||||
+36
-3
@@ -1,12 +1,18 @@
|
|||||||
# Phase 0 stack: Redpanda -> ingest -> ClickHouse -> api -> web.
|
# Phase 0+1 stack: Redpanda -> ingest -> ClickHouse -> api -> web, plus
|
||||||
|
# search (Tantivy full-text indexing, reads the same Redpanda topic
|
||||||
|
# ingest's consumer does).
|
||||||
#
|
#
|
||||||
# Does NOT include the Rust agent — see /agent/README.md: journald
|
# Does NOT include the Rust agent — see /agent/README.md: journald
|
||||||
# sourcing needs the host's journal, which isn't something a container
|
# sourcing needs the host's journal, which isn't something a container
|
||||||
# gets for free. Run the agent natively on the host per
|
# gets for free. Run the agent natively on the host per
|
||||||
# /docs/phase-0-runbook.md, pointed at ingest's mapped port (localhost:4317).
|
# /docs/phase-0-runbook.md, pointed at ingest's mapped port (localhost:4317).
|
||||||
|
# Windows Event Log/ETW sourcing needs a real Windows host regardless —
|
||||||
|
# see /docs/phase-1-runbook.md.
|
||||||
#
|
#
|
||||||
# Before first run: generate dev mTLS certs (hack/dev-certs/generate.sh).
|
# Before first run: generate dev mTLS certs (hack/dev-certs/generate.sh).
|
||||||
# See /docs/phase-0-runbook.md for the full sequence.
|
# See /docs/phase-0-runbook.md (Linux pipeline) and
|
||||||
|
# /docs/phase-1-runbook.md (Windows + full-text search) for the full
|
||||||
|
# sequences.
|
||||||
services:
|
services:
|
||||||
redpanda:
|
redpanda:
|
||||||
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
|
image: docker.redpanda.com/redpandadata/redpanda:v24.2.7
|
||||||
@@ -44,6 +50,10 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
REDPANDA_BROKERS: "redpanda:9092"
|
REDPANDA_BROKERS: "redpanda:9092"
|
||||||
REDPANDA_ADMIN_HOSTS: "redpanda:9644"
|
REDPANDA_ADMIN_HOSTS: "redpanda:9644"
|
||||||
|
# Explicit rather than relying on both this script's and /search's
|
||||||
|
# defaults happening to agree — search consumes this same topic and
|
||||||
|
# needs to know the partition count up front (see /search/README.md).
|
||||||
|
REDPANDA_TOPIC_PARTITIONS: "6"
|
||||||
|
|
||||||
clickhouse:
|
clickhouse:
|
||||||
image: clickhouse/clickhouse-server:24.8
|
image: clickhouse/clickhouse-server:24.8
|
||||||
@@ -109,9 +119,30 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- ./hack/dev-certs/out:/etc/sentry-ingest:ro
|
- ./hack/dev-certs/out:/etc/sentry-ingest:ro
|
||||||
|
|
||||||
|
# Reads the same sentry.logs.raw topic ingest's consumer does (own
|
||||||
|
# offset tracking, own failure domain — see /search/README.md) and
|
||||||
|
# builds a Tantivy full-text index over the message field.
|
||||||
|
search:
|
||||||
|
build:
|
||||||
|
context: . # needs both search/ and proto/
|
||||||
|
dockerfile: search/Dockerfile
|
||||||
|
container_name: sentry-search
|
||||||
|
depends_on:
|
||||||
|
redpanda-provision:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
environment:
|
||||||
|
REDPANDA_BROKERS: "redpanda:9092"
|
||||||
|
REDPANDA_TOPIC_PARTITIONS: "6" # must match redpanda-provision's above
|
||||||
|
# tracing-subscriber's default filter suppresses INFO without this
|
||||||
|
# -- found by actually checking `docker compose logs search` and
|
||||||
|
# seeing nothing, same silent-logging gap the agent had in Phase 0.
|
||||||
|
RUST_LOG: "info"
|
||||||
|
volumes:
|
||||||
|
- search-index-data:/var/lib/sentry-search
|
||||||
|
|
||||||
api:
|
api:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: . # needs both api/ and proto/ (gRPC client to search)
|
||||||
dockerfile: api/Dockerfile
|
dockerfile: api/Dockerfile
|
||||||
container_name: sentry-api
|
container_name: sentry-api
|
||||||
depends_on:
|
depends_on:
|
||||||
@@ -122,6 +153,7 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
CLICKHOUSE_ADDR: "clickhouse:9000"
|
CLICKHOUSE_ADDR: "clickhouse:9000"
|
||||||
CLICKHOUSE_PASSWORD: "sentry-dev-only"
|
CLICKHOUSE_PASSWORD: "sentry-dev-only"
|
||||||
|
SEARCH_GRPC_ADDR: "search:50052"
|
||||||
|
|
||||||
web:
|
web:
|
||||||
build:
|
build:
|
||||||
@@ -141,3 +173,4 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
redpanda-data:
|
redpanda-data:
|
||||||
clickhouse-data:
|
clickhouse-data:
|
||||||
|
search-index-data:
|
||||||
|
|||||||
+17
-11
@@ -5,13 +5,16 @@ ingest, and ClickHouse, to a browser table. This is the actual
|
|||||||
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
|
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
|
||||||
regardless of what any individual component's tests say.
|
regardless of what any individual component's tests say.
|
||||||
|
|
||||||
**This sequence has not been run end-to-end** in the environment that
|
**Update:** this sequence has since been run for real, more than once,
|
||||||
built it (no Docker available there — see the caveats each component's
|
against a live Docker install — not just written and trusted. Two real
|
||||||
summary already flagged). Individual pieces are unit-tested and built
|
bugs turned up doing that (ClickHouse's official image silently disabling
|
||||||
successfully in isolation; this document is the logical sequence to run
|
network access without a password set; `rpk`'s exact flag syntax) and got
|
||||||
for real, not a report that it's been run. Expect to debug something on
|
fixed; see the git history around the "Fix two bugs found by actually
|
||||||
first attempt, and treat the "Troubleshooting" section at the bottom as a
|
running the Phase 0 pipeline end-to-end" commit if you want the details.
|
||||||
starting point, not an exhaustive list.
|
The steps below reflect what was actually run, not just planned. The
|
||||||
|
"Troubleshooting" section below is still worth reading first if something
|
||||||
|
doesn't work — it's not an exhaustive list, but it does reflect real
|
||||||
|
failures encountered, not hypothetical ones.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
@@ -113,12 +116,15 @@ Reading the system journal generally needs root (or membership in the
|
|||||||
by distro, root is the reliable path for this runbook):
|
by distro, root is the reliable path for this runbook):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
sudo ./target/release/sentry-agent
|
sudo RUST_LOG=info ./target/release/sentry-agent
|
||||||
```
|
```
|
||||||
|
|
||||||
Leave it running in this terminal — you should see a `connected to ingest
|
`RUST_LOG=info` matters: `tracing_subscriber`'s default filter is
|
||||||
service` log line. If you see a TLS or connection error instead, stop
|
otherwise strict enough to suppress even the startup log line, and the
|
||||||
here and check the Troubleshooting section before continuing.
|
agent will look like it's silently doing nothing. Leave it running in
|
||||||
|
this terminal — you should see a `connected to ingest service` log line.
|
||||||
|
If you see a TLS or connection error instead, stop here and check the
|
||||||
|
Troubleshooting section before continuing.
|
||||||
|
|
||||||
## 6. Generate a test log line
|
## 6. Generate a test log line
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,243 @@
|
|||||||
|
# Phase 1 runbook
|
||||||
|
|
||||||
|
Extends `/docs/phase-0-runbook.md` with Windows log collection and
|
||||||
|
full-text search. Read that one first — this assumes the Phase 0 stack
|
||||||
|
(dev certs, `docker compose up`, backend sanity check) already works;
|
||||||
|
Phase 1 layers on top of it, doesn't replace it.
|
||||||
|
|
||||||
|
## What's actually been verified vs. what needs real Windows
|
||||||
|
|
||||||
|
Unlike Phase 0's original draft, most of this runbook reflects steps
|
||||||
|
actually run in this session against a live stack, not just planned:
|
||||||
|
|
||||||
|
- **Verified for real:** the full Linux pipeline through search (agent →
|
||||||
|
ingest's `record_id` assignment → Redpanda → both consumers →
|
||||||
|
ClickHouse *and* Tantivy → both `/query` and `/search` → the same
|
||||||
|
`record_id` back from both). The `windows-fixture` generator sending
|
||||||
|
Windows-*shaped* data through the same pipeline and being correctly
|
||||||
|
queryable both ways, including `winevt.*` attributes and severity
|
||||||
|
mapping.
|
||||||
|
- **Not verified, and can't be from the environment this was built in:**
|
||||||
|
the actual Windows agent binary — `EvtSubscribe`, ETW session creation,
|
||||||
|
Windows service registration. No Windows toolchain was available
|
||||||
|
anywhere (confirmed: only the Linux target's std library installed, no
|
||||||
|
rustup, no way to even `cargo check --target x86_64-pc-windows-*`).
|
||||||
|
Part C below is the logical sequence to run on a real or virtualized
|
||||||
|
Windows host, not a report that it's been run.
|
||||||
|
|
||||||
|
## Prerequisites (beyond Phase 0's)
|
||||||
|
|
||||||
|
- A Windows host or VM (Windows 10/11 or Windows Server) for Part C.
|
||||||
|
- `mingw-w64` if cross-compiling the Windows build from Linux (optional —
|
||||||
|
building natively on Windows with `rustup target add
|
||||||
|
x86_64-pc-windows-msvc` works too and needs no extra setup on the Linux
|
||||||
|
side).
|
||||||
|
- Administrator access on the Windows host, for service registration and
|
||||||
|
(if you enable it) ETW.
|
||||||
|
|
||||||
|
## Part A: full-text search (Linux-only, no Windows needed)
|
||||||
|
|
||||||
|
Only needs what Phase 0's runbook already set up.
|
||||||
|
|
||||||
|
### A1. Bring the stack up (if not already)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
Same as Phase 0, now also builds and starts `search` (Tantivy full-text
|
||||||
|
indexing). Confirm it's actually logging — same `RUST_LOG` gap the agent
|
||||||
|
has by default:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose logs search
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see "search gRPC server listening" and rskafka connecting to
|
||||||
|
all of `sentry.logs.raw`'s partitions. If you see nothing at all, check
|
||||||
|
`RUST_LOG=info` is set on the `search` service in `docker-compose.yml`.
|
||||||
|
|
||||||
|
### A2. Generate a log line and confirm both query paths agree
|
||||||
|
|
||||||
|
Follow Phase 0's runbook to get the agent running and generate a test
|
||||||
|
line (steps 4–6 there — mTLS certs, build, run, `logger`). Then, instead
|
||||||
|
of just checking `/query`, check both:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||||
|
-d '{"sql": "SELECT record_id, message FROM logs ORDER BY timestamp DESC LIMIT 1"}'
|
||||||
|
|
||||||
|
curl -X POST http://localhost:8080/search -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "<a distinctive word from your test log line>"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**The `record_id` in both responses should match.** That's the actual
|
||||||
|
Phase 1 exit criterion (`/CLAUDE.md`) for the Linux half: the same
|
||||||
|
record, reachable both ways. If `/search` returns nothing yet, give it a
|
||||||
|
few more seconds — Tantivy commits on a timer (`COMMIT_INTERVAL_MS`,
|
||||||
|
default 2s), so there's a small window where a record is in ClickHouse
|
||||||
|
but not yet searchable.
|
||||||
|
|
||||||
|
### A3. Confirm from the web UI
|
||||||
|
|
||||||
|
Open `http://localhost:3000` — there are now two pages, linked via the
|
||||||
|
top nav: **SQL Query** (unchanged from Phase 0) and **Full-Text Search**
|
||||||
|
(new). Run the same free-text term on the search page and confirm you
|
||||||
|
see the row.
|
||||||
|
|
||||||
|
## Part B: Windows-shaped data without a Windows host
|
||||||
|
|
||||||
|
Still no Windows needed — this tests the pipeline's handling of
|
||||||
|
Windows-*shaped* data, not the real Windows integration (see
|
||||||
|
`/hack/windows-fixture/README.md` for the exact distinction).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd hack/windows-fixture
|
||||||
|
go run . --count 5
|
||||||
|
```
|
||||||
|
|
||||||
|
Then repeat A2's pattern: query for one of the synthetic events by
|
||||||
|
`attributes['winevt.event_id']` via `/query`, and by a distinctive word
|
||||||
|
from its message via `/search`. Both should return it, and `attributes`
|
||||||
|
should carry `winevt.event_id`/`winevt.provider`/`winevt.channel`/
|
||||||
|
`winevt.computer`.
|
||||||
|
|
||||||
|
## Part C: the real Windows agent (needs actual Windows)
|
||||||
|
|
||||||
|
### C1. Build
|
||||||
|
|
||||||
|
On the Windows host itself (simplest — avoids cross-compilation
|
||||||
|
entirely):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
rustup target add x86_64-pc-windows-msvc
|
||||||
|
cd agent
|
||||||
|
cargo build --release --target x86_64-pc-windows-msvc --no-default-features --features windows-eventlog,etw
|
||||||
|
```
|
||||||
|
|
||||||
|
Or cross-compile from Linux, then copy the binary over:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
rustup target add x86_64-pc-windows-gnu
|
||||||
|
cargo build --release --target x86_64-pc-windows-gnu --no-default-features --features windows-eventlog,etw
|
||||||
|
```
|
||||||
|
|
||||||
|
`protoc` needs to be on `PATH` either way (used by `tonic-build` at
|
||||||
|
compile time), same requirement as the Linux build.
|
||||||
|
|
||||||
|
### C2. Get mTLS certs onto the Windows host
|
||||||
|
|
||||||
|
Copy `hack/dev-certs/out/{ca,client,client-key}.pem` from wherever you
|
||||||
|
ran `generate.sh` to `C:\ProgramData\SentryAgent\` on the Windows host
|
||||||
|
(create the directory first). Same dev-only certs Phase 0's Linux agent
|
||||||
|
uses — the CA doesn't care what platform the client is on, only that the
|
||||||
|
client cert was signed by it.
|
||||||
|
|
||||||
|
### C3. Config
|
||||||
|
|
||||||
|
Create `C:\ProgramData\SentryAgent\agent.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[source]
|
||||||
|
kind = "eventlog"
|
||||||
|
channels = ["Application", "System", "Security"]
|
||||||
|
|
||||||
|
[ingest]
|
||||||
|
endpoint = "https://<host-running-docker-compose>:4317"
|
||||||
|
```
|
||||||
|
|
||||||
|
If Docker Compose runs on a different machine than the Windows host,
|
||||||
|
`ingest`'s server cert SAN needs to cover that hostname/IP too — see
|
||||||
|
`hack/dev-certs/generate.sh` and regenerate with an updated SAN if
|
||||||
|
needed (same note as Phase 0's runbook's troubleshooting section).
|
||||||
|
|
||||||
|
### C4. Run it directly first, before installing as a service
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:RUST_LOG="info"
|
||||||
|
.\sentry-agent.exe --config C:\ProgramData\SentryAgent\agent.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirms the Event Log source and mTLS connection work before adding the
|
||||||
|
Windows service layer on top — if something's wrong, it's much easier to
|
||||||
|
diagnose here than after wrapping it in a service.
|
||||||
|
|
||||||
|
### C5. Generate a Windows Event Log entry and confirm it flows through
|
||||||
|
|
||||||
|
From another PowerShell window (or Event Viewer):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
eventcreate /T INFORMATION /ID 1 /L APPLICATION /SO "SentryTest" /D "phase1 windows verification line"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then check both query paths, same pattern as A2.
|
||||||
|
|
||||||
|
### C6. Install as a service
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\sentry-agent.exe install
|
||||||
|
sc.exe start SentryAgent
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify it's running (`sc.exe query SentryAgent`) and generate another
|
||||||
|
test event to confirm it's still flowing through while running as a
|
||||||
|
service, not just in the foreground. **Known gap:** no console under the
|
||||||
|
SCM means `tracing`'s log output currently has nowhere to go — see
|
||||||
|
`/agent/README.md`'s "Running as a Windows service" section. If
|
||||||
|
something goes wrong here, you're debugging blind until that's
|
||||||
|
addressed; C4's foreground run is where to diagnose real problems.
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
sc.exe stop SentryAgent
|
||||||
|
.\sentry-agent.exe uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
### C7 (optional). ETW
|
||||||
|
|
||||||
|
Only if you actually want it running — **read the privilege section in
|
||||||
|
`/agent/README.md` first.** ETW needs elevated privileges (an
|
||||||
|
administrator token or `SeSystemProfilePrivilege`), a real consideration
|
||||||
|
for a log-shipping agent, not a formality. Providers are configured by
|
||||||
|
GUID (`logman query providers "<Name>"` to look one up):
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[source]
|
||||||
|
kind = "etw"
|
||||||
|
providers = ["{22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### C8 (informational). WEF
|
||||||
|
|
||||||
|
No new steps — see `/agent/README.md`'s WEF section. The supported
|
||||||
|
pattern is running this same agent (Event Log source) on a Windows
|
||||||
|
Server already acting as a native Windows Event Collector, pointed at
|
||||||
|
the `ForwardedEvents` channel instead of the usual three. A true
|
||||||
|
agentless WS-Management receiver is explicitly not built in Phase 1.
|
||||||
|
|
||||||
|
## Troubleshooting (Phase 1-specific)
|
||||||
|
|
||||||
|
**`/search` returns nothing but `/query` finds the record.**
|
||||||
|
Check `COMMIT_INTERVAL_MS` hasn't elapsed yet (default 2s) — Tantivy
|
||||||
|
batches commits, same reasoning as ClickHouse batching inserts. If it's
|
||||||
|
been well over that and still nothing: `docker compose logs search` —
|
||||||
|
look for "skipping record with empty record_id" (would mean something
|
||||||
|
upstream isn't assigning IDs — shouldn't happen) or connection errors to
|
||||||
|
Redpanda.
|
||||||
|
|
||||||
|
**Windows agent connects but Event Log entries never show up.**
|
||||||
|
Check the channel name is exactly right (`Application`/`System`/
|
||||||
|
`Security`, case matters to the Windows API) and that the account
|
||||||
|
running the agent has read access to that log — `Security` specifically
|
||||||
|
often needs elevated rights beyond what `Application`/`System` need.
|
||||||
|
|
||||||
|
**Windows build fails to find `protoc`.**
|
||||||
|
Same requirement as the Linux build — install `protoc` and ensure it's
|
||||||
|
on `PATH` before `cargo build`. On Windows, the official protoc release
|
||||||
|
zip plus adding its `bin/` to `PATH` is the simplest route.
|
||||||
|
|
||||||
|
**Nothing in this section covers the problem.**
|
||||||
|
Genuinely possible — this is the least-tested part of the whole Phase 1
|
||||||
|
build (see the caveat at the top). Check `/agent/README.md`'s Windows
|
||||||
|
sections for the specific module involved (`source/windows_eventlog.rs`,
|
||||||
|
`source/etw.rs`, `service.rs`) and their own "UNVERIFIED" comments for
|
||||||
|
what's most likely to need a real fix.
|
||||||
@@ -14,3 +14,8 @@ monorepos (Kubernetes among them).
|
|||||||
- `dev-certs/` — generates a throwaway CA + server/client cert pair for
|
- `dev-certs/` — generates a throwaway CA + server/client cert pair for
|
||||||
local mTLS between the agent and ingest. See `/docs/phase-0-runbook.md`
|
local mTLS between the agent and ingest. See `/docs/phase-0-runbook.md`
|
||||||
for when to run it.
|
for when to run it.
|
||||||
|
- `windows-fixture/` — sends synthetic Windows Event Log-shaped records
|
||||||
|
directly to `ingest`, bypassing the real Windows agent. Tests whether
|
||||||
|
the pipeline handles Windows-shaped data; doesn't test the real
|
||||||
|
`EvtSubscribe`/ETW integration, which needs actual Windows. See
|
||||||
|
`/docs/phase-1-runbook.md`.
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# windows-fixture
|
||||||
|
|
||||||
|
Sends synthetic Windows Event Log-shaped `PushBatchRequest`s directly to
|
||||||
|
`ingest`'s gRPC endpoint, bypassing the actual Windows agent entirely.
|
||||||
|
|
||||||
|
## What this does and doesn't test
|
||||||
|
|
||||||
|
**Tests:** can the pipeline (ingest → ClickHouse → search → api → web)
|
||||||
|
correctly handle Windows-*shaped* data — the `winevt.*` attributes, the
|
||||||
|
`record_id` join between SQL and full-text search, Windows severity
|
||||||
|
levels mapping onto the right column values? This is exactly what's
|
||||||
|
automatable without a Windows host, and it's genuinely exercised: five
|
||||||
|
realistic, well-known Windows events (failed/successful logon, a service
|
||||||
|
state change, an application crash, an unexpected reboot) with real
|
||||||
|
EventIDs and providers.
|
||||||
|
|
||||||
|
**Does not test:** whether the real Windows agent's `EvtSubscribe`/ETW
|
||||||
|
integration actually works, whether Windows service registration
|
||||||
|
succeeds, whether ETW session creation/provider enabling works. Those are
|
||||||
|
fundamentally different questions — they need a real or virtualized
|
||||||
|
Windows host, and nothing here pretends otherwise. See
|
||||||
|
`/docs/phase-1-runbook.md` for exactly which is which.
|
||||||
|
|
||||||
|
## Running
|
||||||
|
|
||||||
|
Requires the docker-compose stack up (`ingest` reachable, dev certs
|
||||||
|
generated):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd hack/windows-fixture
|
||||||
|
go run . --count 5
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
sent 5 synthetic Windows-shaped records, ingest accepted 5
|
||||||
|
[SEVERITY_WARN] An account failed to log on. (event_id=4625 provider=Microsoft-Windows-Security-Auditing)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Then confirm both query paths see it:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -s -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||||
|
-d '{"sql": "SELECT host, severity, message, attributes['"'"'winevt.event_id'"'"'] AS event_id FROM logs WHERE host = '"'"'WIN-FIXTURE-01'"'"' ORDER BY timestamp DESC"}'
|
||||||
|
|
||||||
|
curl -s -X POST http://localhost:8080/search -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "notepad"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Flags: `--addr` (default `localhost:4317`), `--ca`/`--cert`/`--key`
|
||||||
|
(default to `../dev-certs/out/{ca,client,client-key}.pem`), `--count`
|
||||||
|
(default 5, cycles through the fixed event list if higher).
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
module github.com/sentry/sentry/hack/windows-fixture
|
||||||
|
|
||||||
|
go 1.25.0
|
||||||
|
|
||||||
|
replace github.com/sentry/sentry/proto => ../../proto
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||||
|
google.golang.org/grpc v1.83.0
|
||||||
|
)
|
||||||
|
|
||||||
|
require (
|
||||||
|
golang.org/x/net v0.55.0 // indirect
|
||||||
|
golang.org/x/sys v0.45.0 // indirect
|
||||||
|
golang.org/x/text v0.37.0 // indirect
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||||
|
google.golang.org/protobuf v1.36.12 // indirect
|
||||||
|
)
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||||
|
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||||
|
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||||
|
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||||
|
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||||
|
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||||
|
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||||
|
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||||
|
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||||
|
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||||
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||||
|
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||||
|
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||||
|
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||||
|
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||||
|
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
|
||||||
|
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
|
||||||
|
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||||
|
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
|
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||||
|
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||||
|
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||||
|
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||||
|
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||||
|
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||||
|
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||||
|
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
// Command windows-fixture sends synthetic Windows Event Log-shaped
|
||||||
|
// PushBatchRequests directly to ingest's gRPC endpoint, bypassing the
|
||||||
|
// actual Windows agent entirely.
|
||||||
|
//
|
||||||
|
// This tests one specific thing: can the pipeline (ingest -> ClickHouse
|
||||||
|
// -> search -> api -> web) correctly handle Windows-*shaped* data (the
|
||||||
|
// winevt.* attributes, the record_id join, severity mapping)? It does
|
||||||
|
// NOT test whether the real Windows agent's EvtSubscribe/ETW integration
|
||||||
|
// actually works -- that's a fundamentally different question that can
|
||||||
|
// only be answered on a real or virtualized Windows host. See
|
||||||
|
// /docs/phase-1-runbook.md for exactly which is which.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"crypto/x509"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"google.golang.org/grpc"
|
||||||
|
"google.golang.org/grpc/credentials"
|
||||||
|
|
||||||
|
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
addr := flag.String("addr", "localhost:4317", "ingest gRPC address")
|
||||||
|
caFile := flag.String("ca", "../dev-certs/out/ca.pem", "CA cert path")
|
||||||
|
certFile := flag.String("cert", "../dev-certs/out/client.pem", "client cert path")
|
||||||
|
keyFile := flag.String("key", "../dev-certs/out/client-key.pem", "client key path")
|
||||||
|
count := flag.Int("count", 5, "number of synthetic events to send")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
tlsConf, err := loadTLSConfig(*caFile, *certFile, *keyFile)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "loading TLS config:", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(credentials.NewTLS(tlsConf)))
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "dialing ingest:", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer conn.Close()
|
||||||
|
|
||||||
|
client := logsv1.NewLogIngestClient(conn)
|
||||||
|
records := syntheticWindowsRecords(*count)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
resp, err := client.PushBatch(ctx, &logsv1.PushBatchRequest{
|
||||||
|
BatchId: "windows-fixture",
|
||||||
|
Records: records,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "PushBatch failed:", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("sent %d synthetic Windows-shaped records, ingest accepted %d\n", len(records), resp.GetAccepted())
|
||||||
|
for _, rec := range records {
|
||||||
|
fmt.Printf(" [%s] %s (event_id=%s provider=%s)\n",
|
||||||
|
rec.GetSeverity(), rec.GetMessage(), rec.GetAttributes()["winevt.event_id"], rec.GetAttributes()["winevt.provider"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadTLSConfig(caFile, certFile, keyFile string) (*tls.Config, error) {
|
||||||
|
caPEM, err := os.ReadFile(caFile)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("reading CA cert %s: %w", caFile, err)
|
||||||
|
}
|
||||||
|
caPool := x509.NewCertPool()
|
||||||
|
if !caPool.AppendCertsFromPEM(caPEM) {
|
||||||
|
return nil, fmt.Errorf("no valid certificates found in %s", caFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("loading client cert/key: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &tls.Config{
|
||||||
|
RootCAs: caPool,
|
||||||
|
Certificates: []tls.Certificate{cert},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// A handful of realistic, well-known Windows Event Log entries (real
|
||||||
|
// EventIDs/providers/channels), cycled through if --count exceeds the
|
||||||
|
// list length.
|
||||||
|
func syntheticWindowsRecords(count int) []*logsv1.LogRecord {
|
||||||
|
events := []struct {
|
||||||
|
eventID string
|
||||||
|
provider string
|
||||||
|
channel string
|
||||||
|
level logsv1.Severity
|
||||||
|
message string
|
||||||
|
}{
|
||||||
|
{"4625", "Microsoft-Windows-Security-Auditing", "Security", logsv1.Severity_SEVERITY_WARN, "An account failed to log on."},
|
||||||
|
{"7036", "Service Control Manager", "System", logsv1.Severity_SEVERITY_INFO, "The Windows Update service entered the running state."},
|
||||||
|
{"1000", "Application Error", "Application", logsv1.Severity_SEVERITY_ERROR, "Faulting application name: notepad.exe"},
|
||||||
|
{"4624", "Microsoft-Windows-Security-Auditing", "Security", logsv1.Severity_SEVERITY_INFO, "An account was successfully logged on."},
|
||||||
|
{"41", "Microsoft-Windows-Kernel-Power", "System", logsv1.Severity_SEVERITY_FATAL, "The system has rebooted without cleanly shutting down first."},
|
||||||
|
}
|
||||||
|
|
||||||
|
records := make([]*logsv1.LogRecord, 0, count)
|
||||||
|
for i := 0; i < count; i++ {
|
||||||
|
e := events[i%len(events)]
|
||||||
|
records = append(records, &logsv1.LogRecord{
|
||||||
|
TimestampUnixNano: time.Now().UnixNano(),
|
||||||
|
Host: "WIN-FIXTURE-01",
|
||||||
|
Service: "default",
|
||||||
|
Severity: e.level,
|
||||||
|
Message: e.message,
|
||||||
|
Attributes: map[string]string{
|
||||||
|
"winevt.event_id": e.eventID,
|
||||||
|
"winevt.provider": e.provider,
|
||||||
|
"winevt.channel": e.channel,
|
||||||
|
"winevt.computer": "WIN-FIXTURE-01",
|
||||||
|
"winevt.record_number": fmt.Sprintf("%d", 100000+i),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return records
|
||||||
|
}
|
||||||
+15
-3
@@ -4,9 +4,17 @@ Go service sitting between the Rust agent and ClickHouse. Two halves in one
|
|||||||
binary, selected with `--mode`:
|
binary, selected with `--mode`:
|
||||||
|
|
||||||
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
||||||
connect to. Forwards each record, proto-encoded and unchanged, onto
|
connect to. Assigns each record a server-side `record_id` (a UUID,
|
||||||
Redpanda. Does no normalization — kept thin so agent-facing latency isn't
|
overwriting whatever the agent sent — agents always send it empty) and
|
||||||
coupled to ClickHouse write performance.
|
otherwise forwards records proto-encoded onto Redpanda unchanged. Still
|
||||||
|
kept thin — one field assignment, no real normalization — so agent-
|
||||||
|
facing latency isn't coupled to ClickHouse write performance.
|
||||||
|
`record_id` has to be assigned exactly once, here, rather than
|
||||||
|
independently by each downstream consumer: Phase 1's Tantivy indexer
|
||||||
|
and the ClickHouse writer both read the same Redpanda messages and need
|
||||||
|
to agree on the same ID for the same record to join search hits back to
|
||||||
|
rows — two consumers generating their own IDs would produce mismatched
|
||||||
|
ones for what's supposed to be the same record.
|
||||||
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
||||||
shape (`internal/normalize`), and batch-writes via the native protocol
|
shape (`internal/normalize`), and batch-writes via the native protocol
|
||||||
driver. Commits Redpanda offsets only after a successful ClickHouse
|
driver. Commits Redpanda offsets only after a successful ClickHouse
|
||||||
@@ -35,6 +43,10 @@ egress an agent has. See `/docs/architecture.md`.
|
|||||||
protocol, pure Go (no cgo).
|
protocol, pure Go (no cgo).
|
||||||
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
||||||
server and consumer halves concurrently and propagate the first error.
|
server and consumer halves concurrently and propagate the first error.
|
||||||
|
- **github.com/google/uuid** — was already in the dependency graph
|
||||||
|
transitively (via clickhouse-go); promoted to a direct dependency for
|
||||||
|
`record_id` generation in `internal/grpcserver`, so not a new addition
|
||||||
|
to the transitive tree.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -6,6 +6,7 @@ replace github.com/sentry/sentry/proto => ../proto
|
|||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
||||||
|
github.com/google/uuid v1.6.0
|
||||||
github.com/segmentio/kafka-go v0.4.51
|
github.com/segmentio/kafka-go v0.4.51
|
||||||
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||||
golang.org/x/sync v0.22.0
|
golang.org/x/sync v0.22.0
|
||||||
@@ -19,7 +20,6 @@ require (
|
|||||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||||
github.com/go-faster/city v1.0.1 // indirect
|
github.com/go-faster/city v1.0.1 // indirect
|
||||||
github.com/go-faster/errors v0.7.1 // indirect
|
github.com/go-faster/errors v0.7.1 // indirect
|
||||||
github.com/google/uuid v1.6.0 // indirect
|
|
||||||
github.com/klauspost/compress v1.19.1 // indirect
|
github.com/klauspost/compress v1.19.1 // indirect
|
||||||
github.com/paulmach/orb v0.13.0 // indirect
|
github.com/paulmach/orb v0.13.0 // indirect
|
||||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||||
|
|||||||
@@ -41,14 +41,14 @@ func (w *Writer) Close() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (w *Writer) WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error {
|
func (w *Writer) WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error {
|
||||||
batch, err := w.conn.PrepareBatch(ctx, "INSERT INTO logs (timestamp, host, service, severity, message, attributes)")
|
batch, err := w.conn.PrepareBatch(ctx, "INSERT INTO logs (timestamp, host, service, severity, message, attributes, record_id)")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("preparing batch: %w", err)
|
return fmt.Errorf("preparing batch: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, rec := range records {
|
for _, rec := range records {
|
||||||
row := normalize.ToRow(rec)
|
row := normalize.ToRow(rec)
|
||||||
if err := batch.Append(row.Timestamp, row.Host, row.Service, row.Severity, row.Message, row.Attributes); err != nil {
|
if err := batch.Append(row.Timestamp, row.Host, row.Service, row.Severity, row.Message, row.Attributes, row.RecordID); err != nil {
|
||||||
return fmt.Errorf("appending row to batch: %w", err)
|
return fmt.Errorf("appending row to batch: %w", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
// Package grpcserver implements the agent-facing side of ingest: an mTLS
|
// Package grpcserver implements the agent-facing side of ingest: an mTLS
|
||||||
// gRPC server accepting LogIngest.PushBatch calls, which it forwards
|
// gRPC server accepting LogIngest.PushBatch calls. It assigns each record
|
||||||
// unchanged (proto-encoded) onto Redpanda. Normalization into the
|
// a stable record_id (see the proto field comment for why this has to
|
||||||
// ClickHouse row shape happens later, on the consumer side.
|
// happen exactly once, here, rather than in either downstream consumer)
|
||||||
|
// and otherwise forwards records unchanged onto Redpanda — normalization
|
||||||
|
// into the ClickHouse row shape happens later, on the consumer side.
|
||||||
package grpcserver
|
package grpcserver
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -10,6 +12,7 @@ import (
|
|||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
"github.com/segmentio/kafka-go"
|
"github.com/segmentio/kafka-go"
|
||||||
"google.golang.org/grpc"
|
"google.golang.org/grpc"
|
||||||
"google.golang.org/grpc/codes"
|
"google.golang.org/grpc/codes"
|
||||||
@@ -76,6 +79,15 @@ func (s *Server) PushBatch(ctx context.Context, req *logsv1.PushBatchRequest) (*
|
|||||||
|
|
||||||
msgs := make([]kafka.Message, 0, len(req.GetRecords()))
|
msgs := make([]kafka.Message, 0, len(req.GetRecords()))
|
||||||
for _, rec := range req.GetRecords() {
|
for _, rec := range req.GetRecords() {
|
||||||
|
// Assigned here, once, before this record is produced to
|
||||||
|
// Redpanda: the ClickHouse-writer consumer and the Tantivy-
|
||||||
|
// indexer consumer (Phase 1) both read the same Redpanda
|
||||||
|
// messages and need to agree on the same ID for the same
|
||||||
|
// record. Overwrites anything the agent sent (it always sends
|
||||||
|
// empty, per the proto comment, but this is authoritative
|
||||||
|
// regardless).
|
||||||
|
rec.RecordId = uuid.NewString()
|
||||||
|
|
||||||
val, err := proto.Marshal(rec)
|
val, err := proto.Marshal(rec)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, status.Errorf(codes.InvalidArgument, "marshaling record: %v", err)
|
return nil, status.Errorf(codes.InvalidArgument, "marshaling record: %v", err)
|
||||||
|
|||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package grpcserver
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/segmentio/kafka-go"
|
||||||
|
"google.golang.org/protobuf/proto"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/ingest/internal/config"
|
||||||
|
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeProducer struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
written [][]kafka.Message
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeProducer) WriteBatch(_ context.Context, msgs []kafka.Message) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if f.err != nil {
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
batch := make([]kafka.Message, len(msgs))
|
||||||
|
copy(batch, msgs)
|
||||||
|
f.written = append(f.written, batch)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestServer(p batchProducer) *Server {
|
||||||
|
return New(slog.New(slog.NewTextHandler(io.Discard, nil)), config.GRPCConfig{}, config.TLSConfig{}, p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushBatchAssignsRecordID(t *testing.T) {
|
||||||
|
fp := &fakeProducer{}
|
||||||
|
s := newTestServer(fp)
|
||||||
|
|
||||||
|
req := &logsv1.PushBatchRequest{
|
||||||
|
BatchId: "b1",
|
||||||
|
Records: []*logsv1.LogRecord{
|
||||||
|
{Host: "h1", Message: "one"},
|
||||||
|
{Host: "h1", Message: "two"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := s.PushBatch(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PushBatch() error = %v", err)
|
||||||
|
}
|
||||||
|
if resp.GetAccepted() != 2 {
|
||||||
|
t.Fatalf("Accepted = %d, want 2", resp.GetAccepted())
|
||||||
|
}
|
||||||
|
|
||||||
|
fp.mu.Lock()
|
||||||
|
defer fp.mu.Unlock()
|
||||||
|
if len(fp.written) != 1 || len(fp.written[0]) != 2 {
|
||||||
|
t.Fatalf("unexpected written batches: %+v", fp.written)
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := make(map[string]bool)
|
||||||
|
for _, m := range fp.written[0] {
|
||||||
|
var rec logsv1.LogRecord
|
||||||
|
if err := proto.Unmarshal(m.Value, &rec); err != nil {
|
||||||
|
t.Fatalf("unmarshaling produced message: %v", err)
|
||||||
|
}
|
||||||
|
if rec.GetRecordId() == "" {
|
||||||
|
t.Fatalf("record_id was not assigned for message %q", rec.GetMessage())
|
||||||
|
}
|
||||||
|
if seen[rec.GetRecordId()] {
|
||||||
|
t.Fatalf("duplicate record_id %q across records in the same batch", rec.GetRecordId())
|
||||||
|
}
|
||||||
|
seen[rec.GetRecordId()] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushBatchOverwritesAgentSuppliedRecordID(t *testing.T) {
|
||||||
|
fp := &fakeProducer{}
|
||||||
|
s := newTestServer(fp)
|
||||||
|
|
||||||
|
req := &logsv1.PushBatchRequest{
|
||||||
|
Records: []*logsv1.LogRecord{
|
||||||
|
{Host: "h1", Message: "one", RecordId: "agent-supplied-should-be-ignored"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := s.PushBatch(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("PushBatch() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fp.mu.Lock()
|
||||||
|
defer fp.mu.Unlock()
|
||||||
|
var rec logsv1.LogRecord
|
||||||
|
if err := proto.Unmarshal(fp.written[0][0].Value, &rec); err != nil {
|
||||||
|
t.Fatalf("unmarshaling produced message: %v", err)
|
||||||
|
}
|
||||||
|
if rec.GetRecordId() == "agent-supplied-should-be-ignored" {
|
||||||
|
t.Fatal("expected ingest to overwrite any agent-supplied record_id")
|
||||||
|
}
|
||||||
|
if rec.GetRecordId() == "" {
|
||||||
|
t.Fatal("expected a server-assigned record_id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushBatchEmptyRecordsIsANoOp(t *testing.T) {
|
||||||
|
fp := &fakeProducer{}
|
||||||
|
s := newTestServer(fp)
|
||||||
|
|
||||||
|
resp, err := s.PushBatch(context.Background(), &logsv1.PushBatchRequest{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PushBatch() error = %v", err)
|
||||||
|
}
|
||||||
|
if resp.GetAccepted() != 0 {
|
||||||
|
t.Fatalf("Accepted = %d, want 0", resp.GetAccepted())
|
||||||
|
}
|
||||||
|
|
||||||
|
fp.mu.Lock()
|
||||||
|
defer fp.mu.Unlock()
|
||||||
|
if len(fp.written) != 0 {
|
||||||
|
t.Fatalf("expected no batches written for an empty request, got %d", len(fp.written))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,6 +9,8 @@ package normalize
|
|||||||
import (
|
import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
|
||||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -19,6 +21,7 @@ type Row struct {
|
|||||||
Severity string
|
Severity string
|
||||||
Message string
|
Message string
|
||||||
Attributes map[string]string
|
Attributes map[string]string
|
||||||
|
RecordID uuid.UUID
|
||||||
}
|
}
|
||||||
|
|
||||||
func ToRow(rec *logsv1.LogRecord) Row {
|
func ToRow(rec *logsv1.LogRecord) Row {
|
||||||
@@ -26,6 +29,12 @@ func ToRow(rec *logsv1.LogRecord) Row {
|
|||||||
if attrs == nil {
|
if attrs == nil {
|
||||||
attrs = map[string]string{}
|
attrs = map[string]string{}
|
||||||
}
|
}
|
||||||
|
// grpcserver's PushBatch handler always assigns a valid UUID before a
|
||||||
|
// record reaches this point (see its doc comment for why), so a parse
|
||||||
|
// failure here would mean something upstream is bypassing that —
|
||||||
|
// fall back to the nil UUID rather than failing the whole row, same
|
||||||
|
// "never silently drop a record" spirit as the rest of this pipeline.
|
||||||
|
recordID, _ := uuid.Parse(rec.GetRecordId())
|
||||||
return Row{
|
return Row{
|
||||||
Timestamp: time.Unix(0, rec.GetTimestampUnixNano()).UTC(),
|
Timestamp: time.Unix(0, rec.GetTimestampUnixNano()).UTC(),
|
||||||
Host: rec.GetHost(),
|
Host: rec.GetHost(),
|
||||||
@@ -33,6 +42,7 @@ func ToRow(rec *logsv1.LogRecord) Row {
|
|||||||
Severity: severityText(rec.GetSeverity()),
|
Severity: severityText(rec.GetSeverity()),
|
||||||
Message: rec.GetMessage(),
|
Message: rec.GetMessage(),
|
||||||
Attributes: attrs,
|
Attributes: attrs,
|
||||||
|
RecordID: recordID,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,10 +4,13 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
|
||||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
||||||
|
id := uuid.New()
|
||||||
rec := &logsv1.LogRecord{
|
rec := &logsv1.LogRecord{
|
||||||
TimestampUnixNano: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).UnixNano(),
|
TimestampUnixNano: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).UnixNano(),
|
||||||
Host: "host-1",
|
Host: "host-1",
|
||||||
@@ -15,6 +18,7 @@ func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
|||||||
Severity: logsv1.Severity_SEVERITY_ERROR,
|
Severity: logsv1.Severity_SEVERITY_ERROR,
|
||||||
Message: "boom",
|
Message: "boom",
|
||||||
Attributes: map[string]string{"k": "v"},
|
Attributes: map[string]string{"k": "v"},
|
||||||
|
RecordId: id.String(),
|
||||||
}
|
}
|
||||||
|
|
||||||
row := ToRow(rec)
|
row := ToRow(rec)
|
||||||
@@ -31,6 +35,17 @@ func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
|||||||
if !row.Timestamp.Equal(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) {
|
if !row.Timestamp.Equal(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) {
|
||||||
t.Fatalf("unexpected timestamp: %v", row.Timestamp)
|
t.Fatalf("unexpected timestamp: %v", row.Timestamp)
|
||||||
}
|
}
|
||||||
|
if row.RecordID != id {
|
||||||
|
t.Fatalf("RecordID = %v, want %v", row.RecordID, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestToRowInvalidRecordIDFallsBackToNilUUID(t *testing.T) {
|
||||||
|
rec := &logsv1.LogRecord{Host: "h", Service: "s", Message: "m", RecordId: "not-a-uuid"}
|
||||||
|
row := ToRow(rec)
|
||||||
|
if row.RecordID != uuid.Nil {
|
||||||
|
t.Fatalf("expected nil UUID fallback for an invalid record_id, got %v", row.RecordID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestToRowNilAttributesBecomesEmptyMap(t *testing.T) {
|
func TestToRowNilAttributesBecomesEmptyMap(t *testing.T) {
|
||||||
|
|||||||
@@ -103,8 +103,18 @@ type LogRecord struct {
|
|||||||
// requirement in CLAUDE.md.
|
// requirement in CLAUDE.md.
|
||||||
Message string `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
|
Message string `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
|
||||||
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
||||||
// syslog header fields). Empty when the raw-passthrough fallback fires.
|
// syslog header fields), plus source-provided fields (e.g. Windows
|
||||||
Attributes map[string]string `protobuf:"bytes,6,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
// Event Log's winevt.event_id/winevt.provider/winevt.channel). Empty
|
||||||
|
// when the raw-passthrough fallback fires and the source added nothing.
|
||||||
|
Attributes map[string]string `protobuf:"bytes,6,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||||
|
// Stable per-record identifier, used to join Tantivy full-text search
|
||||||
|
// hits back to their ClickHouse row (Phase 1). Always empty as sent by
|
||||||
|
// the agent — ingest's PushBatch handler assigns this server-side,
|
||||||
|
// once, before producing to Redpanda, since both the ClickHouse-writer
|
||||||
|
// consumer and the Tantivy-indexer consumer read the same Redpanda
|
||||||
|
// messages and need to agree on the same ID for the same record. See
|
||||||
|
// /ingest/README.md.
|
||||||
|
RecordId string `protobuf:"bytes,7,opt,name=record_id,json=recordId,proto3" json:"record_id,omitempty"`
|
||||||
unknownFields protoimpl.UnknownFields
|
unknownFields protoimpl.UnknownFields
|
||||||
sizeCache protoimpl.SizeCache
|
sizeCache protoimpl.SizeCache
|
||||||
}
|
}
|
||||||
@@ -181,6 +191,13 @@ func (x *LogRecord) GetAttributes() map[string]string {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (x *LogRecord) GetRecordId() string {
|
||||||
|
if x != nil {
|
||||||
|
return x.RecordId
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
type PushBatchRequest struct {
|
type PushBatchRequest struct {
|
||||||
state protoimpl.MessageState `protogen:"open.v1"`
|
state protoimpl.MessageState `protogen:"open.v1"`
|
||||||
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
|
// Agent-assigned identifier for dedup/idempotency on retry. Ingest may
|
||||||
@@ -287,7 +304,7 @@ var File_sentry_logs_v1_logs_proto protoreflect.FileDescriptor
|
|||||||
|
|
||||||
const file_sentry_logs_v1_logs_proto_rawDesc = "" +
|
const file_sentry_logs_v1_logs_proto_rawDesc = "" +
|
||||||
"\n" +
|
"\n" +
|
||||||
"\x19sentry/logs/v1/logs.proto\x12\x0esentry.logs.v1\"\xc3\x02\n" +
|
"\x19sentry/logs/v1/logs.proto\x12\x0esentry.logs.v1\"\xe0\x02\n" +
|
||||||
"\tLogRecord\x12.\n" +
|
"\tLogRecord\x12.\n" +
|
||||||
"\x13timestamp_unix_nano\x18\x01 \x01(\x03R\x11timestampUnixNano\x12\x12\n" +
|
"\x13timestamp_unix_nano\x18\x01 \x01(\x03R\x11timestampUnixNano\x12\x12\n" +
|
||||||
"\x04host\x18\x02 \x01(\tR\x04host\x12\x18\n" +
|
"\x04host\x18\x02 \x01(\tR\x04host\x12\x18\n" +
|
||||||
@@ -296,7 +313,8 @@ const file_sentry_logs_v1_logs_proto_rawDesc = "" +
|
|||||||
"\amessage\x18\x05 \x01(\tR\amessage\x12I\n" +
|
"\amessage\x18\x05 \x01(\tR\amessage\x12I\n" +
|
||||||
"\n" +
|
"\n" +
|
||||||
"attributes\x18\x06 \x03(\v2).sentry.logs.v1.LogRecord.AttributesEntryR\n" +
|
"attributes\x18\x06 \x03(\v2).sentry.logs.v1.LogRecord.AttributesEntryR\n" +
|
||||||
"attributes\x1a=\n" +
|
"attributes\x12\x1b\n" +
|
||||||
|
"\trecord_id\x18\a \x01(\tR\brecordId\x1a=\n" +
|
||||||
"\x0fAttributesEntry\x12\x10\n" +
|
"\x0fAttributesEntry\x12\x10\n" +
|
||||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"b\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"b\n" +
|
||||||
|
|||||||
@@ -47,8 +47,19 @@ message LogRecord {
|
|||||||
string message = 5;
|
string message = 5;
|
||||||
|
|
||||||
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
// Structured fields extracted by the agent's parser (e.g. RFC 5424
|
||||||
// syslog header fields). Empty when the raw-passthrough fallback fires.
|
// syslog header fields), plus source-provided fields (e.g. Windows
|
||||||
|
// Event Log's winevt.event_id/winevt.provider/winevt.channel). Empty
|
||||||
|
// when the raw-passthrough fallback fires and the source added nothing.
|
||||||
map<string, string> attributes = 6;
|
map<string, string> attributes = 6;
|
||||||
|
|
||||||
|
// Stable per-record identifier, used to join Tantivy full-text search
|
||||||
|
// hits back to their ClickHouse row (Phase 1). Always empty as sent by
|
||||||
|
// the agent — ingest's PushBatch handler assigns this server-side,
|
||||||
|
// once, before producing to Redpanda, since both the ClickHouse-writer
|
||||||
|
// consumer and the Tantivy-indexer consumer read the same Redpanda
|
||||||
|
// messages and need to agree on the same ID for the same record. See
|
||||||
|
// /ingest/README.md.
|
||||||
|
string record_id = 7;
|
||||||
}
|
}
|
||||||
|
|
||||||
message PushBatchRequest {
|
message PushBatchRequest {
|
||||||
|
|||||||
@@ -0,0 +1,192 @@
|
|||||||
|
// Code generated by protoc-gen-go. DO NOT EDIT.
|
||||||
|
// versions:
|
||||||
|
// protoc-gen-go v1.36.12
|
||||||
|
// protoc v7.35.1
|
||||||
|
// source: sentry/search/v1/search.proto
|
||||||
|
|
||||||
|
package searchv1
|
||||||
|
|
||||||
|
import (
|
||||||
|
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
|
||||||
|
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
|
||||||
|
reflect "reflect"
|
||||||
|
sync "sync"
|
||||||
|
unsafe "unsafe"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// Verify that this generated code is sufficiently up-to-date.
|
||||||
|
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
|
||||||
|
// Verify that runtime/protoimpl is sufficiently up-to-date.
|
||||||
|
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
|
||||||
|
)
|
||||||
|
|
||||||
|
type SearchRequest struct {
|
||||||
|
state protoimpl.MessageState `protogen:"open.v1"`
|
||||||
|
// Free-text query passed to Tantivy's query parser as-is. Supports
|
||||||
|
// phrase queries ("exact phrase") and wildcards (foo*) per Tantivy's
|
||||||
|
// own query syntax — see /search/README.md for exactly what that does
|
||||||
|
// and doesn't support in Phase 1.
|
||||||
|
Query string `protobuf:"bytes,1,opt,name=query,proto3" json:"query,omitempty"`
|
||||||
|
// Max results to return. 0 (unset) uses the service's own default.
|
||||||
|
Limit uint32 `protobuf:"varint,2,opt,name=limit,proto3" json:"limit,omitempty"`
|
||||||
|
unknownFields protoimpl.UnknownFields
|
||||||
|
sizeCache protoimpl.SizeCache
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchRequest) Reset() {
|
||||||
|
*x = SearchRequest{}
|
||||||
|
mi := &file_sentry_search_v1_search_proto_msgTypes[0]
|
||||||
|
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||||
|
ms.StoreMessageInfo(mi)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchRequest) String() string {
|
||||||
|
return protoimpl.X.MessageStringOf(x)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (*SearchRequest) ProtoMessage() {}
|
||||||
|
|
||||||
|
func (x *SearchRequest) ProtoReflect() protoreflect.Message {
|
||||||
|
mi := &file_sentry_search_v1_search_proto_msgTypes[0]
|
||||||
|
if x != nil {
|
||||||
|
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||||
|
if ms.LoadMessageInfo() == nil {
|
||||||
|
ms.StoreMessageInfo(mi)
|
||||||
|
}
|
||||||
|
return ms
|
||||||
|
}
|
||||||
|
return mi.MessageOf(x)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deprecated: Use SearchRequest.ProtoReflect.Descriptor instead.
|
||||||
|
func (*SearchRequest) Descriptor() ([]byte, []int) {
|
||||||
|
return file_sentry_search_v1_search_proto_rawDescGZIP(), []int{0}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchRequest) GetQuery() string {
|
||||||
|
if x != nil {
|
||||||
|
return x.Query
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchRequest) GetLimit() uint32 {
|
||||||
|
if x != nil {
|
||||||
|
return x.Limit
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
type SearchResponse struct {
|
||||||
|
state protoimpl.MessageState `protogen:"open.v1"`
|
||||||
|
// record_ids of matching logs, most-relevant first. Callers join these
|
||||||
|
// back against ClickHouse's `logs.record_id` column to get full rows —
|
||||||
|
// this service only ever returns IDs, never row data, so it stays a
|
||||||
|
// pure text index rather than a second copy of the row.
|
||||||
|
RecordIds []string `protobuf:"bytes,1,rep,name=record_ids,json=recordIds,proto3" json:"record_ids,omitempty"`
|
||||||
|
unknownFields protoimpl.UnknownFields
|
||||||
|
sizeCache protoimpl.SizeCache
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchResponse) Reset() {
|
||||||
|
*x = SearchResponse{}
|
||||||
|
mi := &file_sentry_search_v1_search_proto_msgTypes[1]
|
||||||
|
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||||
|
ms.StoreMessageInfo(mi)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchResponse) String() string {
|
||||||
|
return protoimpl.X.MessageStringOf(x)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (*SearchResponse) ProtoMessage() {}
|
||||||
|
|
||||||
|
func (x *SearchResponse) ProtoReflect() protoreflect.Message {
|
||||||
|
mi := &file_sentry_search_v1_search_proto_msgTypes[1]
|
||||||
|
if x != nil {
|
||||||
|
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||||
|
if ms.LoadMessageInfo() == nil {
|
||||||
|
ms.StoreMessageInfo(mi)
|
||||||
|
}
|
||||||
|
return ms
|
||||||
|
}
|
||||||
|
return mi.MessageOf(x)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deprecated: Use SearchResponse.ProtoReflect.Descriptor instead.
|
||||||
|
func (*SearchResponse) Descriptor() ([]byte, []int) {
|
||||||
|
return file_sentry_search_v1_search_proto_rawDescGZIP(), []int{1}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *SearchResponse) GetRecordIds() []string {
|
||||||
|
if x != nil {
|
||||||
|
return x.RecordIds
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var File_sentry_search_v1_search_proto protoreflect.FileDescriptor
|
||||||
|
|
||||||
|
const file_sentry_search_v1_search_proto_rawDesc = "" +
|
||||||
|
"\n" +
|
||||||
|
"\x1dsentry/search/v1/search.proto\x12\x10sentry.search.v1\";\n" +
|
||||||
|
"\rSearchRequest\x12\x14\n" +
|
||||||
|
"\x05query\x18\x01 \x01(\tR\x05query\x12\x14\n" +
|
||||||
|
"\x05limit\x18\x02 \x01(\rR\x05limit\"/\n" +
|
||||||
|
"\x0eSearchResponse\x12\x1d\n" +
|
||||||
|
"\n" +
|
||||||
|
"record_ids\x18\x01 \x03(\tR\trecordIds2\\\n" +
|
||||||
|
"\rSearchService\x12K\n" +
|
||||||
|
"\x06Search\x12\x1f.sentry.search.v1.SearchRequest\x1a .sentry.search.v1.SearchResponseB:Z8github.com/sentry/sentry/proto/sentry/search/v1;searchv1b\x06proto3"
|
||||||
|
|
||||||
|
var (
|
||||||
|
file_sentry_search_v1_search_proto_rawDescOnce sync.Once
|
||||||
|
file_sentry_search_v1_search_proto_rawDescData []byte
|
||||||
|
)
|
||||||
|
|
||||||
|
func file_sentry_search_v1_search_proto_rawDescGZIP() []byte {
|
||||||
|
file_sentry_search_v1_search_proto_rawDescOnce.Do(func() {
|
||||||
|
file_sentry_search_v1_search_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_sentry_search_v1_search_proto_rawDesc), len(file_sentry_search_v1_search_proto_rawDesc)))
|
||||||
|
})
|
||||||
|
return file_sentry_search_v1_search_proto_rawDescData
|
||||||
|
}
|
||||||
|
|
||||||
|
var file_sentry_search_v1_search_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
|
||||||
|
var file_sentry_search_v1_search_proto_goTypes = []any{
|
||||||
|
(*SearchRequest)(nil), // 0: sentry.search.v1.SearchRequest
|
||||||
|
(*SearchResponse)(nil), // 1: sentry.search.v1.SearchResponse
|
||||||
|
}
|
||||||
|
var file_sentry_search_v1_search_proto_depIdxs = []int32{
|
||||||
|
0, // 0: sentry.search.v1.SearchService.Search:input_type -> sentry.search.v1.SearchRequest
|
||||||
|
1, // 1: sentry.search.v1.SearchService.Search:output_type -> sentry.search.v1.SearchResponse
|
||||||
|
1, // [1:2] is the sub-list for method output_type
|
||||||
|
0, // [0:1] is the sub-list for method input_type
|
||||||
|
0, // [0:0] is the sub-list for extension type_name
|
||||||
|
0, // [0:0] is the sub-list for extension extendee
|
||||||
|
0, // [0:0] is the sub-list for field type_name
|
||||||
|
}
|
||||||
|
|
||||||
|
func init() { file_sentry_search_v1_search_proto_init() }
|
||||||
|
func file_sentry_search_v1_search_proto_init() {
|
||||||
|
if File_sentry_search_v1_search_proto != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
type x struct{}
|
||||||
|
out := protoimpl.TypeBuilder{
|
||||||
|
File: protoimpl.DescBuilder{
|
||||||
|
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||||
|
RawDescriptor: unsafe.Slice(unsafe.StringData(file_sentry_search_v1_search_proto_rawDesc), len(file_sentry_search_v1_search_proto_rawDesc)),
|
||||||
|
NumEnums: 0,
|
||||||
|
NumMessages: 2,
|
||||||
|
NumExtensions: 0,
|
||||||
|
NumServices: 1,
|
||||||
|
},
|
||||||
|
GoTypes: file_sentry_search_v1_search_proto_goTypes,
|
||||||
|
DependencyIndexes: file_sentry_search_v1_search_proto_depIdxs,
|
||||||
|
MessageInfos: file_sentry_search_v1_search_proto_msgTypes,
|
||||||
|
}.Build()
|
||||||
|
File_sentry_search_v1_search_proto = out.File
|
||||||
|
file_sentry_search_v1_search_proto_goTypes = nil
|
||||||
|
file_sentry_search_v1_search_proto_depIdxs = nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
syntax = "proto3";
|
||||||
|
|
||||||
|
package sentry.search.v1;
|
||||||
|
|
||||||
|
option go_package = "github.com/sentry/sentry/proto/sentry/search/v1;searchv1";
|
||||||
|
|
||||||
|
// SearchService is the full-text search index (Tantivy-backed) that
|
||||||
|
// `api` calls to resolve a free-text query into matching record_ids,
|
||||||
|
// which `api` then joins back against ClickHouse. Internal service-to-
|
||||||
|
// service call, same gRPC-first convention as agent<->ingest — see
|
||||||
|
// /docs/architecture.md and /search/README.md.
|
||||||
|
service SearchService {
|
||||||
|
rpc Search(SearchRequest) returns (SearchResponse);
|
||||||
|
}
|
||||||
|
|
||||||
|
message SearchRequest {
|
||||||
|
// Free-text query passed to Tantivy's query parser as-is. Supports
|
||||||
|
// phrase queries ("exact phrase") and wildcards (foo*) per Tantivy's
|
||||||
|
// own query syntax — see /search/README.md for exactly what that does
|
||||||
|
// and doesn't support in Phase 1.
|
||||||
|
string query = 1;
|
||||||
|
|
||||||
|
// Max results to return. 0 (unset) uses the service's own default.
|
||||||
|
uint32 limit = 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
message SearchResponse {
|
||||||
|
// record_ids of matching logs, most-relevant first. Callers join these
|
||||||
|
// back against ClickHouse's `logs.record_id` column to get full rows —
|
||||||
|
// this service only ever returns IDs, never row data, so it stays a
|
||||||
|
// pure text index rather than a second copy of the row.
|
||||||
|
repeated string record_ids = 1;
|
||||||
|
}
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
|
||||||
|
// versions:
|
||||||
|
// - protoc-gen-go-grpc v1.6.2
|
||||||
|
// - protoc v7.35.1
|
||||||
|
// source: sentry/search/v1/search.proto
|
||||||
|
|
||||||
|
package searchv1
|
||||||
|
|
||||||
|
import (
|
||||||
|
context "context"
|
||||||
|
grpc "google.golang.org/grpc"
|
||||||
|
codes "google.golang.org/grpc/codes"
|
||||||
|
status "google.golang.org/grpc/status"
|
||||||
|
)
|
||||||
|
|
||||||
|
// This is a compile-time assertion to ensure that this generated file
|
||||||
|
// is compatible with the grpc package it is being compiled against.
|
||||||
|
// Requires gRPC-Go v1.64.0 or later.
|
||||||
|
const _ = grpc.SupportPackageIsVersion9
|
||||||
|
|
||||||
|
const (
|
||||||
|
SearchService_Search_FullMethodName = "/sentry.search.v1.SearchService/Search"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SearchServiceClient is the client API for SearchService service.
|
||||||
|
//
|
||||||
|
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
|
||||||
|
//
|
||||||
|
// SearchService is the full-text search index (Tantivy-backed) that
|
||||||
|
// `api` calls to resolve a free-text query into matching record_ids,
|
||||||
|
// which `api` then joins back against ClickHouse. Internal service-to-
|
||||||
|
// service call, same gRPC-first convention as agent<->ingest — see
|
||||||
|
// /docs/architecture.md and /search/README.md.
|
||||||
|
type SearchServiceClient interface {
|
||||||
|
Search(ctx context.Context, in *SearchRequest, opts ...grpc.CallOption) (*SearchResponse, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type searchServiceClient struct {
|
||||||
|
cc grpc.ClientConnInterface
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewSearchServiceClient(cc grpc.ClientConnInterface) SearchServiceClient {
|
||||||
|
return &searchServiceClient{cc}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *searchServiceClient) Search(ctx context.Context, in *SearchRequest, opts ...grpc.CallOption) (*SearchResponse, error) {
|
||||||
|
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||||
|
out := new(SearchResponse)
|
||||||
|
err := c.cc.Invoke(ctx, SearchService_Search_FullMethodName, in, out, cOpts...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SearchServiceServer is the server API for SearchService service.
|
||||||
|
// All implementations must embed UnimplementedSearchServiceServer
|
||||||
|
// for forward compatibility.
|
||||||
|
//
|
||||||
|
// SearchService is the full-text search index (Tantivy-backed) that
|
||||||
|
// `api` calls to resolve a free-text query into matching record_ids,
|
||||||
|
// which `api` then joins back against ClickHouse. Internal service-to-
|
||||||
|
// service call, same gRPC-first convention as agent<->ingest — see
|
||||||
|
// /docs/architecture.md and /search/README.md.
|
||||||
|
type SearchServiceServer interface {
|
||||||
|
Search(context.Context, *SearchRequest) (*SearchResponse, error)
|
||||||
|
mustEmbedUnimplementedSearchServiceServer()
|
||||||
|
}
|
||||||
|
|
||||||
|
// UnimplementedSearchServiceServer must be embedded to have
|
||||||
|
// forward compatible implementations.
|
||||||
|
//
|
||||||
|
// NOTE: this should be embedded by value instead of pointer to avoid a nil
|
||||||
|
// pointer dereference when methods are called.
|
||||||
|
type UnimplementedSearchServiceServer struct{}
|
||||||
|
|
||||||
|
func (UnimplementedSearchServiceServer) Search(context.Context, *SearchRequest) (*SearchResponse, error) {
|
||||||
|
return nil, status.Error(codes.Unimplemented, "method Search not implemented")
|
||||||
|
}
|
||||||
|
func (UnimplementedSearchServiceServer) mustEmbedUnimplementedSearchServiceServer() {}
|
||||||
|
func (UnimplementedSearchServiceServer) testEmbeddedByValue() {}
|
||||||
|
|
||||||
|
// UnsafeSearchServiceServer may be embedded to opt out of forward compatibility for this service.
|
||||||
|
// Use of this interface is not recommended, as added methods to SearchServiceServer will
|
||||||
|
// result in compilation errors.
|
||||||
|
type UnsafeSearchServiceServer interface {
|
||||||
|
mustEmbedUnimplementedSearchServiceServer()
|
||||||
|
}
|
||||||
|
|
||||||
|
func RegisterSearchServiceServer(s grpc.ServiceRegistrar, srv SearchServiceServer) {
|
||||||
|
// If the following call panics, it indicates UnimplementedSearchServiceServer was
|
||||||
|
// embedded by pointer and is nil. This will cause panics if an
|
||||||
|
// unimplemented method is ever invoked, so we test this at initialization
|
||||||
|
// time to prevent it from happening at runtime later due to I/O.
|
||||||
|
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
|
||||||
|
t.testEmbeddedByValue()
|
||||||
|
}
|
||||||
|
s.RegisterService(&SearchService_ServiceDesc, srv)
|
||||||
|
}
|
||||||
|
|
||||||
|
func _SearchService_Search_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||||
|
in := new(SearchRequest)
|
||||||
|
if err := dec(in); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if interceptor == nil {
|
||||||
|
return srv.(SearchServiceServer).Search(ctx, in)
|
||||||
|
}
|
||||||
|
info := &grpc.UnaryServerInfo{
|
||||||
|
Server: srv,
|
||||||
|
FullMethod: SearchService_Search_FullMethodName,
|
||||||
|
}
|
||||||
|
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||||
|
return srv.(SearchServiceServer).Search(ctx, req.(*SearchRequest))
|
||||||
|
}
|
||||||
|
return interceptor(ctx, in, info, handler)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SearchService_ServiceDesc is the grpc.ServiceDesc for SearchService service.
|
||||||
|
// It's only intended for direct use with grpc.RegisterService,
|
||||||
|
// and not to be introspected or modified (even as a copy)
|
||||||
|
var SearchService_ServiceDesc = grpc.ServiceDesc{
|
||||||
|
ServiceName: "sentry.search.v1.SearchService",
|
||||||
|
HandlerType: (*SearchServiceServer)(nil),
|
||||||
|
Methods: []grpc.MethodDesc{
|
||||||
|
{
|
||||||
|
MethodName: "Search",
|
||||||
|
Handler: _SearchService_Search_Handler,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Streams: []grpc.StreamDesc{},
|
||||||
|
Metadata: "sentry/search/v1/search.proto",
|
||||||
|
}
|
||||||
Generated
+2354
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
|||||||
|
[package]
|
||||||
|
name = "sentry-search"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
license = "AGPL-3.0-only"
|
||||||
|
description = "Sentry Tantivy-backed full-text search service"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "sentry-search"
|
||||||
|
path = "src/main.rs"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "fs", "sync", "signal", "time"] }
|
||||||
|
tonic = "0.12"
|
||||||
|
prost = "0.13"
|
||||||
|
|
||||||
|
tantivy = "0.22"
|
||||||
|
rskafka = "0.6"
|
||||||
|
|
||||||
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
serde_json = "1"
|
||||||
|
|
||||||
|
anyhow = "1"
|
||||||
|
tracing = "0.1"
|
||||||
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
|
|
||||||
|
[build-dependencies]
|
||||||
|
tonic-build = "0.12"
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
# Already in the dependency graph transitively (tantivy/tonic-build both
|
||||||
|
# pull it in); promoted to a direct dev-dependency for test use.
|
||||||
|
tempfile = "3"
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Build context must be the repo root (sentry/), since this needs both
|
||||||
|
# search/ and proto/:
|
||||||
|
# docker build -f search/Dockerfile -t sentry-search .
|
||||||
|
#
|
||||||
|
# Unlike /agent, this doesn't need musl/static linking -- it's a normal
|
||||||
|
# server-side container, not an edge-deployed binary, and both Tantivy and
|
||||||
|
# rskafka are pure Rust (no C deps to link against). distroless/cc rather
|
||||||
|
# than distroless/static: a normal dynamically-linked binary, not a fully
|
||||||
|
# static one.
|
||||||
|
FROM rust:1-slim AS builder
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends protobuf-compiler && rm -rf /var/lib/apt/lists/*
|
||||||
|
WORKDIR /src
|
||||||
|
COPY proto ./proto
|
||||||
|
COPY search ./search
|
||||||
|
WORKDIR /src/search
|
||||||
|
RUN cargo build --release
|
||||||
|
|
||||||
|
FROM gcr.io/distroless/cc-debian12
|
||||||
|
COPY --from=builder /src/search/target/release/sentry-search /sentry-search
|
||||||
|
ENTRYPOINT ["/sentry-search"]
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# search
|
||||||
|
|
||||||
|
Tantivy-backed full-text search over log messages. Phase 1's answer to
|
||||||
|
"grep across everything," separate from ClickHouse's structured/
|
||||||
|
aggregation queries.
|
||||||
|
|
||||||
|
## Why a separate service, not embedded in ingest
|
||||||
|
|
||||||
|
Tantivy is a Rust library with no maintained Go bindings — using it from
|
||||||
|
`ingest` (Go) would mean cgo-bridging to a compiled Rust cdylib, exactly
|
||||||
|
the fragile FFI complexity CLAUDE.md's "prefer boring, well-understood
|
||||||
|
dependencies... operators need to trust it" principle steers away from.
|
||||||
|
It would also couple ClickHouse-write latency to Tantivy-write latency in
|
||||||
|
the same request path. See `/docs/architecture.md` for the fuller
|
||||||
|
tradeoff writeup (dual-write vs. second-consumer-group) from Phase 1
|
||||||
|
planning.
|
||||||
|
|
||||||
|
## How it fits together
|
||||||
|
|
||||||
|
```
|
||||||
|
ingest (gRPC front end) --> Redpanda (sentry.logs.raw) --> ingest's ClickHouse-writer consumer --> ClickHouse
|
||||||
|
\
|
||||||
|
`--> search's own consumer --> Tantivy index
|
||||||
|
```
|
||||||
|
|
||||||
|
`search` reads the *same* Redpanda topic `ingest`'s ClickHouse-writer
|
||||||
|
consumer reads, as an independent consumer in spirit (own offset
|
||||||
|
tracking, own failure domain — see "Offset tracking" below) even though
|
||||||
|
it's a completely separate process/service. If Tantivy indexing lags or
|
||||||
|
crashes, ClickHouse ingestion is completely unaffected.
|
||||||
|
|
||||||
|
`api` calls `search`'s `SearchService.Search` gRPC RPC (see
|
||||||
|
`/proto/sentry/search/v1/search.proto`) to resolve a free-text query into
|
||||||
|
matching `record_id`s, then joins those back against ClickHouse's
|
||||||
|
`logs.record_id` column (see `/storage/migrations/0002_add_record_id.sql`)
|
||||||
|
to get full rows. `search` only ever returns IDs, never row data — it
|
||||||
|
stays a pure text index, not a second copy of the row.
|
||||||
|
|
||||||
|
## Offset tracking: why this isn't a Kafka consumer group
|
||||||
|
|
||||||
|
`rskafka` (chosen for being pure Rust, no cgo — consistent with why
|
||||||
|
`ingest` chose `segmentio/kafka-go` over `confluent-kafka-go`) is a
|
||||||
|
low-level client: it doesn't implement Kafka's broker-side consumer-group
|
||||||
|
coordination protocol the way `kafka-go` or `librdkafka` do. So `search`
|
||||||
|
tracks its own per-partition offsets in a plain JSON file next to the
|
||||||
|
Tantivy index (`offsets.rs`), persisted after each fetched batch.
|
||||||
|
|
||||||
|
This is deliberately best-effort, not exactly-once: if the process dies
|
||||||
|
between processing a record and persisting its offset, that record gets
|
||||||
|
reprocessed after restart. This is safe because `SearchIndex::upsert` is
|
||||||
|
**delete-then-add on `record_id`** — Tantivy segments are immutable, so
|
||||||
|
this is the standard idiom for updates anyway, and it happens to make
|
||||||
|
reprocessing idempotent for free. Partition count is read from
|
||||||
|
`REDPANDA_TOPIC_PARTITIONS` (must match what
|
||||||
|
`/transport/provision-topics.sh` actually created — same kind of
|
||||||
|
documented cross-component contract as the topic name itself), not
|
||||||
|
discovered dynamically.
|
||||||
|
|
||||||
|
## Query syntax
|
||||||
|
|
||||||
|
Whatever Tantivy's own `QueryParser` supports against the `message`
|
||||||
|
field: plain terms, `"exact phrase"` queries, and `foo*` wildcards. Not
|
||||||
|
documented further here because it's Tantivy's syntax, not Sentry's — see
|
||||||
|
[Tantivy's query parser docs](https://docs.rs/tantivy/latest/tantivy/query/struct.QueryParser.html)
|
||||||
|
for the full grammar. No unified query language yet; that's Phase 2.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Environment variables (see `src/config.rs`):
|
||||||
|
|
||||||
|
| Var | Default | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `GRPC_LISTEN_ADDR` | `0.0.0.0:50052` | Full socket address — Rust's parser needs one, unlike Go's `:PORT` shorthand `ingest`/`api` use |
|
||||||
|
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
|
||||||
|
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match `/ingest`'s topic |
|
||||||
|
| `REDPANDA_TOPIC_PARTITIONS` | `6` | Must match what `/transport/provision-topics.sh` created |
|
||||||
|
| `INDEX_PATH` | `/var/lib/sentry-search/index` | Tantivy index directory |
|
||||||
|
| `OFFSETS_PATH` | `/var/lib/sentry-search/offsets.json` | Offset tracking file |
|
||||||
|
| `COMMIT_INTERVAL_MS` | `2000` | How often buffered writes become searchable |
|
||||||
|
|
||||||
|
## Building & testing
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo build --release
|
||||||
|
cargo clippy --all-targets -- -D warnings
|
||||||
|
cargo test
|
||||||
|
```
|
||||||
|
|
||||||
|
`index.rs`'s tests run against a real (temp-directory) Tantivy index —
|
||||||
|
no external service needed, unlike ClickHouse. They cover the
|
||||||
|
delete-then-add idempotency, phrase queries, result limits, and the
|
||||||
|
before-commit/after-commit visibility boundary. `consumer.rs` (the
|
||||||
|
rskafka wiring) is not unit-tested — that needs a real Redpanda, same
|
||||||
|
category of gap as `/ingest`'s `kafka.Reader`/`kafka.Writer` wiring, and
|
||||||
|
is exercised by the docker-compose end-to-end flow instead.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# from the repo root, not search/
|
||||||
|
docker build -f search/Dockerfile -t sentry-search .
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||||
|
// logs.proto is compiled only for the LogRecord message type (to
|
||||||
|
// decode what's read off Redpanda) -- this service never calls or
|
||||||
|
// implements LogIngest. search.proto is compiled for the
|
||||||
|
// SearchService server this binary implements.
|
||||||
|
tonic_build::configure().compile_protos(
|
||||||
|
&[
|
||||||
|
"../proto/sentry/logs/v1/logs.proto",
|
||||||
|
"../proto/sentry/search/v1/search.proto",
|
||||||
|
],
|
||||||
|
&["../proto"],
|
||||||
|
)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
/// All via environment variables, same convention as /ingest and /api —
|
||||||
|
/// no config file format for this service either.
|
||||||
|
pub struct Config {
|
||||||
|
pub grpc_listen_addr: String,
|
||||||
|
pub redpanda_brokers: Vec<String>,
|
||||||
|
pub redpanda_topic: String,
|
||||||
|
pub index_path: PathBuf,
|
||||||
|
pub offsets_path: PathBuf,
|
||||||
|
pub commit_interval: Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Config {
|
||||||
|
pub fn load() -> Result<Self> {
|
||||||
|
let commit_interval_ms: u64 = getenv("COMMIT_INTERVAL_MS", "2000")
|
||||||
|
.parse()
|
||||||
|
.context("COMMIT_INTERVAL_MS must be a number")?;
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
// Rust's SocketAddr parser needs a full address, unlike Go's
|
||||||
|
// net package (ingest/api's ":PORT" convention won't parse
|
||||||
|
// here).
|
||||||
|
grpc_listen_addr: getenv("GRPC_LISTEN_ADDR", "0.0.0.0:50052"),
|
||||||
|
redpanda_brokers: getenv("REDPANDA_BROKERS", "localhost:9092")
|
||||||
|
.split(',')
|
||||||
|
.map(str::to_string)
|
||||||
|
.collect(),
|
||||||
|
redpanda_topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
|
||||||
|
index_path: PathBuf::from(getenv("INDEX_PATH", "/var/lib/sentry-search/index")),
|
||||||
|
offsets_path: PathBuf::from(getenv(
|
||||||
|
"OFFSETS_PATH",
|
||||||
|
"/var/lib/sentry-search/offsets.json",
|
||||||
|
)),
|
||||||
|
commit_interval: Duration::from_millis(commit_interval_ms),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn getenv(key: &str, fallback: &str) -> String {
|
||||||
|
std::env::var(key).unwrap_or_else(|_| fallback.to_string())
|
||||||
|
}
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use prost::Message;
|
||||||
|
use rskafka::client::partition::UnknownTopicHandling;
|
||||||
|
use rskafka::client::ClientBuilder;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
|
||||||
|
use crate::config::Config;
|
||||||
|
use crate::index::SearchIndex;
|
||||||
|
use crate::logsv1;
|
||||||
|
use crate::offsets::OffsetStore;
|
||||||
|
|
||||||
|
/// Reads the same `sentry.logs.raw` topic ingest's ClickHouse-writer
|
||||||
|
/// consumer reads, as an independent consumer group in spirit (its own
|
||||||
|
/// offset tracking, own failure domain) even though rskafka doesn't speak
|
||||||
|
/// Kafka's broker-side consumer-group protocol -- see offsets.rs. One
|
||||||
|
/// task per partition; partition count comes from config rather than
|
||||||
|
/// discovered dynamically, since it has to match what
|
||||||
|
/// /transport/provision-topics.sh actually created anyway (documented
|
||||||
|
/// cross-component contract, same as the topic name already is).
|
||||||
|
pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32) -> Result<()> {
|
||||||
|
let client = ClientBuilder::new(cfg.redpanda_brokers.clone())
|
||||||
|
.build()
|
||||||
|
.await
|
||||||
|
.context("building rskafka client")?;
|
||||||
|
let client = Arc::new(client);
|
||||||
|
|
||||||
|
let offsets = OffsetStore::load(&cfg.offsets_path)
|
||||||
|
.await
|
||||||
|
.context("loading offset store")?;
|
||||||
|
let offsets = Arc::new(Mutex::new(offsets));
|
||||||
|
|
||||||
|
// Periodic Tantivy commit, batched for throughput the same way
|
||||||
|
// ingest's ClickHouse writer batches inserts rather than inserting
|
||||||
|
// per-record.
|
||||||
|
let commit_interval = cfg.commit_interval;
|
||||||
|
let index_for_commit = Arc::clone(&index);
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let mut ticker = tokio::time::interval(commit_interval);
|
||||||
|
loop {
|
||||||
|
ticker.tick().await;
|
||||||
|
if let Err(e) = index_for_commit.commit().await {
|
||||||
|
tracing::error!(error = %e, "periodic tantivy commit failed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut handles = Vec::with_capacity(partition_count as usize);
|
||||||
|
for partition in 0..partition_count {
|
||||||
|
let start_offset = offsets.lock().await.get(partition);
|
||||||
|
let client = Arc::clone(&client);
|
||||||
|
let index = Arc::clone(&index);
|
||||||
|
let offsets = Arc::clone(&offsets);
|
||||||
|
let topic = cfg.redpanda_topic.clone();
|
||||||
|
handles.push(tokio::spawn(async move {
|
||||||
|
consume_partition(client, topic, partition, start_offset, index, offsets).await
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
for handle in handles {
|
||||||
|
handle
|
||||||
|
.await
|
||||||
|
.context("partition consumer task panicked")??;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
async fn consume_partition(
|
||||||
|
client: Arc<rskafka::client::Client>,
|
||||||
|
topic: String,
|
||||||
|
partition: i32,
|
||||||
|
start_offset: i64,
|
||||||
|
index: Arc<SearchIndex>,
|
||||||
|
offsets: Arc<Mutex<OffsetStore>>,
|
||||||
|
) -> Result<()> {
|
||||||
|
let partition_client = client
|
||||||
|
.partition_client(topic.clone(), partition, UnknownTopicHandling::Error)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("creating partition client for {topic}[{partition}]"))?;
|
||||||
|
|
||||||
|
let mut offset = start_offset;
|
||||||
|
loop {
|
||||||
|
let (records, _high_watermark) = partition_client
|
||||||
|
.fetch_records(offset, 1..1_000_000, 5_000)
|
||||||
|
.await
|
||||||
|
.with_context(|| {
|
||||||
|
format!("fetching records from {topic}[{partition}] at offset {offset}")
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if records.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
for record_and_offset in &records {
|
||||||
|
offset = record_and_offset.offset + 1;
|
||||||
|
|
||||||
|
let Some(value) = &record_and_offset.record.value else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let rec = match logsv1::LogRecord::decode(value.as_slice()) {
|
||||||
|
Ok(rec) => rec,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(error = %e, partition, "skipping unparseable message");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if rec.record_id.is_empty() {
|
||||||
|
// Shouldn't happen -- ingest's gRPC front end always
|
||||||
|
// assigns this before producing -- but a message with no
|
||||||
|
// ID can't be joined back to a ClickHouse row, so it's
|
||||||
|
// useless to index.
|
||||||
|
tracing::warn!(partition, "skipping record with empty record_id");
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Err(e) = index.upsert(&rec.record_id, &rec.message).await {
|
||||||
|
tracing::error!(error = %e, record_id = %rec.record_id, "failed to index record");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persisted after each fetched batch, not per-record: worst-case
|
||||||
|
// reprocessing on an unclean restart is one batch, which
|
||||||
|
// `SearchIndex::upsert`'s delete-then-add makes harmless anyway.
|
||||||
|
{
|
||||||
|
let mut offsets = offsets.lock().await;
|
||||||
|
offsets.set(partition, offset);
|
||||||
|
if let Err(e) = offsets.persist().await {
|
||||||
|
tracing::error!(error = %e, partition, "failed to persist offset");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
use std::sync::Arc;
|
||||||
|
use tonic::{Request, Response, Status};
|
||||||
|
|
||||||
|
use crate::index::SearchIndex;
|
||||||
|
use crate::searchv1;
|
||||||
|
|
||||||
|
const DEFAULT_LIMIT: usize = 100;
|
||||||
|
|
||||||
|
pub struct SearchServer {
|
||||||
|
index: Arc<SearchIndex>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SearchServer {
|
||||||
|
pub fn new(index: Arc<SearchIndex>) -> Self {
|
||||||
|
Self { index }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tonic::async_trait]
|
||||||
|
impl searchv1::search_service_server::SearchService for SearchServer {
|
||||||
|
async fn search(
|
||||||
|
&self,
|
||||||
|
request: Request<searchv1::SearchRequest>,
|
||||||
|
) -> Result<Response<searchv1::SearchResponse>, Status> {
|
||||||
|
let req = request.into_inner();
|
||||||
|
if req.query.trim().is_empty() {
|
||||||
|
return Err(Status::invalid_argument("query must not be empty"));
|
||||||
|
}
|
||||||
|
|
||||||
|
let limit = if req.limit == 0 {
|
||||||
|
DEFAULT_LIMIT
|
||||||
|
} else {
|
||||||
|
req.limit as usize
|
||||||
|
};
|
||||||
|
|
||||||
|
let index = Arc::clone(&self.index);
|
||||||
|
let query = req.query.clone();
|
||||||
|
// Tantivy's searcher is synchronous; run it on a blocking thread
|
||||||
|
// so it doesn't stall the async runtime alongside the consumer
|
||||||
|
// tasks.
|
||||||
|
let record_ids = tokio::task::spawn_blocking(move || index.search(&query, limit))
|
||||||
|
.await
|
||||||
|
.map_err(|e| Status::internal(format!("search task panicked: {e}")))?
|
||||||
|
.map_err(|e| Status::invalid_argument(format!("search failed: {e}")))?;
|
||||||
|
|
||||||
|
Ok(Response::new(searchv1::SearchResponse { record_ids }))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::path::Path;
|
||||||
|
use tantivy::collector::TopDocs;
|
||||||
|
use tantivy::query::QueryParser;
|
||||||
|
use tantivy::schema::{Schema, Value, STORED, STRING, TEXT};
|
||||||
|
use tantivy::{doc, Index, IndexReader, IndexWriter, ReloadPolicy, TantivyDocument, Term};
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
|
||||||
|
/// Minimal Tantivy index: a stable `record_id` (stored, exact-match) and
|
||||||
|
/// tokenized `message` text. Everything else (timestamp, host, service,
|
||||||
|
/// severity) is fetched by joining `record_id` back against ClickHouse in
|
||||||
|
/// `/api`'s search handler, not duplicated in here — this stays a pure
|
||||||
|
/// text index, not a second copy of the row.
|
||||||
|
pub struct SearchIndex {
|
||||||
|
index: Index,
|
||||||
|
writer: Mutex<IndexWriter>,
|
||||||
|
reader: IndexReader,
|
||||||
|
record_id_field: tantivy::schema::Field,
|
||||||
|
message_field: tantivy::schema::Field,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 50MB is Tantivy's own suggested minimum writer heap budget; Phase 1
|
||||||
|
/// has no real sizing data yet to tune this against.
|
||||||
|
const WRITER_HEAP_BYTES: usize = 50_000_000;
|
||||||
|
|
||||||
|
impl SearchIndex {
|
||||||
|
pub fn open_or_create(path: &Path) -> Result<Self> {
|
||||||
|
std::fs::create_dir_all(path).context("creating tantivy index directory")?;
|
||||||
|
|
||||||
|
let mut schema_builder = Schema::builder();
|
||||||
|
let record_id_field = schema_builder.add_text_field("record_id", STRING | STORED);
|
||||||
|
let message_field = schema_builder.add_text_field("message", TEXT);
|
||||||
|
let schema = schema_builder.build();
|
||||||
|
|
||||||
|
let dir = tantivy::directory::MmapDirectory::open(path)
|
||||||
|
.context("opening tantivy mmap directory")?;
|
||||||
|
let index =
|
||||||
|
Index::open_or_create(dir, schema).context("opening/creating tantivy index")?;
|
||||||
|
|
||||||
|
let writer = index
|
||||||
|
.writer(WRITER_HEAP_BYTES)
|
||||||
|
.context("creating tantivy index writer")?;
|
||||||
|
|
||||||
|
let reader = index
|
||||||
|
.reader_builder()
|
||||||
|
.reload_policy(ReloadPolicy::OnCommitWithDelay)
|
||||||
|
.try_into()
|
||||||
|
.context("building tantivy index reader")?;
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
index,
|
||||||
|
writer: Mutex::new(writer),
|
||||||
|
reader,
|
||||||
|
record_id_field,
|
||||||
|
message_field,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Upserts one record: delete-then-add on record_id. Tantivy segments
|
||||||
|
/// are immutable, so this delete-then-add is the standard idiom for
|
||||||
|
/// updates, not a workaround -- and it matters here specifically
|
||||||
|
/// because /search's offset tracking is best-effort (see
|
||||||
|
/// consumer.rs's OffsetStore), so the same record can genuinely be
|
||||||
|
/// reprocessed after an unclean shutdown. Without this, that would
|
||||||
|
/// silently duplicate documents instead of just re-writing the same
|
||||||
|
/// one.
|
||||||
|
pub async fn upsert(&self, record_id: &str, message: &str) -> Result<()> {
|
||||||
|
let writer = self.writer.lock().await;
|
||||||
|
let term = Term::from_field_text(self.record_id_field, record_id);
|
||||||
|
writer.delete_term(term);
|
||||||
|
writer
|
||||||
|
.add_document(doc!(
|
||||||
|
self.record_id_field => record_id,
|
||||||
|
self.message_field => message,
|
||||||
|
))
|
||||||
|
.context("adding document to tantivy index")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn commit(&self) -> Result<()> {
|
||||||
|
let mut writer = self.writer.lock().await;
|
||||||
|
writer.commit().context("committing tantivy index")?;
|
||||||
|
// Explicit reload rather than relying solely on ReloadPolicy::
|
||||||
|
// OnCommitWithDelay's background timing: callers of `commit()`
|
||||||
|
// (the periodic ticker in consumer.rs, and tests) expect a
|
||||||
|
// committed document to be immediately searchable, not visible
|
||||||
|
// after some undocumented delay.
|
||||||
|
self.reader.reload().context("reloading tantivy reader after commit")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs a Tantivy query-parser query against the `message` field,
|
||||||
|
/// returning matching record_ids, most-relevant first. Phase 1: no
|
||||||
|
/// pagination, no score exposed to the caller — just IDs for `/api`
|
||||||
|
/// to join against ClickHouse.
|
||||||
|
pub fn search(&self, query: &str, limit: usize) -> Result<Vec<String>> {
|
||||||
|
let searcher = self.reader.searcher();
|
||||||
|
let query_parser = QueryParser::for_index(&self.index, vec![self.message_field]);
|
||||||
|
let parsed_query = query_parser
|
||||||
|
.parse_query(query)
|
||||||
|
.context("parsing search query")?;
|
||||||
|
let top_docs = searcher
|
||||||
|
.search(&parsed_query, &TopDocs::with_limit(limit))
|
||||||
|
.context("executing search")?;
|
||||||
|
|
||||||
|
let mut ids = Vec::with_capacity(top_docs.len());
|
||||||
|
for (_score, doc_address) in top_docs {
|
||||||
|
let retrieved: TantivyDocument = searcher
|
||||||
|
.doc(doc_address)
|
||||||
|
.context("retrieving matched document")?;
|
||||||
|
if let Some(value) = retrieved.get_first(self.record_id_field) {
|
||||||
|
if let Some(s) = value.as_str() {
|
||||||
|
ids.push(s.to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn new_test_index() -> (SearchIndex, tempfile::TempDir) {
|
||||||
|
let dir = tempfile::tempdir().expect("creating temp dir");
|
||||||
|
let index = SearchIndex::open_or_create(dir.path()).expect("opening tantivy index");
|
||||||
|
(index, dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn upsert_and_search_finds_matching_message() {
|
||||||
|
let (index, _dir) = new_test_index();
|
||||||
|
index.upsert("id-1", "hello world").await.unwrap();
|
||||||
|
index.upsert("id-2", "goodbye moon").await.unwrap();
|
||||||
|
index.commit().await.unwrap();
|
||||||
|
|
||||||
|
let results = index.search("hello", 10).unwrap();
|
||||||
|
assert_eq!(results, vec!["id-1".to_string()]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn search_before_commit_finds_nothing() {
|
||||||
|
let (index, _dir) = new_test_index();
|
||||||
|
index.upsert("id-1", "hello world").await.unwrap();
|
||||||
|
// no commit yet
|
||||||
|
let results = index.search("hello", 10).unwrap();
|
||||||
|
assert!(results.is_empty(), "expected no results before commit, got {results:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn upsert_same_id_twice_does_not_duplicate() {
|
||||||
|
let (index, _dir) = new_test_index();
|
||||||
|
index.upsert("id-1", "hello world").await.unwrap();
|
||||||
|
index.commit().await.unwrap();
|
||||||
|
index.upsert("id-1", "hello world again").await.unwrap();
|
||||||
|
index.commit().await.unwrap();
|
||||||
|
|
||||||
|
let results = index.search("hello", 10).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
results.len(),
|
||||||
|
1,
|
||||||
|
"expected exactly one result after re-upserting the same record_id, got {results:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn search_respects_limit() {
|
||||||
|
let (index, _dir) = new_test_index();
|
||||||
|
for i in 0..5 {
|
||||||
|
index
|
||||||
|
.upsert(&format!("id-{i}"), "shared term")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
index.commit().await.unwrap();
|
||||||
|
|
||||||
|
let results = index.search("shared", 2).unwrap();
|
||||||
|
assert_eq!(results.len(), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn search_supports_phrase_queries() {
|
||||||
|
let (index, _dir) = new_test_index();
|
||||||
|
index.upsert("id-1", "the quick brown fox").await.unwrap();
|
||||||
|
index.upsert("id-2", "quick and brown but not adjacent fox").await.unwrap();
|
||||||
|
index.commit().await.unwrap();
|
||||||
|
|
||||||
|
let results = index.search("\"quick brown\"", 10).unwrap();
|
||||||
|
assert_eq!(results, vec!["id-1".to_string()]);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
mod config;
|
||||||
|
mod consumer;
|
||||||
|
mod grpc;
|
||||||
|
mod index;
|
||||||
|
mod offsets;
|
||||||
|
|
||||||
|
pub mod logsv1 {
|
||||||
|
tonic::include_proto!("sentry.logs.v1");
|
||||||
|
}
|
||||||
|
pub mod searchv1 {
|
||||||
|
tonic::include_proto!("sentry.search.v1");
|
||||||
|
}
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use config::Config;
|
||||||
|
use index::SearchIndex;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tonic::transport::Server;
|
||||||
|
|
||||||
|
/// Matches /transport/provision-topics.sh's default
|
||||||
|
/// REDPANDA_TOPIC_PARTITIONS -- documented cross-component contract, not
|
||||||
|
/// discovered dynamically. See consumer.rs.
|
||||||
|
const DEFAULT_PARTITION_COUNT: i32 = 6;
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
tracing_subscriber::fmt()
|
||||||
|
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||||||
|
.init();
|
||||||
|
|
||||||
|
let cfg = Arc::new(Config::load().context("loading config")?);
|
||||||
|
|
||||||
|
let index = Arc::new(
|
||||||
|
SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?,
|
||||||
|
);
|
||||||
|
|
||||||
|
let partition_count: i32 = std::env::var("REDPANDA_TOPIC_PARTITIONS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(DEFAULT_PARTITION_COUNT);
|
||||||
|
|
||||||
|
let consumer_cfg = Arc::clone(&cfg);
|
||||||
|
let consumer_index = Arc::clone(&index);
|
||||||
|
let consumer_handle = tokio::spawn(async move {
|
||||||
|
if let Err(e) = consumer::run(consumer_cfg, consumer_index, partition_count).await {
|
||||||
|
tracing::error!(error = %e, "redpanda consumer exited with error");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let addr = cfg
|
||||||
|
.grpc_listen_addr
|
||||||
|
.parse()
|
||||||
|
.context("parsing GRPC_LISTEN_ADDR")?;
|
||||||
|
tracing::info!(addr = %cfg.grpc_listen_addr, "search gRPC server listening");
|
||||||
|
|
||||||
|
let search_server = grpc::SearchServer::new(Arc::clone(&index));
|
||||||
|
Server::builder()
|
||||||
|
.add_service(searchv1::search_service_server::SearchServiceServer::new(
|
||||||
|
search_server,
|
||||||
|
))
|
||||||
|
.serve(addr)
|
||||||
|
.await
|
||||||
|
.context("gRPC server failed")?;
|
||||||
|
|
||||||
|
consumer_handle.abort();
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
/// Tracks per-partition offsets in a plain JSON file next to the Tantivy
|
||||||
|
/// index, since rskafka is a low-level client with no built-in consumer-
|
||||||
|
/// group coordination/offset-commit protocol (unlike kafka-go on the
|
||||||
|
/// ingest side) -- there's no broker-side group to commit to here, so
|
||||||
|
/// this service owns its own offset bookkeeping.
|
||||||
|
///
|
||||||
|
/// Best-effort, not exactly-once: if the process dies between processing
|
||||||
|
/// a record and persisting its offset, that record gets reprocessed on
|
||||||
|
/// restart. This is fine because `SearchIndex::upsert` is delete-then-add
|
||||||
|
/// on `record_id` -- reprocessing the same record overwrites the same
|
||||||
|
/// document rather than duplicating it.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct OffsetStore {
|
||||||
|
path: PathBuf,
|
||||||
|
offsets: HashMap<i32, i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl OffsetStore {
|
||||||
|
pub async fn load(path: &Path) -> Result<Self> {
|
||||||
|
let offsets = match tokio::fs::read(path).await {
|
||||||
|
Ok(bytes) => serde_json::from_slice(&bytes)
|
||||||
|
.with_context(|| format!("parsing offsets file {}", path.display()))?,
|
||||||
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => HashMap::new(),
|
||||||
|
Err(e) => return Err(e).with_context(|| format!("reading offsets file {}", path.display())),
|
||||||
|
};
|
||||||
|
Ok(Self {
|
||||||
|
path: path.to_path_buf(),
|
||||||
|
offsets,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Next offset to fetch for a partition -- 0 (earliest) if never
|
||||||
|
/// recorded before.
|
||||||
|
pub fn get(&self, partition: i32) -> i64 {
|
||||||
|
self.offsets.get(&partition).copied().unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn set(&mut self, partition: i32, offset: i64) {
|
||||||
|
self.offsets.insert(partition, offset);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn persist(&self) -> Result<()> {
|
||||||
|
if let Some(parent) = self.path.parent() {
|
||||||
|
tokio::fs::create_dir_all(parent)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("creating offsets directory {}", parent.display()))?;
|
||||||
|
}
|
||||||
|
let bytes = serde_json::to_vec_pretty(&self.offsets).context("serializing offsets")?;
|
||||||
|
tokio::fs::write(&self.path, bytes)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("writing offsets file {}", self.path.display()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn get_defaults_to_zero_for_unknown_partition() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let store = OffsetStore::load(&dir.path().join("offsets.json")).await.unwrap();
|
||||||
|
assert_eq!(store.get(0), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn missing_file_loads_as_empty_not_an_error() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("does-not-exist.json");
|
||||||
|
let store = OffsetStore::load(&path).await;
|
||||||
|
assert!(store.is_ok(), "expected a missing offsets file to load as empty, got {store:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn persist_and_reload_round_trips() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("offsets.json");
|
||||||
|
|
||||||
|
let mut store = OffsetStore::load(&path).await.unwrap();
|
||||||
|
store.set(0, 42);
|
||||||
|
store.set(1, 7);
|
||||||
|
store.persist().await.unwrap();
|
||||||
|
|
||||||
|
let reloaded = OffsetStore::load(&path).await.unwrap();
|
||||||
|
assert_eq!(reloaded.get(0), 42);
|
||||||
|
assert_eq!(reloaded.get(1), 7);
|
||||||
|
assert_eq!(reloaded.get(2), 0, "unset partition should still default to 0");
|
||||||
|
}
|
||||||
|
}
|
||||||
+21
-2
@@ -4,7 +4,8 @@ ClickHouse schema and migration tooling for Sentry's analytical store.
|
|||||||
|
|
||||||
## Schema
|
## Schema
|
||||||
|
|
||||||
One table for Phase 0, `logs`:
|
One table, `logs` (Phase 0 columns plus `record_id`, added in
|
||||||
|
`migrations/0002_add_record_id.sql`):
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE TABLE logs
|
CREATE TABLE logs
|
||||||
@@ -14,13 +15,31 @@ CREATE TABLE logs
|
|||||||
`service` String,
|
`service` String,
|
||||||
`severity` LowCardinality(String),
|
`severity` LowCardinality(String),
|
||||||
`message` String,
|
`message` String,
|
||||||
`attributes` Map(String, String)
|
`attributes` Map(String, String),
|
||||||
|
`record_id` UUID DEFAULT generateUUIDv4()
|
||||||
)
|
)
|
||||||
ENGINE = MergeTree
|
ENGINE = MergeTree
|
||||||
PARTITION BY toDate(timestamp)
|
PARTITION BY toDate(timestamp)
|
||||||
ORDER BY (service, timestamp)
|
ORDER BY (service, timestamp)
|
||||||
|
-- plus: INDEX record_id_idx record_id TYPE bloom_filter GRANULARITY 4
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**`record_id`** (Phase 1) is the stable per-record identifier Tantivy's
|
||||||
|
full-text search joins back to this table with — `/ingest`'s gRPC front
|
||||||
|
end assigns it once, server-side, before a record is produced to
|
||||||
|
Redpanda (see `/ingest/README.md` for why it has to happen exactly once,
|
||||||
|
upstream of both the ClickHouse-writer and Tantivy-indexer consumers).
|
||||||
|
Added via `ALTER TABLE ... ADD COLUMN` + `ADD INDEX` rather than changing
|
||||||
|
`ORDER BY`: `ORDER BY (service, timestamp)` is the proven time-range-scan
|
||||||
|
access pattern from Phase 0 and shouldn't be disturbed for a
|
||||||
|
fundamentally different access pattern (point lookups by ID). A
|
||||||
|
data-skipping bloom filter index on `record_id` serves the `WHERE
|
||||||
|
record_id IN (...)` lookup Tantivy-backed search results need, without
|
||||||
|
touching the primary sort order. The `DEFAULT generateUUIDv4()` is a
|
||||||
|
safety net, not the normal path — every row `/ingest` writes explicitly
|
||||||
|
supplies its own `record_id` from the proto message; the default only
|
||||||
|
matters for rows written some other way.
|
||||||
|
|
||||||
Notes on choices that weren't fully specified by the task description:
|
Notes on choices that weren't fully specified by the task description:
|
||||||
|
|
||||||
- **`DateTime64(9, 'UTC')`** (nanosecond precision) rather than second or
|
- **`DateTime64(9, 'UTC')`** (nanosecond precision) rather than second or
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
ALTER TABLE logs
|
||||||
|
ADD COLUMN record_id UUID DEFAULT generateUUIDv4(),
|
||||||
|
ADD INDEX record_id_idx record_id TYPE bloom_filter GRANULARITY 4
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
<script lang="ts">
|
||||||
|
// Shared between the SQL query page and the free-text search page —
|
||||||
|
// both /api endpoints return the same {columns, rows} shape
|
||||||
|
// specifically so this component didn't need to exist twice.
|
||||||
|
let {
|
||||||
|
columns,
|
||||||
|
rows,
|
||||||
|
hasRun = false
|
||||||
|
}: { columns: string[]; rows: unknown[][]; hasRun?: boolean } = $props();
|
||||||
|
|
||||||
|
function formatCell(value: unknown): string {
|
||||||
|
if (value === null || value === undefined) return '';
|
||||||
|
if (typeof value === 'object') return JSON.stringify(value);
|
||||||
|
return String(value);
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
{#if hasRun}
|
||||||
|
<p>{rows.length} row(s)</p>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
{#if columns.length > 0}
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
{#each columns as col (col)}
|
||||||
|
<th>{col}</th>
|
||||||
|
{/each}
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{#each rows as row, i (i)}
|
||||||
|
<tr>
|
||||||
|
{#each row as cell, j (j)}
|
||||||
|
<td>{formatCell(cell)}</td>
|
||||||
|
{/each}
|
||||||
|
</tr>
|
||||||
|
{/each}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
<style>
|
||||||
|
table {
|
||||||
|
border-collapse: collapse;
|
||||||
|
width: 100%;
|
||||||
|
margin-top: 1rem;
|
||||||
|
}
|
||||||
|
th,
|
||||||
|
td {
|
||||||
|
border: 1px solid #ccc;
|
||||||
|
padding: 0.25rem 0.5rem;
|
||||||
|
text-align: left;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
}
|
||||||
|
th {
|
||||||
|
background: #f0f0f0;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
<script lang="ts">
|
<script lang="ts">
|
||||||
import favicon from '$lib/assets/favicon.svg';
|
import favicon from '$lib/assets/favicon.svg';
|
||||||
|
import { page } from '$app/state';
|
||||||
|
|
||||||
let { children } = $props();
|
let { children } = $props();
|
||||||
</script>
|
</script>
|
||||||
@@ -9,4 +10,32 @@
|
|||||||
<link rel="icon" href={favicon} />
|
<link rel="icon" href={favicon} />
|
||||||
</svelte:head>
|
</svelte:head>
|
||||||
|
|
||||||
|
<nav>
|
||||||
|
<a href="/" class:active={page.url.pathname === '/'}>SQL Query</a>
|
||||||
|
<a href="/search" class:active={page.url.pathname === '/search'}>Full-Text Search</a>
|
||||||
|
</nav>
|
||||||
|
|
||||||
{@render children()}
|
{@render children()}
|
||||||
|
|
||||||
|
<style>
|
||||||
|
nav {
|
||||||
|
font-family: system-ui, sans-serif;
|
||||||
|
max-width: 960px;
|
||||||
|
margin: 1rem auto 0;
|
||||||
|
padding: 0 1rem;
|
||||||
|
display: flex;
|
||||||
|
gap: 1rem;
|
||||||
|
border-bottom: 1px solid #ccc;
|
||||||
|
}
|
||||||
|
nav a {
|
||||||
|
padding: 0.5rem 0;
|
||||||
|
text-decoration: none;
|
||||||
|
color: #555;
|
||||||
|
border-bottom: 2px solid transparent;
|
||||||
|
}
|
||||||
|
nav a.active {
|
||||||
|
color: #000;
|
||||||
|
border-bottom-color: #000;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|||||||
@@ -4,6 +4,8 @@
|
|||||||
// This is a placeholder for the real query UI that lands once /api grows
|
// This is a placeholder for the real query UI that lands once /api grows
|
||||||
// a real SPL-like query layer in Phase 2.
|
// a real SPL-like query layer in Phase 2.
|
||||||
|
|
||||||
|
import ResultsTable from '$lib/ResultsTable.svelte';
|
||||||
|
|
||||||
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
||||||
|
|
||||||
let sql = $state('SELECT * FROM logs ORDER BY timestamp DESC LIMIT 100');
|
let sql = $state('SELECT * FROM logs ORDER BY timestamp DESC LIMIT 100');
|
||||||
@@ -13,12 +15,6 @@
|
|||||||
let loading = $state(false);
|
let loading = $state(false);
|
||||||
let hasRun = $state(false);
|
let hasRun = $state(false);
|
||||||
|
|
||||||
function formatCell(value: unknown): string {
|
|
||||||
if (value === null || value === undefined) return '';
|
|
||||||
if (typeof value === 'object') return JSON.stringify(value);
|
|
||||||
return String(value);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function runQuery() {
|
async function runQuery() {
|
||||||
loading = true;
|
loading = true;
|
||||||
error = '';
|
error = '';
|
||||||
@@ -49,10 +45,11 @@
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<main>
|
<main>
|
||||||
<h1>Sentry — Log Query (Phase 0)</h1>
|
<h1>Sentry — Log Query</h1>
|
||||||
<p>
|
<p>
|
||||||
Raw SQL only, SELECT statements against the <code>logs</code> table. No auth, no query
|
Raw SQL only, SELECT statements against the <code>logs</code> table. No auth, no query
|
||||||
builder yet — see <code>/api</code> for what's actually allowed.
|
builder yet — see <code>/api</code> for what's actually allowed. Looking for free-text
|
||||||
|
search instead? See the <a href="/search">Full-Text Search</a> page.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<textarea bind:value={sql} rows="4" cols="100" spellcheck="false"></textarea>
|
<textarea bind:value={sql} rows="4" cols="100" spellcheck="false"></textarea>
|
||||||
@@ -66,30 +63,7 @@
|
|||||||
<p class="error">Error: {error}</p>
|
<p class="error">Error: {error}</p>
|
||||||
{/if}
|
{/if}
|
||||||
|
|
||||||
{#if hasRun && !error}
|
<ResultsTable {columns} {rows} {hasRun} />
|
||||||
<p>{rows.length} row(s)</p>
|
|
||||||
{/if}
|
|
||||||
|
|
||||||
{#if columns.length > 0}
|
|
||||||
<table>
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
{#each columns as col (col)}
|
|
||||||
<th>{col}</th>
|
|
||||||
{/each}
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{#each rows as row, i (i)}
|
|
||||||
<tr>
|
|
||||||
{#each row as cell, j (j)}
|
|
||||||
<td>{formatCell(cell)}</td>
|
|
||||||
{/each}
|
|
||||||
</tr>
|
|
||||||
{/each}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
{/if}
|
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<style>
|
<style>
|
||||||
@@ -110,19 +84,4 @@
|
|||||||
.error {
|
.error {
|
||||||
color: #b00020;
|
color: #b00020;
|
||||||
}
|
}
|
||||||
table {
|
|
||||||
border-collapse: collapse;
|
|
||||||
width: 100%;
|
|
||||||
margin-top: 1rem;
|
|
||||||
}
|
|
||||||
th,
|
|
||||||
td {
|
|
||||||
border: 1px solid #ccc;
|
|
||||||
padding: 0.25rem 0.5rem;
|
|
||||||
text-align: left;
|
|
||||||
font-size: 0.85rem;
|
|
||||||
}
|
|
||||||
th {
|
|
||||||
background: #f0f0f0;
|
|
||||||
}
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
<script lang="ts">
|
||||||
|
// Phase 1: free-text search via POST /search on the api service, hits
|
||||||
|
// the Tantivy-backed search service and returns full rows joined back
|
||||||
|
// against ClickHouse. No unified query experience with the SQL page —
|
||||||
|
// that's Phase 2's job.
|
||||||
|
|
||||||
|
import ResultsTable from '$lib/ResultsTable.svelte';
|
||||||
|
|
||||||
|
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
||||||
|
|
||||||
|
let query = $state('');
|
||||||
|
let columns = $state<string[]>([]);
|
||||||
|
let rows = $state<unknown[][]>([]);
|
||||||
|
let error = $state('');
|
||||||
|
let loading = $state(false);
|
||||||
|
let hasRun = $state(false);
|
||||||
|
|
||||||
|
async function runSearch() {
|
||||||
|
loading = true;
|
||||||
|
error = '';
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${apiBase}/search`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ query })
|
||||||
|
});
|
||||||
|
const body = await res.json();
|
||||||
|
if (!res.ok) {
|
||||||
|
error = body?.error ?? `request failed with status ${res.status}`;
|
||||||
|
columns = [];
|
||||||
|
rows = [];
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
columns = body.columns ?? [];
|
||||||
|
rows = body.rows ?? [];
|
||||||
|
} catch (e) {
|
||||||
|
error = e instanceof Error ? e.message : String(e);
|
||||||
|
columns = [];
|
||||||
|
rows = [];
|
||||||
|
} finally {
|
||||||
|
loading = false;
|
||||||
|
hasRun = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<h1>Sentry — Full-Text Search</h1>
|
||||||
|
<p>
|
||||||
|
Free-text search over the <code>message</code> field, via Tantivy. Supports plain terms,
|
||||||
|
<code>"exact phrases"</code>, and <code>wildcard*</code> — see
|
||||||
|
<code>/search</code> for the full query syntax. Looking for structured/aggregation queries
|
||||||
|
instead? See the <a href="/">SQL Query</a> page.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
bind:value={query}
|
||||||
|
placeholder="e.g. "connection refused" or timeout*"
|
||||||
|
spellcheck="false"
|
||||||
|
onkeydown={(e) => e.key === 'Enter' && runSearch()}
|
||||||
|
/>
|
||||||
|
<div>
|
||||||
|
<button onclick={runSearch} disabled={loading || query.trim() === ''}>
|
||||||
|
{loading ? 'Searching…' : 'Search'}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{#if error}
|
||||||
|
<p class="error">Error: {error}</p>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
<ResultsTable {columns} {rows} {hasRun} />
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
main {
|
||||||
|
font-family: system-ui, sans-serif;
|
||||||
|
max-width: 960px;
|
||||||
|
margin: 2rem auto;
|
||||||
|
padding: 0 1rem;
|
||||||
|
}
|
||||||
|
input {
|
||||||
|
width: 100%;
|
||||||
|
font-family: monospace;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
padding: 0.4rem;
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
button {
|
||||||
|
margin-top: 0.5rem;
|
||||||
|
}
|
||||||
|
.error {
|
||||||
|
color: #b00020;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
// Same reasoning as the root page's +page.ts: no load function (all data
|
||||||
|
// comes from a client-side fetch on submit), so a plain prerender is
|
||||||
|
// enough for the static adapter.
|
||||||
|
export const prerender = true;
|
||||||
Reference in New Issue
Block a user