jcoffey-dev
cd8aa290ca
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
4.5 KiB
search
Tantivy-backed full-text search over log messages. Phase 1's answer to "grep across everything," separate from ClickHouse's structured/ aggregation queries.
Why a separate service, not embedded in ingest
Tantivy is a Rust library with no maintained Go bindings — using it from
ingest (Go) would mean cgo-bridging to a compiled Rust cdylib, exactly
the fragile FFI complexity CLAUDE.md's "prefer boring, well-understood
dependencies... operators need to trust it" principle steers away from.
It would also couple ClickHouse-write latency to Tantivy-write latency in
the same request path. See /docs/architecture.md for the fuller
tradeoff writeup (dual-write vs. second-consumer-group) from Phase 1
planning.
How it fits together
ingest (gRPC front end) --> Redpanda (sentry.logs.raw) --> ingest's ClickHouse-writer consumer --> ClickHouse
\
`--> search's own consumer --> Tantivy index
search reads the same Redpanda topic ingest's ClickHouse-writer
consumer reads, as an independent consumer in spirit (own offset
tracking, own failure domain — see "Offset tracking" below) even though
it's a completely separate process/service. If Tantivy indexing lags or
crashes, ClickHouse ingestion is completely unaffected.
api calls search's SearchService.Search gRPC RPC (see
/proto/sentry/search/v1/search.proto) to resolve a free-text query into
matching record_ids, then joins those back against ClickHouse's
logs.record_id column (see /storage/migrations/0002_add_record_id.sql)
to get full rows. search only ever returns IDs, never row data — it
stays a pure text index, not a second copy of the row.
Offset tracking: why this isn't a Kafka consumer group
rskafka (chosen for being pure Rust, no cgo — consistent with why
ingest chose segmentio/kafka-go over confluent-kafka-go) is a
low-level client: it doesn't implement Kafka's broker-side consumer-group
coordination protocol the way kafka-go or librdkafka do. So search
tracks its own per-partition offsets in a plain JSON file next to the
Tantivy index (offsets.rs), persisted after each fetched batch.
This is deliberately best-effort, not exactly-once: if the process dies
between processing a record and persisting its offset, that record gets
reprocessed after restart. This is safe because SearchIndex::upsert is
delete-then-add on record_id — Tantivy segments are immutable, so
this is the standard idiom for updates anyway, and it happens to make
reprocessing idempotent for free. Partition count is read from
REDPANDA_TOPIC_PARTITIONS (must match what
/transport/provision-topics.sh actually created — same kind of
documented cross-component contract as the topic name itself), not
discovered dynamically.
Query syntax
Whatever Tantivy's own QueryParser supports against the message
field: plain terms, "exact phrase" queries, and foo* wildcards. Not
documented further here because it's Tantivy's syntax, not Sentry's — see
Tantivy's query parser docs
for the full grammar. No unified query language yet; that's Phase 2.
Configuration
Environment variables (see src/config.rs):
| Var | Default | Purpose |
|---|---|---|
GRPC_LISTEN_ADDR |
0.0.0.0:50052 |
Full socket address — Rust's parser needs one, unlike Go's :PORT shorthand ingest/api use |
REDPANDA_BROKERS |
localhost:9092 |
Comma-separated broker list |
REDPANDA_TOPIC |
sentry.logs.raw |
Must match /ingest's topic |
REDPANDA_TOPIC_PARTITIONS |
6 |
Must match what /transport/provision-topics.sh created |
INDEX_PATH |
/var/lib/sentry-search/index |
Tantivy index directory |
OFFSETS_PATH |
/var/lib/sentry-search/offsets.json |
Offset tracking file |
COMMIT_INTERVAL_MS |
2000 |
How often buffered writes become searchable |
Building & testing
cargo build --release
cargo clippy --all-targets -- -D warnings
cargo test
index.rs's tests run against a real (temp-directory) Tantivy index —
no external service needed, unlike ClickHouse. They cover the
delete-then-add idempotency, phrase queries, result limits, and the
before-commit/after-commit visibility boundary. consumer.rs (the
rskafka wiring) is not unit-tested — that needs a real Redpanda, same
category of gap as /ingest's kafka.Reader/kafka.Writer wiring, and
is exercised by the docker-compose end-to-end flow instead.
# from the repo root, not search/
docker build -f search/Dockerfile -t sentry-search .