Files

T

jcoffey-dev cd8aa290ca Phase 1: Windows log collection + full-text search

Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.

- CLAUDE.md: materialized on disk (never existed as a file before) with
  a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
  service wrapper (install/uninstall/run-service), both feature- and
  target_os-gated so Linux builds/tests/clippy stay unaffected. Also
  fixed two pre-existing Phase 0 clippy gaps (dead-code on
  default-features-only builds, a type-inference edge case) found while
  testing every feature combination properly for the first time.
  UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
  the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
  ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
  for the same record.
- storage: record_id column + bloom filter index, verified against a
  live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
  second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
  result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
  ingest, so the pipeline's handling of it is verifiable without a
  Windows host.

Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.

2026-08-13 11:27:35 -07:00

4.0 KiB

Raw Blame History

Project: Sentry — Distributed Log Aggregation & Observability Platform

Mission

Build an open-core, Kubernetes-native centralized logging platform that rivals Splunk on features but wins on cost-per-GB, modern language stack, and honest multi-tenant RBAC. Full architecture spec is in /docs/architecture.md — read it before touching any component. Do not deviate from the storage/query split described there without flagging it to me first.

Non-negotiable constraints

Distro-agnostic Linux agent: must run identically on RHEL/Debian/Arch/SUSE derivatives via a statically-linked musl binary. No glibc runtime deps.
Windows support via native ETW/Event Log API, not a WSL shim.
AGPLv3 for core + agents. Enterprise module (SSO/multi-tenancy/compliance) lives in a separate enterprise/ directory under a commercial license stub — keep the boundary clean from day one, don't let AGPL code import from it.
Schema-on-write with OTel semantic conventions as the default schema, with schema-on-read fallback for unstructured text.
Every UI action must correspond to a documented REST/gRPC call. No UI-only logic. CLI (sentryctl) and Terraform provider are first-class, not afterthoughts.

Tech stack (pinned — do not substitute without discussion)

Component	Language/Tool
Edge agent	Rust, musl target
Transport	Redpanda (Kafka API)
Ingest/parse	Go
Analytical store	ClickHouse
Full-text index	Tantivy (Rust)
Control plane/API	Go, gRPC + REST gateway
Frontend	SvelteKit + TypeScript
Deployment	Kubernetes Operator (Go, kubebuilder), Helm, docker-compose for local/homelab

Repo conventions

Monorepo, one top-level dir per component (see structure below).
Rust: workspace-based, cargo clippy --all-targets -- -D warnings must pass.
Go: standard go vet + golangci-lint, no globals for shared state.
Every component ships with: unit tests, a README.md, and a Dockerfile using distroless or scratch base images where feasible.
Conventional commits. Every PR-sized change should be a logically complete, independently revertible unit.
Prefer boring, well-understood dependencies over novel ones. This is infrastructure software; operators need to trust it.

What "done" looks like for Phase 0 (MVP)

Status: shipped. A single log line, generated on a Linux host by the Rust agent, flows: agent → Redpanda → Go ingest service → ClickHouse, and is queryable via a minimal SQL endpoint and visible in a bare-bones SvelteKit table view. Verified end-to-end on real hardware, not just in CI — see /docs/phase-0-runbook.md. No alerting, no multi-tenancy, no dashboards — that discipline held for the whole phase.

What "done" looks like for Phase 1

A Windows Event Log entry and a Linux journald entry should both be queryable via SQL (the ClickHouse path) and via free-text search (the Tantivy path), from the same UI, within a few seconds of being generated.

Non-goals for this phase (same "resist scope creep" discipline as Phase 0): no alerting, no dashboards, no SPL-like query layer, no multi-tenancy, and no unified query experience — two separate boxes on two separate pages is correct for Phase 1; unifying them is Phase 2's job.

ETW and WEF (Windows Event Forwarding) are designed in this phase but not required to be running for "done": ETW ships behind a feature flag most environments won't enable (it needs elevated privileges), and WEF's receiver-side is explicitly deferred rather than built now — see /docs/phase-1-runbook.md for both. Only the Event Log source needs to actually be running end-to-end for this phase to count as done.

When in doubt

Ask before: changing the pinned stack, adding a new external dependency that pulls in a large transitive tree, or making an architectural decision that isn't already specified in /docs/architecture.md.

4.0 KiB Raw Blame History