Phase 1: Windows log collection + full-text search

Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.

- CLAUDE.md: materialized on disk (never existed as a file before) with
  a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
  service wrapper (install/uninstall/run-service), both feature- and
  target_os-gated so Linux builds/tests/clippy stay unaffected. Also
  fixed two pre-existing Phase 0 clippy gaps (dead-code on
  default-features-only builds, a type-inference edge case) found while
  testing every feature combination properly for the first time.
  UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
  the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
  ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
  for the same record.
- storage: record_id column + bloom filter index, verified against a
  live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
  second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
  result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
  ingest, so the pipeline's handling of it is verifiable without a
  Windows host.

Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.
This commit is contained in:
2026-08-13 11:27:35 -07:00
parent fe854b1091
commit cd8aa290ca
66 changed files with 6084 additions and 171 deletions
+41 -19
View File
@@ -1,17 +1,19 @@
# api
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
Sentry's query API: two intentionally crude endpoints — raw SQL and
free-text search — that Phase 2's real query layer replaces outright.
## Why plain REST, not gRPC + REST gateway
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
service is plain `net/http` instead — a deliberate Phase 0 simplification,
not a change to the pinned stack. Wiring up a `.proto` service,
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
single endpoint that Phase 2 replaces outright with a real SPL-like query
service is plain `net/http` instead — a deliberate simplification, not a
change to the pinned stack. Wiring up `.proto` services,
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
two endpoints that Phase 2 replaces outright with a real SPL-like query
layer would be exactly the kind of premature machinery this project's
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
a second real, durable endpoint.
conventions warn against. `api` *does* speak gRPC internally though — to
`/search` (see below) — this simplification is specifically about the
public-facing surface, not a blanket avoidance of gRPC.
## Endpoints
@@ -20,10 +22,21 @@ a second real, durable endpoint.
SELECT-only, single-statement, basic keyword-based injection guarding
(see `internal/queryapi/validate.go` for exactly what that does and
doesn't catch — it's not a SQL parser).
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
resolve the free-text query into matching `record_id`s, then joins
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
(...)`) to return full rows — so both endpoints return the same
`{columns, rows}` shape and `/web` can reuse one table component for
both. Every `record_id` is validated as a real UUID before being
embedded in the generated SQL (defense in depth: `record_id`s come from
an internal, trusted service, not raw user input, but a value that
fails to parse as a UUID can't contain SQL-breaking characters either
way).
- `GET /healthz` — for docker-compose/k8s liveness checks.
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
dev/homelab network.
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
network.
## Configuration
@@ -34,9 +47,15 @@ Environment variables (see `internal/config/config.go`):
| `HTTP_LISTEN_ADDR` | `:8080` | |
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
trust boundary as `api`'s existing plain-TCP connection to ClickHouse.
mTLS in this project is specifically the agent↔ingest edge boundary, not
every internal hop.
## Building & testing
```sh
@@ -52,12 +71,15 @@ docker build -f api/Dockerfile -t sentry-api .
## Testing notes
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
one-method `queryExecutor` interface, so routing, validation, JSON
encoding, and error-status mapping are all unit-tested against a fake —
no live ClickHouse needed. `Executor` itself (the reflection-based row
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
`driver.Rows` interface fully would be significant test-only scaffolding
for a Phase 0 placeholder, and the driver package's own docs note it isn't
meant to be implemented by adopters. It's exercised end-to-end via the
docker-compose flow in `/docs/phase-0-runbook.md` instead.
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
only through narrow interfaces (`queryExecutor`, `searchClient`), so
routing, validation, JSON encoding, error-status mapping, and the
record_id-to-SQL query building are all unit-tested against fakes — no
live ClickHouse or `/search` instance needed. `Executor` itself (the
reflection-based row scanning against `driver.Rows`) and
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
former because faking ClickHouse's `driver.Rows` interface fully would be
significant test-only scaffolding the driver's own docs say isn't meant
to be implemented by adopters; the latter because it's a thin wrapper
with nothing but wiring to test. Both are exercised end-to-end via the
docker-compose flow in `/docs/phase-1-runbook.md` instead.