Phase 2: unified query language spanning ClickHouse and Tantivy
Replaces the separate SQL-only /query and text-only /search endpoints with one pipe-syntax query language (plus raw SQL escape hatch) that compiles to a single IR and execution plan across both backends, so a query like `message:"connection refused" | stats count by host` runs as one request instead of two disjoint tools. - api/internal/querylang: lexer -> ast -> parser -> ir -> planner -> executor, each layer independently tested. - Execution generalizes Phase 1's proven Tantivy-prefilter pattern into a 4-way routing table (pure ClickHouse / text-only / text + aggregation / raw SQL passthrough). - Unified web query page and `sentryctl query`, both hitting the same POST /query endpoint. - Benchmarked against a real 1,022,000-row dataset (hack/benchmark-fixture); caught and fixed a real bug where the Tantivy prefilter cap (10,000) produced an IN-clause exceeding ClickHouse's default max_query_size -- lowered to 5,000, documented in docs/query-language-design.md and docs/phase-2-runbook.md. - docs/query-language-reference.md: customer-facing syntax reference.
This commit is contained in:
+42
-39
@@ -1,39 +1,39 @@
|
||||
# api
|
||||
|
||||
Sentry's query API: two intentionally crude endpoints — raw SQL and
|
||||
free-text search — that Phase 2's real query layer replaces outright.
|
||||
Sentry's query API: a single `POST /query` endpoint accepting either the
|
||||
pipe syntax or raw SQL, compiled and routed across ClickHouse and Tantivy
|
||||
by `internal/querylang`. Replaces Phase 0/1's two separate placeholder
|
||||
endpoints (raw-SQL-only `/query`, free-text-only `/search`) — see
|
||||
`/docs/query-language-design.md` for the grammar, IR, and routing design,
|
||||
and `/docs/query-language-reference.md` for the user-facing syntax.
|
||||
|
||||
## Why plain REST, not gRPC + REST gateway
|
||||
|
||||
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
||||
service is plain `net/http` instead — a deliberate simplification, not a
|
||||
change to the pinned stack. Wiring up `.proto` services,
|
||||
change to the pinned stack. Wiring up a `.proto` service,
|
||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
||||
two endpoints that Phase 2 replaces outright with a real SPL-like query
|
||||
layer would be exactly the kind of premature machinery this project's
|
||||
conventions warn against. `api` *does* speak gRPC internally though — to
|
||||
`/search` (see below) — this simplification is specifically about the
|
||||
public-facing surface, not a blanket avoidance of gRPC.
|
||||
one endpoint doesn't buy much at this size. `api` *does* speak gRPC
|
||||
internally — to `/search` — this simplification is about the
|
||||
public-facing surface only.
|
||||
|
||||
## Endpoints
|
||||
## Endpoint
|
||||
|
||||
- `POST /query` — body `{"sql": "SELECT ..."}`, response
|
||||
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
||||
SELECT-only, single-statement, basic keyword-based injection guarding
|
||||
(see `internal/queryapi/validate.go` for exactly what that does and
|
||||
doesn't catch — it's not a SQL parser).
|
||||
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
|
||||
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
|
||||
resolve the free-text query into matching `record_id`s, then joins
|
||||
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
|
||||
(...)`) to return full rows — so both endpoints return the same
|
||||
`{columns, rows}` shape and `/web` can reuse one table component for
|
||||
both. Every `record_id` is validated as a real UUID before being
|
||||
embedded in the generated SQL (defense in depth: `record_id`s come from
|
||||
an internal, trusted service, not raw user input, but a value that
|
||||
fails to parse as a UUID can't contain SQL-breaking characters either
|
||||
way).
|
||||
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
||||
`POST /query` — body `{"query": "...", "language": ""}`, response
|
||||
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
||||
|
||||
- `query` is either pipe syntax (`service=api | where status>=500 |
|
||||
stats count by host`) or raw SQL (`SELECT ...`). Auto-detected by
|
||||
whether the query starts with `SELECT` (case-insensitive).
|
||||
- `language` optionally overrides detection: `"sql"` or `"spl"`. Exists
|
||||
for the rare case a pipe query legitimately starts with the literal
|
||||
word "select" as a bare search term.
|
||||
- Both syntaxes compile to the same `querylang/ir.Plan` and execute
|
||||
through the same code path — see `internal/querylang/executor` for the
|
||||
four routing cases (pure ClickHouse; Tantivy prefilter + ClickHouse
|
||||
rows; Tantivy prefilter + ClickHouse aggregation; raw SQL passthrough).
|
||||
|
||||
`GET /healthz` — for docker-compose/k8s liveness checks.
|
||||
|
||||
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
||||
network.
|
||||
@@ -48,7 +48,7 @@ Environment variables (see `internal/config/config.go`):
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
|
||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout |
|
||||
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
||||
|
||||
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
||||
@@ -71,15 +71,18 @@ docker build -f api/Dockerfile -t sentry-api .
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
|
||||
only through narrow interfaces (`queryExecutor`, `searchClient`), so
|
||||
routing, validation, JSON encoding, error-status mapping, and the
|
||||
record_id-to-SQL query building are all unit-tested against fakes — no
|
||||
live ClickHouse or `/search` instance needed. `Executor` itself (the
|
||||
reflection-based row scanning against `driver.Rows`) and
|
||||
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
||||
former because faking ClickHouse's `driver.Rows` interface fully would be
|
||||
significant test-only scaffolding the driver's own docs say isn't meant
|
||||
to be implemented by adopters; the latter because it's a thin wrapper
|
||||
with nothing but wiring to test. Both are exercised end-to-end via the
|
||||
docker-compose flow in `/docs/phase-1-runbook.md` instead.
|
||||
`internal/queryapi`'s HTTP handler depends on ClickHouse and `/search`
|
||||
only through the narrow interfaces `querylang/executor` defines
|
||||
(`SQLRunner`, `SearchClient`), so routing, compilation, JSON encoding,
|
||||
and error-status mapping are all unit-tested against fakes — no live
|
||||
ClickHouse or `/search` instance needed, and the real lexer/parser/
|
||||
planner run unmocked in these tests, only the backends are faked. See
|
||||
`internal/querylang`'s own package docs for how compilation and
|
||||
execution are tested independently of each other. `executor.ChRunner`
|
||||
(the reflection-based row scanning against ClickHouse's `driver.Rows`)
|
||||
and `internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
||||
former because faking `driver.Rows` fully would be significant
|
||||
test-only scaffolding the driver's own docs say isn't meant to be
|
||||
implemented by adopters; the latter because it's a thin wrapper with
|
||||
nothing but wiring to test. Both are exercised end-to-end via the
|
||||
docker-compose flow in `/docs/phase-2-runbook.md`.
|
||||
|
||||
Reference in New Issue
Block a user