coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Phase 2: unified query language spanning ClickHouse and Tantivy

Browse Source 
Replaces the separate SQL-only /query and text-only /search endpoints
with one pipe-syntax query language (plus raw SQL escape hatch) that
compiles to a single IR and execution plan across both backends, so a
query like `message:"connection refused" | stats count by host` runs
as one request instead of two disjoint tools.

- api/internal/querylang: lexer -> ast -> parser -> ir -> planner ->
  executor, each layer independently tested.
- Execution generalizes Phase 1's proven Tantivy-prefilter pattern
  into a 4-way routing table (pure ClickHouse / text-only / text +
  aggregation / raw SQL passthrough).
- Unified web query page and `sentryctl query`, both hitting the same
  POST /query endpoint.
- Benchmarked against a real 1,022,000-row dataset
  (hack/benchmark-fixture); caught and fixed a real bug where the
  Tantivy prefilter cap (10,000) produced an IN-clause exceeding
  ClickHouse's default max_query_size -- lowered to 5,000, documented
  in docs/query-language-design.md and docs/phase-2-runbook.md.
- docs/query-language-reference.md: customer-facing syntax reference.
This commit is contained in:
jcoffey-dev
2026-08-13 12:21:42 -07:00
parent cd8aa290ca
commit fb5049a747
36 changed files with 4119 additions and 613 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-2-runbook.md
+178
View File
@@ -0,0 +1,178 @@
# Phase 2 runbook
Extends `/docs/phase-0-runbook.md` and `/docs/phase-1-runbook.md` with
the unified query language and the "modest dataset" query-latency
benchmark. Read those first — this assumes the stack already works;
Phase 2 replaces the two placeholder query endpoints/pages with one, and
adds a way to actually measure query performance at volume.
## What's actually been verified
Unlike the phrasing "define 'modest', put a rough benchmark in the
runbook" might suggest, this isn't an estimate — the numbers below are
from actually generating 1,022,000 rows in a live ClickHouse (via
`/hack/benchmark-fixture`, pushed through the real agent-facing gRPC
endpoint, so both ClickHouse and Tantivy have real, matching data) and
timing real queries against it. One real bug turned up doing this (see
"What the benchmark caught" below) that wouldn't have been found any
other way.
## 1. Bring up the stack
```sh
docker compose up -d --build
```
Same as Phase 0/1. Confirm the unified endpoint works for both syntaxes:
```sh
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
-d '{"query": "SELECT 1"}'
# -> {"columns":["1"],"rows":[[1]]}
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
-d '{"query": "service=api"}'
# -> {"columns":[...],"rows":[]} (empty is fine -- no data ingested yet)
```
## 2. Generate the "modest" benchmark dataset
"Modest" is defined here as **1,000,000 rows** — enough to be a real
volume test, small enough to generate and query interactively rather
than needing a dedicated load-testing pass.
```sh
cd hack/benchmark-fixture
go run . --count 1000000 --batch-size 1000 --concurrency 16
```
Sends batched `PushBatch` calls directly to `ingest`'s gRPC endpoint —
the same path the real agent uses, just generating synthetic data
instead of reading journald, so it exercises the *real* ingest →
Redpanda → ClickHouse-writer-consumer and → search-indexer-consumer
paths, not a shortcut that bypasses them. Concurrency matters:
sequential single-batch calls topped out around 500 records/sec (would
take ~33 minutes for 1M rows); 16 concurrent `PushBatch` calls over one
shared gRPC connection (HTTP/2 multiplexes concurrent RPCs over it
natively) got over 400,000 records/sec, so the actual measured run took
about 2 seconds for the ingest call itself.
Wait for both consumers to fully drain before benchmarking — ingestion
finishing doesn't mean ClickHouse/Tantivy are caught up yet:
```sh
# poll until this stops climbing
curl -s -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
-d '{"query": "SELECT count() FROM logs", "language": "sql"}'
```
In the run this runbook was written from, ClickHouse settled at
1,022,000 rows a few seconds after ingestion finished (the extra 22,000
were earlier smaller test runs from developing the benchmark tool itself
— harmless, still real rows).
## 3. Run the benchmark queries
```sh
# structured filter + aggregation -- pure ClickHouse path
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"query": "service=api | where status>=500 | stats count by host | sort -count"}'
# free-text + aggregation -- Tantivy prefilter feeding a ClickHouse GROUP BY,
# the case /docs/query-language-design.md called "the hardest part"
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
-H 'Content-Type: application/json' \
-d '{"query": "message:\"connection refused\" | stats count by host | sort -count"}'
```
**Measured results against the 1,022,000-row dataset** (this exact run,
not a projection):
| Query | Path | Wall time |
|---|---|---|
| `service=api \| where status>=500 \| stats count by host \| sort -count` | Pure ClickHouse | 19.7ms |
| `message:"connection refused" \| stats count by host \| sort -count` | Tantivy prefilter + ClickHouse aggregate | 46.4ms |
| `message:"connection refused" \| head 50` | Tantivy prefilter, no aggregation | 49.2ms |
| `SELECT count() FROM logs WHERE service='web'` (raw SQL) | Pure ClickHouse | 17.5ms |
All four "well under a second" — the Phase 2 exit criteria in
`/CLAUDE.md`. The combined text+aggregation case (the one everyone should
be nervous about, since it's a two-backend query) came in at 46ms, not
meaningfully slower than the pure-ClickHouse case — the Tantivy prefilter
step is fast, and 5,000 UUIDs (see below) is a small `IN` clause by
ClickHouse's standards.
## 4. What the benchmark caught
The original design capped the Tantivy prefilter at **10,000** matching
`record_id`s before feeding them into ClickHouse's `WHERE record_id IN
(...)`. Running the actual combined-query benchmark against real data
(where "connection refused" matched a large fraction of the 1M rows,
by design — the fixture generator seeds that phrase deliberately) hit
this immediately:
```
query failed: executing query: code: 62, message: Syntax error:
failed at position 262116 [...] Max query size exceeded
```
10,000 quoted UUIDs (~39 bytes each including the separating comma)
produces a ~390KB query string, which exceeds ClickHouse's *default*
`max_query_size` (262144 bytes / 256KiB) — a much lower ceiling than
"multi-million-entry" suggested before anyone had actually tried it. The
cap is now **5,000** (~195KB, safely under the default with headroom) —
see `api/internal/querylang/executor/executor.go`'s `textSearchLimit`
and `/docs/query-language-design.md`'s "Known scaling limitation"
section, both updated with this exact finding rather than a
first-principles guess.
## 5. Confirm the web UI and CLI both work end to end
Open `http://localhost:3000` — one page now (Phase 0/1's two pages are
gone), run `service=api | where status>=500 | stats count by host | sort
-count` in the query bar, confirm results render and the query appears
in the session-local history panel.
```sh
cd cli
go run ./cmd/sentryctl query 'service=api | where status>=500 | stats count by host | sort -count'
```
Both hit the exact same `POST /query` endpoint — there's no separate
query logic to drift out of sync between the CLI, the web UI, and
whatever else calls this API later.
## Tearing down
```sh
docker compose down -v # also wipes the 1M-row benchmark dataset
```
## Troubleshooting
**A text-search query with aggregation fails with a ClickHouse syntax
error / "Max query size exceeded."**
If you've raised `textSearchLimit` above 5,000, you've likely
reintroduced the exact failure this runbook's benchmark caught (see
"What the benchmark caught" above) — lower it back down, or raise
ClickHouse's `max_query_size` server-side with matching memory sizing if
you genuinely need a larger prefilter.
**Benchmark ingestion is much slower than ~400K records/sec.**
Check `--concurrency` wasn't left at a low value, and that `ingest`
isn't CPU-starved (`docker stats`) — the numbers above are from a single
machine running the whole stack including Redpanda/ClickHouse/search
concurrently, not a dedicated load-test environment, so your exact
throughput will vary with hardware. What matters is "well under a
second" for the actual query latency, not the ingestion throughput
number itself, which is just how the test data gets there.
**`stats sum(field)`/`avg(field)` etc. on an attributes-map field
returns 0 for everything.**
Check the field's values are actually numeric strings — non-numeric
attribute values silently cast to 0 via `toFloat64OrZero` (see
`/docs/query-language-reference.md`'s "Field mapping" section). This is
documented behavior, not a bug, but it's easy to trip over with a typo'd
field name (which also "succeeds" with all zeros, since a missing map
key reads as an empty string, which also casts to 0).
docs/query-language-design.md
+250
View File
@@ -0,0 +1,250 @@
# Query language design
> **Status:** Design, approved 2026-08-14, not yet implemented (that's
> Task 3). This is the reference Task 3's implementation is built against
> — if implementation reveals this design is wrong somewhere, fix this
> doc in the same change, don't let them drift apart.
## Why this design, in one paragraph
Phases 0–1 shipped two disconnected, placeholder query paths: raw SQL
against ClickHouse, and free-text against Tantivy. Phase 2 needs one
query language that can express both filter/aggregation and free-text
search in a single query, without picking a winner between "give up
structured querying" and "give up full-text search." The approach below
does that by keeping parsing and execution strictly separate (a small
pipe-syntax grammar and an "opaque SQL" passthrough both compile to the
same IR) and by generalizing a mechanism Phase 1 already built and proved
works (Tantivy-prefilter → ClickHouse `IN (...)`) rather than inventing a
new cross-backend join strategy from scratch.
## Grammar
Pipe syntax, SPL-inspired, EBNF-ish:
```
query := base_search ("|" pipe_stage)*
base_search := bool_expr // implicit filter/search, SPL convention
pipe_stage := "where" bool_expr
| "stats" agg_call ("," agg_call)* ["by" field ("," field)*]
| "sort" sort_field ("," sort_field)*
| "fields" field ("," field)*
| "head" [INT]
| "tail" [INT]
bool_expr := term (("and" | "or") term)*
term := field comparator value // structured filter -> ClickHouse
| "earliest" "=" time_expr // time range lower bound
| "latest" "=" time_expr // time range upper bound
| STRING | QUOTED_STRING // bare term -> free-text (Tantivy) on `message`
| "message" ":" QUOTED_STRING // explicit free-text (Tantivy phrase/wildcard syntax passed through)
comparator := "=" | "!=" | ">" | ">=" | "<" | "<="
agg_call := IDENT "(" [field] ")" ["as" IDENT] // count(), sum(field), avg(field), min(field), max(field)
sort_field := ["-" | "+"] field // "-" = desc (default), "+" = asc
time_expr := QUOTED_STRING // absolute RFC3339
| "-" INT ("s"|"m"|"h"|"d"|"w") // relative to query time, e.g. -1h, -7d
field := IDENT
```
### Worked examples
- `service=api | where status>=500 | stats count by host | sort -count`
— `service=api` is the base filter (structured, top-level column);
`where status>=500` filters on `status`, which isn't a top-level
column (see field mapping below); `stats count by host` aggregates;
`sort -count` orders descending by the aggregate's implicit `count`
alias.
- `message:"connection refused" | stats count by host` — free-text
predicate feeding a ClickHouse aggregation. This is the case task 2
called "the hardest part" — see Execution below.
- `SELECT host, count(*) FROM logs GROUP BY host` — detected as SQL (see
Detection below), executed directly against ClickHouse.
## Parser: hand-written recursive descent, no new dependency
This grammar is small and stable — seven pipe-stage kinds, one
expression grammar for filters. A hand-written lexer + recursive-descent
parser beats a combinator library (e.g. `participle`) or a generator
(`goyacc`) here:
- **No new dependency.** Consistent with "ask before adding a new
external dependency, there's no case for one at this grammar size.
- **Error messages matter** for a user-facing query language in a way
they don't for most internal parsing — "expected `by` after `stats
count`, got `sort`" is easy to produce by hand, harder to get right
through a combinator or generated parser.
- Generator tooling (`goyacc`) adds a codegen build step disproportionate
to a grammar this size.
- This is the standard approach for small, real query DSLs at this
scope — not a novel choice.
## The SQL escape hatch: not parsed, wrapped as opaque IR
"Both syntaxes compile to the same IR" does not mean writing a SQL
parser — reimplementing ClickHouse's SQL dialect would be a large,
pointless undertaking when ClickHouse already parses its own SQL. A query
that starts with `SELECT` (case-insensitive — see Detection) skips the
pipe-syntax parser entirely and produces an IR value that wraps the raw
SQL string as an opaque passthrough node. Both syntaxes still flow
through the same `Plan` type and the same executor code path — that's
what "same IR" actually buys (one execution and testing surface), not a
shared abstract syntax tree. The existing SELECT-only / single-statement
/ keyword-blocklist validation (`api/internal/queryapi/validate.go`) is
reused unchanged as the guard before wrapping.
## IR (`Plan`)
```go
type Plan struct {
RawSQL string // set => everything else is ignored; opaque ClickHouse passthrough
TextSearch []TextPredicate // bare terms / message: clauses -> routed to Tantivy
Filters []FilterPredicate // structured comparisons -> ClickHouse WHERE
TimeRange *TimeRange
Aggregation *Aggregation // nil => raw rows, no GROUP BY
Sort []SortField
Fields []string // projection; empty => all columns
Limit *Limit // head/tail
}
type TextPredicate struct {
Query string // passed to Tantivy's query parser as-is
}
type FilterPredicate struct {
Field string
Op string // "=", "!=", ">", ">=", "<", "<="
Value string
}
type Aggregation struct {
Funcs []AggFunc // count/sum/avg/min/max, each with an optional field + alias
GroupBy []string
}
type AggFunc struct {
Func string
Field string // empty for count()
Alias string
}
type SortField struct {
Field string
Desc bool
}
type Limit struct {
N int
Tail bool // true = last N (by time), false = first N
}
type TimeRange struct {
From, To time.Time // relative expressions (-1h etc.) resolved to absolute at compile time
}
```
## Field mapping: top-level columns vs. `attributes`
`logs`' real columns (per `/storage`) are `timestamp, host, service,
severity, message, attributes, record_id`. Any field name in a query
that isn't one of those maps to `attributes['<field>']` — e.g.
`status>=500` compiles to a comparison against `attributes['status']`,
not a top-level column, since `status` isn't promoted (Phase 1's decision
not to promote anything without real usage data still holds). Because
`attributes` is `Map(String,String)`, every stored value is a string;
numeric comparators against a non-top-level field cast via
`toFloat64OrZero(attributes['field'])` when the compared value looks
numeric, otherwise compare as string. This is what makes `where
status>=500` work against the existing schema with no migration —
querying an unpromoted field is always slightly more expensive than a
top-level column, which is worth knowing, not hiding.
## Execution: routing between ClickHouse and Tantivy
The core mechanism already exists and is proven: Phase 1's `/search`
endpoint (`api/internal/queryapi/search.go`, `recordIDsQuery`) already
does exactly steps 1–2 below for text-only queries. Phase 2 generalizes
it into four cases:
1. **No `TextSearch` predicates** → pure ClickHouse path. Build one SQL
statement directly from `Filters`/`TimeRange`/`Aggregation`/`Sort`/
`Fields`/`Limit`. The common case, and the fast path.
2. **`TextSearch` predicates, no `Aggregation`** → Phase 1's `/search`
behavior, generalized: Tantivy resolves matching `record_id`s, then
`SELECT ... WHERE record_id IN (...)` for the rows, with `Filters`/
`TimeRange`/`Sort`/`Fields`/`Limit` folded into that same statement.
3. **`TextSearch` predicates *and* `Aggregation`** — the genuinely new
case (`message:"connection refused" | stats count by host`): Tantivy
resolves matching `record_id`s as a *prefilter*, not a join, then one
ClickHouse statement does `WHERE record_id IN (...) AND <Filters>
GROUP BY <...>`. Aggregation always happens in ClickHouse; Tantivy
only ever narrows which rows are eligible before that.
4. **`RawSQL` set** → executed as-is against ClickHouse, no Tantivy
involvement regardless of what the SQL contains. The escape hatch is
opaque by design — no attempt to detect free-text intent inside raw
SQL.
### Known scaling limitation
Steps 2/3's `record_id IN (...)` approach breaks down if a text search
matches a large number of rows — the `IN` clause is a literal, quoted
UUID list embedded in the query string. Phase 2's mitigation: cap the
Tantivy prefilter at **5,000** results. Tantivy's `TopDocs` already
returns most-relevant-first, so the cap keeps the *best* matches rather
than an arbitrary truncation, but it's a real limitation on result
completeness for very broad text searches combined with aggregation.
Documented in `/docs/query-language-reference.md`, not silently
swallowed.
This number isn't a first-principles estimate — running the Phase 2
benchmark against a real 1M-row dataset (`/docs/phase-2-runbook.md`)
caught the original 10,000 cap failing outright: 10,000 quoted UUIDs
(~39 bytes each) produces a ~390KB query string, which exceeds
ClickHouse's *default* `max_query_size` (262144 bytes / 256KiB) and
fails with a syntax error rather than degrading gracefully — a much
lower ceiling than "multi-million-entry" suggested before anyone had
actually tried it. 5,000 UUIDs (~195KB) stays safely under that default
with headroom. The real long-term fix (streaming `record_id` batches,
ClickHouse-side text indexing, a different join strategy, or simply
raising `max_query_size` server-side with matching memory sizing) is
explicitly future work, out of scope for Phase 2.
## Where this lives: `api/internal/querylang/`
Not a new top-level component. This subsystem always executes in-process
within `/api` — it doesn't run standalone, doesn't get its own Docker
image, and needs both connections `/api` already holds (the ClickHouse
driver, the search gRPC client). A new top-level directory would imply a
new deployable service, which this isn't.
```
api/internal/querylang/
lexer/ tokenizer
ast/ parsed pipe-syntax tree
parser/ tokens -> ast (recursive descent)
ir/ Plan and supporting types
planner/ ast -> Plan (field-mapping rule, SQL-passthrough detection)
executor/ Plan -> results (the four-case routing above)
```
Mirrors the existing `internal/queryapi`, `internal/searchclient`
convention already in `/api`. Each layer is independently testable per
task 3's requirement: "pipe syntax X compiles to IR Y" tests live in
`parser`/`planner` against fixture ASTs/Plans, no backend needed; "IR Y
executes correctly" tests live in `executor` against fakes for both the
ClickHouse and search-client interfaces, same pattern already used
throughout `/ingest` and `/api`.
## `/query` endpoint: auto-detect, with an explicit override
Detection: a request body's query starting with `SELECT`
(case-insensitive, same rule `validateSelectOnly` already applies) is
SQL; otherwise pipe syntax. Covers the overwhelming common case with no
extra field required. An optional `"language": "sql" | "spl"` field in
the request body overrides detection, for the rare case a pipe query
legitimately starts with the literal word "select" as a bare search
term. Auto-detect-with-override matches the shape of other
inference-with-explicit-override choices already made in this stack
(e.g. severity hints winning over parsed values when present) — good
default ergonomics, no silent ambiguity once a caller cares enough to be
explicit.
docs/query-language-reference.md
+322
View File
@@ -0,0 +1,322 @@
# Query language reference
Sentry has one query language for everything: filtering, free-text
search, and aggregation, in a single query, against a single endpoint
(`POST /query`), from a single query bar in the web UI or `sentryctl
query` on the command line. You don't pick a "search mode" or a
"reporting mode" first — you write one query, and Sentry figures out
which parts need ClickHouse, which parts need the full-text index, and
combines them.
If you already know Splunk's SPL, most of this will feel immediately
familiar: a base search, piped through a sequence of processing stages.
Sentry's language is a deliberately smaller subset — the operators
people actually use day to day, not SPL's full surface area — plus raw
SQL as an escape hatch for anything the pipe syntax doesn't (yet) cover.
## The shape of a query
```
<base search> | <stage> | <stage> | ...
```
Everything before the first `|` is the base search — a filter and/or a
free-text search. Everything after each `|` is a processing stage that
narrows, reshapes, or summarizes what came before it.
```
service=api | where status>=500 | stats count by host | sort -count
```
Read left to right: start with everything logged by the `api` service,
keep only the entries with `status >= 500`, count how many there are per
`host`, and show the busiest hosts first.
## Filtering
```
field=value
field!=value
field>value
field>=value
field<value
field<=value
```
```
service=api
status>=500
host!=host-03
```
Multiple filters combine with `and` (the default when you don't write a
conjunction at all — see "Combining terms" below):
```
service=api status>=500
service=api and status>=500 (equivalent)
```
## Free-text search
Three ways to search the `message` field's text:
```
timeout a single bare word
"connection refused" a quoted phrase
message:"connection refused" the same thing, explicit
```
Free-text search is powered by Sentry's full-text index (Tantivy), which
supports phrase matching and wildcards:
```
message:"exact phrase"
message:"time*"
```
Bare words and quoted phrases can be mixed freely with structured filters
in the same query — that's the whole point of having one language:
```
service=api "connection refused"
message:"connection refused" | stats count by host
```
## Combining terms: `and` / `or`
Adjacent terms with nothing between them are implicitly `and`ed, matching
what most people expect from a search bar:
```
error timeout same as: error and timeout
```
`or` works between free-text terms, and Sentry's full-text index handles
it natively:
```
error or timeout
```
**Current limitation:** `or` is not supported between structured filters
(`service=api or service=web` returns a clear error rather than silently
being treated as `and`). If you need this, use two separate queries for
now, or the raw SQL escape hatch. This is a known gap, not an oversight —
full boolean-tree support for structured filters is on the list for a
future release once there's real usage data on how much it's needed.
## Time ranges
```
earliest=-1h relative: last hour
earliest=-15m latest=-5m relative window
earliest="2026-08-14T00:00:00Z" absolute (RFC 3339)
```
Relative offsets: a number followed by `s` (seconds), `m` (minutes), `h`
(hours), `d` (days), or `w` (weeks), always relative to when the query
runs.
## Pipe stages
### `where` — additional filtering after the base search
Same syntax as the base search's filter terms:
```
service=api | where status>=500
```
### `stats` — aggregation
```
stats count by host
stats count(), avg(latency_ms) as avg_latency by host, service
```
Supported functions: `count`, `sum`, `avg`, `min`, `max`. `count` doesn't
need a field (`count`, `count()`, and `count(*)` are all equivalent);
every other function requires one (`sum(latency_ms)`). Give a result an
explicit name with `as`, or accept the default (the function name, or
`count` for a bare count).
```
stats sum(bytes_sent) as total_bytes by host
```
### `sort` — ordering
```
sort -count descending by count (the default direction)
sort +host ascending by host
sort -severity, +host descending by severity, then ascending by host
```
`-` and `+` mean the same thing they do in most search tools: `-` for
descending, `+` for ascending. No sign at all also means descending.
You can sort by any field from the base data, or by a `stats` result's
column name/alias.
### `fields` — choosing which columns come back
```
fields host, message, severity
```
Without `fields`, you get every column.
### `head` / `tail` — limiting results
```
head first 100 (the default) results
head 20 first 20
tail 50 last 50, chronologically
```
## Field mapping: what's a "real" column vs. an attribute
Sentry's structured columns are `timestamp`, `host`, `service`,
`severity`, `message`, and `record_id`. Anything else you reference by
name — `status`, `latency_ms`, `winevt.event_id`, whatever your logs
happen to carry — is looked up in the per-record attributes, which are
always stored as text.
This matters for comparisons: `status>=500` only makes sense as a number,
so Sentry casts the attribute's text value to a number for you
automatically when the value you're comparing against looks numeric.
`status="unknown"` compares as text instead, since `"unknown"` isn't a
number. You don't need to do anything differently — this happens based
on what you write on the right-hand side of the comparison — but it's
worth knowing that:
- A field that's missing entirely, or whose value isn't actually numeric,
reads as `0` in a numeric comparison or aggregation (`toFloat64OrZero`
semantics) rather than erroring. A typo'd field name will "succeed"
with everything showing as `0` — if a `stats sum(...)` looks
suspiciously empty, double-check the field name.
- `stats min()`/`max()` on an attribute field always compares
numerically, not alphabetically, in this release.
- Querying an attribute is always a little more work for ClickHouse than
querying a real column — if a field turns out to be central to how you
query your logs, that's a signal it might be worth promoting to a real
column in a future schema change (not something you can do yourself
today).
## Raw SQL
Anything starting with `SELECT` is treated as raw ClickHouse SQL and run
directly, no pipe-syntax parsing involved:
```
SELECT host, count(*) FROM logs WHERE service = 'api' GROUP BY host
```
SELECT-only, single statement — Sentry allowlists this at the API level.
Use this for anything the pipe syntax doesn't cover yet: window
functions, `WITH` clauses, ClickHouse-specific functions, joins across
other tables you've added, and so on. There's no performance penalty for
using SQL over the pipe syntax or vice versa — both compile to the same
execution plan internally.
## Which syntax am I using?
Sentry detects automatically: a query starting with `SELECT` runs as
SQL, anything else runs as the pipe syntax. This covers the overwhelming
majority of real queries with no extra step. If you're writing a pipe
query that happens to start with the literal word "select" as a search
term, set the language explicitly instead of relying on detection:
```json
{"query": "select", "language": "spl"}
```
`language` accepts `"sql"`, `"spl"`, or can be omitted entirely (the
default, auto-detect). The web UI's query bar shows which one it
detected next to the query box, with a dropdown to override it.
## Combining free-text search with aggregation
This is the case that makes Sentry's query language more than "SQL with
extra steps" — free text and aggregation, together, in one query:
```
message:"connection refused" | stats count by host
```
Under the hood: the full-text index resolves which records match the
text search first, then ClickHouse does the counting and grouping over
just those records. You don't need to know this to use it — it's
mentioned here because of the one limitation it implies:
**A single free-text search is capped at 5,000 matching records** when
it's combined with a `stats`/filter stage that needs to know exactly
which records matched (the most-relevant 5,000, not an arbitrary
truncation). A text search alone, with no aggregation, isn't affected by
this cap. If your combined query's text search is broad enough to match
more than 5,000 records, narrow it — a more specific phrase, an added
`where` filter, or a tighter time range — the same way you'd narrow an
overly broad search in any tool.
## Response shape
Every query, regardless of syntax or which backend(s) it touched,
returns the same shape:
```json
{"columns": ["host", "count"], "rows": [["api-01", 42], ["api-02", 17]]}
```
or, on error:
```json
{"error": "a description of what went wrong"}
```
## Quick reference
| Syntax | Meaning |
|---|---|
| `field=value` | equals |
| `field!=value` | not equals |
| `field>value` / `>=` / `<` / `<=` | comparison |
| `"phrase"` / bare word | free-text search on `message` |
| `message:"phrase"` | explicit free-text search |
| `earliest=-1h` / `latest=...` | time range |
| `\| where ...` | additional filter |
| `\| stats count by field` | aggregate |
| `\| sort -field` / `+field` | sort desc / asc |
| `\| fields a, b` | choose columns |
| `\| head N` / `\| tail N` | limit results |
| `SELECT ...` | raw SQL |
## Examples
```
service=api | where status>=500 | stats count by host | sort -count
```
Which hosts are producing the most 5xx errors from the `api` service?
```
message:"connection refused" | stats count by host
```
Where are connection-refused errors coming from?
```
earliest=-24h severity=ERROR | stats count by service | sort -count
```
Error volume by service over the last day.
```
winevt.event_id=4625 | fields host, message | head 20
```
Recent failed Windows logon attempts (a `winevt.*` attribute from the
Windows Event Log source — see `/docs/phase-1-runbook.md`).
```
SELECT host, avg(toFloat64OrZero(attributes['latency_ms'])) AS avg_latency
FROM logs WHERE service = 'api' GROUP BY host ORDER BY avg_latency DESC
```
The same kind of query the pipe syntax's `stats avg(latency_ms) by host`
would produce, written by hand — useful as a starting point if you need
something the pipe syntax doesn't support yet.