No code changes -- alert rules already accepted query_language: "sql"
with zero validation restricting it to the pipe syntax, and the web
UI's rule-creation form already auto-detects SQL vs. pipe syntax via
the shared QueryBar component. This was simply never exercised in
this specific combination before.
Live-verified with three agents: a threshold rule on
count(DISTINCT host) against an expected fleet size correctly
evaluated ok with all three healthy, then correctly fired when one
was killed and its heartbeat rows aged out of the window -- one rule
covering a whole named group of hosts instead of one rule per host.
Documented as an honest aggregate check, not true per-host alerting:
that would need the alerting engine's own per-group state tracking,
already named a Phase 3 non-goal for the whole engine, not something
specific to agents -- explicitly out of scope here rather than
quietly built as a side effect.
Agents now send an independent "still alive" record on a configurable
schedule (seconds/minutes/hours, [heartbeat] in agent.toml), separate
from real log traffic and tagged with a sentry.heartbeat attribute.
No new wire protocol -- it's an ordinary record through the same
PushBatch RPC/mTLS identity every log line already uses. Unavailability
alerting reuses the existing absence-condition alert rule type
unchanged; no new alerting code was needed. See
/docs/agent-heartbeat-monitoring.md for the design and how to build the
alert rule.
While verifying the alert rule live, found that the query language's
lexer never treated '-' as part of an identifier, so any unquoted
hyphenated filter value -- including the reference doc's own canonical
example, `host!=host-03` -- failed to parse at all. Fixed in
api/internal/querylang/lexer/lexer.go with regression tests; a leading
'-' still lexes as its own token so earliest=-1h/sort -count are
unaffected.
jcoffey-dev