jcoffey-dev
c920e0f2c4
The rename commit before this one covered module paths and the obvious user-facing strings; this is the rest of it -- the places where "sentry" was a default value, a filename, or a picture rather than a word in a sentence. Defaults that changed: CLICKHOUSE_DATABASE (sentry -> cairnobs), POSTGRES_DATABASE (sentry_metadata -> cairnobs_metadata), and POSTGRES_USERNAME (sentry -> cairnobs), across api/alerting/ingest and the enterprise binaries, plus the compose files and migrate scripts that create those objects. These are *defaults*, so a deployment that sets them explicitly is unaffected -- but any deployment relying on the old defaults must have its environment updated before it picks this up, or it will come up pointing at a database that doesn't exist. Also: the light-mode logo variants (the dark ones existed alone, so the landing page and sidebar rendered a dark mark on a light background), regenerated favicons, and the docs/README/threat-model prose that still said Sentry.
99 lines
3.9 KiB
Markdown
99 lines
3.9 KiB
Markdown
# alerting
|
|
|
|
Alert rule CRUD, the ticker-driven evaluator, and webhook/Slack/PagerDuty
|
|
delivery. See `/docs/phase-3-alerting-design.md` for the full design
|
|
(data model, the ok/pending/firing state machine, and the four
|
|
correctness properties this implementation follows exactly).
|
|
|
|
## Running
|
|
|
|
```sh
|
|
POSTGRES_PASSWORD=cairnobs-dev-only API_QUERY_URL=http://localhost:8080 go run ./cmd/alerting
|
|
```
|
|
|
|
Talks to the same `cairnobs_metadata` Postgres database as `/api`
|
|
(different tables — see `/metadata/README.md`), and to `/api`'s
|
|
`POST /query` over plain HTTP for rule evaluation. Never connects to
|
|
ClickHouse or Tantivy directly.
|
|
|
|
## HTTP API
|
|
|
|
```
|
|
POST /rules create a rule
|
|
GET /rules list rules (with current state)
|
|
GET /rules/{id} get a rule (with current state)
|
|
DELETE /rules/{id}
|
|
GET /rules/{id}/deliveries delivery log for a rule, most recent first
|
|
|
|
POST /targets create a notification target
|
|
GET /targets
|
|
GET /targets/{id}
|
|
DELETE /targets/{id}
|
|
|
|
GET /healthz
|
|
```
|
|
|
|
A rule's `condition_type` is `"threshold"` (requires `comparator` +
|
|
`threshold_value`, and the query must resolve to exactly one row) or
|
|
`"absence"` (fires when the query returns zero rows in its own
|
|
`earliest=`/`latest=` window — no separate window field). A notification
|
|
target's `kind` is `"webhook"`, `"slack"`, or `"pagerduty"` — all three
|
|
deliver via the same HTTP POST + retry/backoff mechanism
|
|
(`internal/delivery/webhook.go`); slack/pagerduty are payload formatters
|
|
only, not separate delivery paths.
|
|
|
|
## Environment variables
|
|
|
|
| Var | Default |
|
|
|---|---|
|
|
| `HTTP_LISTEN_ADDR` | `:8081` |
|
|
| `POSTGRES_ADDR` | `localhost:5432` |
|
|
| `POSTGRES_DATABASE` | `cairnobs_metadata` |
|
|
| `POSTGRES_USERNAME` | `cairnobs` |
|
|
| `POSTGRES_PASSWORD` | (empty — must be set) |
|
|
| `API_QUERY_URL` | `http://localhost:8080` |
|
|
| `CORS_ALLOWED_ORIGIN` | `*` |
|
|
| `EVALUATOR_TICK_SECONDS` | `5` — how often the scheduler checks for due rules |
|
|
| `EVALUATOR_CLAIM_BATCH_SIZE` | `1000` — how many due rules one tick can pull off the queue |
|
|
| `EVALUATOR_WORKER_POOL_SIZE` | `20` — bounded concurrency for `/query` calls within a claimed batch |
|
|
| `EVALUATOR_QUERY_TIMEOUT_SECONDS` | `30` — per-evaluation `POST /query` timeout |
|
|
|
|
`EVALUATOR_CLAIM_BATCH_SIZE` and `EVALUATOR_WORKER_POOL_SIZE` are
|
|
deliberately separate knobs, not the same number — see
|
|
`internal/config/config.go`'s doc comment for the real bug this
|
|
separation fixes (found by `hack/alert-load-test`, see
|
|
`/docs/phase-3-runbook.md`): with both capped at 20, 500 rules due at
|
|
once took 125s to cycle through instead of the configured 60s.
|
|
|
|
## Package layout
|
|
|
|
```
|
|
cmd/alerting/ wires config, Postgres pool, api client; runs the
|
|
HTTP server + evaluator + delivery worker concurrently (errgroup)
|
|
internal/httpapi/ REST handlers -- Handler/RegisterRoutes, same shape as api/internal/dashboards
|
|
internal/rulestore/ pgx CRUD for alert_rules + alert_state; ClaimDueRules
|
|
(fix 1's atomic claim) and ApplyTransition (fix 2's transactional outbox)
|
|
internal/notifystore/ pgx CRUD for notification_targets
|
|
internal/queryclient/ thin HTTP client to api's POST /query -- no querylang import here
|
|
internal/evaluator/ the ticker + worker pool; transitions.go is the pure,
|
|
exhaustively-tested ok/pending/firing state machine;
|
|
condition.go implements fixes 3/4 (errors never
|
|
coerced to "condition false"; threshold zero-rows
|
|
is an error, not a 0)
|
|
internal/delivery/ webhook.go is the claim-and-send worker (all three
|
|
kinds go through it); slack.go/pagerduty.go are
|
|
payload formatters only
|
|
```
|
|
|
|
## Building & testing
|
|
|
|
```sh
|
|
go build ./...
|
|
go vet ./...
|
|
go test ./...
|
|
```
|
|
|
|
```sh
|
|
docker build -f Dockerfile -t cairnobs-alerting . # context is alerting/, not the repo root -- no /proto needed
|
|
```
|