jcoffey-dev
13cf9a30cb
Full rebrand across cosmetic branding, code identifiers, and infrastructure/data-plane naming, using the supplied Cairn OBS logo package. Cosmetic: favicon/logo swap (also closes a stale license-audit finding -- the old favicon was SvelteKit's unreplaced scaffold logo), new centered welcome landing page, larger/legible sidebar logo, page titles, CLAUDE.md/README/docs prose. Code identifiers: Go module path github.com/sentry/sentry -> github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc regenerated); Rust crates sentry-agent/sentry-parser/sentry-search -> cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type, env vars); every session/auth cookie name; agent config paths and Windows service identity. Deliberately preserved: the gRPC wire protocol's protobuf packages (sentry.logs.v1, sentry.agent.v1) and their Go import directory (proto/sentry/...) -- renaming the wire-level package would break every currently-deployed agent binary (confirmed two real hosts, including mail.inbuxa.com, are actively streaming through this exact contract) until rebuilt and redeployed in lockstep with an ingest cutover. Only the Go module path wrapping the generated code changes. Infrastructure: every docker-compose container name (root and three component-level compose files); the Helm chart (directory, Chart.yaml, named-template helpers, all templates, values.yaml image repos); Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd package. Caught and fixed real path-coupling bugs along the way: the Helm chart's search/ingest volume mounts and the dev-only-credential detection constant vs. docker-compose.yml's literal values had to move together or a security warning would have silently stopped firing. Data plane: Postgres database sentry_metadata -> cairnobs_metadata and role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups. Source-level defaults, docker-compose.yml, and every migrate.sh/ provision script default updated together; already-applied migration files left untouched per this repo's immutable-migration convention. Verified at every layer: all 13 Go modules build/vet/test clean, both Rust workspaces (agent, search) build/clippy/test clean, npm run check/ build clean, docker compose config validates on all four compose files. Live-verified against a real docker stack multiple times through this work, including a final fresh-volume run confirming the actual renamed Postgres database/role, ClickHouse database, and Kafka topic all work end to end with a real login and query, zero console errors.
99 lines
3.9 KiB
Markdown
99 lines
3.9 KiB
Markdown
# alerting
|
|
|
|
Alert rule CRUD, the ticker-driven evaluator, and webhook/Slack/PagerDuty
|
|
delivery. See `/docs/phase-3-alerting-design.md` for the full design
|
|
(data model, the ok/pending/firing state machine, and the four
|
|
correctness properties this implementation follows exactly).
|
|
|
|
## Running
|
|
|
|
```sh
|
|
POSTGRES_PASSWORD=cairnobs-dev-only API_QUERY_URL=http://localhost:8080 go run ./cmd/alerting
|
|
```
|
|
|
|
Talks to the same `sentry_metadata` Postgres database as `/api`
|
|
(different tables — see `/metadata/README.md`), and to `/api`'s
|
|
`POST /query` over plain HTTP for rule evaluation. Never connects to
|
|
ClickHouse or Tantivy directly.
|
|
|
|
## HTTP API
|
|
|
|
```
|
|
POST /rules create a rule
|
|
GET /rules list rules (with current state)
|
|
GET /rules/{id} get a rule (with current state)
|
|
DELETE /rules/{id}
|
|
GET /rules/{id}/deliveries delivery log for a rule, most recent first
|
|
|
|
POST /targets create a notification target
|
|
GET /targets
|
|
GET /targets/{id}
|
|
DELETE /targets/{id}
|
|
|
|
GET /healthz
|
|
```
|
|
|
|
A rule's `condition_type` is `"threshold"` (requires `comparator` +
|
|
`threshold_value`, and the query must resolve to exactly one row) or
|
|
`"absence"` (fires when the query returns zero rows in its own
|
|
`earliest=`/`latest=` window — no separate window field). A notification
|
|
target's `kind` is `"webhook"`, `"slack"`, or `"pagerduty"` — all three
|
|
deliver via the same HTTP POST + retry/backoff mechanism
|
|
(`internal/delivery/webhook.go`); slack/pagerduty are payload formatters
|
|
only, not separate delivery paths.
|
|
|
|
## Environment variables
|
|
|
|
| Var | Default |
|
|
|---|---|
|
|
| `HTTP_LISTEN_ADDR` | `:8081` |
|
|
| `POSTGRES_ADDR` | `localhost:5432` |
|
|
| `POSTGRES_DATABASE` | `sentry_metadata` |
|
|
| `POSTGRES_USERNAME` | `sentry` |
|
|
| `POSTGRES_PASSWORD` | (empty — must be set) |
|
|
| `API_QUERY_URL` | `http://localhost:8080` |
|
|
| `CORS_ALLOWED_ORIGIN` | `*` |
|
|
| `EVALUATOR_TICK_SECONDS` | `5` — how often the scheduler checks for due rules |
|
|
| `EVALUATOR_CLAIM_BATCH_SIZE` | `1000` — how many due rules one tick can pull off the queue |
|
|
| `EVALUATOR_WORKER_POOL_SIZE` | `20` — bounded concurrency for `/query` calls within a claimed batch |
|
|
| `EVALUATOR_QUERY_TIMEOUT_SECONDS` | `30` — per-evaluation `POST /query` timeout |
|
|
|
|
`EVALUATOR_CLAIM_BATCH_SIZE` and `EVALUATOR_WORKER_POOL_SIZE` are
|
|
deliberately separate knobs, not the same number — see
|
|
`internal/config/config.go`'s doc comment for the real bug this
|
|
separation fixes (found by `hack/alert-load-test`, see
|
|
`/docs/phase-3-runbook.md`): with both capped at 20, 500 rules due at
|
|
once took 125s to cycle through instead of the configured 60s.
|
|
|
|
## Package layout
|
|
|
|
```
|
|
cmd/alerting/ wires config, Postgres pool, api client; runs the
|
|
HTTP server + evaluator + delivery worker concurrently (errgroup)
|
|
internal/httpapi/ REST handlers -- Handler/RegisterRoutes, same shape as api/internal/dashboards
|
|
internal/rulestore/ pgx CRUD for alert_rules + alert_state; ClaimDueRules
|
|
(fix 1's atomic claim) and ApplyTransition (fix 2's transactional outbox)
|
|
internal/notifystore/ pgx CRUD for notification_targets
|
|
internal/queryclient/ thin HTTP client to api's POST /query -- no querylang import here
|
|
internal/evaluator/ the ticker + worker pool; transitions.go is the pure,
|
|
exhaustively-tested ok/pending/firing state machine;
|
|
condition.go implements fixes 3/4 (errors never
|
|
coerced to "condition false"; threshold zero-rows
|
|
is an error, not a 0)
|
|
internal/delivery/ webhook.go is the claim-and-send worker (all three
|
|
kinds go through it); slack.go/pagerduty.go are
|
|
payload formatters only
|
|
```
|
|
|
|
## Building & testing
|
|
|
|
```sh
|
|
go build ./...
|
|
go vet ./...
|
|
go test ./...
|
|
```
|
|
|
|
```sh
|
|
docker build -f Dockerfile -t sentry-alerting . # context is alerting/, not the repo root -- no /proto needed
|
|
```
|