Files
cairnobs/hack/alert-load-test
jcoffey-dev 13cf9a30cb Rebrand: Sentry -> Cairn OBS
Full rebrand across cosmetic branding, code identifiers, and
infrastructure/data-plane naming, using the supplied Cairn OBS logo
package. Cosmetic: favicon/logo swap (also closes a stale license-audit
finding -- the old favicon was SvelteKit's unreplaced scaffold logo),
new centered welcome landing page, larger/legible sidebar logo, page
titles, CLAUDE.md/README/docs prose.

Code identifiers: Go module path github.com/sentry/sentry ->
github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc
regenerated); Rust crates sentry-agent/sentry-parser/sentry-search ->
cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully
renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type,
env vars); every session/auth cookie name; agent config paths and
Windows service identity.

Deliberately preserved: the gRPC wire protocol's protobuf packages
(sentry.logs.v1, sentry.agent.v1) and their Go import directory
(proto/sentry/...) -- renaming the wire-level package would break every
currently-deployed agent binary (confirmed two real hosts, including
mail.inbuxa.com, are actively streaming through this exact contract)
until rebuilt and redeployed in lockstep with an ingest cutover. Only
the Go module path wrapping the generated code changes.

Infrastructure: every docker-compose container name (root and three
component-level compose files); the Helm chart (directory, Chart.yaml,
named-template helpers, all templates, values.yaml image repos);
Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML
files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd
package. Caught and fixed real path-coupling bugs along the way: the
Helm chart's search/ingest volume mounts and the dev-only-credential
detection constant vs. docker-compose.yml's literal values had to move
together or a security warning would have silently stopped firing.

Data plane: Postgres database sentry_metadata -> cairnobs_metadata and
role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka
topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups.
Source-level defaults, docker-compose.yml, and every migrate.sh/
provision script default updated together; already-applied migration
files left untouched per this repo's immutable-migration convention.

Verified at every layer: all 13 Go modules build/vet/test clean, both
Rust workspaces (agent, search) build/clippy/test clean, npm run check/
build clean, docker compose config validates on all four compose files.
Live-verified against a real docker stack multiple times through this
work, including a final fresh-volume run confirming the actual renamed
Postgres database/role, ClickHouse database, and Kafka topic all work
end to end with a real login and query, zero console errors.
2026-08-21 20:53:32 -07:00
..
2026-08-21 20:53:32 -07:00
2026-08-13 17:29:38 -07:00
2026-08-13 17:29:38 -07:00

alert-load-test

Seeds a realistic number of concurrent alert rules via /alerting's real create API (not a direct DB insert) and measures whether the evaluator's claim scheduling keeps up under load. See /docs/phase-3-alerting-design.md's "Load-testing plan" and /docs/phase-3-runbook.md for the methodology and real measured results.

# 1. Push real data so rule queries have real work to do (reuses
#    hack/benchmark-fixture):
cd ../benchmark-fixture
go run . --count 500000

# 2. Run a webhook-sink so the (never-firing, by design) rules have a
#    valid notification target to point at:
docker run -d --name sentry-webhook-sink --network sentry_default \
  -p 9099:9099 -v $(pwd)/../webhook-sink:/src -w /src golang:1.25-alpine go run .

# 3. Run the load test:
cd ../alert-load-test
go run . --rule-count 500 --eval-interval-seconds 60 --duration 3m30s

Each rule queries a different host's count over the last minute (earliest=-1m host="host-01" | stats count) against real ClickHouse data, with threshold_value set unreachably high so rules stay ok -- this isolates evaluator/ClickHouse scheduling throughput from delivery-worker load (a query that never fires still exercises the exact same claim → /query → evaluate → ApplyTransition path every tick).

The report shows, per rule, the observed intervals between consecutive last_evaluated_at changes (polled at --poll-interval), compared against the configured eval_interval_seconds. A real, significant finding from actually running this: the evaluator's claim batch size and worker-pool concurrency limit defaulted to the same number (20), so 500 rules all due at once took 125s to cycle through instead of the configured 60s. Fixed by separating EVALUATOR_CLAIM_BATCH_SIZE from EVALUATOR_WORKER_POOL_SIZE (see alerting/internal/config/config.go).

Cleans up the seeded rules and notification target on exit unless --no-cleanup is passed.