49dd05068909725c1aa6893935050673caf988e9
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
49dd050689 |
Start the Terraform provider: sentry_dashboard, the first resource
CLAUDE.md names the Terraform provider a first-class deliverable
alongside sentryctl ("CLI and Terraform provider are first-class, not
afterthoughts"), but no phase before this one had actually built any of
it -- no terraform/ directory existed. This is a first slice, not a
finished provider: one resource, scoped and confirmed with the project
owner before starting (a new pinned external dependency and an
architectural decision not covered in /docs/architecture.md are both
things CLAUDE.md's own "When in doubt" section says to ask about).
New Go module (terraform/, github.com/sentry/sentry/terraform) built on
HashiCorp's terraform-plugin-framework -- the actively-developed
library, not the legacy SDKv2, since there's no existing provider code
to migrate and no reason to start new on the framework HashiCorp itself
steers people away from.
internal/provider/client.go talks the exact same JSON contract
sentryctl's "dashboards apply" and web's Export JSON button already use
against api/dashboards.Handler (POST/GET/PUT/DELETE /dashboards[/{id}]) --
cli/README.md already named this "the seed of a future Terraform
provider: one JSON contract, multiple callers," this is that third
caller, not a new contract invented for Terraform's sake.
sentry_dashboard's schema deliberately leaves default_earliest/
default_latest Optional+Computed with no Terraform-side static default,
even though the API defaults them to "-1h"/"now" when empty -- letting
the API stay the one source of truth for what "unset" means rather than
duplicating that default in two places that could drift. tenant_id is
Computed-only, matching api/dashboards.Handler's own tenantID() doc
comment that a client-supplied value is always overridden server-side.
Panels are not modeled by this resource -- a genuinely separate resource
shape (own lifecycle, own endpoints, own validation needs), scoped out
deliberately, not an oversight. Alert rules, notification targets, and
tenant/RBAC resources are the same: real, disclosed future work, not
attempted in this pass. See terraform/README.md for the full accounting.
Verified: client_test.go runs real HTTP round trips against httptest.
Server (request construction, response parsing, the 404-vs-other-error
distinction Read/Delete need for Terraform's out-of-band-deletion
convention) -- same pattern cli/cmd/sentryctl's own tests already use
against the same api/dashboards endpoints. provider_test.go validates
both schemas are internally well-formed without needing a Terraform
binary. dashboard_resource_test.go's TestAccDashboardResource_basic is a
real acceptance test (terraform-plugin-testing), skip-gated by TF_ACC=1
per that framework's own convention -- even with TF_ACC set it would
still need a live api service (Postgres+ClickHouse) to apply against,
which this environment has no Docker access to bring up, so it has not
actually run here, same disclosed gap as every other live-infra test in
this repo.
|
||
|
jcoffey-dev
|
9435115ab7 |
Phase 3: dashboards and alerting
Saved, shareable multi-panel dashboards (table/line/bar/single-stat panels via gridstack + uPlot, global + per-panel time range, JSON export/import) and threshold/absence alert rules with an ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery. - New /metadata component: Postgres control-plane store for dashboards, panels, notification targets, alert rules/state, and delivery log -- see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree family isn't a fit for this access pattern (needs real row-level locking and read-your-writes consistency). - api/internal/dashboards: dashboard/panel CRUD, pure -- panel query execution stays client-side, reusing the existing /query endpoint. - New /alerting service: rule/target CRUD, a ticker-driven evaluator (claim-then-evaluate concurrency control, transactional-outbox delivery, query errors and threshold zero-rows never coerced into a false transition) and webhook/Slack/PagerDuty delivery with retry/backoff. See docs/phase-3-alerting-design.md for the full state-machine design and the four correctness properties it implements. - web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts list/get/apply, seeding a future Terraform provider's JSON contract. - hack/alert-load-test: 500 rules against real ClickHouse data, real measured results in docs/phase-3-runbook.md. Five real bugs found by actually running this against a live stack (documented in the runbook, not just fixed silently): a latent Phase 2 bug where ClickHouse rejected the timestamp format used for earliest=/latest= queries; a "now" literal token injected into query text; a GridStack/uPlot layout-timing race; JS's Date.parse being too lenient to use as a timestamp-detection heuristic; a rule's "enabled" field silently defaulting to false when omitted; and the evaluator's claim-batch-size and worker-pool-concurrency defaulting to the same value, causing 500 concurrently-due rules to take 125s to cycle through instead of the configured 60s. |
||
|
jcoffey-dev
|
fb5049a747 |
Phase 2: unified query language spanning ClickHouse and Tantivy
Replaces the separate SQL-only /query and text-only /search endpoints with one pipe-syntax query language (plus raw SQL escape hatch) that compiles to a single IR and execution plan across both backends, so a query like `message:"connection refused" | stats count by host` runs as one request instead of two disjoint tools. - api/internal/querylang: lexer -> ast -> parser -> ir -> planner -> executor, each layer independently tested. - Execution generalizes Phase 1's proven Tantivy-prefilter pattern into a 4-way routing table (pure ClickHouse / text-only / text + aggregation / raw SQL passthrough). - Unified web query page and `sentryctl query`, both hitting the same POST /query endpoint. - Benchmarked against a real 1,022,000-row dataset (hack/benchmark-fixture); caught and fixed a real bug where the Tantivy prefilter cap (10,000) produced an IN-clause exceeding ClickHouse's default max_query_size -- lowered to 5,000, documented in docs/query-language-design.md and docs/phase-2-runbook.md. - docs/query-language-reference.md: customer-facing syntax reference. |
||
|
jcoffey-dev
|
cd8aa290ca |
Phase 1: Windows log collection + full-text search
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore. |
||
|
jcoffey-dev
|
b6b092c912 |
Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is. |