coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Give the demo a live synthetic fleet, dashboards, and alert rules

Browse Source 
The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.

/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.

Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.

Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.

/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.

No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
This commit is contained in:
jcoffey-dev
2026-08-22 16:12:35 -07:00
parent e6a58f58ea
commit bcb9a01cd6
31 changed files with 3031 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hack/demo-seed/README.md
+79
View File
@@ -0,0 +1,79 @@
# demo-seed
Everything the public demo deployment (`demo.cairnobs.org`) is built out
of, kept in the repo rather than only on the box so a demo can be rebuilt
from scratch and so its dashboards and rules are reviewable like any
other code.
- `reset-demo.sh` — the nightly reset. Wipes every volume, brings the
stack back up, re-seeds users, notification targets, a week of
synthetic history, dashboards, and alert rules. Run from cron at 04:00
on the demo host.
- `dashboards/*.json` — one file per dashboard, in the shape
`POST /dashboards/import` consumes (identical to what
`GET /dashboards/{id}/export` and the web UI's Export JSON button
produce, so a dashboard edited in the UI can be exported straight back
into this directory).
- `alerts/*.json.template` — one file per alert rule, in the shape
`POST /rules` consumes. `__TARGET_OPS__` / `__TARGET_SECURITY__` /
`__TARGET_PLATFORM__` are substituted at apply time with the IDs of the
three notification targets `reset-demo.sh` creates: every reset starts
from an empty database, so the IDs can't be baked in.
- `cairnobs-demo-simulator.service` — systemd unit for the live half of
the demo, `/hack/demo-simulator`. Installed at `/etc/systemd/system/`
on the demo box.
## Prefilled login
The demo's login page comes up with the read-only `demo` account already
in both fields, so a visitor doesn't need credentials handed to them.
That's a build-time opt-in, off everywhere else: the web image is built
with `VITE_DEMO_USERNAME`/`VITE_DEMO_PASSWORD` (set in the demo host's
`docker-compose.override.yml`), and the login page prefills only when it
has both. Any deployment that doesn't set them gets the ordinary empty
form -- see `web/src/lib/api.ts`'s `demoUsername`.
The password is baked into the static bundle, which is fine for exactly
this case and nothing else: a Viewer-role account on a deployment whose
database is wiped and reseeded nightly. It has to match `DEMO_PASSWORD`
in `reset-demo.sh`, and changing that means rebuilding the web image.
## Why the demo needs a long-running process
Three things the demo has to show are only true if data keeps arriving,
and no amount of one-shot seeding fixes any of them:
- **Agents.** The Agents page is populated by the `AgentControl.CheckIn`
RPC, and marks a host stale once it stops calling in. A fleet seeded
once at 04:00 is entirely stale by 04:10.
- **Alerts.** Rules evaluate over trailing windows (`earliest=-5m`).
Against a frozen dataset every rule settles into a permanent state
within minutes and the Alerts page never moves again.
- **Recent views.** A "last 15 minutes" dashboard, or a query for what
just happened, is empty on a dataset that stopped growing overnight.
So the demo runs `demo-simulator` continuously, and `reset-demo.sh` only
handles the parts that genuinely are one-shot: the history behind the
present, and the dashboards and rules themselves.
## Editing a dashboard
Change it in the web UI, hit Export JSON, and drop the file in
`dashboards/` — the export shape and the import shape are the same one.
The next reset picks it up. (Note that panel IDs and the dashboard ID are
not part of that shape: every reset creates them fresh.)
## What the queries can't do yet
Every panel here uses `table`, `bar`, `top_n`, `single_stat`, or
`heatmap`. None uses `line`, because a line chart needs a time axis and
the query language has no time-bucketing function — `stats count by
<field>` groups by literal column values, so there's no equivalent of
Splunk's `bin`/`timechart` to group by hour or minute. Raw SQL could
express it (`toStartOfHour(timestamp)`), but dashboard panels reject the
SQL escape hatch by design, since the time-range picker works by
prepending `earliest=`/`latest=` terms to a pipe-syntax query
(see `api/dashboards/types.go`'s `validatePanel`).
That gap is the one real thing standing between these dashboards and a
conventional observability overview screen.
hack/demo-seed/alerts/agent-legacy-01-unavailable.json.template
+11
View File
@@ -0,0 +1,11 @@
{
"name": "legacy-01 agent unavailable",
"description": "No heartbeat from legacy-01 within its heartbeat window -- the agent or the host is gone",
"query": "earliest=-5m host=\"legacy-01\" cairnobs.heartbeat=true",
"query_language": "spl",
"condition_type": "absence",
"eval_interval_seconds": 60,
"for_minutes": 0,
"notification_target_id": "__TARGET_PLATFORM__",
"enabled": true
}
hack/demo-seed/alerts/api-5xx-rate.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "High API 5xx rate",
"description": "The API tier is returning server errors well above its normal background rate",
"query": "service=api earliest=-5m | where status>=500 | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 60,
"for_minutes": 1,
"notification_target_id": "__TARGET_OPS__",
"enabled": true,
"comparator": "gt",
"threshold_value": 4
}
hack/demo-seed/alerts/api-latency.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "API latency degraded",
"description": "Average API response time over the last 10 minutes is above the service objective",
"query": "service=api earliest=-10m | stats avg(latency_ms) as avg_latency_ms",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 60,
"for_minutes": 2,
"notification_target_id": "__TARGET_OPS__",
"enabled": true,
"comparator": "gt",
"threshold_value": 130
}
hack/demo-seed/alerts/edge-5xx-surge.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Edge 5xx surge",
"description": "nginx is serving 5xx to clients -- either upstreams are failing or the edge itself is",
"query": "service=nginx earliest=-5m | where status>=500 | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 60,
"for_minutes": 1,
"notification_target_id": "__TARGET_OPS__",
"enabled": true,
"comparator": "gt",
"threshold_value": 4
}
hack/demo-seed/alerts/firewall-block-surge.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Firewall block surge",
"description": "UFW is dropping far more inbound connections than usual -- typically a scan in progress",
"query": "service=system ufw_action=BLOCK earliest=-15m | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 120,
"for_minutes": 0,
"notification_target_id": "__TARGET_SECURITY__",
"enabled": true,
"comparator": "gt",
"threshold_value": 20
}
hack/demo-seed/alerts/mail-auth-failures.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Mail authentication failures",
"description": "Repeated SMTP auth failures on the mail host -- credential stuffing against the mail server",
"query": "service=smtp result=auth_failed earliest=-15m | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 120,
"for_minutes": 0,
"notification_target_id": "__TARGET_SECURITY__",
"enabled": true,
"comparator": "gt",
"threshold_value": 6
}
hack/demo-seed/alerts/postgres-slow-queries.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Slow database queries",
"description": "Statements taking over a second are piling up, which usually shows up as API latency next",
"query": "service=postgres earliest=-10m | where duration_ms>1000 | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 60,
"for_minutes": 2,
"notification_target_id": "__TARGET_PLATFORM__",
"enabled": true,
"comparator": "gt",
"threshold_value": 5
}
hack/demo-seed/alerts/ssh-brute-force.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "SSH brute-force attempt",
"description": "A burst of failed SSH authentications across the fleet, well past normal background probing",
"query": "service=system auth_result=failed earliest=-10m | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 60,
"for_minutes": 0,
"notification_target_id": "__TARGET_SECURITY__",
"enabled": true,
"comparator": "gt",
"threshold_value": 12
}
hack/demo-seed/alerts/windows-account-lockout.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Windows account lockout",
"description": "Any 4740 on the Windows hosts: an account was locked out after repeated failures",
"query": "winevt.event_id=4740 earliest=-15m | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 120,
"for_minutes": 0,
"notification_target_id": "__TARGET_SECURITY__",
"enabled": true,
"comparator": "gte",
"threshold_value": 1
}
hack/demo-seed/alerts/worker-disk-filling.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "worker-02 disk nearly full",
"description": "worker-02's data volume has passed 175 GiB of 200 GiB and is still climbing",
"query": "host=\"worker-02\" cairnobs.metrics=true earliest=-15m | stats max(disk_used_bytes) as disk_used_bytes",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 300,
"for_minutes": 0,
"notification_target_id": "__TARGET_PLATFORM__",
"enabled": true,
"comparator": "gt",
"threshold_value": 187904819200
}
hack/demo-seed/alerts/worker-job-failures.json.template
+13
View File
@@ -0,0 +1,13 @@
{
"name": "Background job failures",
"description": "Worker jobs are failing after their retries are exhausted",
"query": "service=worker result=failed earliest=-15m | stats count",
"query_language": "spl",
"condition_type": "threshold",
"eval_interval_seconds": 120,
"for_minutes": 0,
"notification_target_id": "__TARGET_PLATFORM__",
"enabled": true,
"comparator": "gt",
"threshold_value": 12
}
hack/demo-seed/cairnobs-demo-simulator.service
+39
View File
@@ -0,0 +1,39 @@
# The live half of the demo deployment: /hack/demo-simulator kept running
# so the demo's agents keep checking in, its hosts keep reporting metrics,
# and its logs keep arriving. -backfill 0 because history is seeded once
# per night by reset-demo.sh, which stops this unit for the duration and
# starts it again afterwards.
#
# Installed on the demo box at /etc/systemd/system/, not built into any
# image -- this is demo scaffolding, not part of the product.
[Unit]
Description=Cairn OBS demo data simulator
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=simple
User=john
WorkingDirectory=/home/john/cairnobs-demo
ExecStart=/home/john/cairnobs-demo/bin/demo-simulator \
-addr 127.0.0.1:4317 \
-ca /home/john/cairnobs-demo/hack/dev-certs/out/ca.pem \
-cert /home/john/cairnobs-demo/hack/dev-certs/out/client.pem \
-key /home/john/cairnobs-demo/hack/dev-certs/out/client-key.pem \
-backfill 0 -live
Restart=always
RestartSec=10
# Same unprivileged posture the real agents' units were moved to during
# the 2026-08-19 security remediation -- a data generator has no business
# with write access to anything on this box.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
[Install]
WantedBy=multi-user.target
hack/demo-seed/dashboards/database-cache.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Database & cache",
"description": "Postgres statement latency and Redis memory pressure",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Slow queries (>1s)",
"query": "service=postgres | where duration_ms>1000 | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Avg statement (ms)",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=postgres | stats avg(duration_ms) as avg_duration_ms",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Redis evictions",
"query": "service=redis op=evict | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Statements by kind",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=postgres | stats count by query_kind | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "query_kind",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Slowest tables (avg ms)",
"query": "service=postgres | stats avg(duration_ms) as avg_duration_ms by table | sort -avg_duration_ms | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "table",
"value_column": "avg_duration_ms"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Statement kind by table",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=postgres | stats count by table, query_kind",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "table",
"y_column": "query_kind",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Redis events by operation",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=redis | stats count by op | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "op",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Slowest statements",
"query": "service=postgres | where duration_ms>1000 | sort -duration_ms | head 50 | fields timestamp, table, query_kind, duration_ms, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/errors-reliability.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Errors & reliability",
"description": "Where failures are concentrated right now -- 5xx, fatal events, and failed background jobs",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Error events",
"query": "severity=ERROR | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Fatal events",
"query": "severity=FATAL | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Failed jobs",
"query": "service=worker result=failed | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Errors by service",
"query": "severity=ERROR | stats count by service | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "service",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Top API error codes",
"query": "service=api | where status>=500 | stats count by error_code | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "error_code",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "5xx by host and status",
"query": "status>=500 | stats count by host, status",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"y_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Failed jobs by queue",
"query": "service=worker result=failed | stats count by queue | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "queue",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Recent 5xx responses",
"query": "status>=500 | sort -timestamp | head 50 | fields timestamp, host, service, status, route, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/infrastructure.json
+112
View File
@@ -0,0 +1,112 @@
{
"name": "Infrastructure",
"description": "Agent-reported CPU, memory, and disk for every host in the fleet",
"default_earliest": "-6h",
"default_latest": "now",
"panels": [
{
"title": "Metric samples",
"query": "cairnobs.metrics=true | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Peak CPU (%)",
"query": "cairnobs.metrics=true | stats max(cpu_percent) as max_cpu_percent",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Agent heartbeats",
"query": "cairnobs.heartbeat=true | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Average CPU by host",
"query": "cairnobs.metrics=true | stats avg(cpu_percent) as avg_cpu_percent by host | sort -avg_cpu_percent",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"value_column": "avg_cpu_percent"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Disk used by host (bytes)",
"query": "cairnobs.metrics=true | stats max(disk_used_bytes) as disk_used_bytes by host | sort -disk_used_bytes | head 12",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "host",
"value_column": "disk_used_bytes"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Memory used by host (bytes)",
"query": "cairnobs.metrics=true | stats max(mem_used_bytes) as mem_used_bytes by host | sort -mem_used_bytes",
"viz_type": "bar",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"value_column": "mem_used_bytes"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Heartbeats by host",
"query": "cairnobs.heartbeat=true | stats count by host | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Latest sample per host",
"query": "cairnobs.metrics=true | stats max(uptime_seconds) as uptime_seconds, avg(cpu_percent) as avg_cpu_percent, max(cpu_cores) as cores by host | sort -avg_cpu_percent",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/mail-delivery.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Mail delivery",
"description": "SMTP delivery outcomes, authentication failures, and spam rejections on the mail host",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Delivered",
"query": "service=smtp result=delivered | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Auth failures",
"query": "service=smtp result=auth_failed | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Spam rejections",
"query": "service=smtp result=spam_reject | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Outcomes",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=smtp | stats count by result | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "result",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Top sender domains",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=smtp | stats count by sender_domain | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "sender_domain",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Outcome by sender domain",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=smtp | stats count by sender_domain, result",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "sender_domain",
"y_column": "result",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Auth failures by source address",
"query": "service=smtp result=auth_failed | stats count by remote_addr | sort -count | head 10",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "remote_addr",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Recent rejections",
"query": "service=smtp result=spam_reject | sort -timestamp | head 50 | fields timestamp, remote_addr, sender_domain, spam_score, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/security.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Security",
"description": "SSH authentication, firewall blocks, and Windows logon failures across the fleet",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Failed SSH logins",
"query": "service=system auth_result=failed | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Firewall blocks",
"query": "service=system ufw_action=BLOCK | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Windows logon failures",
"query": "winevt.event_id=4625 | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Top source addresses",
"query": "service=system auth_result=failed | stats count by remote_addr | sort -count | head 10",
"viz_type": "top_n",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "remote_addr",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Blocked destination ports",
"query": "service=system ufw_action=BLOCK | stats count by dst_port | sort -count | head 12",
"viz_type": "bar",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "dst_port",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Source address by blocked port",
"query": "service=system ufw_action=BLOCK | stats count by remote_addr, dst_port",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "remote_addr",
"y_column": "dst_port",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Failed logins by host",
"query": "service=system auth_result=failed | stats count by host | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Recent authentication failures",
"query": "service=system auth_result=failed | sort -timestamp | head 50 | fields timestamp, host, ssh_user, remote_addr, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/service-overview.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Service overview",
"description": "Every service at a glance: volume, error mix, and the hosts carrying the load",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Log events",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Errors",
"query": "severity=ERROR | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "5xx responses",
"query": "status>=500 | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Events by service",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true | stats count by service | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "service",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Busiest hosts",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true | stats count by host | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "host",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Severity by service",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true | stats count by service, severity",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "service",
"y_column": "severity",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Errors by host",
"query": "severity=ERROR | stats count by host | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Most recent errors",
"query": "severity=ERROR | sort -timestamp | head 50 | fields timestamp, host, service, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/web-api-traffic.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Web & API traffic",
"description": "Edge (nginx) and application (api) tiers: status mix, hot routes, and latency",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Edge requests",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=nginx | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Avg API latency (ms)",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=api | stats avg(latency_ms) as avg_latency_ms",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "API 5xx",
"query": "service=api | where status>=500 | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Edge responses by status",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=nginx | stats count by status | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Busiest routes",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=nginx | stats count by route | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "route",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Status by edge host",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=nginx | stats count by host, status",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"y_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Slowest API routes (avg ms)",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=api | stats avg(latency_ms) as avg_latency_ms by route | sort -avg_latency_ms | head 10",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "route",
"value_column": "avg_latency_ms"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Slow API requests (>800ms)",
"query": "service=api | where latency_ms>800 | sort -timestamp | head 50 | fields timestamp, host, route, status, latency_ms, trace_id",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/dashboards/windows-events.json
+113
View File
@@ -0,0 +1,113 @@
{
"name": "Windows events",
"description": "Security, System, and Application channels from the Windows hosts",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Windows events",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=eventlog | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Failed logons (4625)",
"query": "winevt.event_id=4625 | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Account lockouts (4740)",
"query": "winevt.event_id=4740 | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Events by ID",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=eventlog | stats count by winevt.event_id | sort -count | head 12",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "winevt.event_id",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Top providers",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=eventlog | stats count by winevt.provider | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "winevt.provider",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "Channel by computer",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=eventlog | stats count by winevt.computer, winevt.channel",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "winevt.computer",
"y_column": "winevt.channel",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Severity mix",
"query": "cairnobs.metrics!=true cairnobs.heartbeat!=true service=eventlog | stats count by severity | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "severity",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Recent security-channel events",
"query": "service=eventlog winevt.channel=Security | sort -timestamp | head 50 | fields timestamp, winevt.computer, winevt.event_id, winevt.target_user, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}
hack/demo-seed/reset-demo.sh
Executable
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env bash
# Nightly reset for the demo.cairnobs.org stack: wipes every data volume
# and re-seeds from scratch, so the demo's timestamps stay recent, its
# incidents stay at the same recent offsets, and storage doesn't grow
# forever. Mirrors the teardown docs/phase-0-runbook.md and
# docs/phase-2-runbook.md already document (`docker compose down -v`),
# plus the seed sequence below.
#
# Seeding is data-first, config-second: dashboards and alert rules are
# applied after the backfill exists, so nothing renders an empty panel or
# evaluates against an empty table on its first pass.
#
# The live half of the demo (/hack/demo-simulator running as
# cairnobs-demo-simulator.service) is stopped for the duration and
# started again at the end -- it must not be pushing records into a stack
# that's being torn down, and it must re-register its agents against the
# fresh, empty `agents` table afterwards.
set -euo pipefail
DEMO_ROOT=${DEMO_ROOT:-/home/john/cairnobs-demo}
SEED_DIR="$DEMO_ROOT/hack/demo-seed"
CERTS="$DEMO_ROOT/hack/dev-certs/out"
BACKFILL=${BACKFILL:-168h}
RATE_SCALE=${RATE_SCALE:-0.5}
SIMULATOR_UNIT=cairnobs-demo-simulator.service
cd "$DEMO_ROOT"
ADMIN_PASSWORD_FILE="$DEMO_ROOT/.admin-password"
# Changing DEMO_PASSWORD means rebuilding the web image too: the login
# page prefills this account's credentials, and they're baked into the
# bundle at build time from VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD in the
# demo host's docker-compose.override.yml. Change one without the other
# and the demo's own login form stops working.
DEMO_PASSWORD='CairnDemo_2026!'
EVALUATOR_PASSWORD='REDACTED_ROTATED_CREDENTIAL'
echo "=== $(date -u +%FT%TZ) reset starting ==="
sudo systemctl stop "$SIMULATOR_UNIT" || true
docker compose down -v
docker compose up -d
echo "waiting for api to report healthy..."
until [ "$(docker inspect -f '{{.State.Health.Status}}' cairnobs-api 2>/dev/null)" = "healthy" ]; do sleep 2; done
# -seed-admin is idempotent and prints the password once -- capture it
# fresh each run rather than reusing a stale one from a prior reset.
ADMIN_PASSWORD=$(docker compose run --rm api -seed-admin 2>&1 | grep '^ password:' | awk '{print $2}')
echo "$ADMIN_PASSWORD" > "$ADMIN_PASSWORD_FILE"
chmod 600 "$ADMIN_PASSWORD_FILE"
export CAIRNOBSCTL_API_URL=http://localhost:8080
export CAIRNOBSCTL_ALERTING_API_URL=http://localhost:8081
ADMIN_TOKEN=$(echo "$ADMIN_PASSWORD" | ./bin/cairnobsctl users login admin)
export CAIRNOBSCTL_TOKEN="$ADMIN_TOKEN"
# The password reaches the CLI on stdin only -- `--password <value>` was
# removed deliberately (see cli/cmd/cairnobsctl/cmd_users.go).
echo "$DEMO_PASSWORD" | ./bin/cairnobsctl users create demo --role viewer >/dev/null
echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users create alerting-evaluator --role viewer >/dev/null
SVCTOKEN=$(echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users login alerting-evaluator)
printf 'COMPOSE_PROFILES=single-tenant\nALERTING_SERVICE_TOKEN=%s\n' "$SVCTOKEN" > .env && chmod 600 .env
docker compose up -d alerting
# Three notification targets so the Alerts page shows rules routed to
# different destinations, the way a real deployment splits ops/security/
# platform. The URLs are deliberately inert placeholders on a domain
# reserved for documentation -- nothing is actually notified.
create_target() {
curl -s -X POST http://localhost:8081/targets \
-H "Authorization: Bearer $ADMIN_TOKEN" -H 'Content-Type: application/json' \
-d "{\"name\":\"$1\",\"kind\":\"webhook\",\"webhook_url\":\"$2\"}" \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'
}
TARGET_OPS=$(create_target 'Ops on-call (placeholder)' 'https://example.com/webhooks/ops-oncall')
TARGET_SECURITY=$(create_target 'Security team (placeholder)' 'https://example.com/webhooks/security')
TARGET_PLATFORM=$(create_target 'Platform team (placeholder)' 'https://example.com/webhooks/platform')
echo "backfilling $BACKFILL of synthetic history..."
./bin/demo-simulator \
-addr 127.0.0.1:4317 -ca "$CERTS/ca.pem" -cert "$CERTS/client.pem" -key "$CERTS/client-key.pem" \
-backfill "$BACKFILL" -rate-scale "$RATE_SCALE" -live=false
# A handful of Windows-shaped records from the dedicated fixture as well:
# it's the tool the Windows ingest path is actually verified with, so
# keeping its output present means the demo and that check agree.
docker run --rm --network host -v "$DEMO_ROOT":/src -w /src/hack/windows-fixture -e GOCACHE=/tmp/gocache golang:1.25-bookworm \
go run . --addr 127.0.0.1:4317 --ca /src/hack/dev-certs/out/ca.pem --cert /src/hack/dev-certs/out/client.pem --key /src/hack/dev-certs/out/client-key.pem --count 5
echo "applying dashboards..."
for f in "$SEED_DIR"/dashboards/*.json; do
./bin/cairnobsctl dashboards apply "$f" >/dev/null
echo " $(basename "$f")"
done
echo "applying alert rules..."
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
for f in "$SEED_DIR"/alerts/*.json.template; do
out="$TMP/$(basename "${f%.template}")"
sed -e "s/__TARGET_OPS__/$TARGET_OPS/" \
-e "s/__TARGET_SECURITY__/$TARGET_SECURITY/" \
-e "s/__TARGET_PLATFORM__/$TARGET_PLATFORM/" "$f" > "$out"
./bin/cairnobsctl alerts apply "$out" >/dev/null
echo " $(basename "$out")"
done
sudo systemctl start "$SIMULATOR_UNIT"
echo "=== $(date -u +%FT%TZ) reset complete ==="