From d05ebe046cf8537941855649462971cedba152f7 Mon Sep 17 00:00:00 2001 From: John Coffey Date: Fri, 4 Sep 2026 15:55:43 -0700 Subject: [PATCH] Make the demo reset survive a slow start, and drop the SQL panels Two faults, both found by running the reset against the demo rather than by reading it, and both fixed on the box before this commit existed. The reset raced its own alerting container. Writing .env changes alerting's environment, so `docker compose up -d alerting` recreates it -- and the next line posted notification targets to it with no wait. On a busy box that lost: curl returned nothing, json.load threw on an empty string, and set -e killed the script. The damage is in the ordering: `docker compose down -v` runs near the top, so any failure after it leaves the public demo up, empty, and with the simulator still stopped, because the unit is only restarted on the last line. It has been surviving nightly on timing alone. It now polls /healthz for up to 60 seconds and fails loudly before the seed rather than after the wipe. Five dashboard panels were raw ClickHouse SQL, which dashboards refuse: validatePanel rejects query_language "sql" outright, because the time-range picker is injected as leading query terms and a SELECT has nowhere to put them. They were written that way because the pipe syntax has no time bucketing -- no timechart, no bin -- so a genuine time series is not available to a dashboard panel at all. Each is now the breakdown the panel was actually asking for: kubelet events by kind and host, DNS by type and result, queue depth by queue and host, IIS by site and host, HAProxy by backend and balancer. Both mistakes were mine and both were avoidable by reading: the rule about pipe-syntax-only dashboards is stated in terraform/README.md, and I had already read the line that says it. Verified on the demo: 50 hosts, 24 services, 1,922,128 records, all 13 dashboards and all 11 alert rules applied, simulator active. `system` appears on exactly 31 hosts, which is the Linux count -- no Windows host was given a journald stream. --- hack/demo-seed/dashboards/directory-dns.json | 13 +++++++------ hack/demo-seed/dashboards/edge-proxy.json | 13 +++++++------ .../demo-seed/dashboards/messaging-search.json | 14 +++++++------- .../dashboards/platform-kubernetes.json | 13 +++++++------ .../dashboards/windows-server-estate.json | 13 +++++++------ hack/demo-seed/reset-demo.sh | 18 ++++++++++++++++++ 6 files changed, 53 insertions(+), 31 deletions(-) diff --git a/hack/demo-seed/dashboards/directory-dns.json b/hack/demo-seed/dashboards/directory-dns.json index 9ce69ec..6e787ed 100644 --- a/hack/demo-seed/dashboards/directory-dns.json +++ b/hack/demo-seed/dashboards/directory-dns.json @@ -49,19 +49,20 @@ "sort_order": 3 }, { - "title": "DNS query volume by type", - "query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['dns_type'] AS dns_type, count(*) AS count FROM logs WHERE service = 'bind' GROUP BY bucket, dns_type ORDER BY bucket", - "viz_type": "line", + "title": "DNS queries by type and result", + "query": "service=bind | stats count by dns_type, dns_rcode | sort -count", + "viz_type": "bar", "position_x": 0, "position_y": 3, "width": 12, "height": 5, "viz_config": { - "x_column": "bucket", + "x_column": "dns_type", "value_column": "count", - "series_column": "dns_type" + "series_column": "dns_rcode", + "stacked": "true" }, - "query_language": "sql", + "query_language": "spl", "sort_order": 4 }, { diff --git a/hack/demo-seed/dashboards/edge-proxy.json b/hack/demo-seed/dashboards/edge-proxy.json index 944f4d2..f0071b5 100644 --- a/hack/demo-seed/dashboards/edge-proxy.json +++ b/hack/demo-seed/dashboards/edge-proxy.json @@ -49,19 +49,20 @@ "sort_order": 3 }, { - "title": "Requests per backend over time", - "query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['backend'] AS backend, count(*) AS count FROM logs WHERE service = 'haproxy' GROUP BY bucket, backend ORDER BY bucket", - "viz_type": "line", + "title": "Requests per backend, by balancer", + "query": "service=haproxy | stats count by backend, host | sort -count", + "viz_type": "bar", "position_x": 0, "position_y": 3, "width": 12, "height": 5, "viz_config": { - "x_column": "bucket", + "x_column": "backend", "value_column": "count", - "series_column": "backend" + "series_column": "host", + "stacked": "true" }, - "query_language": "sql", + "query_language": "spl", "sort_order": 4 }, { diff --git a/hack/demo-seed/dashboards/messaging-search.json b/hack/demo-seed/dashboards/messaging-search.json index 47c3d5d..4b0af6a 100644 --- a/hack/demo-seed/dashboards/messaging-search.json +++ b/hack/demo-seed/dashboards/messaging-search.json @@ -49,19 +49,19 @@ "sort_order": 3 }, { - "title": "Queue depth over time", - "query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['queue'] AS queue, max(toFloat64OrZero(attributes['queue_depth'])) AS depth FROM logs WHERE service = 'rabbitmq' GROUP BY bucket, queue ORDER BY bucket", - "viz_type": "line", + "title": "Queue depth by queue and host", + "query": "service=rabbitmq | stats max(queue_depth) as max_depth by queue, host | sort -max_depth", + "viz_type": "bar", "position_x": 0, "position_y": 3, "width": 12, "height": 5, "viz_config": { - "x_column": "bucket", - "value_column": "depth", - "series_column": "queue" + "x_column": "queue", + "value_column": "max_depth", + "series_column": "host" }, - "query_language": "sql", + "query_language": "spl", "sort_order": 4 }, { diff --git a/hack/demo-seed/dashboards/platform-kubernetes.json b/hack/demo-seed/dashboards/platform-kubernetes.json index 677ca2d..7fe7b74 100644 --- a/hack/demo-seed/dashboards/platform-kubernetes.json +++ b/hack/demo-seed/dashboards/platform-kubernetes.json @@ -49,19 +49,20 @@ "sort_order": 3 }, { - "title": "Kubelet events over time", - "query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['event_kind'] AS event_kind, count(*) AS count FROM logs WHERE service = 'kubelet' GROUP BY bucket, event_kind ORDER BY bucket", - "viz_type": "line", + "title": "Kubelet events by kind", + "query": "service=kubelet | stats count by event_kind, host | sort -count", + "viz_type": "bar", "position_x": 0, "position_y": 3, "width": 12, "height": 5, "viz_config": { - "x_column": "bucket", + "x_column": "host", "value_column": "count", - "series_column": "event_kind" + "series_column": "event_kind", + "stacked": "true" }, - "query_language": "sql", + "query_language": "spl", "sort_order": 4 }, { diff --git a/hack/demo-seed/dashboards/windows-server-estate.json b/hack/demo-seed/dashboards/windows-server-estate.json index 662847a..97a5c1c 100644 --- a/hack/demo-seed/dashboards/windows-server-estate.json +++ b/hack/demo-seed/dashboards/windows-server-estate.json @@ -49,19 +49,20 @@ "sort_order": 3 }, { - "title": "IIS traffic by site", - "query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['site'] AS site, count(*) AS count FROM logs WHERE service = 'iis' GROUP BY bucket, site ORDER BY bucket", - "viz_type": "line", + "title": "IIS requests by site and host", + "query": "service=iis | stats count by site, host | sort -count", + "viz_type": "bar", "position_x": 0, "position_y": 3, "width": 12, "height": 5, "viz_config": { - "x_column": "bucket", + "x_column": "site", "value_column": "count", - "series_column": "site" + "series_column": "host", + "stacked": "true" }, - "query_language": "sql", + "query_language": "spl", "sort_order": 4 }, { diff --git a/hack/demo-seed/reset-demo.sh b/hack/demo-seed/reset-demo.sh index b34af0e..49e2236 100755 --- a/hack/demo-seed/reset-demo.sh +++ b/hack/demo-seed/reset-demo.sh @@ -77,6 +77,24 @@ SVCTOKEN=$(echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users login alerting-e printf 'COMPOSE_PROFILES=single-tenant\nALERTING_SERVICE_TOKEN=%s\n' "$SVCTOKEN" > .env && chmod 600 .env docker compose up -d alerting +# Wait for it to answer before posting to it. Writing the .env above +# changes alerting's environment, so `up -d` *recreates* the container +# rather than leaving it running -- and the next line used to curl it +# immediately. On a busy box that lost the race: curl returned nothing, +# `json.load` got an empty string, and the reset died after `down -v` had +# already wiped everything, leaving the public demo up, empty, and with +# the simulator still stopped because the unit only restarts at the very +# end. Cheap to wait; expensive not to. +echo "waiting for alerting to answer..." +for i in $(seq 1 60); do + if curl -sf -o /dev/null http://localhost:8081/healthz; then break; fi + if [ "$i" -eq 60 ]; then + echo "!! alerting did not become ready within 60s -- stopping before the seed" >&2 + exit 1 + fi + sleep 1 +done + # Three notification targets so the Alerts page shows rules routed to # different destinations, the way a real deployment splits ops/security/ # platform. The URLs are deliberately inert placeholders on a domain