coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge pull request #23 from Coffey-Labs/feat/processing-conformance-suite

Browse Source 
Build the Phase 8 conformance corpus
This commit is contained in:
Coffey Labs
2026-09-04 20:53:43 -07:00
committed by GitHub
parent 8787c1d087 7fabc6a067
commit bd0d71c45b
43 changed files with 2360 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/processing-conformance.yml
+29
View File
@@ -0,0 +1,29 @@
name: Processing conformance corpus
# /processing/conformance/cases IS the Phase 8 rule language spec: two
# implementations (Rust in the agent, Go at ingest) will each be held to
# it, and two hand-written implementations of one language diverge
# unless something shared pins them.
#
# Neither implementation exists yet, so nothing here executes rules.
# What this checks is that the corpus stays well-formed in the meantime
# -- known actions, addressable fields, compilable patterns, no case
# quietly depending on record_id. Cheap insurance against the spec
# rotting during the gap between writing it and building against it.
#
# Its own workflow, following web-routes.yml's reasoning: license-
# compliance.yml already carries one unrelated check for historical
# reasons, and that is not worth compounding.
on:
push:
branches: [master, main]
pull_request:
jobs:
conformance-corpus:
name: processing conformance corpus check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: python3 processing/conformance/validate.py
README.md
+1
View File
@@ -79,6 +79,7 @@ metadata/ PostgreSQL schema and migrations
web/ SvelteKit frontend
cli/ cairnobsctl
proto/ gRPC service definitions
processing/ Phase 8 rule language spec and conformance corpus (no implementation yet)
terraform/ Terraform provider
enterprise/ SSO, multi-tenancy, per-tenant provisioning (off the roadmap)
deploy/ Helm charts and Kubernetes operator
docs/phase-8-processing-design.md
+12 -3
View File
@@ -153,15 +153,24 @@ the same thing in both places" is the whole promise, and two
hand-written implementations will diverge — not maybe, eventually.
The deliverable that prevents it is a **language-neutral conformance
suite**: a directory of cases, each an input record, a rule set, and the
expected output record, in JSON. Both implementations run it in their own
CI. A case is added for every bug found in either.
suite**: a directory of cases, each a rule set, a sequence of input
records, and the records expected out, in JSON. Both implementations run
it in their own CI. A case is added for every bug found in either.
This is the same discipline `/hack`'s fixtures already apply to ingest
shapes, applied to semantics instead. It should be built **first**, not
last — the suite is the specification, and the prose above is a summary
of it.
**Built:** [`/processing`](../processing/README.md), 38 cases. Nothing
executes them yet, since neither implementation exists; a structural
validator runs in CI so the corpus cannot rot in the meantime. Writing
the cases first has already paid for itself — it forced two determinism
decisions that prose had left vague (see that README's "Two determinism
decisions the suite forces"), and it made the absence of an
`aggregate_count` answer concrete enough that the validator rejects any
case using it.
## Decision 4: distribution reuses the channel that exists
Fleet management already delivers desired state
processing/README.md
+150
View File
@@ -0,0 +1,150 @@
# processing
The Phase 8 processing rule language: its specification, and the
conformance suite that *is* that specification.
> **Status:** the suite exists; the implementations do not. Nothing in
> Phase 8 is built. See
> [`/docs/phase-8-processing-design.md`](../docs/phase-8-processing-design.md)
> for the design and its open questions.
## Why this is a top-level directory
Rules run in two places, in two languages: the Rust agent
(`/agent`) and the Go ingest tier (`/ingest`). Neither owns the
definition. This is the same shape as `/proto` — a language-neutral
contract that several modules consume and none of them is the source of
truth for.
## Why the cases are the specification
Two hand-written implementations of one language will diverge. Not
maybe, eventually. The only thing that reliably prevents it is a shared
corpus both must satisfy, written down before either exists.
So the prose below is a summary of `conformance/cases/`, not the other
way round. When they disagree, the cases win, and the prose is the bug.
## Running it
Nothing executes the cases yet, because neither implementation exists.
What runs today is a structural check that every case is well-formed —
valid JSON, known actions, no unknown keys, declared fields that exist:
```sh
python3 processing/conformance/validate.py
```
That is worth having on its own: it stops the corpus rotting between now
and the first implementation, and it fails loudly if someone adds a case
using an action the spec does not define.
Each implementation is expected to add a test that walks
`conformance/cases/*.json`, feeds `inputs` through `rules`, and asserts
the emitted records equal `expect`. Those runners are part of building
each side, not part of this directory.
## Case format
One JSON object per file:
```json
{
"name": "drop_discards_the_record",
"description": "A matched drop emits nothing at all.",
"rules": [
{
"match": [{"field": "message", "op": "eq", "value": "noise"}],
"actions": [{"action": "drop"}]
}
],
"inputs": [
{"timestamp_unix_nano": 1000, "host": "h1", "service": "s1",
"severity": "SEVERITY_INFO", "message": "noise", "attributes": {}}
],
"expect": []
}
```
`inputs` is always a list, even for a single record, because windowed
actions need a sequence. `expect` is the records emitted, in order.
### Records
Mirrors `LogRecord` in `/proto/sentry/logs/v1/logs.proto`, snake_case,
with `severity` as the enum's name string.
`record_id` is deliberately absent from every case. The agent never sets
it, and whether an ingest-side rule can see one depends on where in the
ingest path rules run — an open question. No case may depend on it, so
that decision stays free.
### Matching
A rule matches when **every** clause in `match` holds. An empty `match`
matches every record.
| `op` | Meaning |
|---|---|
| `eq` / `ne` | exact string equality |
| `contains` | substring |
| `prefix` / `suffix` | string boundary |
| `regex` | linear-time regex, unanchored |
| `exists` / `not_exists` | field present (no `value`) |
Addressable fields: `message`, `host`, `service`, `severity`, and
`attributes.<key>`. A field that does not exist compares as absent, not
as empty string — `eq ""` does not match a missing attribute.
### Actions
Applied in order. A `drop` ends processing for that record immediately;
no later action or rule runs.
| Action | Parameters |
|---|---|
| `drop` | — |
| `drop_fields` | `fields` |
| `keep_fields` | `fields` (attributes only; never removes top-level fields) |
| `rename` | `from`, `to` |
| `derive` | `field`, and one of `value` / `from_field` |
| `mask` | `field`, `pattern`, `replacement` |
| `parse_json` | `field` (default `message`), optional `prefix` |
| `parse_regex` | `field`, `pattern` (named captures become attributes) |
| `sample` | `keep_one_in` |
| `suppress_duplicates` | `window_ms`, optional `key_fields` |
Rules are evaluated in the order given. Every matching rule's actions
apply, to the record as left by the rule before it.
### Two determinism decisions the suite forces
Conformance testing cannot assert on nondeterminism, so two things that
are usually left vague are pinned here:
**`sample` is counter-based, not random.** `keep_one_in: 3` keeps the
first record and every third thereafter, per rule, per process. Random
sampling is statistically nicer and untestable; a counter is testable and
close enough at volume.
**Windows are measured on record timestamps, not wall-clock.** A
`suppress_duplicates` window of 5000ms compares
`timestamp_unix_nano` values, so replaying the same input always gives
the same output regardless of how fast the test runs. This also means the
behaviour is correct under backfill, which wall-clock windows are not.
### `aggregate_count` has no cases, deliberately
The design lists it as an action but does not answer what it emits, or
what a query that is not expecting a synthetic record sees. Writing cases
now would invent that answer by accident and freeze it. It stays
unspecified until that question is decided —
[`phase-8-processing-design.md`](../docs/phase-8-processing-design.md)
open question 5.
## Adding a case
One behaviour per file, named for the behaviour rather than the action.
A case is added for every bug found in either implementation — that is
the mechanism that keeps the two from drifting, and it only works if the
case lands with the fix.
processing/conformance/cases/actions_apply_in_order.json
+43
View File
@@ -0,0 +1,43 @@
{
"name": "actions_apply_in_order",
"description": "The second action sees what the first produced: parse then drop one of the parsed fields.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_json",
"field": "message"
},
{
"action": "drop_fields",
"fields": [
"attributes.secret"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"keep\":\"k\",\"secret\":\"s\"}",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"keep\":\"k\",\"secret\":\"s\"}",
"attributes": {
"keep": "k"
}
}
]
}
processing/conformance/cases/derive_copies_from_another_field.json
+38
View File
@@ -0,0 +1,38 @@
{
"name": "derive_copies_from_another_field",
"description": "derive with from_field copies, leaving the source in place.",
"rules": [
{
"match": [],
"actions": [
{
"action": "derive",
"field": "attributes.origin_host",
"from_field": "host"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "web-01",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "web-01",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"origin_host": "web-01"
}
}
]
}
processing/conformance/cases/derive_overwrites_an_existing_value.json
+40
View File
@@ -0,0 +1,40 @@
{
"name": "derive_overwrites_an_existing_value",
"description": "derive is a set, not an insert-if-absent.",
"rules": [
{
"match": [],
"actions": [
{
"action": "derive",
"field": "attributes.env",
"value": "dev"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"env": "prod"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"env": "dev"
}
}
]
}
processing/conformance/cases/derive_sets_a_literal_value.json
+38
View File
@@ -0,0 +1,38 @@
{
"name": "derive_sets_a_literal_value",
"description": "derive with value writes a constant.",
"rules": [
{
"match": [],
"actions": [
{
"action": "derive",
"field": "attributes.env",
"value": "dev"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"env": "dev"
}
}
]
}
processing/conformance/cases/drop_fields_on_absent_attribute_is_not_an_error.json
+41
View File
@@ -0,0 +1,41 @@
{
"name": "drop_fields_on_absent_attribute_is_not_an_error",
"description": "Removing something that is not there succeeds and changes nothing.",
"rules": [
{
"match": [],
"actions": [
{
"action": "drop_fields",
"fields": [
"attributes.nope"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"keep": "k"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"keep": "k"
}
}
]
}
processing/conformance/cases/drop_fields_removes_only_named_attributes.json
+44
View File
@@ -0,0 +1,44 @@
{
"name": "drop_fields_removes_only_named_attributes",
"description": "drop_fields removes the named attributes and leaves the rest alone.",
"rules": [
{
"match": [],
"actions": [
{
"action": "drop_fields",
"fields": [
"attributes.secret",
"attributes.gone"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"secret": "s",
"gone": "g",
"keep": "k"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"keep": "k"
}
}
]
}
processing/conformance/cases/drop_stops_later_actions_in_the_same_rule.json
+30
View File
@@ -0,0 +1,30 @@
{
"name": "drop_stops_later_actions_in_the_same_rule",
"description": "Nothing after a drop runs; the record is gone, so a subsequent derive cannot resurrect it.",
"rules": [
{
"match": [],
"actions": [
{
"action": "drop"
},
{
"action": "derive",
"field": "attributes.late",
"value": "x"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
}
],
"expect": []
}
processing/conformance/cases/drop_stops_later_rules.json
+60
View File
@@ -0,0 +1,60 @@
{
"name": "drop_stops_later_rules",
"description": "A drop in rule 1 means rule 2 never sees the record.",
"rules": [
{
"match": [
{
"field": "message",
"op": "eq",
"value": "gone"
}
],
"actions": [
{
"action": "drop"
}
]
},
{
"match": [],
"actions": [
{
"action": "derive",
"field": "attributes.seen",
"value": "yes"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "gone",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "stays",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "stays",
"attributes": {
"seen": "yes"
}
}
]
}
processing/conformance/cases/empty_rule_set_is_a_pass_through.json
+45
View File
@@ -0,0 +1,45 @@
{
"name": "empty_rule_set_is_a_pass_through",
"description": "No rules means every record is emitted exactly as it arrived. This is the state an agent falls back to when it refuses a rule set.",
"rules": [],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"k": "v"
}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"k": "v"
}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
]
}
processing/conformance/cases/keep_fields_is_an_attribute_allowlist.json
+43
View File
@@ -0,0 +1,43 @@
{
"name": "keep_fields_is_an_attribute_allowlist",
"description": "keep_fields drops every attribute not named, and never touches top-level fields.",
"rules": [
{
"match": [],
"actions": [
{
"action": "keep_fields",
"fields": [
"attributes.keep"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h9",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "message survives",
"attributes": {
"keep": "k",
"a": "1",
"b": "2"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h9",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "message survives",
"attributes": {
"keep": "k"
}
}
]
}
processing/conformance/cases/mask_leaves_non_matching_records_untouched.json
+37
View File
@@ -0,0 +1,37 @@
{
"name": "mask_leaves_non_matching_records_untouched",
"description": "A mask whose pattern does not hit changes nothing at all.",
"rules": [
{
"match": [],
"actions": [
{
"action": "mask",
"field": "message",
"pattern": "[0-9]{16}",
"replacement": "[REDACTED]"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "nothing sensitive here",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "nothing sensitive here",
"attributes": {}
}
]
}
processing/conformance/cases/mask_replaces_every_occurrence.json
+37
View File
@@ -0,0 +1,37 @@
{
"name": "mask_replaces_every_occurrence",
"description": "mask replaces all matches in the field, not just the first -- redaction that stops at the first hit is a leak.",
"rules": [
{
"match": [],
"actions": [
{
"action": "mask",
"field": "message",
"pattern": "[0-9]{16}",
"replacement": "[REDACTED]"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "card 4111111111111111 and 5500000000000004 seen",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "card [REDACTED] and [REDACTED] seen",
"attributes": {}
}
]
}
processing/conformance/cases/match_absent_attribute_is_not_empty_string.json
+50
View File
@@ -0,0 +1,50 @@
{
"name": "match_absent_attribute_is_not_empty_string",
"description": "A missing attribute must not compare equal to \"\" -- absence and emptiness are different.",
"rules": [
{
"match": [
{
"field": "attributes.tier",
"op": "eq",
"value": ""
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "missing",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "empty",
"attributes": {
"tier": ""
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "missing",
"attributes": {}
}
]
}
processing/conformance/cases/match_all_clauses_must_hold.json
+69
View File
@@ -0,0 +1,69 @@
{
"name": "match_all_clauses_must_hold",
"description": "Multiple clauses are AND, never OR.",
"rules": [
{
"match": [
{
"field": "host",
"op": "eq",
"value": "h1"
},
{
"field": "severity",
"op": "eq",
"value": "SEVERITY_ERROR"
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_ERROR",
"message": "both",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "host only",
"attributes": {}
},
{
"timestamp_unix_nano": 3000,
"host": "h2",
"service": "s1",
"severity": "SEVERITY_ERROR",
"message": "sev only",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "host only",
"attributes": {}
},
{
"timestamp_unix_nano": 3000,
"host": "h2",
"service": "s1",
"severity": "SEVERITY_ERROR",
"message": "sev only",
"attributes": {}
}
]
}
processing/conformance/cases/match_contains_prefix_suffix.json
+112
View File
@@ -0,0 +1,112 @@
{
"name": "match_contains_prefix_suffix",
"description": "The three substring operators, each matching exactly one of three records.",
"rules": [
{
"match": [
{
"field": "message",
"op": "prefix",
"value": "start"
}
],
"actions": [
{
"action": "derive",
"field": "attributes.hit",
"value": "prefix"
}
]
},
{
"match": [
{
"field": "message",
"op": "suffix",
"value": "end"
}
],
"actions": [
{
"action": "derive",
"field": "attributes.hit",
"value": "suffix"
}
]
},
{
"match": [
{
"field": "message",
"op": "contains",
"value": "middle"
}
],
"actions": [
{
"action": "derive",
"field": "attributes.hit",
"value": "contains"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "start of line",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "line to end",
"attributes": {}
},
{
"timestamp_unix_nano": 3000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a middle b",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "start of line",
"attributes": {
"hit": "prefix"
}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "line to end",
"attributes": {
"hit": "suffix"
}
},
{
"timestamp_unix_nano": 3000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a middle b",
"attributes": {
"hit": "contains"
}
}
]
}
processing/conformance/cases/match_empty_matcher_matches_everything.json
+33
View File
@@ -0,0 +1,33 @@
{
"name": "match_empty_matcher_matches_everything",
"description": "An empty match list is 'always', not 'never'.",
"rules": [
{
"match": [],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "anything",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "anything else",
"attributes": {}
}
],
"expect": []
}
processing/conformance/cases/match_eq_is_exact_not_substring.json
+48
View File
@@ -0,0 +1,48 @@
{
"name": "match_eq_is_exact_not_substring",
"description": "eq compares the whole value; a superstring does not match.",
"rules": [
{
"match": [
{
"field": "message",
"op": "eq",
"value": "noise"
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "noise",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "noise and more",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "noise and more",
"attributes": {}
}
]
}
processing/conformance/cases/match_exists_on_attribute.json
+62
View File
@@ -0,0 +1,62 @@
{
"name": "match_exists_on_attribute",
"description": "exists/not_exists test presence and take no value.",
"rules": [
{
"match": [
{
"field": "attributes.trace_id",
"op": "exists"
}
],
"actions": [
{
"action": "derive",
"field": "attributes.traced",
"value": "yes"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"trace_id": "abc"
}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"trace_id": "abc",
"traced": "yes"
}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
]
}
processing/conformance/cases/match_ne_matches_when_different.json
+48
View File
@@ -0,0 +1,48 @@
{
"name": "match_ne_matches_when_different",
"description": "ne is the complement of eq, including for absent fields.",
"rules": [
{
"match": [
{
"field": "service",
"op": "ne",
"value": "keep-me"
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "keep-me",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "other",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "keep-me",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
}
]
}
processing/conformance/cases/match_on_severity_by_enum_name.json
+48
View File
@@ -0,0 +1,48 @@
{
"name": "match_on_severity_by_enum_name",
"description": "severity is matched by its enum name string, not its number.",
"rules": [
{
"match": [
{
"field": "severity",
"op": "eq",
"value": "SEVERITY_DEBUG"
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_DEBUG",
"message": "debug",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "info",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "info",
"attributes": {}
}
]
}
processing/conformance/cases/match_regex_is_unanchored.json
+56
View File
@@ -0,0 +1,56 @@
{
"name": "match_regex_is_unanchored",
"description": "regex matches anywhere in the value unless the pattern anchors itself.",
"rules": [
{
"match": [
{
"field": "message",
"op": "regex",
"value": "conn(ect|ected)"
}
],
"actions": [
{
"action": "drop"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "client connected ok",
"attributes": {}
},
{
"timestamp_unix_nano": 2000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "disconnected",
"attributes": {}
},
{
"timestamp_unix_nano": 3000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "unrelated",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 3000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "unrelated",
"attributes": {}
}
]
}
processing/conformance/cases/non_matching_rule_leaves_the_record_untouched.json
+47
View File
@@ -0,0 +1,47 @@
{
"name": "non_matching_rule_leaves_the_record_untouched",
"description": "A rule that does not match is not a rule that empties things.",
"rules": [
{
"match": [
{
"field": "host",
"op": "eq",
"value": "other"
}
],
"actions": [
{
"action": "drop_fields",
"fields": [
"attributes.keep"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"keep": "k"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"keep": "k"
}
}
]
}
processing/conformance/cases/parse_json_lifts_top_level_keys_to_attributes.json
+38
View File
@@ -0,0 +1,38 @@
{
"name": "parse_json_lifts_top_level_keys_to_attributes",
"description": "parse_json on message adds string attributes and leaves message intact.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_json",
"field": "message"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"level\":\"warn\",\"code\":\"503\"}",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"level\":\"warn\",\"code\":\"503\"}",
"attributes": {
"level": "warn",
"code": "503"
}
}
]
}
processing/conformance/cases/parse_json_on_invalid_json_is_a_no_op.json
+35
View File
@@ -0,0 +1,35 @@
{
"name": "parse_json_on_invalid_json_is_a_no_op",
"description": "A record that is not JSON passes through unchanged rather than being dropped or erroring -- same schema-on-read fallback the agent parser already uses.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_json",
"field": "message"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "this is not json {",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "this is not json {",
"attributes": {}
}
]
}
processing/conformance/cases/parse_json_prefixes_keys_when_asked.json
+41
View File
@@ -0,0 +1,41 @@
{
"name": "parse_json_prefixes_keys_when_asked",
"description": "prefix namespaces the lifted keys so they cannot collide with existing attributes.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_json",
"field": "message",
"prefix": "json."
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"code\":\"503\"}",
"attributes": {
"code": "keep-me"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "{\"code\":\"503\"}",
"attributes": {
"code": "keep-me",
"json.code": "503"
}
}
]
}
processing/conformance/cases/parse_regex_named_captures_become_attributes.json
+39
View File
@@ -0,0 +1,39 @@
{
"name": "parse_regex_named_captures_become_attributes",
"description": "Only named groups are lifted; unnamed groups are ignored.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_regex",
"field": "message",
"pattern": "user=(?P<user>[a-z]+) code=(?P<code>[0-9]+)"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "user=alice code=403 extra",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "user=alice code=403 extra",
"attributes": {
"user": "alice",
"code": "403"
}
}
]
}
processing/conformance/cases/parse_regex_that_does_not_match_is_a_no_op.json
+36
View File
@@ -0,0 +1,36 @@
{
"name": "parse_regex_that_does_not_match_is_a_no_op",
"description": "A non-matching pattern adds nothing and does not drop the record.",
"rules": [
{
"match": [],
"actions": [
{
"action": "parse_regex",
"field": "message",
"pattern": "user=(?P<user>[a-z]+)"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "no user here",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "no user here",
"attributes": {}
}
]
}
processing/conformance/cases/rename_from_absent_field_is_a_no_op.json
+40
View File
@@ -0,0 +1,40 @@
{
"name": "rename_from_absent_field_is_a_no_op",
"description": "Renaming something absent does not create an empty destination.",
"rules": [
{
"match": [],
"actions": [
{
"action": "rename",
"from": "attributes.old",
"to": "attributes.new"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"other": "v"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"other": "v"
}
}
]
}
processing/conformance/cases/rename_moves_an_attribute.json
+40
View File
@@ -0,0 +1,40 @@
{
"name": "rename_moves_an_attribute",
"description": "rename removes the source key and creates the destination.",
"rules": [
{
"match": [],
"actions": [
{
"action": "rename",
"from": "attributes.old",
"to": "attributes.new"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"old": "v"
}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {
"new": "v"
}
}
]
}
processing/conformance/cases/rules_apply_in_sequence_to_the_same_record.json
+55
View File
@@ -0,0 +1,55 @@
{
"name": "rules_apply_in_sequence_to_the_same_record",
"description": "Every matching rule runs, each seeing the record as the previous rule left it.",
"rules": [
{
"match": [],
"actions": [
{
"action": "derive",
"field": "attributes.a",
"value": "1"
}
]
},
{
"match": [
{
"field": "attributes.a",
"op": "eq",
"value": "1"
}
],
"actions": [
{
"action": "derive",
"field": "attributes.b",
"value": "2"
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "x",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "x",
"attributes": {
"a": "1",
"b": "2"
}
}
]
}
processing/conformance/cases/sample_counts_only_matching_records.json
+81
View File
@@ -0,0 +1,81 @@
{
"name": "sample_counts_only_matching_records",
"description": "The counter advances on records the rule matched, so an unrelated record between two matches does not consume a slot.",
"rules": [
{
"match": [
{
"field": "service",
"op": "eq",
"value": "noisy"
}
],
"actions": [
{
"action": "sample",
"keep_one_in": 2
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "noisy",
"severity": "SEVERITY_INFO",
"message": "n1",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "quiet",
"severity": "SEVERITY_INFO",
"message": "other",
"attributes": {}
},
{
"timestamp_unix_nano": 3000000,
"host": "h1",
"service": "noisy",
"severity": "SEVERITY_INFO",
"message": "n2",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000,
"host": "h1",
"service": "noisy",
"severity": "SEVERITY_INFO",
"message": "n3",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "noisy",
"severity": "SEVERITY_INFO",
"message": "n1",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "quiet",
"severity": "SEVERITY_INFO",
"message": "other",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000,
"host": "h1",
"service": "noisy",
"severity": "SEVERITY_INFO",
"message": "n3",
"attributes": {}
}
]
}
processing/conformance/cases/sample_keeps_the_first_then_every_nth.json
+99
View File
@@ -0,0 +1,99 @@
{
"name": "sample_keeps_the_first_then_every_nth",
"description": "sample is counter-based, not random: keep_one_in 3 keeps records 1, 4, 7. Random sampling is untestable; a counter is testable and close enough at volume.",
"rules": [
{
"match": [],
"actions": [
{
"action": "sample",
"keep_one_in": 3
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m1",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m2",
"attributes": {}
},
{
"timestamp_unix_nano": 3000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m3",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m4",
"attributes": {}
},
{
"timestamp_unix_nano": 5000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m5",
"attributes": {}
},
{
"timestamp_unix_nano": 6000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m6",
"attributes": {}
},
{
"timestamp_unix_nano": 7000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m7",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m1",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m4",
"attributes": {}
},
{
"timestamp_unix_nano": 7000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "m7",
"attributes": {}
}
]
}
processing/conformance/cases/sample_of_one_keeps_everything.json
+51
View File
@@ -0,0 +1,51 @@
{
"name": "sample_of_one_keeps_everything",
"description": "keep_one_in 1 is a no-op, not an off switch.",
"rules": [
{
"match": [],
"actions": [
{
"action": "sample",
"keep_one_in": 1
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
]
}
processing/conformance/cases/suppress_duplicates_collapses_within_the_window.json
+51
View File
@@ -0,0 +1,51 @@
{
"name": "suppress_duplicates_collapses_within_the_window",
"description": "Identical messages inside the window collapse to the first; the window is measured on record timestamps, not wall-clock, so replay is deterministic and backfill behaves.",
"rules": [
{
"match": [],
"actions": [
{
"action": "suppress_duplicates",
"window_ms": 5000
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
},
{
"timestamp_unix_nano": 4999000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
}
]
}
processing/conformance/cases/suppress_duplicates_emits_again_after_the_window.json
+51
View File
@@ -0,0 +1,51 @@
{
"name": "suppress_duplicates_emits_again_after_the_window",
"description": "A duplicate outside the window is a new first occurrence.",
"rules": [
{
"match": [],
"actions": [
{
"action": "suppress_duplicates",
"window_ms": 5000
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
},
{
"timestamp_unix_nano": 5000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
},
{
"timestamp_unix_nano": 5000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
}
]
}
processing/conformance/cases/suppress_duplicates_honours_key_fields.json
+63
View File
@@ -0,0 +1,63 @@
{
"name": "suppress_duplicates_honours_key_fields",
"description": "With key_fields, identical messages from different hosts are not duplicates of each other.",
"rules": [
{
"match": [],
"actions": [
{
"action": "suppress_duplicates",
"window_ms": 5000,
"key_fields": [
"host",
"message"
]
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000,
"host": "h2",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000,
"host": "h2",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "same",
"attributes": {}
}
]
}
processing/conformance/cases/suppress_duplicates_keys_on_message_by_default.json
+59
View File
@@ -0,0 +1,59 @@
{
"name": "suppress_duplicates_keys_on_message_by_default",
"description": "Two different messages never suppress each other.",
"rules": [
{
"match": [],
"actions": [
{
"action": "suppress_duplicates",
"window_ms": 5000
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "a",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "b",
"attributes": {}
}
]
}
processing/conformance/cases/workstation_noise_reduction.json
+103
View File
@@ -0,0 +1,103 @@
{
"name": "workstation_noise_reduction",
"description": "The v1 acceptance shape, from real measured data: Discord repeating a byte-identical version string and a polling daemon repeating a fixed message, against one kernel message worth keeping. See /docs/phase-8-processing-design.md.",
"rules": [
{
"match": [
{
"field": "message",
"op": "eq",
"value": "Discord 1.0.155"
}
],
"actions": [
{
"action": "drop"
}
]
},
{
"match": [
{
"field": "message",
"op": "eq",
"value": "Checking active tasks"
}
],
"actions": [
{
"action": "suppress_duplicates",
"window_ms": 60000
}
]
}
],
"inputs": [
{
"timestamp_unix_nano": 0,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
},
{
"timestamp_unix_nano": 1000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Checking active tasks",
"attributes": {}
},
{
"timestamp_unix_nano": 2000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
},
{
"timestamp_unix_nano": 3000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Checking active tasks",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_WARN",
"message": "[UFW BLOCK] IN=wlan0 SRC=172.16.16.1",
"attributes": {}
},
{
"timestamp_unix_nano": 5000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Discord 1.0.155",
"attributes": {}
}
],
"expect": [
{
"timestamp_unix_nano": 1000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_INFO",
"message": "Checking active tasks",
"attributes": {}
},
{
"timestamp_unix_nano": 4000000000,
"host": "h1",
"service": "s1",
"severity": "SEVERITY_WARN",
"message": "[UFW BLOCK] IN=wlan0 SRC=172.16.16.1",
"attributes": {}
}
]
}
processing/conformance/validate.py
Executable
+217
View File
@@ -0,0 +1,217 @@
#!/usr/bin/env python3
"""Structural check for the processing conformance corpus.
The cases are the specification (see ../README.md), so they have to stay
well-formed even while neither implementation exists to execute them.
This validates shape only -- it does not run rules against records,
because there is nothing yet to run them with.
Stdlib only, on purpose: this must be runnable anywhere without a
dependency install, including from a bare CI container.
Exit 0 if every case is valid, 1 otherwise.
"""
import json
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
CASES = os.path.join(HERE, "cases")
CASE_KEYS = {"name", "description", "rules", "inputs", "expect"}
RECORD_KEYS = {"timestamp_unix_nano", "host", "service", "severity",
"message", "attributes"}
SEVERITIES = {"SEVERITY_UNSPECIFIED", "SEVERITY_TRACE", "SEVERITY_DEBUG",
"SEVERITY_INFO", "SEVERITY_WARN", "SEVERITY_ERROR",
"SEVERITY_FATAL"}
OPS_WITH_VALUE = {"eq", "ne", "contains", "prefix", "suffix", "regex"}
OPS_WITHOUT_VALUE = {"exists", "not_exists"}
TOP_LEVEL_FIELDS = {"message", "host", "service", "severity"}
# action -> (required params, optional params)
ACTIONS = {
"drop": (set(), set()),
"drop_fields": ({"fields"}, set()),
"keep_fields": ({"fields"}, set()),
"rename": ({"from", "to"}, set()),
"derive": ({"field"}, {"value", "from_field"}),
"mask": ({"field", "pattern", "replacement"}, set()),
"parse_json": ({"field"}, {"prefix"}),
"parse_regex": ({"field", "pattern"}, set()),
"sample": ({"keep_one_in"}, set()),
"suppress_duplicates": ({"window_ms"}, {"key_fields"}),
}
# Deliberately absent from ACTIONS. The design does not say what it emits
# or what a query that is not expecting a synthetic record sees, so a
# case using it would invent that answer and freeze it by accident.
UNSPECIFIED_ACTIONS = {"aggregate_count"}
def valid_field(name):
return name in TOP_LEVEL_FIELDS or name.startswith("attributes.")
def check_record(rec, where, err):
if not isinstance(rec, dict):
return err(f"{where}: record must be an object")
missing = RECORD_KEYS - set(rec)
unknown = set(rec) - RECORD_KEYS
if missing:
err(f"{where}: record missing {sorted(missing)}")
if "record_id" in unknown:
err(f"{where}: record_id must never appear -- no case may depend "
f"on it (see ../README.md)")
if unknown - {"record_id"}:
err(f"{where}: record has unknown keys {sorted(unknown - {'record_id'})}")
if rec.get("severity") not in SEVERITIES:
err(f"{where}: unknown severity {rec.get('severity')!r}")
if not isinstance(rec.get("timestamp_unix_nano"), int):
err(f"{where}: timestamp_unix_nano must be an integer")
attrs = rec.get("attributes")
if not isinstance(attrs, dict):
err(f"{where}: attributes must be an object")
else:
for k, v in attrs.items():
if not isinstance(k, str) or not isinstance(v, str):
err(f"{where}: attributes must be string->string, got {k!r}={v!r}")
def check_action(a, where, err):
if not isinstance(a, dict) or "action" not in a:
return err(f"{where}: action must be an object with an 'action' key")
name = a["action"]
if name in UNSPECIFIED_ACTIONS:
return err(f"{where}: '{name}' is deliberately unspecified -- decide "
f"its output shape in the design doc before writing cases "
f"for it")
if name not in ACTIONS:
return err(f"{where}: unknown action {name!r}")
required, optional = ACTIONS[name]
given = set(a) - {"action"}
if required - given:
err(f"{where}: action {name!r} missing {sorted(required - given)}")
if given - required - optional:
err(f"{where}: action {name!r} has unknown params "
f"{sorted(given - required - optional)}")
if name == "derive" and not ({"value", "from_field"} & given):
err(f"{where}: derive needs exactly one of value/from_field")
if name == "derive" and {"value", "from_field"} <= given:
err(f"{where}: derive needs exactly one of value/from_field, not both")
for key in ("field", "from", "to"):
if key in a and not valid_field(a[key]):
err(f"{where}: {key}={a[key]!r} is not an addressable field")
for key in ("fields", "key_fields"):
for f in a.get(key, []):
if not valid_field(f):
err(f"{where}: {key} entry {f!r} is not an addressable field")
for key in ("pattern",):
if key in a:
try:
re.compile(a[key])
except re.error as exc:
err(f"{where}: {key} is not a valid regex: {exc}")
if name == "sample":
n = a.get("keep_one_in")
if not isinstance(n, int) or n < 1:
err(f"{where}: keep_one_in must be an integer >= 1")
if name == "suppress_duplicates":
w = a.get("window_ms")
if not isinstance(w, int) or w < 1:
err(f"{where}: window_ms must be an integer >= 1")
def check_case(path, err):
with open(path) as fh:
try:
case = json.load(fh)
except json.JSONDecodeError as exc:
return err(f"invalid JSON: {exc}")
missing = CASE_KEYS - set(case)
unknown = set(case) - CASE_KEYS
if missing:
err(f"missing top-level keys {sorted(missing)}")
if unknown:
err(f"unknown top-level keys {sorted(unknown)}")
expected_name = os.path.basename(path)[:-len(".json")]
if case.get("name") != expected_name:
err(f"name {case.get('name')!r} does not match filename "
f"{expected_name!r}")
if not case.get("description"):
err("description must not be empty -- it is what the case is for")
for i, rule in enumerate(case.get("rules", [])):
where = f"rules[{i}]"
if not isinstance(rule, dict) or set(rule) != {"match", "actions"}:
err(f"{where}: a rule is exactly {{match, actions}}")
continue
for j, clause in enumerate(rule["match"]):
cw = f"{where}.match[{j}]"
if not isinstance(clause, dict):
err(f"{cw}: clause must be an object")
continue
op = clause.get("op")
if op in OPS_WITHOUT_VALUE:
if "value" in clause:
err(f"{cw}: {op} takes no value")
allowed = {"field", "op"}
elif op in OPS_WITH_VALUE:
if "value" not in clause:
err(f"{cw}: {op} requires a value")
allowed = {"field", "op", "value"}
else:
err(f"{cw}: unknown op {op!r}")
continue
if set(clause) - allowed:
err(f"{cw}: unknown keys {sorted(set(clause) - allowed)}")
if not valid_field(clause.get("field", "")):
err(f"{cw}: {clause.get('field')!r} is not an addressable field")
if op == "regex":
try:
re.compile(clause["value"])
except re.error as exc:
err(f"{cw}: not a valid regex: {exc}")
if not rule["actions"]:
err(f"{where}: a rule with no actions does nothing")
for j, action in enumerate(rule["actions"]):
check_action(action, f"{where}.actions[{j}]", err)
for i, rec in enumerate(case.get("inputs", [])):
check_record(rec, f"inputs[{i}]", err)
if not case.get("inputs"):
err("inputs must not be empty -- a case with no input asserts nothing")
for i, rec in enumerate(case.get("expect", [])):
check_record(rec, f"expect[{i}]", err)
def main():
if not os.path.isdir(CASES):
print(f"no cases directory at {CASES}", file=sys.stderr)
return 1
files = sorted(f for f in os.listdir(CASES) if f.endswith(".json"))
if not files:
print("no cases found", file=sys.stderr)
return 1
failures = 0
for name in files:
errors = []
check_case(os.path.join(CASES, name), errors.append)
if errors:
failures += 1
print(f"FAIL {name}")
for e in errors:
print(f" {e}")
print(f"\n{len(files) - failures}/{len(files)} cases valid")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())