cairnobs/docs/agent-management-design.md

Agent inventory, management, and remote config

Extends /docs/agent-heartbeat-monitoring.md (heartbeat + absence alerting). This adds three things to the web UI: an inventory view of every agent that's checked in, read-only visibility into each agent's actual running config, and the ability to remotely edit a narrow, deliberately-scoped subset of that config.

Scope decision (confirmed before building)

"Agent management" could mean several different things with very different risk profiles. Confirmed up front: this covers inventory, config visibility, and remote config editing — explicitly not remote lifecycle commands (restart/stop/uninstall). That's a real command-and-control channel across every managed host and deserves its own security design (strict RBAC, full audit trail) before it's built, not something to fold in as a side effect of a config-editing feature.

Update — restart is now built (punch-list item 1, see the "Lifecycle commands" section below); stop and uninstall remain deliberately out of scope, for reasons specific to each. Building restart confirmed the "strict RBAC, full audit trail" precondition mattered in practice, not just as a stated principle: it's gated at RoleAdmin (stricter than config editing's RoleEditor) and logged into the same audit_log table Phase 7's AI interactions use.

Why this is still pull, not push, on the wire

The agent's transport is unchanged and still push-only: it dials out to ingest over mTLS; nothing in the platform ever reaches into an agent. See /docs/agent-heartbeat-monitoring.md's "Design: why this is a heartbeat, not a true pull" section for the full reasoning (NAT/ dynamic-IP tolerance, no inbound port on any remote host). Remote config editing extends that same posture: an operator's edit doesn't get pushed to the agent at the moment it's saved. It sits in Postgres until the agent's own next scheduled check-in asks "what should I be running" and picks it up. The web UI's "pending" vs. "applied" badge exists specifically to make that asynchrony visible rather than implying something more immediate.

Wire shape: AgentControl.CheckIn

New proto (proto/sentry/agent/v1/agent_control.proto), a second gRPC service on the exact same mTLS channel/listener LogIngest.PushBatch already uses — not a second protocol or connection the agent has to maintain. Called on the agent's own heartbeat ticker (see agent/cairnobs-agent/src/main.rs's heartbeat_ticker arm), independently of whether the heartbeat log record itself is enabled — CheckIn keeps running even with heartbeat.enabled = false, since that's an agent's only path to ever receive a remote override that re-enables it.

• Request: host/service identity, a ReportedConfig snapshot of what the agent is actually running (version, source kind/detail, batch settings, heartbeat settings), and the version of the last override this agent successfully applied (empty if never).
• Response: has_override plus, when true, a DesiredOverride — every field optional (unset = "no change to this field, keep local config"), each independently overridable.

What's remotely editable, and what deliberately isn't

Editable: batch.max_size, batch.flush_interval_ms, heartbeat.enabled, heartbeat.interval, and — only when the agent's local source is journald — the unit filter.

Never editable, permanently: TLS material and the ingest endpoint. ReportedConfig doesn't even carry these fields, and DesiredOverride has no fields for them at all — this isn't a validation rule that could be relaxed later, it's a shape decision. Two reasons, both serious enough that this needed to be a design boundary rather than a judgment call per edit:

1. A bad edit could permanently strand an agent. Point an agent's ingest.endpoint at an address that doesn't exist, or corrupt its TLS config, and it can never call CheckIn again to receive a correction — the one channel capable of fixing the mistake would be exactly what broke. Every other editable field is safe by construction: even a bad heartbeat interval or an over-aggressive batch size degrades the agent's behavior without ever cutting off its ability to receive the next correction.
2. A compromised web session/API credential must not be able to redirect where an agent's logs go. If ingest.endpoint were editable, an attacker with write access to this feature could point agents at an address they control and exfiltrate log data. Keeping connection details local-file-only means this feature's blast radius is "an agent's operational tuning gets messed with," never "an agent's data goes somewhere else."

validateOverride (api/agents/handler.go) additionally floors batch_max_size >= 1, batch_flush_interval_ms >= 100, and heartbeat_interval_ms >= 5000 — the same kind of real, found-by- building-it floor as alerting's eval_interval_seconds >= 30, here to stop a fat-fingered edit from telling an agent to flush constantly or heartbeat constantly.

Merge semantics: an override is a live layer, not a rewrite

The agent's local agent.toml is never rewritten. A remote override lives only in the running process's memory (agent/cairnobs-agent/src/main.rs's apply_override) and is re-applied fresh on every check-in that returns one — a restarted agent boots from agent.toml alone and re-syncs whatever override is still set on its next successful check-in. This was a deliberate simplicity choice over persisting the override to disk: it avoids needing filesystem write access on every managed host (not guaranteed, e.g. read-only base images) and avoids a whole "reconcile a locally-cached override against a freshly-fetched one at startup" state machine. The cost is that an agent's effective config isn't fully recoverable from agent.toml alone while an override is active — acceptable, since the web UI's GET /agents/{host} is the source of truth for "what is this agent actually running" regardless.

Applying an override that changes batch_max_size/ batch_flush_interval_ms rebuilds the Batcher/flush ticker outright. Whatever was buffered under the old settings is flushed first (Batcher::flush_all, new) rather than dropped — building this exposed a real, independent, pre-existing bug: agent shutdown was calling poll_timeout(), which only drains when flush_interval has already elapsed, meaning records buffered more recently than that were silently lost on every graceful shutdown that happened to land between flushes. Fixed alongside this feature (flush_all() now used at both shutdown and hot-reload) since it's the exact same correctness property in both places.

Changing the journald unit filter is the one override that can't just swap a struct field — there's no way to change what source::journald::run is tailing without restarting that task. Applying it aborts the current source task and respawns a fresh one with the new filter, swapping the channel main.rs reads from. Source kind (journald vs. file vs. eventlog vs. etw) is never remotely switchable — only narrowing/widening the filter within whatever source the host is already configured for.

Data model

metadata/migrations/0037_create_agents.sql: one agents table, one row per (tenant_id, host), in the same cairnobs_metadata Postgres dashboards/alert_rules already live in — not a new database, matching this project's established "shared schema, different services own different tables" shape. tenant_id defaults to 'default' for single-tenant deployments with no TenantResolver configured on ingest, same as every other tenant-scoped table since Phase 3.

Two services write to it, cleanly split by concern:

• ingest/internal/agentregistry (new) upserts on every CheckIn: last_seen_at, the reported_* columns, and applied_override_version (echoed from the agent). Reads back whatever desired_override/desired_override_version is currently stored to answer the RPC. Gated on AGENT_REGISTRY_POSTGRES_ADDR being set — nil/off by default, same "off unless configured" shape as TenantResolver; a deployment that hasn't opted in still accepts CheckIn calls (agents never see an error), it just doesn't record anything or ever return an override.
• api/agents (new) is the web-facing read/write side: GET /agents, GET /agents/{host}, PUT /agents/{host}/config (replaces the whole stored override — the web UI's edit form always reads the current override first and submits the complete merged set, same "PUT replaces the resource" convention every other edit form in this codebase already uses), DELETE /agents/{host}/config (clears it, reverting the agent to its local agent.toml).

ConfigOverride's JSON shape is duplicated three times — Go structs in ingest/internal/agentregistry and api/agents, a proto message for the wire — deliberately, matching this codebase's established convention for shapes shared across module boundaries (see grpcserver.TenantIDHeaderKey, enterprise/internal/apiconfig.AIConfig) rather than coupling independently deployable services' builds together. Keep the three in sync by hand.

RBAC

Viewing inventory is RoleViewer (same bar as viewing a dashboard); editing an agent's remote config is RoleEditor — treated as an operational-tuning action matching alert rules/notification targets, not an admin-only capability like user/role management. Issuing a lifecycle command is RoleAdmin — stricter, matching the RBAC matrix's treatment of similarly consequential actions (e.g. deleting a notification target) rather than day-to-day tuning.

Lifecycle commands (punch-list item 1: restart)

A one-shot action, not a persistent desired state like DesiredOverride -- AgentCommand (proto enum, agent_control.proto) is delivered at-most-once: ingest/internal/agentregistry.Registry.CheckIn clears agents.pending_command in the same transaction that hands it to the agent, before the agent has any chance to confirm it executed the command. This is a deliberate, disclosed asymmetry from DesiredOverride's "keep re-offering until the agent's applied_override_version matches" semantics: a restarting agent's process is gone before it could ever send that confirmation, so waiting for one isn't possible. A command lost to a network blip between the response and the agent acting on it is simply lost -- re-issuing (PUT /agents/{host}/command again) is the operator's recourse, the same as it would be for a systemctl restart that silently failed to reach its target.

Only restart exists today. stop and uninstall remain deliberately deferred -- both need real OS service-manager integration (systemd's Restart=/RestartPreventExitStatus= semantics vs. Windows SCM recovery options differ enough per platform that hand-waving them would be dishonest), which restart doesn't: the agent does a graceful shutdown (flushing whatever's buffered via Batcher::flush_all, aborting the source task) and then a clean std::process::exit(0), relying entirely on whatever restart policy the host's service manager already has configured -- the same contract any well-behaved service already expects. api/agents.Store.IssueCommand/the agents table's pending_command column are written generically enough that adding stop/uninstall later is a real per-platform agent-side implementation, not a data-model change.

Logged into the same append-only audit_log table as everything else privileged in this codebase (event_type = 'agent_command', metadata/migrations/0039), via enterprise/internal/audit. AgentCommandLogger (api/agents.CommandLogger, nil-by-default in core same as every other optional audit hook) -- fail-open, same posture as Phase 7's AI-interaction logging: a write failure is logged server-side and never turns a legitimate restart into a 500.

A real bug was found and fixed while verifying this live. The first version of agentregistry.Registry.CheckIn tried to read-and- clear pending_command atomically using a single INSERT ... ON CONFLICT statement with a sibling read-only CTE referenced only from RETURNING, on the assumption that Postgres evaluates every part of a WITH query against the same pre-statement snapshot. That assumption is wrong specifically for FOR UPDATE: it always locks (and therefore reads) the latest row version to do its job, including a version written earlier in the very same statement -- confirmed empirically against a live Postgres (the CTE's FOR UPDATE was reading its own sibling UPDATE's just-cleared NULL, so pending_command always came back empty even when a command was genuinely pending, and the agent never received it). Live verification caught this immediately: a restart command showed pending_command: "restart" in the API response, but the agent never logged receiving it and never exited. Fixed by splitting into two real, ordered statements inside one explicit transaction (SELECT ... FOR UPDATE strictly happens-before the clearing UPDATE) -- unambiguous, no snapshot-timing subtlety to get wrong.

Verified live

A real agent binary, its heartbeat/CheckIn cadence pointed at a live ingest with AGENT_REGISTRY_POSTGRES_ADDR configured, confirming: the agent appears in GET /agents after its first check-in; an edit made via PUT /agents/{host}/config shows pending: true immediately and pending: false after the agent's next check-in; the edited setting (heartbeat interval) visibly takes effect in the agent's own behavior, confirmed by the change in cadence of new heartbeat rows landing in ClickHouse; a journald-unit override triggers a real source-task restart, reflected in the next reported source_detail; and (after the bug above was fixed) a restart command issued via PUT /agents/{host}/ command is picked up on the agent's very next check-in, logged ("received remote restart command, shutting down gracefully"), and the process exits cleanly -- pending_command confirmed cleared and ps confirming the process gone.

CLI surface (punch-list item 3)

cairnobsctl agents (cli/cmd/cairnobsctl/cmd_agents.go), same list/get shape as dashboards/alerts, plus a config sub-subcommand (mirroring dashboards permissions) since an override has its own get/set/clear lifecycle distinct from the agent resource itself:

cairnobsctl agents list|get <host>
cairnobsctl agents config get <host>|clear <host>
cairnobsctl agents config set <host> [--batch-max-size N] [--batch-flush-interval-ms N]
                            [--heartbeat-enabled true|false] [--heartbeat-interval-ms N]
                            [--journald-unit UNIT]
cairnobsctl agents restart <host> [--yes]

config set is the one command with real logic beyond a thin HTTP wrapper: since PUT /agents/{host}/config replaces the whole stored override (not a per-field patch), the command fetches the agent's current effective config first and merges only the flags actually passed on top of it -- whatever's already overridden stays overridden, whatever isn't falls back to the agent's reported value -- exactly mirroring web/src/routes/agents/[host]/+page.svelte's edit form logic in Go instead of Svelte. restart requires explicit confirmation (interactive y/N, or --yes for scripted use) and refuses outright on a non-interactive stdin without --yes, the same posture cmd_query.go's --nl/--execute already established for anything that can actually change what's running.

Verified live, including the merge logic specifically (the part most likely to have a real bug): config set --heartbeat-interval-ms 20000 on an agent with no existing override correctly carried forward its reported batch_max_size/batch_flush_interval_ms/ heartbeat_enabled; a second config set --batch-max-size 750 call correctly carried forward the first call's heartbeat_interval_ms: 20000 override rather than resetting it to the reported 5000 -- confirming the read-current-then-merge step actually reads the override, not just the reported baseline. config clear and restart --yes both round-tripped against the live stack; the restart was picked up by the real agent process on its next check-in, logged, and the process exited cleanly, closing the loop from a single CLI command all the way to real process behavior.

Punch list: complete

All three items from /docs/agent-heartbeat-monitoring.md's original punch list are done: lifecycle commands (restart), fleet-wide alerting (via the existing raw-SQL escape hatch, no engine changes), and this CLI surface. Real, disclosed remaining gaps, not oversights: stop/ uninstall lifecycle commands (need genuine per-platform OS service- manager integration), true per-host multi-row alerting from one rule (needs the alerting engine's own per-group state tracking, a Phase 3 non-goal for the whole engine, not agent-specific), and a scripted rule-per-host generator (a real alternative to the fleet-wide aggregate check, not built since it's tooling that could live under this same CLI surface if ever wanted).