coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Scope log retention deletion and floors to (host, service), not host alone

Browse Source 
logs rows carry a real per-record `service` (nginx, smtp, ufw, ...) --
already true of the schema (storage/migrations/0001) and wire protocol,
not something this feature invents. Both the deletion picker and the
retention floor now operate on (host, service) pairs instead of whole
hosts, so an operator can delete just one noisy log type from an agent
without touching everything else it ships, and can protect one service
(e.g. keep smtp a year) longer than the rest of that host's default.

api/agents.ConfigOverride gains ServiceLogRetentionDays (map[string]int),
owner-only to change like LogRetentionDays -- a service listed there
overrides the host's LogRetentionDays default for that service only.
Agent config page gets a matching "Per-service log retention overrides"
add/remove list next to the existing host-level field.

api/logretention: Store's count/delete now take []HostService and build
a ClickHouse tuple IN ((?,?),...) over (host, service); AgentRetentionStore.
FloorsByHost returns each host's default plus its per-service map, with
HostFloor.Effective(service) resolving which one applies. preview/delete
moved from GET/DELETE-with-query-params to POST-with-JSON-body (a list of
targets needs a real body, not a repeated compound query param), and
partitionTargets checks the floor per target so one protected service
never blocks deleting a different, unprotected one in the same request.

Settings' Log retention section is a two-level picker now: each host
row (with a "select all services" checkbox and its default floor badge)
expands to its services, each with its own count and effective
protected-days badge.

Verified live against real ClickHouse/Postgres and in-browser: a host
with a 7-day default plus a 365-day smtp override -- deleting nginx+
smtp+ufw together correctly removed nginx and ufw, left smtp's 10
records untouched, and confirmed via a follow-up owner delete that
bypassing the floor works. Also verified the full click-through (add a
service override on the agent page, see it reflected in Settings'
picker, select/preview/cancel) and confirmed no regression from the
prior host-only version's tests.
This commit is contained in:
jcoffey-dev
2026-08-21 15:54:58 -07:00
parent 087c52a64f
commit 5ff2e5bb60
10 changed files with 878 additions and 410 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web/src/lib/api.ts
+32 -20
View File
@@ -483,40 +483,47 @@ export function setUserRole(id: string, role: string): Promise<LocalUser> {
}
// --- log retention (owner/admin only, see api/logretention) -----------
// Deletion is host-scoped, not wholesale: a caller must name which
// hosts' logs to target (listRetentionHosts is how the UI discovers
// what to offer), and api/logretention never treats an omitted host
// list as "every host."
// Deletion is scoped to specific (host, service) targets, not wholesale
// -- a caller must name which agents' *and* which log types' logs to
// target (listRetentionHosts is how the UI discovers what to offer,
// grouped by host with each host's services underneath), and
// api/logretention never treats an omitted target list as "everything."
export type BlockedHost = { host: string; protected_days: number };
export type RetentionHost = { host: string; count: number; protected_days?: number };
export type HostService = { host: string; service: string };
export type BlockedTarget = { host: string; service: string; protected_days: number };
export type RetentionService = { service: string; count: number; protected_days?: number };
export type RetentionHost = { host: string; protected_days?: number; services: RetentionService[] };
export type RetentionHostsResult = { hosts: RetentionHost[]; cutoff: string };
export type LogRetentionPreview = { count: number; cutoff: string; hosts: string[]; blocked_hosts?: BlockedHost[] };
export type LogRetentionPreview = {
count: number;
cutoff: string;
targets: HostService[];
blocked_targets?: BlockedTarget[];
};
export type LogRetentionDeleteResult = {
deleted_count: number;
cutoff: string;
deleted_hosts: string[];
blocked_hosts?: BlockedHost[];
deleted_targets: HostService[];
blocked_targets?: BlockedTarget[];
};
function hostsQuery(hosts: string[]): string {
return hosts.map((h) => `host=${encodeURIComponent(h)}`).join('&');
}
export function listRetentionHosts(olderThanHours: number): Promise<RetentionHostsResult> {
return request(`/logs/retention/hosts?older_than_hours=${olderThanHours}`, { credentials: 'include' });
}
export function previewLogDeletion(olderThanHours: number, hosts: string[]): Promise<LogRetentionPreview> {
return request(`/logs/retention/preview?older_than_hours=${olderThanHours}&${hostsQuery(hosts)}`, {
credentials: 'include'
export function previewLogDeletion(olderThanHours: number, targets: HostService[]): Promise<LogRetentionPreview> {
return request('/logs/retention/preview', {
method: 'POST',
credentials: 'include',
body: JSON.stringify({ older_than_hours: olderThanHours, targets })
});
}
export function deleteLogsOlderThan(olderThanHours: number, hosts: string[]): Promise<LogRetentionDeleteResult> {
return request(`/logs/retention?older_than_hours=${olderThanHours}&${hostsQuery(hosts)}`, {
method: 'DELETE',
credentials: 'include'
export function deleteLogsOlderThan(olderThanHours: number, targets: HostService[]): Promise<LogRetentionDeleteResult> {
return request('/logs/retention/delete', {
method: 'POST',
credentials: 'include',
body: JSON.stringify({ older_than_hours: olderThanHours, targets })
});
}
@@ -617,6 +624,11 @@ export type ConfigOverride = {
// reads as a protective floor, not something the agent process itself
// ever sees or applies.
log_retention_days?: number;
// service_log_retention_days is log_retention_days' per-service
// refinement, also owner-only -- a service present here overrides
// log_retention_days for that service only; every other service on
// this host still falls back to log_retention_days.
service_log_retention_days?: Record<string, number>;
};
export type Agent = {
web/src/routes/agents/[host]/+page.svelte
+54 -1
View File
@@ -38,6 +38,11 @@
// one either. Kept as a string ('' = no override) so the input can be
// empty rather than defaulting to some arbitrary number of days.
let logRetentionDays = $state('');
// Per-service overrides of logRetentionDays -- a service named here
// (e.g. "smtp") keeps its own retention floor instead of falling back
// to the host default above. Rows with an empty service name are
// dropped at save(), same as extraFilePaths drops blank paths.
let serviceRetention = $state<{ service: string; days: string }[]>([]);
function resetForm(a: Agent) {
const o = a.desired_override;
@@ -48,6 +53,9 @@
journaldUnit = o?.journald_unit ?? '';
extraFilePaths = o?.extra_file_paths ? [...o.extra_file_paths] : [];
logRetentionDays = o?.log_retention_days != null ? String(o.log_retention_days) : '';
serviceRetention = o?.service_log_retention_days
? Object.entries(o.service_log_retention_days).map(([service, days]) => ({ service, days: String(days) }))
: [];
}
function addExtraFilePath() {
@@ -58,6 +66,14 @@
extraFilePaths = extraFilePaths.filter((_, i) => i !== index);
}
function addServiceRetention() {
serviceRetention = [...serviceRetention, { service: '', days: '' }];
}
function removeServiceRetention(index: number) {
serviceRetention = serviceRetention.filter((_, i) => i !== index);
}
async function load() {
loading = true;
error = '';
@@ -72,6 +88,22 @@
}
load();
// Always sent, even when empty ({}) -- unlike logRetentionDays'
// conditional omit, this mirrors extraFilePaths' unconditional style
// (a collection field), so clearing every row genuinely clears the
// stored overrides rather than leaving stale ones behind.
function buildServiceRetentionMap(): Record<string, number> {
const out: Record<string, number> = {};
for (const row of serviceRetention) {
const service = row.service.trim();
const days = String(row.days).trim();
if (service !== '' && days !== '') {
out[service] = Number(days);
}
}
return out;
}
async function save() {
saving = true;
saveError = '';
@@ -91,7 +123,8 @@
// the moment someone edits this field. batch_max_size etc.
// above never hit this because Number(x) doesn't care
// whether x is already a number.
...(String(logRetentionDays).trim() !== '' ? { log_retention_days: Number(logRetentionDays) } : {})
...(String(logRetentionDays).trim() !== '' ? { log_retention_days: Number(logRetentionDays) } : {}),
service_log_retention_days: buildServiceRetentionMap()
});
} catch (e) {
saveError = e instanceof Error ? e.message : String(e);
@@ -242,6 +275,23 @@
</p>
</div>
<div class="field extra-paths">
<span class="field-label">Per-service log retention overrides</span>
<p class="hint">
Owner only. Keep a specific log type from this host longer (or shorter) than the general retention
above -- e.g. protect "smtp" for a year while everything else on this host uses the default. A
service not listed here just uses the host default.
</p>
{#each serviceRetention as _, i}
<div class="path-row service-row">
<Input placeholder="Service (e.g. smtp)" bind:value={serviceRetention[i].service} />
<Input type="number" min="1" max="3650" placeholder="Days" bind:value={serviceRetention[i].days} />
<Button variant="secondary" onclick={() => removeServiceRetention(i)}>Remove</Button>
</div>
{/each}
<Button variant="secondary" onclick={addServiceRetention}>Add service override</Button>
</div>
{#if saveError}<p class="error">Error: {saveError}</p>{/if}
<div class="actions">
@@ -366,6 +416,9 @@
.path-row :global(input) {
flex: 1;
}
.service-row :global(input:last-of-type) {
flex: 0 0 6rem;
}
.actions {
display: flex;
gap: var(--space-3);
web/src/routes/settings/+page.svelte
+138 -50
View File
@@ -12,6 +12,7 @@
type LocalSession,
type CurrentSession,
type RetentionHost,
type HostService,
type LogRetentionPreview,
type LogRetentionDeleteResult
} from '$lib/api';
@@ -70,10 +71,12 @@
return !localAuthEnabled;
});
// --- log retention deletion -- host-scoped, not wholesale: a caller
// picks which hosts to target from what listRetentionHosts reports for
// the chosen age, same "select specific agents, not delete everything"
// requirement api/logretention's own parseHosts enforces server-side. ---
// --- log retention deletion -- scoped to specific (host, service)
// targets, not wholesale: a caller picks which agents' *and* which log
// types' logs to target from what listRetentionHosts reports for the
// chosen age, same "select specific targets, not delete everything"
// requirement api/logretention's own parseTargets enforces
// server-side. ---
const retentionOptions: { label: string; hours: number }[] = [
{ label: '7 days', hours: 24 * 7 },
{ label: '30 days', hours: 24 * 30 },
@@ -86,7 +89,10 @@
let hostsLoading = $state(false);
let hosts = $state<RetentionHost[]>([]);
let hostsError = $state('');
let selectedHosts = $state<Set<string>>(new Set());
// Selection keyed by a composite "host service" string for O(1)
// membership checks -- targetKey/targetsFromKeys convert to/from the
// {host, service} objects the API actually wants.
let selectedTargets = $state<Set<string>>(new Set());
let previewing = $state(false);
let preview = $state<LogRetentionPreview | null>(null);
@@ -94,6 +100,17 @@
let deleteResult = $state<LogRetentionDeleteResult | null>(null);
let retentionError = $state('');
function targetKey(host: string, service: string): string {
return `${host} ${service}`;
}
function targetsFromKeys(keys: Iterable<string>): HostService[] {
return [...keys].map((key) => {
const [host, service] = key.split(' ');
return { host, service };
});
}
// Reloads whenever retentionHours changes (including on mount, since
// $effect runs once immediately too) -- selection/preview/result all
// reset because they were scoped to the previous age's host list.
@@ -102,7 +119,7 @@
hostsError = '';
preview = null;
deleteResult = null;
selectedHosts = new Set();
selectedTargets = new Set();
try {
const result = await listRetentionHosts(hours);
hosts = result.hosts;
@@ -117,32 +134,56 @@
loadHosts(retentionHours);
});
function toggleHost(host: string) {
const next = new Set(selectedHosts);
if (next.has(host)) next.delete(host);
else next.add(host);
selectedHosts = next;
function toggleTarget(host: string, service: string) {
const key = targetKey(host, service);
const next = new Set(selectedTargets);
if (next.has(key)) next.delete(key);
else next.add(key);
selectedTargets = next;
preview = null;
deleteResult = null;
}
function selectAllHosts() {
selectedHosts = new Set(hosts.map((h) => h.host));
function isHostFullySelected(host: RetentionHost): boolean {
return host.services.length > 0 && host.services.every((s) => selectedTargets.has(targetKey(host.host, s.service)));
}
// Toggles every service under one host together -- selects all of
// them if any are currently unselected, otherwise clears all of them.
function toggleHostAll(host: RetentionHost) {
const next = new Set(selectedTargets);
const selectAll = !isHostFullySelected(host);
for (const s of host.services) {
const key = targetKey(host.host, s.service);
if (selectAll) next.add(key);
else next.delete(key);
}
selectedTargets = next;
preview = null;
deleteResult = null;
}
function selectAllTargets() {
const next = new Set<string>();
for (const h of hosts) {
for (const s of h.services) next.add(targetKey(h.host, s.service));
}
selectedTargets = next;
preview = null;
}
function selectNoHosts() {
selectedHosts = new Set();
function selectNoTargets() {
selectedTargets = new Set();
preview = null;
}
async function handlePreview() {
if (selectedHosts.size === 0) return;
if (selectedTargets.size === 0) return;
previewing = true;
retentionError = '';
deleteResult = null;
try {
preview = await previewLogDeletion(retentionHours, [...selectedHosts]);
preview = await previewLogDeletion(retentionHours, targetsFromKeys(selectedTargets));
} catch (e) {
retentionError = e instanceof Error ? e.message : String(e);
} finally {
@@ -159,10 +200,10 @@
deleting = true;
retentionError = '';
try {
// preview.hosts, not [...selectedHosts] -- already excludes any
// host the preview found protected, so this never re-asks for a
// host the response just said would be skipped.
const result = await deleteLogsOlderThan(retentionHours, preview.hosts);
// preview.targets, not the current selection -- already excludes
// any target the preview found protected, so this never re-asks
// for a target the response just said would be skipped.
const result = await deleteLogsOlderThan(retentionHours, preview.targets);
preview = null;
// loadHosts resets deleteResult as part of its "fresh state" load
// (stale counts/now-empty hosts shouldn't linger), so it runs
@@ -176,6 +217,10 @@
}
}
function formatTarget(t: HostService): string {
return `${t.host}/${t.service}`;
}
function formatCutoff(iso: string): string {
return new Date(iso).toLocaleString(undefined, {
year: 'numeric',
@@ -244,8 +289,8 @@
<section>
<h2>Log retention</h2>
<p class="note">
Permanently delete log records from specific hosts, older than a chosen age. Visible to owners and
admins only.
Permanently delete specific log types from specific hosts, older than a chosen age. Visible to
owners and admins only.
</p>
<div class="retention-controls">
@@ -265,32 +310,54 @@
{:else}
<div class="host-picker">
<div class="host-picker-actions">
<button type="button" class="link" onclick={selectAllHosts} disabled={deleting}>Select all</button>
<button type="button" class="link" onclick={selectNoHosts} disabled={deleting}>Select none</button>
<button type="button" class="link" onclick={selectAllTargets} disabled={deleting}>Select all</button>
<button type="button" class="link" onclick={selectNoTargets} disabled={deleting}>Select none</button>
</div>
<ul class="host-list">
{#each hosts as h (h.host)}
<li>
<label>
<li class="host-group">
<label class="host-row">
<input
type="checkbox"
checked={selectedHosts.has(h.host)}
checked={isHostFullySelected(h)}
disabled={deleting}
onchange={() => toggleHost(h.host)}
onchange={() => toggleHostAll(h)}
/>
<span class="host-name">{h.host}</span>
<span class="host-count">{h.count.toLocaleString()} records</span>
{#if h.protected_days != null}
<span class="protected-badge">protected {h.protected_days}d</span>
<span class="protected-badge">host default {h.protected_days}d</span>
{/if}
</label>
<ul class="service-list">
{#each h.services as s (s.service)}
<li>
<label>
<input
type="checkbox"
checked={selectedTargets.has(targetKey(h.host, s.service))}
disabled={deleting}
onchange={() => toggleTarget(h.host, s.service)}
/>
<span class="service-name">{s.service}</span>
<span class="host-count">{s.count.toLocaleString()} records</span>
{#if s.protected_days != null}
<span class="protected-badge">protected {s.protected_days}d</span>
{/if}
</label>
</li>
{/each}
</ul>
</li>
{/each}
</ul>
<button type="button" onclick={handlePreview} disabled={selectedHosts.size === 0 || previewing || deleting}>
<button
type="button"
onclick={handlePreview}
disabled={selectedTargets.size === 0 || previewing || deleting}
>
{previewing
? 'Checking…'
: `Delete logs from ${selectedHosts.size} host${selectedHosts.size === 1 ? '' : 's'}…`}
: `Delete logs from ${selectedTargets.size} target${selectedTargets.size === 1 ? '' : 's'}…`}
</button>
</div>
{/if}
@@ -299,28 +366,29 @@
{#if preview}
<div class="confirm-panel">
{#if preview.hosts.length > 0}
{#if preview.targets.length > 0}
<p>
This will <strong>permanently delete {preview.count.toLocaleString()}</strong>
log record{preview.count === 1 ? '' : 's'} older than {formatCutoff(preview.cutoff)} from
{preview.hosts.length} host{preview.hosts.length === 1 ? '' : 's'} ({preview.hosts.join(', ')}).
This cannot be undone.
{preview.targets.length} target{preview.targets.length === 1 ? '' : 's'} ({preview.targets
.map(formatTarget)
.join(', ')}). This cannot be undone.
</p>
{:else}
<p>Every selected host is protected by a retention policy -- nothing to delete.</p>
<p>Every selected target is protected by a retention policy -- nothing to delete.</p>
{/if}
{#if preview.blocked_hosts?.length}
{#if preview.blocked_targets?.length}
<p class="note">
Protected by a retention policy, skipped: {preview.blocked_hosts
.map((b) => `${b.host} (${b.protected_days}d)`)
Protected by a retention policy, skipped: {preview.blocked_targets
.map((b) => `${formatTarget(b)} (${b.protected_days}d)`)
.join(', ')}.
</p>
{/if}
<div class="confirm-actions">
<button type="button" onclick={cancelPreview} disabled={deleting}>
{preview.hosts.length > 0 ? 'Cancel' : 'Close'}
{preview.targets.length > 0 ? 'Cancel' : 'Close'}
</button>
{#if preview.hosts.length > 0}
{#if preview.targets.length > 0}
<button type="button" class="danger" onclick={confirmDelete} disabled={deleting}>
{deleting ? 'Deleting…' : 'Yes, delete permanently'}
</button>
@@ -334,10 +402,12 @@
Deleted {deleteResult.deleted_count.toLocaleString()} log record{deleteResult.deleted_count === 1
? ''
: 's'} older than {formatCutoff(deleteResult.cutoff)}
{#if deleteResult.deleted_hosts.length}from {deleteResult.deleted_hosts.join(', ')}{/if}.
{#if deleteResult.blocked_hosts?.length}
Skipped (protected): {deleteResult.blocked_hosts
.map((b) => `${b.host} (${b.protected_days}d)`)
{#if deleteResult.deleted_targets.length}from {deleteResult.deleted_targets
.map(formatTarget)
.join(', ')}{/if}.
{#if deleteResult.blocked_targets?.length}
Skipped (protected): {deleteResult.blocked_targets
.map((b) => `${formatTarget(b)} (${b.protected_days}d)`)
.join(', ')}.
{/if}
</p>
@@ -488,27 +558,45 @@
list-style: none;
margin: 0 0 var(--space-3);
padding: 0;
max-height: 14rem;
max-height: 20rem;
overflow-y: auto;
}
.host-list li {
.host-group {
border-bottom: 1px solid var(--color-border);
}
.host-list li:last-child {
.host-group:last-child {
border-bottom: none;
}
.host-list label {
.host-row {
display: flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-2) var(--space-1);
font-size: var(--text-sm);
font-weight: var(--font-weight-medium);
cursor: pointer;
}
.service-list {
list-style: none;
margin: 0;
padding: 0 0 var(--space-2);
}
.service-list label {
display: flex;
align-items: center;
gap: var(--space-2);
padding: var(--space-1) var(--space-1) var(--space-1) var(--space-5);
font-size: var(--text-sm);
cursor: pointer;
}
.host-name {
font-family: var(--font-mono);
color: var(--color-text);
}
.service-name {
font-family: var(--font-mono);
color: var(--color-text-muted);
}
.host-count {
color: var(--color-text-muted);
font-size: var(--text-xs);