jcoffey-dev
4f0da1ae5e
Extends the heartbeat mechanism with a second gRPC service on the same mTLS channel (AgentControl.CheckIn, agent-initiated on the existing heartbeat ticker -- still push-only, no inbound port on any agent) so an agent reports its running config and can pick up an operator-set override. A new web UI section (/agents) lists every agent that's checked in, shows its reported config, and lets an operator edit a narrow, deliberately-scoped subset remotely: batch/heartbeat tuning, and (journald sources only) the unit filter. TLS material and the ingest endpoint are never reportable or remotely editable, by proto shape rather than a validation rule -- a bad or malicious edit there could permanently strand an agent or redirect where its logs go, unlike every other editable field, which only degrades behavior. An override lives only in the agent's memory (agent.toml is never rewritten) and re-syncs on the agent's own schedule; changing the journald filter aborts and respawns the source task since there's no other way to change what's being tailed. Building the hot-reload path surfaced a real, independent, pre-existing bug: shutdown was using poll_timeout(), which only drains once flush_interval has elapsed, silently dropping anything buffered more recently on every graceful shutdown that landed between flushes -- fixed with a new unconditional Batcher::flush_all(), now used at both shutdown and hot-reload. Verified live end-to-end against a real stack: an edited heartbeat interval changed a running agent's actual send cadence within one check-in cycle (confirmed by the real timestamps landing in ClickHouse), and an edited journald filter triggered a real source restart, both reflected back in the next reported-config snapshot. See /docs/agent-management-design.md.
139 lines
4.6 KiB
Rust
139 lines
4.6 KiB
Rust
use crate::pb::LogRecord;
|
|
use std::time::{Duration, Instant};
|
|
|
|
/// Buffers `LogRecord`s and signals when to flush, either because the
|
|
/// buffer hit `max_size` (checked on every push) or because
|
|
/// `flush_interval` elapsed since the last flush (checked by the caller via
|
|
/// `poll_timeout` on a timer tick). Not thread-safe by design — one
|
|
/// batcher per agent, driven from a single async task's select loop.
|
|
pub struct Batcher {
|
|
max_size: usize,
|
|
flush_interval: Duration,
|
|
buf: Vec<LogRecord>,
|
|
last_flush: Instant,
|
|
}
|
|
|
|
impl Batcher {
|
|
pub fn new(max_size: usize, flush_interval: Duration) -> Self {
|
|
Self {
|
|
max_size,
|
|
flush_interval,
|
|
buf: Vec::with_capacity(max_size),
|
|
last_flush: Instant::now(),
|
|
}
|
|
}
|
|
|
|
/// Push a record. Returns the drained batch if this push filled the
|
|
/// buffer to `max_size`.
|
|
pub fn push(&mut self, record: LogRecord) -> Option<Vec<LogRecord>> {
|
|
self.buf.push(record);
|
|
if self.buf.len() >= self.max_size {
|
|
Some(self.drain())
|
|
} else {
|
|
None
|
|
}
|
|
}
|
|
|
|
/// Call periodically (e.g. from a timer tick). Returns the drained
|
|
/// batch if the flush interval has elapsed and there's anything
|
|
/// buffered.
|
|
pub fn poll_timeout(&mut self) -> Option<Vec<LogRecord>> {
|
|
if !self.buf.is_empty() && self.last_flush.elapsed() >= self.flush_interval {
|
|
Some(self.drain())
|
|
} else {
|
|
None
|
|
}
|
|
}
|
|
|
|
/// Unconditionally drains whatever is buffered, ignoring both
|
|
/// `max_size` and `flush_interval` -- for shutdown and for a
|
|
/// config hot-reload replacing this `Batcher` outright (Phase:
|
|
/// agent management's remote config editing), neither of which
|
|
/// should silently drop records just because the timeout hadn't
|
|
/// elapsed yet.
|
|
pub fn flush_all(&mut self) -> Option<Vec<LogRecord>> {
|
|
if self.buf.is_empty() {
|
|
None
|
|
} else {
|
|
Some(self.drain())
|
|
}
|
|
}
|
|
|
|
fn drain(&mut self) -> Vec<LogRecord> {
|
|
self.last_flush = Instant::now();
|
|
std::mem::replace(&mut self.buf, Vec::with_capacity(self.max_size))
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
fn rec(msg: &str) -> LogRecord {
|
|
LogRecord {
|
|
timestamp_unix_nano: 0,
|
|
host: "h".into(),
|
|
service: "s".into(),
|
|
severity: 0,
|
|
message: msg.into(),
|
|
attributes: Default::default(),
|
|
record_id: String::new(),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn flushes_on_size() {
|
|
let mut b = Batcher::new(2, Duration::from_secs(999));
|
|
assert!(b.push(rec("a")).is_none());
|
|
let batch = b.push(rec("b")).expect("should flush at max_size");
|
|
assert_eq!(batch.len(), 2);
|
|
assert_eq!(batch[0].message, "a");
|
|
assert_eq!(batch[1].message, "b");
|
|
}
|
|
|
|
#[test]
|
|
fn buffer_empty_after_size_flush() {
|
|
let mut b = Batcher::new(1, Duration::from_secs(999));
|
|
b.push(rec("a")).expect("flush at max_size 1");
|
|
assert!(b.poll_timeout().is_none(), "buffer should be empty post-flush");
|
|
}
|
|
|
|
#[test]
|
|
fn flushes_on_timeout() {
|
|
let mut b = Batcher::new(100, Duration::from_millis(10));
|
|
assert!(b.push(rec("a")).is_none());
|
|
std::thread::sleep(Duration::from_millis(30));
|
|
let batch = b.poll_timeout().expect("should flush after timeout");
|
|
assert_eq!(batch.len(), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn no_flush_when_buffer_empty() {
|
|
let mut b = Batcher::new(10, Duration::from_millis(1));
|
|
std::thread::sleep(Duration::from_millis(5));
|
|
assert!(b.poll_timeout().is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn no_flush_before_timeout_elapsed() {
|
|
let mut b = Batcher::new(10, Duration::from_secs(999));
|
|
b.push(rec("a"));
|
|
assert!(b.poll_timeout().is_none());
|
|
}
|
|
|
|
// Regression test: shutdown used to call poll_timeout(), which
|
|
// silently drops anything buffered before flush_interval elapses --
|
|
// real data loss on a graceful shutdown that happened to land
|
|
// between flushes. flush_all() is what shutdown (and hot-reload)
|
|
// must use instead.
|
|
#[test]
|
|
fn flush_all_drains_regardless_of_timeout() {
|
|
let mut b = Batcher::new(10, Duration::from_secs(999));
|
|
b.push(rec("a"));
|
|
assert!(b.poll_timeout().is_none(), "sanity: timeout hasn't elapsed");
|
|
let batch = b.flush_all().expect("flush_all should drain unconditionally");
|
|
assert_eq!(batch.len(), 1);
|
|
assert!(b.flush_all().is_none(), "buffer should be empty after draining");
|
|
}
|
|
}
|