[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

improving signal to noise ratio from centralized network syslogs

On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:

> Hey All,
> Centralized logging is a good thing. However, what happens is that every
> repetitive, annoying but not (usually) important thing fills up the log
> with reams of what you are not looking for.
> Networks are a noisy place and silencing every logged condition is
> impractical and sometimes undesirable.
> What I am interested in is an automated zoom-in zoom-out tool to mask
> the repetition of "normal" events and allow the unusual to stand out.
> Add to that an ability to identify gaps in the background noise. (The
> dog that didnt bark)
> What I am not interested in are solutions based upon preconfigured
> filters and definitions and built in analysis for supported
> (prepopulated definitions) platforms, this is all about pattern
> mining/masking and should be self discoverable. Ideally a command tool
> to generate static versions of the analysis coupled with a web platform
> (with zoom +- buttons)  for realtime.
> I made a crude run of it with SLCT, using its generated patterns to grep
> -v, and that in and of itself was useful, but needs a bit of work. Also,
> its not quite real time.
> Any ideas would be greatly appreciated.

Not cheap, but Splunk comes to mind.

> Joe

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler