[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
improving signal to noise ratio from centralized network syslogs
   ELK stack.
   Java RAM devoring monster but Kibana makes indexing easy.
-----
Alain Hebert ahebert at pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
On 01/26/18 01:01, Michael Loftis wrote:
> On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:
>
>> Hey All,
>>
>> Centralized logging is a good thing. However, what happens is that every
>> repetitive, annoying but not (usually) important thing fills up the log
>> with reams of what you are not looking for.
>>
>> Networks are a noisy place and silencing every logged condition is
>> impractical and sometimes undesirable.
>>
>> What I am interested in is an automated zoom-in zoom-out tool to mask
>> the repetition of "normal" events and allow the unusual to stand out.
>>
>> Add to that an ability to identify gaps in the background noise. (The
>> dog that didnt bark)
>>
>> What I am not interested in are solutions based upon preconfigured
>> filters and definitions and built in analysis for supported
>> (prepopulated definitions) platforms, this is all about pattern
>> mining/masking and should be self discoverable. Ideally a command tool
>> to generate static versions of the analysis coupled with a web platform
>> (with zoom +- buttons) for realtime.
>>
>> I made a crude run of it with SLCT, using its generated patterns to grep
>> -v, and that in and of itself was useful, but needs a bit of work. Also,
>> its not quite real time.
>>
>> Any ideas would be greatly appreciated.
>
> Not cheap, but Splunk comes to mind.
>
>>
>> Joe
>>