LogAn is an open source log analysis tool developed by IBM Research with Red Hat contributions that addresses the scalability problem of using LLMs on large production log datasets. It combines the Drain template mining algorithm with small language model (SLM) classification to compress millions of log lines into unique structural templates. In a real case, 199,181 log lines compressed to just 97 unique templates, with only 41 requiring analysis. A label broadcasting technique keeps inference time flat at 10-12 seconds regardless of log volume. LogAn classifies logs using Google SRE golden signals (availability, error, latency, saturation, traffic) and fault categories, supports bring-your-own-model, and can be run as a CLI tool or Podman container.

6m read timeFrom developers.redhat.com
Post cover image
Table of contents
The limits of LLMs in log analysisThe key insight: Logs are massively redundantDrain: Compressing logs into templatesGet started with LogAnExplore the code and contribute
166 Impressions