An error log is meant to signal something needs fixing. Unfortunately, in many legacy systems, error logs are noisy enough that they stop being useful.
My work as a Senior Software Engineer involves maintaining, refactoring and migrating legacy services — often those that had their fair share of ownership change in the past. They are heavy-hit production systems and so I thoroughly test before shipping: unit tests, docker isolated component tests, system tests and AB Diff tests. Post production, I lean heavily on error logs to determine system health, in addition to baseline APM metrics for the application.
Common Properties of Low Quality Error Logs#
- Error logs at source locations that don’t have a real problem with the application. For instance, invalid input to the API that is well handled, or a cache miss that eventually uses a golden copy of data.
- Misconfigured alerts. An error log implies a human needs to act at some point. For such legacy applications you likely find both: missing alerts on genuine errors (blindsided) and alerts set up on noisy error logs (alert fatigue).
- The error rate is noisy. With so much noise, it’s tedious to weed out genuine failures. The error rate is a simple SLO of maximum acceptable frequency of failed requests in a given window.
From Low to High Quality Error Logs#
Conversely, high quality error logs can help you a lot with proactive monitoring of your application. In order to fix a legacy application that has low quality error logs, we must weed out the low quality noise.
Step 1: Error log catalog#
One approach I have been using is to start with an error log catalog. I set out to build a list of all unique error logs emitted by my application along with their daily counts and 7-day volume using the DRAIN algorithm. DRAIN clusters raw log lines into templates using a fixed-depth parse tree, matching on token position and length. I faced some challenges on step one:
- Semi-structured or unstructured logs contain high cardinality fields that are difficult to parse and systematically handle during aggregation to log templates.
- Linked libraries log into your application’s stream. Lacking your application’s context, they flag conditions at ERROR level that aren’t real errors for you.
- Since algorithms like DRAIN are sensitive to token position, detecting new error log patterns leads to false positives. To reliably tell if an existing template has drifted or a new error has been emitted is a non-trivial problem.
Why This is Important (Even in the AI-Era)#
- Logs are read by machine (AI/LLMs): many agentic tools allow connecting your log vendor to debug production issues. A noisy error log stops being merely a human annoyance. It’s garbage in — garbage out for AI and you get to pay for the noisy investigations.
- Logs are written by machine (AI/LLMs): agentic coding tools ship a lot of code fast, and they instrument inconsistently: over-logging, wrong levels, duplicated catch-and-log. The rate at which low-quality log sites enter a codebase is going up, not down.
- Alert coverage: having stable error log patterns means you have a catalog of all the different errors occurring in your system. This not only makes alerting on each category/pattern of error straightforward, it also gives a good indication of alert coverage on different types of errors.
What’s Next#
The catalog is Step 1 of 3. Step 2 is building a harness to allow AI to fix the noise at the source in the code rather than filtering it downstream, with AI-generated replication tests that reproduce the error condition before any log site is touched. Step 3 is measuring alert coverage against the catalog of genuine errors. Those are the next two posts.