Error Logs and Missing Traces

An ERROR in your logs doesn’t automatically mean an ERROR in your traces.

The sampling gap#

In Python, it’s common to catch an exception, log the failure and return a response. The operation failed, but unless something also updates the OpenTelemetry span status, the span can finish without an error recorded.

This matters when you use tail sampling: waiting for traces to finish, then deciding which ones to store. A common policy keeps traces exceeding a given duration or containing an ERROR span, and a small percentage of everything else. If the failure only reached your logs, the trace you need to debug it might get dropped.

Animated walkthrough of tail sampling, error logs marking the active span as ERROR, and YAML rules that exclude known noise. — Tail sampling keeps traces that contain an ERROR span. A failure that only reaches your logs leaves its trace unmarked, so it can be dropped.

Manually setting span status in every error-handling block is tedious. Automatically setting span status through a generic log handler for every log at ERROR level or higher (CRITICAL/FATAL) has its own shortcoming. A retry that eventually succeeds or a noisy library can cause traces to be kept unnecessarily. You end up paying to store the noise.

What the library does#

I built otel-log-span-status to connect the two:

Python

import otel_log_span_status

otel_log_span_status.install("otel-log-span-status.yaml")

It adds a handler to Python’s logging system. An ERROR-level or higher log marks the active recording span as ERROR. A YAML ignore list lets each service exclude known noise:

otel-log-span-status.yaml

ignore:
  - logger: "myapp.legacy_cache"
    message_template: "Cache miss for user %s, falling back to DB"

Here, both the logger name and the unformatted message template must match. The log is still emitted; the handler leaves the span alone.

The library doesn’t configure your sampler or fix noisy logging. It gives your existing sampling policy a signal that would otherwise exist only in your logs, with service-specific rules for what deserves that signal.

I’ve written more about dealing with noisy error logs in Noise Tax on ERROR Logs.