Structured Logging — JSON Output for Log Aggregators

Mental Model

Python's json.dumps is like a meticulous chef who only knows how to prepare a few basic ingredients (strings, numbers, lists, dictionaries). When you give it a datetime object, it's like handing the chef a raw fish and expecting a perfectly grilled fillet – it simply doesn't know how to transform it without explicit instructions (an encoder).

Rule: Always use specialized structured logging tools like structlog or provide strict fallback encoders to prevent logger exceptions from taking down your app.

The Setup

To make logs easy for aggregators like Datadog to parse, you build a custom JSON logger. Everything works in staging, but the first unhandled exception in production crashes the logging loop itself.

What Does This Print?

⚠ Broken code

Python

import json
import logging
from datetime import datetime

class SimpleJSONFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            "time": datetime.utcnow(), # Complex object type
            "message": record.getMessage(),
            "level": record.levelname
        }
        # Attempting standard serialization
        return json.dumps(payload)

logger = logging.getLogger("api")
record = logging.LogRecord("api", logging.INFO, "api.py", 42, "Success", (), None)
try:
    formatter = SimpleJSONFormatter()
    print(formatter.format(record))
except TypeError as err:
    print(f"Logging Failure: {err}")

Predict what happens when this logging setup processes a standard log record containing a custom timestamp.

The Output

What actually happens

Logging Failure: Object of type datetime is not JSON serializable

Standard JSON serialization via json.dumps fails immediately when encountering objects it cannot natively translate, such as datetime, exceptions, or decimal values. If your custom structured logging formatter relies on raw json.dumps without error handling, logging an unsupported object will crash your application threads.

Why Python Does This

Python's json module is built to map strict JSON types to corresponding basic Python types (dict, list, str, int, float, bool, None). Because datetime is a compound type from the standard library rather than a primitive, the json.JSONEncoder has no default representation for it, throwing a TypeError. When implementing logging, any exception raised during formatting halts execution or drops logs. To safely serialize structured payloads, you must supply a custom serializer or serialize dynamic fields to ISO 8601 strings first.

The Fix

✓ Corrected pattern

Python

import json
import logging
from datetime import datetime

class SafeJSONFormatter(logging.Formatter):
    def format(self, record):
        # Fix: Explicitly serialize dates or implement a custom JSON encoder fall-back
        payload = {
            "time": datetime.utcnow().isoformat(), # Convert datetime to string
            "message": record.getMessage(),
            "level": record.levelname
        }
        return json.dumps(payload)

logger = logging.getLogger("api")
record = logging.LogRecord("api", logging.INFO, "api.py", 42, "Success", (), None)
formatter = SafeJSONFormatter()
print(formatter.format(record)) # Succeeds safely

Specialized structured logging libraries or custom JSON encoders provide mechanisms to convert non-primitive types (like datetime objects) into JSON-compatible formats (e.g., ISO 8601 strings) before serialization. This prevents TypeError exceptions, ensuring that logs are consistently emitted and don't interrupt application flow.

How This Fails in Real Systems

A high-volume payment processor encountered uncaught exceptions that should have been logged. However, the logger itself crashed while trying to parse the tracebacks into a custom JSON structure. The resulting logging failure cascaded into thread starvation, hiding the payment failures for 6 hours.

Key Takeaway

Always use specialized structured logging tools like structlog or provide strict fallback encoders to prevent logger exceptions from taking down your app.

Common mistake: Developers assume that Python's standard json module can automatically serialize complex or custom object types like datetime objects into a JSON-compatible format, leading to TypeError exceptions that crash the logging process itself.