Circuit Breaker Pattern for Microservices

Why It Matters

How it Works

The Intuition: The Electrical Analogy

Imagine your home’s electrical system. If a device malfunctions and draws too much current, the circuit breaker "trips," cutting off power to that specific circuit to prevent a fire. In microservices, the "current" is the stream of incoming requests. If an ML inference service starts failing—perhaps due to a memory leak or an overloaded GPU—continuing to send requests to it is counterproductive. It wastes resources and risks crashing the entire system. The Circuit Breaker pattern acts as a safety switch, automatically blocking traffic to a failing service so it can recover, while providing a fallback response to the user.

How the States Work

The pattern operates through three distinct states: 1. Closed: Everything is functioning normally. Requests flow to the service as expected. The breaker monitors the failure rate. 2. Open: The failure threshold has been exceeded. The breaker "trips," and all subsequent calls to the service fail immediately without even attempting to reach the service. This gives the service a "cooldown" period. 3. Half-Open: After a set timeout, the breaker allows a limited number of "test" requests to pass through. If these succeed, the breaker assumes the service has recovered and transitions back to Closed. If they fail, it returns to Open.

Why ML Systems Need This

Machine learning models are often computationally expensive. Unlike a simple database lookup, a model inference request might require significant CPU/GPU time. If a model service becomes slow, the upstream services (like a web gateway) will start queuing requests. This leads to thread exhaustion. By implementing a circuit breaker, you prevent the "thundering herd" problem, where all clients retry simultaneously, further overwhelming the struggling model.

Advanced Considerations: Adaptive Thresholds

In sophisticated MLOps environments, simple static thresholds (e.g., "trip after 5 errors") are often insufficient. Advanced implementations use adaptive thresholds based on the moving average of latency or error rates. For instance, if your model inference service is deployed on a Kubernetes cluster, the circuit breaker can integrate with metrics from Prometheus. If the P99 latency exceeds a threshold defined by your Service Level Objective (SLO), the breaker trips. This ensures that the system is not just reacting to hard crashes, but also to "gray failures" where the service is technically alive but performing too poorly to be useful.

Common Pitfalls

Sample Code

import time
import random

class CircuitBreaker:
    """A simple state machine for circuit breaking."""
    def __init__(self, threshold=3, recovery_time=5):
        self.threshold = threshold  # Max failures before opening
        self.recovery_time = recovery_time
        self.failures = 0
        self.state = "CLOSED"
        self.last_failure_time = None

    def call(self, func, *args):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.recovery_time:
                self.state = "HALF-OPEN"
            else:
                return "Service Unavailable (Circuit Open)"

        try:
            result = func(*args)
            self.reset()
            return result
        except Exception:
            self.handle_failure()
            return "Service Error (Fallback)"

    def handle_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.state = "OPEN"
            self.last_failure_time = time.time()

    def reset(self):
        self.failures = 0
        self.state = "CLOSED"

# Usage Example
def model_inference():
    if random.random() < 0.7: raise Exception("GPU Timeout")
    return "Prediction: Class A"

cb = CircuitBreaker()
for _ in range(10):
    print(cb.call(model_inference))
# Output:
# Prediction: Class A
# Service Error (Fallback)
# Service Error (Fallback)
# Service Error (Fallback)
# Service Unavailable (Circuit Open)

Key Terms

Related Concepts