Resilience: Failure Is Normal

Assume failure

In a distributed system, a call to another service will sometimes be slow, fail, or never return. A resilient system expects this and keeps working – a single slow service must never cascade into a total outage.

Timeouts

Never wait forever. Every network call needs a timeout so a stuck dependency does not tie up your threads:

// fail fast instead of hanging
client.get("/users/42").timeout(2, SECONDS)

Retries – carefully

Transient failures often succeed on a second try. Retry, but with exponential backoff and a small limit, and only for operations that are safe to repeat (idempotent). Blind, immediate retries can turn a hiccup into a self-inflicted denial of service.

Circuit breakers

A circuit breaker watches for repeated failures to a dependency. After a threshold it “opens” and fails calls immediately for a while, giving the struggling service time to recover instead of hammering it:

Closed  -> calls flow normally
Open    -> calls fail fast (dependency is unhealthy)
Half-open -> let a few through to test recovery

Libraries like Resilience4j implement this pattern for you.

Graceful degradation

When a dependency is down, return a sensible fallback rather than an error: show cached prices, hide a recommendations panel, or queue the work for later. A partly-working page beats a broken one.

Observability

You cannot fix what you cannot see. Distributed systems need centralised logging, metrics, and distributed tracing (a request id followed across services) to understand where time goes and what failed.

Key points

• Treat slow and failed calls as normal; stop them from cascading.
• Put a timeout on every network call to fail fast.
• Retry only idempotent operations, with backoff and a limit.
• Circuit breakers and fallbacks keep the system usable when a dependency fails; tracing makes failures visible.