Resilience Patterns
Retry with Exponential Backoff
async function withRetry<T>(
fn: () => Promise<T>,
maxRetries = 3,
baseDelay = 1000,
): Promise<T> {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
if (attempt === maxRetries) throw error;
if (!isRetryable(error)) throw error;
const delay = baseDelay * Math.pow(2, attempt) + Math.random() * 1000;
await new Promise((r) => setTimeout(r, delay));
}
}
throw new Error('Unreachable');
}
function isRetryable(error: unknown): boolean {
if (error instanceof Error && 'status' in error) {
const status = (error as { status: number }).status;
return status === 429 || status >= 500;
}
return true;
}
Circuit Breaker
class CircuitBreaker {
private failures = 0;
private lastFailure = 0;
private state: 'closed' | 'open' | 'half-open' = 'closed';
constructor(
private threshold = 5,
private resetTimeout = 30000,
) {}
async execute<T>(fn: () => Promise<T>, fallback?: () => T): Promise<T> {
if (this.state === 'open') {
if (Date.now() - this.lastFailure > this.resetTimeout) {
this.state = 'half-open';
} else {
if (fallback) return fallback();
throw new Error('Circuit breaker is open');
}
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
if (fallback) return fallback();
throw error;
}
}
private onSuccess() { this.failures = 0; this.state = 'closed'; }
private onFailure() {
this.failures++;
this.lastFailure = Date.now();
if (this.failures >= this.threshold) this.state = 'open';
}
}
// Usage
const breaker = new CircuitBreaker(5, 30000);
const result = await breaker.execute(
() => externalApi.call(),
() => cachedResult, // fallback
);
resilience4j (Java/Spring)
@CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
@Retry(name = "paymentService")
@TimeLimiter(name = "paymentService")
public CompletableFuture<Payment> processPayment(Order order) {
return CompletableFuture.supplyAsync(() -> paymentClient.charge(order));
}
public CompletableFuture<Payment> paymentFallback(Order order, Throwable t) {
return CompletableFuture.completedFuture(Payment.pending(order.getId()));
}
# application.yml
resilience4j:
circuitbreaker:
instances:
paymentService:
slidingWindowSize: 10
failureRateThreshold: 50
waitDurationInOpenState: 30s
retry:
instances:
paymentService:
maxAttempts: 3
waitDuration: 1s
exponentialBackoffMultiplier: 2
Timeout Pattern
async function withTimeout<T>(fn: () => Promise<T>, ms: number): Promise<T> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), ms);
try {
return await fn();
} finally {
clearTimeout(timer);
}
}
Pattern Selection
| Pattern | Use When |
|---|
| Retry | Transient failures (network, 5xx, 429) |
| Circuit breaker | Repeated failures to same dependency |
| Timeout | Preventing hung connections |
| Bulkhead | Isolating resource pools per dependency |
| Fallback | Degraded but available service |
Anti-Patterns
| Anti-Pattern | Fix |
|---|
| Retrying non-idempotent operations | Only retry idempotent requests (GET, PUT with same key) |
| No jitter on retry delay | Add random jitter to prevent thundering herd |
| Circuit breaker per request (not per service) | Share breaker instance across requests to same service |
| No fallback for open circuit | Provide cached/degraded response |
| Retrying 400 errors | Only retry transient errors (429, 5xx, network) |
Production Checklist
- Retry with exponential backoff + jitter
- Circuit breaker on all external service calls
- Timeouts on all outbound requests
- Fallback responses for critical paths
- Health check endpoints
- Monitoring: circuit breaker state, retry counts
