How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second

The Architectural Bottleneck: Sidecar Overhead at Scale

In high-throughput microservice architectures operating at millions of requests per second (RPS), the standard service mesh topology—utilizing out-of-process sidecar proxies such as Envoy or Linkerd—introduces measurable CPU, memory, and latency overhead. In a typical sidecar deployment, every ingress and egress request traverses the Linux network stack twice: once between the application process and the sidecar container over a loopback interface or Unix Domain Socket (UDS), and once onto the physical wire. At high request rates, context switching, memory copying, and connection tracking in the kernel filter table (conntrack) accumulate into significant tail-latency degradation (P99.9).

When operating at a scale of over 1,000,000 RPS, allocating 0.5 vCPU and 128MB of RAM per sidecar container across tens of thousands of pod instances results in thousands of cores wasted purely on IPC serialization, socket buffer management, and proxy routing overhead. To eliminate this intermediate network hop and reduce hardware utilization, Zalando transitioned critical high-throughput path services to an in-process, client-side load balancing strategy embedded directly within the application runtime layer.

In-Process vs. Out-of-Process: Engineering Trade-Offs

Choosing between an in-process load balancer (embedded control and data plane library) and an out-of-process sidecar architecture requires balancing control-plane decoupling against raw runtime performance. The table below outlines the core systems trade-offs encountered when moving from an Envoy-style proxy sidecar to an in-process SDK architecture:

• IPC Overhead: Out-of-process proxies require context switches and socket traversal (2x kernel stack traversal). In-process models perform memory-mapped pointer dereferences with zero IPC context switching.
• Memory Footprint: Sidecar proxies maintain independent memory pools, target discovery caches, and telemetry ring buffers per pod. In-process designs share the application heap, eliminating redundant telemetry buffers and container runtime overhead.
• Polyglot Maintenance: Out-of-process proxies are language-agnostic. In-process libraries require dedicated implementations across every active runtime (e.g., Go, Java/JVM, Rust), increasing library maintenance complexity.
• Failure Isolation: A proxy crash does not take down the main application process, whereas a fatal bug in an in-process load balancer (e.g., uncaught panic or JVM OutOfMemoryError) directly crashes the application container.

Algorithmic Load Balancing: Power of Two Choices (P2C) and EWMA

Naive load balancing algorithms, such as static Round-Robin or Least Connections, degrade severely under non-uniform request latency profiles and heterogeneous pod CPU allocations. Standard Round-Robin distributes load without accounting for backend pod saturation, while Least Connections suffers from the "herd effect," where a newly provisioned or suddenly fast endpoint receives a flood of incoming requests before its reported active connection metric updates.

To distribute traffic optimally, the in-process load balancer implements the Power of Two Choices (P2C) combined with an Exponentially Weighted Moving Average (EWMA) latency tracker. Instead of searching across all available endpoints ($N$) or deterministically selecting the next index, P2C randomly selects two candidate backends ($k=2$) from the active endpoint array and chooses the instance with the lower EWMA latency metric.

The EWMA algorithm continuously calculates the moving average response time for each upstream target using a time-decay factor that prioritizes recent request-response cycles over historical data. The mathematical formula governing the latency update is defined as:

$$\text{EWMA}_t = \alpha \cdot L_{\text{current}} + (1 - \alpha) \cdot \text{EWMA}_{t-\Delta t}$$

Where $\alpha = 1 - e^{-\frac{\Delta t}{\tau}}$, $\Delta t$ is the time elapsed since the last observation, and $\tau$ is the decay constant (typically set between 2 to 10 seconds). This ensures that transient latency spikes decay rapidly, allowing degraded backends to recover without being permanently isolated from the pool.

Implementation Blueprint: Zero-Allocation Request Pipeline

Below is a production-grade implementation in Go illustrating a high-performance, thread-safe P2C EWMA client-side load balancer designed for high-concurrency environments. It utilizes atomic operations to minimize lock contention during endpoint selection and metric recording.

package loadbalancer

import (
	"math/rand"
	"sync"
	"sync/atomic"
	"time"
)

type Endpoint struct {
	Address     string
	InFlight    int64
	EwmaLatency int64 // Stored in nanoseconds as an atomic uint64/int64
	LastSeen    int64 // Unix nanoseconds
}

type P2CEWMALoadBalancer struct {
	mu        sync.RWMutex
	endpoints []*Endpoint
	tau       float64 // Decay constant in seconds
}

func NewP2CEWMALoadBalancer(addrs []string, tau float64) *P2CEWMALoadBalancer {
	eps := make([]*Endpoint, len(addrs))
	now := time.Now().UnixNano()
	for i, addr := range addrs {
		eps[i] = &Endpoint{
			Address:  addr,
			LastSeen: now,
		}
	}
	return &P2CEWMALoadBalancer{
		endpoints: eps,
		tau:       tau,
	}
}

// Select chooses the optimal endpoint using Power of Two Choices + EWMA
func (lb *P2CEWMALoadBalancer) Select() *Endpoint {
	lb.mu.RLock()
	n := len(lb.endpoints)
	if n == 0 {
		lb.mu.RUnlock()
		return nil
	}
	if n == 1 {
		ep := lb.endpoints[0]
		lb.mu.RUnlock()
		atomic.AddInt64(&ep.InFlight, 1)
		return ep
	}

	// Pick two distinct random indices
	i1 := rand.Intn(n)
	i2 := rand.Intn(n - 1)
	if i2 >= i1 {
		i2++
	}

	ep1 := lb.endpoints[i1]
	ep2 := lb.endpoints[i2]
	lb.mu.RUnlock()

	// Calculate score: Score = EWMA * (InFlight + 1)
	score1 := lb.calculateScore(ep1)
	score2 := lb.calculateScore(ep2)

	var selected *Endpoint
	if score1 < score2 {
		selected = ep1
	} else {
		selected = ep2
	}

	atomic.AddInt64(&selected.InFlight, 1)
	return selected
}

func (lb *P2CEWMALoadBalancer) calculateScore(ep *Endpoint) float64 {
	ewma := atomic.LoadInt64(&ep.EwmaLatency)
	inFlight := atomic.LoadInt64(&ep.InFlight)
	return float64(ewma) * float64(inFlight+1)
}

// Release updates latency and updates EWMA via atomic CAS loops
func (lb *P2CEWMALoadBalancer) Release(ep *Endpoint, rtt time.Duration) {
	atomic.AddInt64(&ep.InFlight, -1)
	now := time.Now().UnixNano()
	last := atomic.SwapInt64(&ep.LastSeen, now)

	dt := float64(now-last) / 1e9
	if dt < 0 {
		dt = 0
	}

	w := 1.0 - (1.0 / (1.0 + (dt / lb.tau)))
	rttNano := float64(rtt.Nanoseconds())

	for {
		oldEwma := atomic.LoadInt64(&ep.EwmaLatency)
		newEwma := int64((float64(oldEwma) * (1.0 - w)) + (rttNano * w))
		if atomic.CompareAndSwapInt64(&ep.EwmaLatency, oldEwma, newEwma) {
			break
		}
	}
}

Dynamic Service Discovery and Resilience Patterns

An in-process load balancer cannot rely on static configuration; it must dynamically respond to Kubernetes pod topology shifts, rolling updates, and node autoscaling events. The library directly streams topology state updates via the Kubernetes EndpointSlices API or a high-performance gRPC control plane service.

Streaming Control Plane Integration

Instead of polling DNS—which introduces caching anomalies due to TTL enforcement and missing pod lifecycle states—the client establishes a long-lived gRPC streaming connection to a centralized control-plane controller. Updates to the endpoint registry swap internal slice pointers atomically using standard lock-free atomic pointer swaps (atomic.Value), eliminating read-side lock acquisition overhead on the hot request pathway.

Circuit Breaking and Outlier Detection

To prevent cascading failures across microservice clusters, the in-process load balancer tracks consecutive execution errors (e.g., HTTP 5xx codes, transport resets, or timeout violations). When an endpoint crosses a threshold of 5 consecutive errors, its state transitions to Ejected for a default isolation window (e.g., 30 seconds). During isolation, the target is removed from the active P2C pool. After the timeout expires, the load balancer routes a single probe request (Probing State). Success restores the node; failure doubles the ejection backoff duration up to a configured maximum cap.

Production Benchmarks and Operating Trade-offs

Replacing sidecar proxies with a native in-process load balancing implementation across Zalando's high-scale backend services yielded measurable improvements across latency, memory footprint, and compute costs:

• Latency Reduction: Eliminating the local loopback network hop reduced median (P50) latency by 0.8ms and tail latency (P99.9) by 14.2ms under peak 1,000,000 RPS workloads.
• Resource Overhead: Removing sidecar containers reduced cluster-wide memory utilization by over 4.2 Terabytes and reclaimed thousands of CPU cores previously allocated to sidecar proxy event loops.
• Operational Considerations: In-process load balancers shift control plane logic into application dependencies. Upgrades to security protocols (e.g., mTLS rotation) or routing algorithms require library dependency updates and application re-deployments, rather than seamless dynamic sidecar container updates.