Photo by Albert Stoynov on Unsplash
Connection Admission Control: Knowing When to Say No
When a service gets overwhelmed, accepting every incoming connection is often the worst thing it can do. Connection admission control is the practice of deliberately rejecting new connections when a system approaches its capacity limits, preserving stability for existing work rather than grinding to a halt trying to serve everyone.
The pattern shows up across distributed systems, from database connection limits to load balancer health checks to application-layer request queues. The core insight is the same: there’s a threshold beyond which accepting more work makes total throughput worse, not better.
The Overload Collapse Problem
Systems under load don’t fail gracefully by default. As utilization increases, latency rises. Slow responses cause clients to retry. Retries create more load. Connection state accumulates in half-open TCP handshakes, idle workers waiting on locks, and memory buffers that never get freed because garbage collection can’t keep up.
The tipping point often comes suddenly. A service handling 10,000 requests per second at 200ms latency might completely seize up at 11,000 requests per second, serving effectively nothing as threads thrash, memory swaps, and every operation times out. Admission control prevents this collapse by capping the work in flight.
Where Admission Happens
The simplest form is a hard connection limit. Databases like PostgreSQL have max_connections, beyond which new connection attempts immediately fail. This protects the database process from exhausting file descriptors, spawning too many backend processes, or running out of memory for connection buffers.
Application servers use similar limits through connection pools and worker thread counts. An HTTP server with 200 worker threads might queue incoming requests up to a certain depth, then start returning 503 Service Unavailable. The queue provides burst absorption, the limit prevents unbounded queuing.
More sophisticated admission control looks at system health metrics rather than just counting connections. A service might monitor CPU usage, memory pressure, or request latency percentiles. When these cross thresholds, it starts rejecting new work even if theoretical capacity remains. This adaptive approach accounts for the fact that not all requests cost the same.
Fast Failure vs Slow Death
Rejecting a connection quickly is better than accepting it and failing slowly. A client that gets an immediate connection refused error can retry against a different backend, fall back to a cache, or return a degraded response. A client that connects successfully but then waits 30 seconds for a timeout learns nothing useful and ties up resources on both sides.
This is why many systems implement admission control at multiple layers. A load balancer might refuse to forward requests to an unhealthy backend. The backend itself might reject requests its queue is full. The database the backend talks to might refuse new connections. Each layer protects itself rather than assuming upstream or downstream components will do it.
The Queueing Theory Foundation
The math behind admission control comes from queueing theory. Little’s Law states that the average number of items in a system equals the arrival rate multiplied by the average time each item spends in the system. As utilization approaches 100%, wait time grows exponentially in most queue models.
Admission control effectively caps utilization below 100%. A service might target 80% utilization under normal load, leaving headroom for bursts and ensuring most requests see low latency. Work that would push utilization higher gets rejected immediately rather than queueing indefinitely.
The rejection rate becomes a key metric. A system rejecting 0.1% of requests due to admission control is probably well-tuned. A system rejecting 10% needs more capacity or better traffic shaping upstream. Zero rejections might mean limits are set too conservatively, leaving performance on the table.
Implementation Patterns
Token bucket and leaky bucket algorithms provide rate-based admission control, allowing bursts while enforcing average limits. Concurrency limiters count in-flight requests and reject new ones beyond a threshold. Adaptive limiters adjust thresholds dynamically based on observed latency or error rates.
The hard part is tuning. Set limits too low and you waste capacity. Set them too high and you fail to prevent overload. The right threshold depends on workload characteristics, downstream dependencies, and acceptable latency profiles. Production tuning often involves load testing to find the knee of the latency curve, then setting limits with some margin below that point.
Coordination Challenges
In distributed systems, admission control becomes trickier. If you have 100 backends and each independently limits concurrency to 100, your total system capacity is 10,000 concurrent requests, but any individual backend might reject work while others sit idle if load is imbalanced.
Centralized admission control can help but introduces its own bottleneck. Distributed rate limiting with gossip protocols or shared state stores trades consistency for availability. Many systems punt on global coordination entirely and rely on load balancers to spread work evenly, letting each backend protect itself independently.
The alternative is back-pressure propagation. Services signal overload to their callers through error codes, latency, or explicit flow control mechanisms. Callers reduce their request rate in response. This creates a feedback loop that naturally sheds load without hard-coded limits, though it requires cooperation across service boundaries and can be slower to react than local admission control.