Every API you rely on, from payment processors to social platforms, has a rate limiter sitting somewhere between the internet and its core logic. It rarely gets attention until it rejects a request with a 429 status code, but rate limiting is one of the few pieces of infrastructure that has to be correct, fast, and fair all at once. Understanding how it actually works reveals a lot about the tradeoffs baked into distributed systems.

Why Rate Limiting Exists

At its core, rate limiting solves a resource allocation problem. Any shared system, whether it’s a database, a third-party API, or a compute cluster, has a finite capacity. Without limits, a single misbehaving client, a runaway retry loop, or a traffic spike can degrade service for everyone else. Rate limiting enforces fairness and protects the system’s tail latency, since a handful of greedy clients can otherwise dominate resource usage in ways that hurt the p99 experience for well-behaved ones.

It also serves as a defense mechanism. Aggressive scraping, credential stuffing, and denial-of-service attempts all look, at a basic level, like “too many requests too fast.” A well-tuned rate limiter catches a lot of this before it ever reaches application logic.

The Core Algorithms

Most rate limiting implementations boil down to a handful of well-known approaches, each with different tradeoffs around burstiness, memory usage, and precision.

Fixed window counters are the simplest: count requests in a time window (say, one minute) and reset the counter when the window ends. They’re cheap to implement but have an edge problem. A client can send a full quota of requests at the tail end of one window and another full quota at the start of the next, effectively doubling their allowed rate for a brief period.

Sliding window approaches fix that by tracking requests over a rolling interval rather than a fixed boundary, often using a weighted combination of the current and previous window’s counts. This smooths out the edge-burst problem at the cost of slightly more bookkeeping.

Token bucket is probably the most widely used algorithm in production systems. Each client has a bucket that fills with tokens at a steady rate up to some maximum capacity. Every request consumes a token, and requests are rejected once the bucket is empty. The appeal is that it naturally allows short bursts (as long as tokens have accumulated) while still enforcing a long-term average rate. This matches real traffic patterns better than a strict per-second cap.

Leaky bucket is the token bucket’s mirror image: requests enter a queue (the bucket) and are processed at a fixed, steady rate, with excess requests either queued or dropped once the bucket overflows. It trades burst tolerance for smoother, more predictable output, which matters when the downstream system genuinely cannot handle spikes regardless of average load.

Where It Gets Hard

The algorithm is the easy part. The hard part is doing this correctly across a distributed fleet of servers. A single-node in-memory counter is trivial, but as soon as requests for the same client can land on any of dozens of servers, you need a shared source of truth. This usually means a fast, centralized (or replicated) store like Redis, using atomic operations to avoid race conditions where two servers both think they have quota left.

That centralized store introduces its own latency and availability tradeoffs. Some systems accept eventual consistency and slightly loose enforcement in exchange for lower latency; others accept the extra network round trip to a shared store for strict accuracy. There’s also the question of what to rate limit by: IP address, API key, user ID, or some combination, each with different implications for how easily limits can be evaded or how legitimate shared-IP traffic (like corporate NATs) gets penalized.

The Practical Tradeoff

Choosing a rate limiting strategy is really about deciding what kind of traffic pattern you want to tolerate. Token buckets are the default choice for APIs that expect bursty, human-driven traffic. Leaky buckets fit systems with hard downstream throughput ceilings, like queue-backed workers. Sliding windows suit cases where precise, gradual enforcement matters more than implementation simplicity.

None of these algorithms are new or exotic, but the way they’re combined, tuned, and distributed is where the real engineering work happens. It’s a good reminder that some of the most important infrastructure decisions are invisible right up until the moment they aren’t.