Design an API Rate Limiter

A rate limiter caps how many requests a client can make in a window of time, such as 100 requests per minute per API key. It protects services from abuse and runaway clients, keeps costs predictable, and enforces pricing tiers. The interesting part is doing it consistently across many servers without adding latency to every call.

Requirements#

Functional

• Limit requests per client (user ID, API key or IP), per endpoint if needed.
• Support different rules: 10 logins per minute, 1,000 API calls per hour.
• Rejected requests get 429 Too Many Requests with a Retry-After header.

Non-functional

• Adds under a millisecond or two to each request.
• Works across a fleet of API servers, so limits are global, not per box.
• If the limiter itself breaks, the API should keep working (fail open) for most endpoints.

Where the limiter lives#

Putting the check in the API gateway (or a sidecar) means services don't each reimplement it. Counters live in Redis, so every gateway instance sees the same numbers.

Algorithms#

Token bucket

Each client has a bucket holding up to b tokens, refilled at r tokens per second. A request takes one token; if the bucket is empty, it's rejected.

It allows short bursts (good for real users) and is cheap: store just tokens and lastRefill per client. Stripe and AWS use variants of this.

Leaky bucket

Requests join a queue that drains at a fixed rate. It smooths traffic into a steady stream, which suits jobs that call a fragile downstream system, but it adds queueing delay and handles bursts poorly.

Fixed window counter

Count requests in each calendar minute: key user:42:2026-10-31T10:15. Simple, but a client can send 100 requests at 10:15:59 and 100 more at 10:16:00, which is double the limit in two seconds.

Sliding window

The sliding log stores a timestamp per request and counts those in the last 60 seconds. It's exact but uses memory per request. The sliding window counter blends the current and previous fixed windows:

estimate = current_count + previous_count × (overlap of previous window)

It is nearly as accurate as the log and as cheap as fixed windows, which makes it a popular default.

Making it atomic#

Two gateway servers reading and writing the same counter can race: both read 99, both allow, both write 100. Run the check-and-update as one atomic step with a Redis Lua script:

-- KEYS[1] = bucket key, ARGV = capacity, refill_per_ms, now_ms
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local capacity, rate, now = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local tokens = tonumber(b[1]) or capacity
local ts = tonumber(b[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * rate)
local allowed = tokens >= 1
if allowed then tokens = tokens - 1 end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate))
return allowed and 1 or 0

Deep dives#

• Latency: one Redis round trip per request is usually under a millisecond in the same region. For extreme traffic, keep a small local token allowance per gateway and sync with Redis in batches, accepting slight overshoot.
• Multiple regions: global exact limits need cross-region coordination, which is slow. Most systems enforce limits per region, or split a client's quota between regions.
• Failure: if Redis is down, fail open for normal endpoints and fail closed for sensitive ones like login or payments.
• Headers: return X-RateLimit-Limit, X-RateLimit-Remaining and Retry-After so well-behaved clients can back off.