Designing Rate Limiting for Distributed APIs

Rate limiting is often introduced as a simple rule: allow a client a fixed number of requests per minute. In production, however, the design affects availability, fairness, cost, and customer experience. A limit that is too permissive may fail to protect an expensive dependency; one that is too strict can reject legitimate traffic during normal bursts.

Distributed deployments add another challenge. If each application instance tracks requests independently, a client can exceed its intended quota by sending traffic to multiple instances. A reliable design needs a clearly defined policy, shared enforcement where necessary, and an explicit plan for when the rate-limiting system itself is unavailable.

Start with the policy, not the algorithm

Before choosing a counter or data store, define what is being limited and why. Common policies include:

• Per identity: Limits requests by authenticated user, API key, or tenant.
• Per network source: Limits unauthenticated traffic by IP address or network range.
• Per operation: Applies a stricter quota to expensive endpoints, such as report generation.
• Global protection: Caps aggregate traffic to a dependency or service.

These dimensions can be combined. For example, an API may enforce a generous per-tenant request quota, a stricter per-endpoint limit, and a global concurrency cap for calls to a costly supplier. Keeping these controls distinct makes the policy easier to explain and tune.

Define the response to each policy as well. A rejected request should usually return HTTP 429 Too Many Requests, with a Retry-After value when the service can estimate when the client may try again. Document whether a quota is shared across a tenant's users, when it resets, and whether unused capacity carries forward.

Choose a model that matches the traffic

No single rate-limiting algorithm is ideal for every workload:

Fixed window

A fixed-window counter allows a maximum number of requests during each interval—for example, 100 requests per minute. It is simple and inexpensive to implement, but requests can cluster at a window boundary. A client might send nearly 100 requests just before the minute ends and another 100 immediately after it.

Sliding window

A sliding-window log tracks request timestamps and counts those inside the current interval. It provides precise enforcement but stores more state and requires cleanup. A sliding-window counter approximates the result using counts from the current and previous windows, reducing storage at the cost of some approximation.

Token bucket

A token bucket refills at a configured rate up to a maximum capacity. Each request consumes a token. The refill rate controls sustained traffic, while the bucket capacity allows short bursts. This often maps well to APIs where clients need occasional bursts but should not maintain a high request rate indefinitely.

Leaky bucket

A leaky bucket processes requests at a controlled rate, smoothing bursts into a more uniform flow. This is useful for shaping work sent to a downstream service, though requests may need to wait in a queue rather than receive an immediate allow-or-reject decision.

Pick the simplest model that captures the intended policy. If the main goal is basic abuse protection, a fixed window may be sufficient. If burst tolerance matters, a token bucket may be a better fit.

Make distributed decisions atomic

In a multi-instance service, the enforcement decision must account for requests handled by all instances. A shared store such as Redis can hold quota state, but a separate read followed by a write is not safe: concurrent requests can all observe the same remaining capacity and all be allowed.

Perform the state update and allow-or-reject decision atomically. For a fixed window, this means incrementing the counter and setting its expiry as one operation. For a token bucket, calculate the refill, consume a token if available, and save the new state as one atomic operation, often through a Redis script. Store only the data required for the decision, and set expirations so inactive keys do not accumulate indefinitely.

Use stable, bounded keys such as a policy name, tenant identifier, and route category. Avoid using raw, attacker-controlled strings without validation: unbounded unique keys can create excessive storage and turn the limiter into a resource-exhaustion target. Hashing sensitive identifiers can also reduce exposure in operational tooling.

If the implementation uses timestamps, be deliberate about the clock source and precision. Inconsistent application-server clocks can skew a distributed decision. Keeping the calculation within the shared store or using a trusted time source can make behavior more predictable.

Put identity and proxy handling on a firm footing

The rate-limit key is only as trustworthy as the identity used to construct it. For authenticated requests, use a verified account, API key, or tenant identifier rather than a client-supplied header. For unauthenticated requests, IP-based limits can help, but proxy configuration must be correct.

Do not blindly trust X-Forwarded-For or similar headers from the public internet. Configure the edge proxy to remove untrusted incoming values and set the headers that the application is permitted to use. Otherwise, clients may spoof their source address and evade limits—or cause other clients to share an incorrectly derived key.

Consider both fairness and cardinality. A single per-IP limit may penalize users behind a shared corporate network, while a per-user limit may not protect an unauthenticated login endpoint. Use different identities and policies where the threat model and product behavior differ.

Decide what happens when the limiter is unavailable

A shared rate-limit store is part of the request path and can itself fail. Choose a failure policy by endpoint and risk:

• Fail open: Continue serving requests if the limiter is unavailable. This favors availability but temporarily removes quota enforcement.
• Fail closed: Reject requests when a decision cannot be made. This protects sensitive operations but can turn a limiter outage into an application outage.
• Degraded local limits: Apply a conservative per-instance fallback. This can preserve some protection, but aggregate limits become approximate and vary with the number of instances.

There is no universally correct choice. A public read endpoint may fail open, while a costly operation or sensitive authentication flow may need stricter protection. Make the choice explicit, document it, and test it. Keep limiter timeouts bounded so a slow shared store does not consume all application capacity.

Return useful signals and operate the policy

Clients need enough information to recover without guessing. In addition to a 429 response and Retry-After, APIs may return standardized rate-limit metadata describing the applicable limit and remaining quota. Ensure that the values are consistent with the enforced policy and do not expose internal identifiers.

Operationally, monitor allowed and rejected requests, store latency and errors, key cardinality, and limiter timeouts. Break metrics down by policy and route without adding every user or API key as a metric label; unbounded labels can overwhelm the monitoring system. Logs and traces can carry a suitably protected identity for diagnosis when needed.

Roll out new limits gradually. Start with measurement or shadow evaluation where practical, compare expected rejections with actual traffic, and communicate policy changes to API consumers. Revisit limits as usage patterns and dependency capacity change.

Build tests around guarantees

Tests should verify more than the happy path. Check behavior at window boundaries, burst capacity, concurrent requests, expiry, invalid identities, and limiter-store failures. For shared enforcement, run concurrency tests against the actual atomic operation or a realistic integration environment; a mock that increments a local variable cannot prove distributed correctness.

The important guarantees are understandable: a request is allowed only when the applicable policy permits it, concurrent requests cannot silently overrun a strict quota, and a limiter failure produces the documented service behavior.

The practical balance

An effective rate limiter is not just a counter. It is a policy boundary between clients, application capacity, and downstream dependencies. Define the identity and scope first, select an algorithm that matches burst behavior, make shared decisions atomic, and decide how the system behaves when enforcement is impaired.

Most importantly, treat limits as part of the API contract. When clients understand their quotas and receive useful retry guidance, rate limiting can protect a service without feeling arbitrary. When operators can observe and safely adjust the policy, it becomes a reliability tool rather than an emergency switch.