Who does your API limit protect? Designing rate limits by unit, concurrency, and communication

EDITORIAL PRACTICE

Decide first whom the limit protects

A rate limit serves more than one purpose: protecting infrastructure capacity, preserving fairness among clients, deterring abuse and brute force, and containing expensive operations. Which purpose dominates changes both the unit and the value. A limit set vaguely, say 100 requests per second, without that decision tends to protect nothing important while blocking traffic that did not need blocking.

Stripe's design shows the separation clearly. Request rates are measured per account per second, with a global limit and per-endpoint limits combined. Separately, long-running requests are governed by a concurrency limit, a different yardstick. Bounding flow and occupancy independently prevents both floods of short requests and accumulation of heavy ones.

GitHub's design runs on a different clock. Authenticated users get a primary limit of 5,000 requests per hour, inside which secondary limits constrain concurrency and content-generating requests. Smoothing bursts per second and managing totals per hour assume different usage patterns. The provider's first job is to articulate which shape its own API has: is the fear a short spike, or total volume growth?

Values should come from measurement: the current usage distribution, the patterns of top clients, and the cost of each operation. The principle is to place limits slightly above normal use. A value chosen without data will either reject legitimate traffic or be too loose to protect anything. Take the distribution before launch and set initial values with evidence.

A limit also becomes a contract with clients the moment it exists. What is returned when it is hit, how that is communicated, and how the client can resolve it are all part of the specification. The experience of touching a limit often shapes the API's reputation more than the limit itself.

Purpose also carries an accountability dimension. When a limited client asks why they were stopped, being able to answer immediately, capacity, fairness, or cost, is support quality itself. A limit whose purpose is not shared internally gets explained ad hoc on every ticket, and the answers drift. A limit value is a number and also a script for explaining it.

Internal and external use can also collide. Your own batch jobs consuming the capacity and squeezing out external clients is a failure of governance. Either subject internal clients to the same limit discipline, or reserve an internal allowance inside the capacity plan. Provider-side design starts by counting internal systems as clients too.

Sources: [1][2]

EDITORIAL PRACTICE

Split the units: global, per-endpoint, per-resource

Limit units divide into at least three layers: an account-wide limit, per-endpoint limits, and limits on specific resources. The global layer prevents total runaway, per-endpoint limits protect expensive operations, and per-resource limits cool down hot objects. Any single layer leaves some threat uncovered.

At Stripe, the live global limit is 100 requests per second, individual endpoints generally allow 25 per second, and resource-specific limits also exist, as the reason codes reveal. Operations differ in cost, so there is no reason their limits should be uniform; creating a payment and reading one back need different protection.

Unit choice also interacts with billing and contracts. Account-level limits map naturally to fairness per contracting party. But when several systems share one account, one runaway system can starve the others. Finer units per API key or per use case prevent collateral damage at the price of more management and explanation. How fine to go should follow how clients actually use the API.

Measurement mechanics need design too. Fixed windows, sliding windows, and token buckets behave differently near boundaries. Per-second limits make burst tolerance the user experience; hourly limits make the wait for recovery the issue. Decide measurement and recovery as a pair, in a form you can explain to clients.

The table of units and values belongs in public documentation, not internal notes. Only clients who know the limits can design their own throttling and retries. A hidden limit is indistinguishable from an unexplained failure.

Per-resource limits matter most for APIs with popular objects. When reads or writes concentrate on one target, total traffic can look normal while the store holding that target screams. Bounding concurrency or rate per object cools local hotspots without lowering global numbers. Design it together with hotspot detection.

Per-plan limits are also part of unit design. Differentiating free and paid tiers by value is standard, but the operational differences, how fast increases are granted and what support covers, belong in the design too. A limit is part of the price list and must be speakable in product language. Limits decided by engineering alone tend to become numbers that sales and support cannot explain.

Sources: [1][2]

EDITORIAL PRACTICE

Rate and concurrency are different limits

An easily missed point: request frequency and concurrency are different things. Low frequency can still pile up concurrency when each request runs long; short requests can run at high frequency with little occupancy. Binding both with one number over- or under-protects against one of the two threats.

Stripe makes the separation explicit. Concurrency limits are distinct from rate limits, with a live-mode global default of 50 in-progress requests. The reason code returned on rejection tells the client whether the cause was concurrency, so the client can distinguish 'sending too fast' from 'holding too long'. Being able to name the cause directly speeds correction.

GitHub's secondary limits carry the same idea: concurrent requests are capped at 100, combined with a points system and per-minute and per-hour caps on content-generating requests. Weighting heavy operations with points is another answer to reflecting cost differences in limits.

Translated to your own API, the procedure is: inventory the operations and measure average and worst-case durations. Give duration-dominated operations a concurrency limit, count-dominated ones a rate limit, and both where both bite. Mapping limits to the finite resources behind them, in-memory queues and database connections, turns limit values from arbitrary numbers into expressions of a capacity plan.

Concurrency limits interact awkwardly with clients' parallel batch jobs. For clients with bulk workloads, the structural answer is an asynchronous job API or bulk endpoints. Forcing all bulk demand through synchronous limits guarantees permanent friction.

Concurrency control interacts with timeout design. When the concurrency budget fills with long requests, followers wait and eventually time out. If the client's timeout is shorter than your processing horizon, clients retry requests with unknown outcomes and squeeze concurrency further, a vicious cycle. Publish the limit, the processing time ceiling, and the recommended timeout as a set of three.

Whether to queue is another decision. Briefly holding requests until a slot frees absorbs momentary concentration but adds latency variance. Rejecting immediately with 429 hands control to the client but makes retry design the client's problem. Which fits depends on the API's use. At minimum, state the policy: do you hold, or do you reject?

Sources: [1][2]

EDITORIAL PRACTICE

Communication is the specification: headers, reasons, retry guidance

The response at the moment a limit is hit is core specification. A bare 429 tells the client nothing. The limit type, the ceiling, the remaining quota, the recovery time, and whether retrying is allowed should come back machine-readably in headers and body; that is what makes automatic client recovery possible.

GitHub's disclosure is thorough: x-ratelimit-limit, remaining, used, reset, and resource headers expose ceiling, consumption, and recovery time, letting clients self-throttle before exhaustion. Just as important, the documentation states these headers are the authoritative source, privileging runtime values over prose, which stabilizes client implementations.

Stripe's hallmark is disclosing the reason. The Stripe-Rate-Limited-Reason header on a 429 identifies whether the limit was global rate, endpoint rate, global concurrency, endpoint concurrency, or resource-specific. Note also that a 429 without that header is not a rate limit at all; it may be a lock timeout. The same status code with different causes demands different responses, and returning the reason breaks chains of wrong remedies.

Include recovery guidance as well. Secondary limits are accompanied by a retry-after header. If clients cannot learn when to retry, they guess and keep hammering, prolonging the limit. Telling them how long to wait serves the client and lowers your own load at the same time.

The error body should pair a human explanation with pointers to durable remedies: where to request an increase, how to check usage, whom to consult when usage exceeds assumptions. The moment of hitting a limit is the moment the client most needs guidance. Handle that moment carelessly and distrust accumulates quietly.

Sources: [2][1]

EDITORIAL PRACTICE

Separate fairness design from abuse handling

Fairness for normal clients and abuse deterrence are separate designs. Fairness is served by per-contract limits grounded in measurement. Abuse is handled with different tools: tight limits on unauthenticated access, immediate key shutdown on leakage, anomaly detection. Mixing abuse-grade strictness into everyday limits makes legitimate clients pay for attackers.

Limits differing sharply by authentication state are common across providers. GitHub allows unauthenticated clients only 60 requests per hour per IP, while authenticated users get 5,000. Wide for identified clients, narrow for anonymous ones: this asymmetry is one of the basic shapes of limit design.

Adaptive limits are another option. Stripe's fraud tooling describes dynamically tightening limits in response to detected attacks. Keep normal limits generous for experience, and tighten only under anomaly, but decide activation conditions, release conditions, and client notification in advance, because dynamic changes cost explainability.

Key-level blocking and emergency limit changes belong in operational procedures. When a key leaks or a client runs away, who can flip the limit, with what authority, and how fast? Limit design is value design and emergency-operation design at once. Without the procedure, the designed limit will not function in an emergency.

Audit fairness as well. Check that limits do not unfairly squeeze one usage pattern, or reject the legitimate peaks of large clients. Review the breakdown of limited requests periodically and track the false-positive share. A limit is not finished when deployed; auditing how it lands is part of operating it.

Leaked keys sit on the boundary between limits and authentication. Revoking a key is the last resort because it stops the business. Graduated responses, temporarily tightening the limit, blocking one source, let you contain damage while coordinating a rotation with the client. Build the emergency staircase into the calm-time design.

Decide in advance how to handle clients who try to evade limits. Multiplying accounts or sharding keys to escape ceilings intersects with your terms of service. Technical detection, contractual remedies, and a ladder from warning to suspension should not be improvised case by case, or consistency will be questioned. A limit is a technology and also a discipline of operating the service.

Sources: [2][1]

EDITORIAL PRACTICE

Increases, changes, and notice: operating with clients

Limits are not fixed. Provide a path to raise them as legitimate use grows: an application channel, review criteria, and an expected response time. Without a legitimate path, clients resort to workarounds like key sharding, and you lose visibility into real usage. Designing increases as a formal procedure is also a matter of control.

For predictable peaks such as business seasons, temporary increases on request are worth offering. In seasonal domains like payments and media, aligning limits with peaks reduces both outages and lost opportunity. Publish application deadlines and required information so clients can build it into their plans.

Treat limit changes, especially decreases or measurement changes, as breaking changes: advance notice, a transition period, publication of the new values. Tightening silently multiplies sudden failures in client production systems and floods support with root-cause questions. Derive the notice list from the clients actually touching the limits.

Visualizing usage is also the provider's job. Dashboards or APIs where clients can see their consumption and their distance to the ceiling enable self-throttling and timely increase requests. As with GitHub's rate-limit status endpoint, making consumption visible is one of the cheapest ways to reduce limit friction.

Design the support path too. On limit inquiries, response speed is decided by how quickly the client's consumption and the limit breakdown can be pulled up. Reason codes are an investment not only in client self-service but also in support triage.

In change practice, do not schedule by code convenience alone. Landing a limit change in a client's busy season or fiscal closing makes impact analysis miserable. Draw the change calendar against the clients' business calendars. The habit of scheduling by provider release convenience causes the same trouble for changes beyond limits.

Documentation should keep history, not just current values: when each value changed and why. Clients investigating their own incidents sometimes need the limit that was in force at a past moment. A public change history enables that investigation and reduces inquiries to the provider. Transparency is also a support-cost measure.

Sources: [1][2]

EDITORIAL PRACTICE

Test, document, and audit the limits

Limit implementations are test targets. Does the boundary return 429 correctly, does the reason code match the cause, is the recovery time computed right, and are there paths that bypass the limit? Avoid discovering limit behavior for the first time in production; include limit activation and recovery in load-test scenarios.

Documentation should publish more than values: units, measurement, recovery, the meaning of reason codes, and the increase procedure. Where values differ by version or plan, state the difference. Divergence between documentation and actual behavior silently breaks client implementations, so behavior changes and document changes must ship together.

Records of limited requests are material for audit and improvement. Which clients touch which limits, and how often? Clients permanently hugging the ceiling are candidates for proactive increase offers and inputs to capacity planning. Read the limit log not as a record of refusals but as a record of demand.

Dogfooding helps as well. If your own systems and monitoring use the same API, their consumption is a representative sample of normal use. If the limits reject your own legitimate traffic, external clients are suffering the same. Internal complaints are a free audit of limit design.

In sum, provider-side rate limiting is a bundle of designs: unit, concurrency, communication, fairness, operations, and verification. It is not finished by picking a number. A limit is complete only when a client who touches it can understand the reason, recover, and ask for help when needed; that state protects stability and trust at the same time.

Testing from the client's seat is also valuable. Verify that major SDKs and usage patterns recover as expected when they touch a limit. Misread reason codes and overlooked headers are common in client implementations. Checking representative client implementations on the provider side also exposes ambiguities in your own documentation early.

Finally, put the limit values themselves under periodic review. Usage shifts, and optimization lowers processing costs. Do not canonize launch-day values; move them when the measured distribution moves. A limit is not an immutable decision but an operational artifact that tracks capacity plans and real usage. That posture turns a limit from an annoying wall into a trustworthy promise.

Sources: [1][2]