Rate Limiting Third-Party API Calls from Workers

A worker fleet is a distributed system pointed at someone else's quota. This guide implements the throttling model from Rate Limiting & Throttling Jobs, part of Queue Fundamentals & Architecture, for outbound third-party calls.

Problem Statement

Forty workers call a payment provider limited to 100 requests per second across your whole account. Each worker's client library enforces a local limit of 10/s, which was correct when you ran four workers and is now four times the quota. The provider returns 429s in bursts, your retry logic amplifies them, and twice this month the provider has temporarily suspended the account. You need a limit enforced across the fleet rather than per process, and behaviour when the limit is reached that does not make things worse.

Prerequisites

  • Redis (or another shared store with atomic operations) reachable from every worker.
  • The vendor's documented limits — requests per second, per minute, and any burst allowance.
  • Retry logic that classifies 429 as throttled rather than transient, per retrying only transient errors by exception type.
  • A dead-letter path, because shedding is one of the valid responses to a full bucket.

Step 1 — Understand Why Per-Worker Limits Fail

A per-process limit multiplies by the number of processes, so the effective rate changes every time the fleet scales — including automatically, at exactly the moment load is highest.

Per-worker limits scale with the fleet; a shared bucket does not The upper line shows the fleet's aggregate request rate under per-worker limits: it tracks worker count upward, crossing the vendor quota as the autoscaler adds capacity. The lower line shows a shared token bucket, where aggregate rate stays flat at the quota no matter how many workers are running. Aggregate request rate as the fleet scales vendor quota: 100 req/s per-worker 10/s × N workers — 429s and suspension shared bucket — flat at the quota 4 workers 40 workers The autoscaler is doing its job — the limit is in the wrong place.

Step 2 — Implement a Shared Token Bucket

A token bucket refills at a fixed rate up to a burst capacity, and each request takes a token. Implemented in Lua, the check and the take are atomic, so no two workers can both spend the last token.

-- token_bucket.lua
-- KEYS[1] = bucket key
-- ARGV[1] = refill rate (tokens/sec)  ARGV[2] = capacity
-- ARGV[3] = now (unix seconds, float)  ARGV[4] = tokens requested
local rate, capacity = tonumber(ARGV[1]), tonumber(ARGV[2])
local now, want = tonumber(ARGV[3]), tonumber(ARGV[4])

local state = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now

-- Refill for the elapsed interval, never above capacity.
tokens = math.min(capacity, tokens + (now - ts) * rate)

local allowed = 0
local wait = 0
if tokens >= want then
  tokens = tokens - want
  allowed = 1
else
  wait = (want - tokens) / rate      -- seconds until enough tokens exist
end

redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / rate) * 2)
return {allowed, tostring(wait)}
BUCKET = r.register_script(open("token_bucket.lua").read())

def take_token(name: str, rate: float, capacity: int, n: int = 1):
    allowed, wait = BUCKET(keys=[f"rl:{name}"],
                           args=[rate, capacity, time.time(), n])
    return bool(allowed), float(wait)

Set capacity below the vendor's burst allowance — typically 50–80% of the sustained rate — so a full bucket draining at once still does not trip their limiter.

Step 3 — Decide What Happens When the Bucket Is Empty

This is the design decision, not the algorithm. There are three responses, and the right one depends on the job.

Response Mechanism Use when
Block Sleep for wait, then retake Wait is short (< 1s) and the worker slot is cheap
Defer Re-enqueue with a delay Wait is long, or the slot is scarce — the default
Shed Fail to the DLQ immediately The work is worthless if late (a stale price check) 
@shared_task(bind=True, acks_late=True, max_retries=None)
def sync_invoice(self, invoice_id: str):
    allowed, wait = take_token("payments-api", rate=80, capacity=40)
    if not allowed:
        if wait < 1.0:
            time.sleep(wait)                       # block: cheap and simple
        else:
            # Defer: give the slot back to the pool rather than holding it idle.
            raise self.retry(countdown=min(wait * 1.2, 60))
    call_payments_api(invoice_id)

Blocking is the wrong default at scale: forty workers sleeping for thirty seconds is forty slots doing nothing while other queues back up. Deferring returns the slot to the fleet, at the cost of an extra broker round trip.

Step 4 — Add Per-Tenant Sub-Limits

A global bucket keeps you inside the vendor's quota but lets one tenant consume all of it. Nest a per-tenant bucket inside the global one.

def acquire(tenant: str) -> tuple[bool, float]:
    # Tenant first: a tenant over its share should not consume a global token.
    ok, wait = take_token(f"payments-api:t:{tenant}", rate=8, capacity=8)
    if not ok:
        return False, wait
    ok, wait = take_token("payments-api", rate=80, capacity=40)
    if not ok:
        # Global bucket empty — refund the tenant token so it is not lost.
        r.hincrbyfloat(f"rl:payments-api:t:{tenant}", "tokens", 1)
        return False, wait
    return True, 0.0

The refund matters: without it, a tenant is charged for a request that never happened, so a busy global bucket slowly starves every tenant even after it clears. The fairness reasoning is the same as in preventing tenant starvation with weighted queues.

Step 5 — Trust the Vendor's Headers Over Your Own Model

Your bucket is a model of the vendor's limiter, and the vendor knows better. Feed their responses back into it.

def call_with_limit(tenant: str, fn, *args):
    ok, wait = acquire(tenant)
    if not ok:
        raise Throttled(wait)
    resp = fn(*args)
    remaining = resp.headers.get("X-RateLimit-Remaining")
    if remaining is not None and int(remaining) < 10:
        # Vendor says we are close: drain our bucket to match reality.
        r.hset("rl:payments-api", "tokens", 0)
    if resp.status_code == 429:
        retry_after = float(resp.headers.get("Retry-After", 30))
        r.hset("rl:payments-api", "tokens", 0)
        r.expire("rl:payments-api", int(retry_after) + 1)
        raise Throttled(retry_after)
    return resp

A 429 means the model was wrong. Zeroing the bucket converts the vendor's rejection into fleet-wide backpressure within one call, instead of letting the other thirty-nine workers each discover it independently.

Step 6 — Observe the Limiter, Not Just the Errors

What a healthy rate limiter looks like Three stacked series. Tokens available dips during bursts but recovers. Deferred job count rises during those dips, showing the limiter is absorbing load. Vendor 429 responses remain at zero throughout, which is the goal: the local limiter should be the binding constraint, not the vendor's. Your limiter should bind before theirs does tokens available deferred jobs bursts absorbed by deferral vendor 429s flat at zero — the goal Any sustained 429 means your rate or capacity is set above the vendor's real limit. 
# Deferrals are healthy; 429s are not. Alert on the latter, watch the former.
sum(rate(ratelimit_deferred_total{limiter="payments-api"}[5m]))
sum(rate(http_client_responses_total{code="429"}[5m])) > 0

Step 7 — Give the Limited Work Its Own Queue

Rate-limited jobs behave differently from everything else: they defer often, they sit in the queue longer, and their throughput is capped by an external number you do not control. Mixing them into a shared queue lets those deferrals crowd out unrelated work, and makes the queue's depth impossible to interpret — a growing backlog might mean insufficient workers or simply a vendor quota doing exactly what it should.

Route them to a dedicated queue with its own worker pool sized to the quota rather than to the backlog. If the vendor allows 100 requests per second and each job makes one call taking 200ms, twenty concurrent workers saturate the quota and any more are pure waste — an autoscaler pointed at that queue's depth will otherwise keep adding workers that immediately defer. Cap the pool at the quota-derived number and let the depth grow; the backlog is real work waiting for a limit, not a capacity problem.

This also makes the SLO honest. A queue whose throughput is externally capped cannot promise the same latency as one bounded by your own workers, and giving it a separate objective — measured against the quota rather than against wall-clock expectations — stops it from consuming the error budget of every other queue it used to share.

Per-tenant bucket nested inside the global quota A request first takes a token from the tenant's own bucket, then from the global bucket that represents the vendor quota. If the global bucket is empty, the tenant token is refunded so the tenant is not charged for a call that never happened, and the job is deferred. Tenant bucket first, global bucket second job for tenant A wants one call tenant bucket 8/s, burst 8 global bucket 80/s = vendor quota call the API global empty → refund the tenant token, defer the job Without the refund, a busy global bucket slowly starves every tenant even after it clears.

Verification

  1. Aggregate rate holds under scaling. Run 4, then 40 workers against a mock endpoint; measured requests per second must stay at the configured rate both times.
  2. Atomicity holds under contention. Drive 200 concurrent acquires against a 10-token bucket; exactly 10 must succeed.
  3. 429s reach zero. After deployment, the vendor's 429 count should fall to zero in steady state.
  4. Deferral does not starve. Confirm deferred jobs eventually run rather than cycling indefinitely — the retry countdown must be at least the reported wait.

Gotchas & Edge Cases

Clock skew across workers. The bucket uses each caller's clock. A worker several seconds ahead refills the bucket early and effectively raises the rate. Use TIME from Redis inside the script if skew is a real risk in your fleet.

Sleeping inside a task holds a worker slot. Blocking is fine for sub-second waits and expensive beyond that. Prefer deferral, and never sleep longer than the visibility timeout allows.

Per-minute and per-second limits both apply. Many vendors enforce both. Model each as its own bucket and require a token from every one before calling.

Retries multiply the request rate. A retry is another request against the same quota. Take a token on retry as well, or your effective rate silently exceeds the limit whenever failures spike.

A limiter outage must fail closed, not open. If Redis is unreachable, code that treats the error as "no limit configured" sends the whole fleet at the vendor unthrottled — turning a cache incident into an account suspension. Treat an unavailable limiter as an empty bucket and defer, with a short circuit-breaker so the queue does not spin.

The bucket key is a single hot key. At very high rates it becomes a Redis hotspot. Shard it into N sub-buckets each with rate/N, and have workers pick one at random — accepting slightly lumpier pacing for much better scaling.

Related