Pattern · last reviewed 2026-08-14

The spend ceiling your code can read

Context & problem

Every LLM feature ships with a budget question and almost every team answers it in a vendor dashboard, because that is where the field is. You set a monthly figure, the provider stops serving you when you cross it, and the risk feels handled.

It is handled in the sense that the bill is bounded. It is not handled in any sense your architecture can use. The limit lives outside your process; your code cannot read the current spend, cannot know it is at 80%, cannot tell a user what happens next, and cannot choose to degrade to a cheaper model instead of failing. The first and only thing it ever observes is a request that did not work — and by then the decision about what the user sees has already been made for you, badly, by whatever your error path happens to do.

The managed products are clear about this if you read the mechanics rather than the promise. Cloudflare's AI Gateway spend limits are configured on the gateway via the dashboard or the API, and when one is crossed the gateway "returns a 429 Too Many Requests response". That is the entire interface to your application: one status code, after the fact. The same page adds a second detail worth more than it looks — "Cost tracking is a best-effort estimation based on token counts and model pricing. Refer to your provider's dashboard for exact billing amounts." So the ceiling is not only invisible to your code, it is approximate. If you need the number to be exact, no gateway-side limit is going to give you that, and a limit you cannot reconcile is a limit you will not trust during an incident.

AWS shipped the same class of control on the agent side. AgentCore Gateway rate limiting covers three dimensions — requests per second and per minute, connections per second, and, for inference targets only, "Token rate limits, measured in tokens per minute (TPM)". Its accounting is the interesting part, and it is the opposite of what most hand-rolled caps do: the gateway "uses a general-purpose tokenizer to estimate the incoming tokens for a request and deducts it from the rate-limit bucket upfront before the gateway dispatches the inference call", then "reconciles the limit by accounting for the true token consumption" once the provider reports usage. Reserve, then settle. Note also what it keys on: scope is expressed through dimensions like $.context.jwt.<claim> and $.context.iam.principal rather than through any per-user primitive of its own — which is a precise statement of the constraint at the centre of this pattern. A ceiling is only as meaningful as the identity it is keyed to.

Forces

• Where the limit lives decides who can respond to it. In the dashboard, the provider responds — by refusing. In your code, you respond — by warning, by degrading, by explaining. Only one of those is a product decision you get to make.
• Ceilings are per-principal; costs are per-call. Attributing spend needs an identity at the moment of the call. Anonymous, batch and background work has no obvious principal, and that is precisely where a runaway is hardest to notice.
• Check-then-spend is a race with money in it. You read the balance, then you spend an unknown amount. Overshoot is guaranteed; the only question is by how much, and whether you reserve up front or reconcile afterwards.
• Accuracy versus availability. An exact ledger on the request path is a dependency that can fail. Deciding whether a broken ledger blocks spending or permits it is a real decision, and it will be made by default if you do not make it.
• The message is the product. Two users hit two different limits and both see "something went wrong". The engineering is worthless if the moment it bites is indistinguishable from an outage.

The pattern

Four decisions, and one rule that follows from them.

• Key on a principal you control, and write down what has none. A user id, a tenant, an API key — something durable enough that starting a new session does not reset the ledger. Then enumerate the call paths that have no such key. Those are not covered, however low you set the number, and pretending otherwise is the most common form of this control being decorative.
• Make the state readable before the call, not only after it. Spent, cap, over/under, and the instant it resets. A ceiling the application can query is what turns "the request failed" into "you have used this month's allowance; it is available again on the 3rd" — and the second one is answerable by the product rather than by support.
• Separate the read from the write. The pre-call check must not mutate the ledger. If a check can move the window, then a user who merely opens the page has changed their own accounting period, and the cap becomes a function of traffic rather than of spend.
• Choose reserve-then-reconcile or record-after, and state the overshoot you accepted. Reserving an estimated cost up front, as AgentCore does with tokens, bounds the overshoot to an estimation error. Recording actual cost after the turn is simpler and exactly right in hindsight, at the price of overshooting by up to one call. Both are defensible. Only one of them is defensible silently, and it is neither.

The rule: warn before you refuse. A ceiling that gives no notice converts a routine limit into an incident, because the first signal the user gets is the failure. A threshold warning is three lines of code and it is the difference between a fair-use policy and a trap.

Reference implementation notes

The mechanism is small either way. What differs is who owns the ledger and, therefore, who owns the moment it bites.

Managed: gateway spend rules and gateway rate limits

Use them. They are the correct backstop, they cost nothing to configure, and they cap the blast radius of a bug your own ledger has no opinion about — a credential leak, a runaway job, a misrouted model. What they are not is a control your application participates in. Cloudflare's spend limits offer two behaviours when a budget is crossed: block the request until the window resets, or use Dynamic Routes to fall back to a cheaper model. The fallback is genuinely useful and it is worth noting that it is still a gateway-side decision — your code does not know the cheaper model is now answering, so anything downstream that reasons about model quality is reasoning about the wrong model.

On the AgentCore side, read the limitation the AWS post states about its own control before you lean on it: "The gateway uses fail-open semantics for rate limit evaluation. Because of fail-open behavior, do not rely solely on rate limits as a security boundary." That sentence is more useful than most of the feature description, and it generalises past AWS. A gateway control that fails open is a cost control, not a containment control — a distinction worth carrying into any review where someone offers a rate limit as an answer to an abuse question.

Rolling your own

A rolling per-principal cap is one table and two functions: a read that never writes, and a write that records actual spend. The subtleties are all in the second one.

Anchor the window on the first real spend, not on the first check. If the read can lazily reset an expired window, then a user who loads the page starts a fresh accounting period without spending anything, and a user who never opens the page keeps an old one. Put the reset inside the write.

Reject non-positive amounts before they touch the ledger. A zero-cost turn — an offline mock, a blocked turn, an error that billed nothing — must not anchor a window. Write the guard as "is this greater than zero", not "is this less than or equal to zero", because the two differ on one input that matters: NaN is not less than or equal to zero, so the sloppy form lets a NaN through, and spent + NaN is NaN forever. Every later "is spent over cap" comparison is then false, and the cap is silently off for that principal until someone edits the database.

// the guard is the whole control, and it is easy to get backwards
if (!(usd > 0)) return;   // rejects 0, negatives AND NaN
if (usd <= 0) return;     // NaN <= 0 is false — NaN gets through

Decide the ledger's failure mode explicitly. If the store is unreachable, do you spend or refuse? Neither answer is free, and the fact that it is a money decision rather than a correctness one does not make it optional. Note that an "is the binding present" check does not settle this: a present dependency whose call throws is a different failure from an absent one, and only a try/catch covers it.

Trade-offs

• You are buying a database read on every paid request. It is small, but it is on the hot path and it is a new dependency for a feature whose entire job is to not be noticed. In exchange you get a control you can test, an admin surface you can build on it, and the ability to raise a single user's cap without a vendor console.
• Your ledger will disagree with your invoice. Costs are computed from token counts and a price table you maintain. Rates change; your table lags. When it lags high you cut users off early, when it lags low you overspend, and neither shows up as an error. This is not a hand-rolled failing — the same page that documents Cloudflare's spend limits calls its own cost tracking "a best-effort estimation". Own it: treat the ledger as a fair-use instrument, and reconcile against the provider's own numbers on a schedule.
• Check-then-spend always overshoots. You authorise before you know the price. Reserve-and-reconcile bounds the error to a mis-estimate; record-after bounds it to one call. Pick, then write the bound down where the next person will find it, because someone will eventually read a $15 cap that stopped at $15.40 as a bug.
• A per-principal cap invites per-principal exceptions, and exceptions are a product surface. The first support request for "raise my limit" arrives sooner than you expect. An override column is cheap; the policy about who gets one, and whether raising it also resets the window, is not, and it will be decided in a hurry if you have not decided it in advance.
• It is a ceiling, not a governor. It bounds a total over a window and says nothing about the rate inside that window. A single principal can legitimately spend the entire month's allowance in one afternoon, and the cap will be perfectly satisfied right up to the moment it is exhausted.

The failure mode: every ceiling returns the same status code

This is the one that survives all your careful design, and it is not in any vendor's documentation because no single vendor owns both ends of it.

A gateway sits between your application and the model. A spend rule is over budget: 429. A rate-limit rule fires: 429. The upstream provider is throttling everyone and the gateway passes it through: 429. Three different causes, three different remedies — wait two seconds, wait until next month, this is not about you at all — and at the call site they are the same response with no distinguishing header or error code to separate them.

The consequence lands on the user. Whatever you write in that error handler is a guess about which limit fired, and the confident version of the message is the wrong one. Tell a rate-limited user they have exhausted their monthly budget and you have given them a false cause and a false remedy: they will come back next month when two minutes would have done. Tell a genuinely-out-of-budget user to try again shortly and they will, repeatedly, and then contact you.

There are only two honest responses, and you want both. Hedge the cause and state the observable — say that an AI usage limit was reached, name the retry that costs nothing to attempt, and mention the monthly budget as the explanation if it persists. And move the limit you actually care about to where you can read it, so the message that matters most is not the ambiguous one. Ambiguity at the gateway is unavoidable; ambiguity about your own product's fair-use policy is a choice.

When not to use this

When the risk is one runaway turn, not one expensive month. This is the most common misapplication and it is dangerous precisely because it feels like coverage. An agent stuck in a tool-call loop can burn a monthly allowance in a minute or two; a cap evaluated once before the turn began will pass, and the next check — after the damage — will correctly report the budget gone. The instrument for a runaway is a bounded agentic loop: a hard turn count, a tool-call ceiling, and a maximum output size on every call. Build that first. A spend ceiling on top of it is a fair-use policy; a spend ceiling instead of it is a false sense of safety with a monthly reset.

When there is no principal to key on. A per-user cap requires a user. A public, unauthenticated surface that calls a model has nobody to bill and nobody to remember, so the ceiling has nothing to attach to. The controls that do work there are different in kind: a per-IP rate limit, a hard cap on output tokens, and a cheap model. They are weaker — an IP is not an identity — and that is the honest position, not a gap to paper over by inventing a pseudo-principal from a cookie an abuser controls.

When hitting the ceiling is the expected path rather than an incident. On a trial or free tier the cap is not an operational control at all, it is the shape of the product: it defines how much of the thing a prospect gets before deciding. Treating it as ops gets you a hard cut-off where the product needed a conversion moment, wired to a threshold somebody picked to feel safe rather than to be the right amount of value. That is a pricing decision with a number attached, and it should be made by whoever owns pricing.

When the spend is not attributable to anyone. Scheduled jobs, ingestion pipelines and background agents spend on behalf of the system, not a user. A per-principal ledger has no row for them, and quietly bolting them onto a shared synthetic principal makes one user's cap the whole platform's cap. Bound that work where it is generated instead — a fixed job budget, a bounded work queue, or simply fewer scheduled calls.

And never on its own. Keep the gateway rule as the backstop. Your ledger protects the product; the vendor's rule protects you from your ledger being wrong, which it eventually will be. A cap hit rendered as a blank panel is also the defect fail empty exists to name — the ceiling is only finished when the moment it bites reads as a policy rather than as a bug.

As-built: what this platform runs

The coach runs a per-user rolling cap in src/lib/spendCap.ts — $15 over a 30-day window, with a warning at 80%. checkSpendCap reads before a turn and never writes; recordSpend writes after it, and the window's lazy reset lives inside the write for the reason given above, so a check with no spend behind it cannot slide the window. The guard on the write is if (!(usd > 0)) return, and the NaN case is the reason, not a stylistic preference. We record after the turn rather than reserving before it, so the cap overshoots by up to one turn. That is deliberate and it is the trade we accepted for a ledger that is exactly right in hindsight.

The moment it bites is the part worth comparing. In src/coach/CoachAgent.ts the over-limit branch sends an in-band reply naming the date the coach returns, and it walks the same persistence tail as a normal turn rather than throwing — a limit is a conversation, not an exception. At 80% it sends a heads-up annotation, deliberately kept out of the path that persists a reply, so a warning can never be saved as the assistant's answer.

Now the contrast, which is the reason this page exists. A few files away, src/lib/coach.ts handles the gateway's 429 — and it cannot name a date, because it does not know which limit fired. That handler hedges on purpose: it tells the learner an AI usage limit was reached, suggests trying again in a few minutes, and only mentions the monthly budget as the explanation if it persists. The mapping in src/lib/llm.ts is where the ambiguity is recorded in full: a gateway 429 may be a spend rule, a rate-limit rule or a passed-through provider 429, all three want the same "stop and degrade" handling, and none is distinguishable at that point. We do not state that any spend rule is configured on our gateway, and we cannot: that is dashboard state, and no code in this repository can read it — which is exactly the property this page is about. The only spend ceiling we can evidence is the one in src/lib/spendCap.ts.

Two admissions, because the "when not to use this" section above is not hypothetical for us. The cap is coach-only. Short-answer and practical grading are bounded by a durable per-user daily counter in src/lib/rateLimit.ts instead, and our scheduled vendor-watch drafter spends with no per-principal ledger at all, because it has no principal — it degrades on a budget error and that is the whole of its protection. And the public guide coach has no ceiling by construction. It is unauthenticated, so src/routes/guideCoach.ts falls back to a per-IP limiter of 8 questions per 5 minutes plus a hard 400-token reply cap. That is weaker than a cap and we prefer stating it to implying coverage we do not have.

The runaway case is covered elsewhere and by different means: the coach loop is turn-bounded and tool-call-bounded, which is the bounded agentic loop doing the job this pattern cannot.

Changelog

• 2026-08-14 — Initial publication. Cloudflare AI Gateway spend limits and AWS AgentCore Gateway rate limiting both retrieved and quoted 2026-08-14. As-built claims verified against the tree the same day.