Skip to main content

💰 Budgets Management

Budgets management for LLMs with a gateway means monitoring and managing the costs of using different LLMs through an API gateway.

Our Otoroshi LLM extension helps you optimize usage, control your budget, and improve cost efficiency across models.

Configuration​

budgets {
enabled = true
enabled = ${?CLOUD_APIM_EXTENSIONS_LLM_EXTENSION_BUDGETS_ENABLED}
embed-budgets-in-responses = false
embed-budgets-in-responses = ${?CLOUD_APIM_EXTENSIONS_LLM_EXTENSION_BUDGETS_EMBED_BUGETS_IN_RESPONSES}
}

Once it's enabled, you can create budgets from the UI or the admin. API and enforce budget limits to any consumer

Budgets​

A budget is defined between a start date and an end date. Limits are applied for a duration that is renewed between the start and end date. Limits can be expressed in USD or in tokens.

A budget can defined limits that can be global or per usage (inference, image, audio, video, embedding, moderation, decision)

Every modality feeds those counters, in tokens and in dollars: an image generation, a transcription, a voice and an extracted page are priced the way cost tracking describes, and a call is refused once the limit it falls under is reached. The only calls that move tokens without moving dollars are the ones priced in a unit the gateway cannot measure — a voice billed per second of audio.

The counters of a cycle stay available for a week after it ends, so the calls a worker of a cluster counted in its last minutes are never lost, then they are cleaned up automatically, like the counters of a deleted budget.

A budget is defined for a scope based on pretty much whatever you want (apikey, api group, user, provider, model, request, etc). You can reference a budget in entity metadata using ai_budget_ref (apikey, api group, user, provider, auth. module)

Scope options​

ScopeDescription
apikeysList of API key IDs. Everything the gateway calls to answer a request counts for its caller: the answer, and with it the embeddings of a semantic cache, the model a router asks to pick a candidate, the panel and the judge of a fusion
usersList of user emails or IDs (regexes). Calls made with an API key count for the owner of the key (its ai_studio_owner metadata) when no user is authenticated
groupsList of API group IDs
providersList of LLM provider IDs. A request handed over to another provider (its fallback, the target of a load balancer, the candidate of a router) counts for the provider that received it and for the one that served it: put a budget on a load balancer to cap everything going through it, on a target to cap what that target serves. A budget with both in its scope counts the request once
modelsList of model names
extractFromApikeyMetaExtract budget ref from API key metadata
extractFromApikeyGroupMetaExtract budget ref from API key group metadata
extractFromUserMetaExtract budget ref from user metadata
extractFromUserAuthModuleMetaExtract budget ref from auth module metadata
extractFromProviderMetaExtract budget ref from provider metadata
rulesJsonPath-based rules for advanced scoping
rulesMatchModeRule matching mode: All (all rules must match) or Any (at least one rule must match)
alwaysApplyRulesWhen true, rules are always evaluated even if explicit lists match
partition_byAn expression of the Otoroshi expression language: the budget counts apart each value it takes, each one getting the whole limit. See budgets per consumer

Duration units​

Budget limits are applied for a renewable duration between the start and end date:

UnitDescription
hourLimits reset every hour
dayLimits reset every day
yearLimits reset every year

Budgets per consumer​

One budget can give each of your consumers their own limit. Set partition_by to an expression of the Otoroshi expression language: the budget counts apart every value the expression takes for a call, and each value has the whole limit for itself. A team of 200 people gets "200 $ per person per month" with a single budget, and everyone keeps their own counter.

{
"limits": { "total_usd": 200 },
"duration": { "value": 30, "unit": "day" },
"scope": {
"always_apply_rules": true,
"rules": [ { "path": "$.provider.metadata.ai_studio_workspace", "value": "v4e46mctfxxw" } ],
"partition_by": "${user.email || apikey.metadata.ai_studio_owner}"
}
}

Pick what a consumer is with the expression:

ExpressionEach consumer is
${user.email || apikey.metadata.ai_studio_owner}a person: their chats and the calls of the API keys they own share one limit
${consumer.id}each user, and each API key
${user.email || apikey.metadata.ai_studio_owner || apikey.clientId}a person, and each API key that has no owner
${apikey.metadata.team}a team, from the metadata of its API keys
${user.email || apikey.metadata.ai_studio_owner}/${ctx.ai.model}a person, for each model

Besides everything the expression language reads (the user, the API key and their metadata, the route), the provider and the model a call counts for are available as ${ctx.ai.provider.id}, ${ctx.ai.provider.name} and ${ctx.ai.model}.

A call the expression finds nothing for, like a call made with an API key that has no owner, is not counted by that budget: it stays under the other budgets of its scope. To count those calls together, give the expression a default value, ${apikey.metadata.team:none}, or ${user.email || apikey.metadata.ai_studio_owner :: shared} for a chain. Values are compared in lower case, so Jane@Acme.io and jane@acme.io share their counter.

caution

Count consumers by what they cannot choose: their identity, their API key, their route. A value the caller sets freely, like a request header, would let anyone start over with a whole new limit.

tip

In a cluster, every node counts the consumers of a budget apart once it runs a version that knows partition_by: upgrade your nodes, then create these budgets.

The consumption of a budget counted per consumer is the one of a consumer: name them with ?partition= on the consumption endpoints, and list them with the admin API. Alerts carry the consumer too: AiBudgetAlmostExceeded has a partition field, AiBudgetExceeded a partitions object with the consumer of each budget it names.

Action on exceed​

A budget can block requests on exceeded limits or just emit alerts.

ParameterTypeDefaultDescription
modestring—"soft" (alerts only) or "block" (deny requests with HTTP 429)
alert_on_exceedboolean—Emit an alert when the budget is exceeded
alert_on_almost_exceedboolean—Emit an alert when the budget is almost exceeded
alert_on_almost_exceed_percentagenumber80Percentage threshold for "almost exceeded" alerts (e.g. 80 means alert at 80% consumption)
error_messagestringbudget exceededError returned to the client of a call blocked by the budget

A blocked call tells your users what happened in your own words, for instance who to contact to get more budget. The response always carries budget_exceeded, so clients, metrics and dashboards recognize it whatever the message:

{
"error": "Your AI budget is exhausted, please contact your administrator",
"budget_exceeded": true
}

When several budgets block the same call, the message of one of them is returned. Budgets in soft mode never block a call, so their message is never returned.

Extensions​

Someone used their whole budget and needs to finish what they started? Let them take an extension. The administrator of a budget decides what an extension adds and how many can be taken in a cycle; the person the budget counts takes one when they need it, in AI Studio or through the admin API, without waiting for anyone.

"extensions": {
"enabled": true,
"max_per_cycle": 1,
"limits": { "total_usd": 10 }
}

With this budget of 200 $ per person and per month, Michel, who spent his 200 $, takes his extension of the month: he can go on up to 210 $, and the budget blocks him again past it. His consumption stays the one he really had, 200 $ out of 210 $, in every report.

ParameterTypeDefaultDescription
enabledbooleanfalseWhether extensions can be taken
max_per_cyclenumber1How many extensions can be taken in a cycle, by each consumer for a budget per consumer
limitsobject—What one extension adds, by limit (total_usd, total_tokens, inference_usd…): only the limits of the budget
  • An extension counts for the current cycle only: the next one starts with the limit of the budget.
  • Extensions add up, up to max_per_cycle. Two taken at the very same time never go past it.
  • On a budget per consumer, each person takes their own, once they consumed in the cycle. On a budget counted as a whole, the extension raises a limit everyone shares, so it is taken by an administrator.
  • An administrator can cancel the extensions of a cycle: their amounts are removed, and they are not given back. A reset starts the cycle over, extensions included.
  • Every extension taken or cancelled is audited (AiBudgetExtended, AiBudgetExtensionCancelled), with who it was for and who took it.

The consumption of a budget then tells what was extended: allowed_total_usd includes the extensions, extra_total_usd is what they added, extensions_taken and extensions_max how many were and can be taken, and extension what the next one would add.

Admin API​

in addition to the classic admin API endpoints for the ai-gateway.extensions.cloud-apim.com/AiBudget entity, you have

  • GET /api/extensions/cloud-apim/extensions/ai-extension/budgets/:id/consumption
  • POST /api/extensions/cloud-apim/extensions/ai-extension/budgets/:id/consumption/_reset
  • GET /api/extensions/cloud-apim/extensions/ai-extension/budgets/:id/partitions
  • POST /api/extensions/cloud-apim/extensions/ai-extension/budgets/:id/consumption/_extend
  • DELETE /api/extensions/cloud-apim/extensions/ai-extension/budgets/:id/consumption/_extend

For a budget per consumer, name the consumer with ?partition=jane@acme.io to read their consumption, reset it, take or cancel their extensions. Without it, the reset starts the cycle over for everyone. /partitions lists the consumers of the current cycle, the ones that consumed most first, with limit, offset and a q filter:

{
"total": 2,
"items": [
{ "partition": "jane@acme.io", "consumption": { "consumed_total_usd": 182.4, "allowed_total_usd": 210, "extra_total_usd": 10 } },
{ "partition": "john@acme.io", "consumption": { "consumed_total_usd": 37.9, "allowed_total_usd": 200 } }
]
}

_extend takes one of the extensions the budget allows, with no body, and returns the consumption it leads to; DELETE cancels the extensions of the cycle.

the consumption endpoint returns a response like:

{
"consumed_total_usd": 0.00188655,
"consumed_total_tokens": 8036,
"consumed_inference_usd": 0.00188655,
"consumed_inference_tokens": 8036,
"consumed_image_usd": 0,
"consumed_image_tokens": 0,
"consumed_audio_usd": 0,
"consumed_audio_tokens": 0,
"consumed_video_usd": 0,
"consumed_video_tokens": 0,
"consumed_embedding_usd": 0,
"consumed_embedding_tokens": 0,
"consumed_moderation_usd": 0,
"consumed_moderation_tokens": 0,
"consumed_decision_usd": 0,
"consumed_decision_tokens": 0,
"remaining_total_usd": 199.99811345,
"remaining_total_tokens": 9991964,
"remaining_inference_usd": 0,
"remaining_inference_tokens": 0,
"remaining_image_usd": 0,
"remaining_image_tokens": 0,
"remaining_audio_usd": 0,
"remaining_audio_tokens": 0,
"remaining_video_usd": 0,
"remaining_video_tokens": 0,
"remaining_embedding_usd": 0,
"remaining_embedding_tokens": 0,
"remaining_moderation_usd": 0,
"remaining_moderation_tokens": 0,
"remaining_decision_usd": 0,
"remaining_decision_tokens": 0,
"allowed_total_usd": 200,
"allowed_total_tokens": 10000000,
"allowed_inference_usd": null,
"allowed_inference_tokens": null,
"allowed_image_usd": null,
"allowed_image_tokens": null,
"allowed_audio_usd": null,
"allowed_audio_tokens": null,
"allowed_video_usd": null,
"allowed_video_tokens": null,
"allowed_embedding_usd": null,
"allowed_embedding_tokens": null,
"allowed_moderation_usd": null,
"allowed_moderation_tokens": null,
"allowed_decision_usd": null,
"allowed_decision_tokens": null
}

Embed in responses​

The budget embedded in a response is the one of the caller: on a budget per consumer, partition names them, and the extra_* and extensions_* fields tell the extensions they took.

curl --request POST \
--url 'http://proxy.oto.tools:8080/v1/chat/completions?embed_budget=true' \
--header 'content-type: application/json' \
--data '{
"model": "anthropic/claude-sonnet-4-5-20250929",
"stream": false,
"messages": [
{
"role": "user",
"content": "tell me a joke ?"
}
]
}'

{
"id": "chatcmpl-9ODWHjtFVS5qzpU1nc79TprfRhZElq0e",
"object": "chat.completion",
"created": 1762441013,
"model": "claude-sonnet-4-5-20250929",
"system_fingerprint": "fp-tEQXbmpSPELEh3fYd5Rs9Jf4HIaI3E6s",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Why don't scientists trust atoms?\n\nBecause they make up everything! 😄"
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 20,
"total_tokens": 32,
"completion_tokens_details": {
"reasoning_tokens": -1
}
},
"budget": {
"consumed_total_usd": 0.00394755,
"consumed_total_tokens": 8231,
"consumed_inference_usd": 0.00394755,
"consumed_inference_tokens": 8231,
"consumed_image_usd": 0,
"consumed_image_tokens": 0,
"consumed_audio_usd": 0,
"consumed_audio_tokens": 0,
"consumed_video_usd": 0,
"consumed_video_tokens": 0,
"consumed_embedding_usd": 0,
"consumed_embedding_tokens": 0,
"consumed_moderation_usd": 0,
"consumed_moderation_tokens": 0,
"consumed_decision_usd": 0,
"consumed_decision_tokens": 0,
"remaining_total_usd": 199.99605245,
"remaining_total_tokens": 9991769,
"remaining_inference_usd": 0,
"remaining_inference_tokens": 0,
"remaining_image_usd": 0,
"remaining_image_tokens": 0,
"remaining_audio_usd": 0,
"remaining_audio_tokens": 0,
"remaining_video_usd": 0,
"remaining_video_tokens": 0,
"remaining_embedding_usd": 0,
"remaining_embedding_tokens": 0,
"remaining_moderation_usd": 0,
"remaining_moderation_tokens": 0,
"remaining_decision_usd": 0,
"remaining_decision_tokens": 0,
"allowed_total_usd": 200,
"allowed_total_tokens": 10000000,
"allowed_inference_usd": null,
"allowed_inference_tokens": null,
"allowed_image_usd": null,
"allowed_image_tokens": null,
"allowed_audio_usd": null,
"allowed_audio_tokens": null,
"allowed_video_usd": null,
"allowed_video_tokens": null,
"allowed_embedding_usd": null,
"allowed_embedding_tokens": null,
"allowed_moderation_usd": null,
"allowed_moderation_tokens": null,
"allowed_decision_usd": null,
"allowed_decision_tokens": null
}
}