Resilience

Resilience ensures that LLM interactions remain highly available and fault-tolerant, even when providers experience outages or failures.

Key Features

• Load Balancing: Distribute requests across multiple providers to optimize performance and availability. See Load Balancing.
• Smart Routing: Let the gateway pick the model for each request, and cascade to the next best one on failure. See Otoroshi Router.
• Fallback Mechanism: Automatically switch to alternative LLM providers in case of failures. See Fallback.
• Rate Limiting & Quotas: Prevent overloading a single provider by distributing usage effectively. See Quotas.

Load Balancing

The load balancer is a virtual provider that distributes requests across multiple providers using configurable strategies (round robin, random, best response time).

Read the full Load Balancing documentation

Smart routing

The Otoroshi Router is a virtual provider that picks the model for each request: the cheapest model that codes well enough (code-router), the best model for each prompt (auto-router), or a panel of models answering together (fusion-router). When the selected model fails, the next best candidate answers.

Read the full Otoroshi Router documentation

Fallback

The fallback mechanism automatically retries failed requests on an alternative provider, ensuring service continuity.

Read the full Fallback documentation

Provider errors

When a provider rejects a call, the client gets the provider's own error, with its message and details, instead of a generic gateway error. This holds for streamed calls too, whatever the size of the error or the pace at which the provider sends it, and for errors that are not json (an html page from a load balancer in front of the provider, for instance), which are returned with their status:

{
  "status": 503,
  "body": "<html><body><h1>503 Service Unavailable</h1></body></html>"
}

The error is also logged, so you can see why a provider refuses a request. The errors of streamed calls are read up to 4 MB, enough for validation errors that echo a whole conversation, and the rest is dropped:

providers {
  max-error-body-size = 4M
  max-error-body-size = ${?CLOUD_APIM_EXTENSIONS_LLM_EXTENSION_PROVIDERS_MAX_ERROR_BODY_SIZE}
}

Parameter	Type	Default	Description
max-error-body-size	size	4M	Maximum size of an error body read from a streamed provider call