Resilience
Resilience ensures that LLM interactions remain highly available and fault-tolerant, even when providers experience outages or failures.
Key Features
- Load Balancing: Distribute requests across multiple providers to optimize performance and availability. See Load Balancing.
- Fallback Mechanism: Automatically switch to alternative LLM providers in case of failures. See Fallback.
- Rate Limiting & Quotas: Prevent overloading a single provider by distributing usage effectively. See Quotas.
Load Balancing
The load balancer is a virtual provider that distributes requests across multiple providers using configurable strategies (round robin, random, best response time).
Read the full Load Balancing documentation
Fallback
The fallback mechanism automatically retries failed requests on an alternative provider, ensuring service continuity.