LLM Gateway

📄️ Philosophy

Today, generative AI is simply unavoidable. It’s transforming industries, unlocking new possibilities, and accelerating innovation

📄️ Supported LLM Providers

Here is a list of all LLM chat providers available in the Otoroshi LLM Extension :

Now that your provider is fully set up, you can expose it to your organization. The idea is to do it through an Otoroshi route with a plugin of type backend that will handle passing incoming requests to the actual LLM provider.

📄️ 🔐 Secure your LLM provider endpoint

Video Turorial

📄️ Anthropic Messages API Proxy

The LLM Anthropic Messages Proxy plugin exposes any LLM provider managed by Otoroshi through an Anthropic Messages API-compatible endpoint. This means any client that speaks the Anthropic API format (including Claude Code) can seamlessly use any LLM provider proxied by Otoroshi, whether it's OpenAI, Mistral, Ollama, Azure OpenAI, Cohere, or any other supported provider.

📄️ Open Responses API Proxy

The LLM OpenResponse Proxy plugin exposes any LLM provider managed by Otoroshi through an Open Responses-compatible API endpoint. Open Responses is an open-source specification for multi-provider, interoperable LLM interfaces, backed by NVIDIA, Vercel, OpenRouter, Hugging Face, Databricks, Red Hat, Ollama, OpenAI, and others.

📄️ Managing secrets

You can use a secret vault to save your LLM provider tokens. The main advantage here is to avoid spreading those token and giving otoroshi apikeys to your own people instead.

📄️ Load Balancing

Load balancing distributes LLM requests across multiple providers, optimizing performance, ensuring availability, and preventing any single provider from being overloaded.