Overview

Otoroshi LLM Extension is set of Otoroshi plugins and resources to interact with LLMs, let's discover it !

Connect, setup, secure and seamlessly manage LLM models using an Universal/OpenAI compatible API

• Unified interface: Simplify interactions and minimize integration hassles

• Use multiple providers: 50+ LLM providers supported right now, a lot more coming

• Load balancing: Ensure optimal performance by distributing workloads across multiple providers

• Fallbacks: Automatically switch LLMs during failures to deliver uninterrupted & accurate performance

• Automatic retries: LLM APIs often have inexplicable failures. You can rescue a substantial number of your requests with our in-built automatic retries feature.

• Semantic cache: Speed up repeated queries, enhance response times, and reduce costs

• Custom quotas: Manage LLM tokens quotas per consumer and optimise costs

• Key vault: securely store your LLM API keys in Otoroshi vault or any other secret vault supported by Otoroshi.

• Observability and reporting: every LLM request is audited with details about the consumer, the LLM provider and usage. All those audit events are exportable using multiple methods for further reporting

• Fine grained authorizations: Use Otoroshi advanced fine grained authorizations capabilities to constrains model usage based on whatever you want: user identity, apikey, consumer metadata, request details, etc

• Prompt engineering: enhance your experience by providing contextual information to your prompts, storing them in a library for reusability, and using prompt templates for increased efficiency

• Guardrails: Validate your prompts and prompts responses to avoid sensitive or personal informations leakage, irrelevant or unhelpful responses, gibberish content, etc

• Cost Tracking

• MCP (Model Context protocol)

• MCP Connectors

  • 💸 Stripe
  • Slack
  • Make
  • ScreenshotOne
  • Redis
  • Perplexity Ask
  • Mailgun
  • Grafana
  • Gitlab
  • Firecrawl
  • Brave Search
  • AWS KB Retrieval
  • EverART
  • Puppeteer
  • Google Maps

• MCP Server exposition

• Wasm functions (tool calls)

• Http functions (tool calls)

• Embeddings models

• Workflows: Use AI-powered workflow functions for LLM calls, audio, image/video generation, embeddings, vector stores, memory, guardrails, and more

• AI Agents: Build agentic workflows with tool calling, agent handoffs, persistent memory, and LLM-based routing

Supported LLM providers

All supported LLM providers are listed here

• Anthropic
• Azure OpenAI
• Azure AI Foundry
• Cloud Temple 🇫🇷 🇪🇺
• Cloudflare
• Cohere
• Deepseek
• Gemini
• Groq
• Huggingface 🇫🇷 🇪🇺
• Mistral 🇫🇷 🇪🇺
• Ollama (Local Models)
• OpenAI
• OVH AI Endpoints 🇫🇷 🇪🇺
• Scaleway 🇫🇷 🇪🇺
• X.ai

And 37 more including Abliteration, AI/ML API, Apertis, AssemblyAI, Cerebras, Chutes, CometAPI, CompactifAI, DeepInfra, Empower, Featherless AI, Fireworks AI, Friendli AI, Galadriel, GMI, Helicone, Hyperbolic, Lambda AI, LlamaGate, Meta Llama API, Minimax, Morph, Nano GPT, Nebius AI Studio, Novita AI, Nscale, Nvidia NIM, OpenRouter, Perplexity, Poe, SambaNova, Sarvam, Synthetic, Together AI, Venice AI, Xiaomi Mimo, Z.AI

MCP Servers

MCP Servers

Cost Optimization

Cost optimization

Ecological impacts

Ecological impacts

Introduction video