Plugins
The Decision models backend plugin exposes decision models on any Otoroshi route. It speaks the System One API of TypeSafe, on the path of your choice.
The same endpoint is also part of the LLM OpenAI Compatible API plugin, on /systemone, next to chat completions, embeddings and the rest. The OpenAI decisions backend plugin and the /decisions path of the same plugin serve the decision models with the decisions API of OpenAI.
Plugin configuration
Add the plugin to your route:
{
"enabled": true,
"plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.DecisionModels",
"config": {
"refs": ["decision-model-entity-id"]
}
}
| Parameter | Type | Default | Description |
|---|---|---|---|
refs | array | [] | List of Decision Model entity IDs |
Usage
curl --request POST \
--url http://myroute.oto.tools:8080/v1/systemone \
--header 'content-type: application/json' \
--data '{
"state": "The checkout has been failing for every customer for the last hour.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this support request urgent?"
}
}
}'
Response
{
"model": "jev-1.13.0",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.95
}
},
"usage": {
"input_tokens": 84,
"output_tokens": 0
}
}
With the TypeSafe SDKs
The official SDKs work against the gateway: give them the URL of your route as base URL and an Otoroshi API key. They call <base url>/v1/systemone.
from typesafe_sdk import TypeSafeClient, Noul, Choice
client = TypeSafeClient(
api_key="your-otoroshi-api-key",
base_url="https://decisions.my-domain.com",
)
result = client.system_one(
"The checkout has been failing for every customer for the last hour.",
{
"urgent": Noul(instructions="Is this support request urgent?"),
"team": Choice(
instructions="Which team should handle this request?",
criteria={"billing": "Invoices and payments", "technical": "Outages and bugs"},
),
},
)
print(result.nouls["urgent"].noul)
print(result.choices["team"].choice)
From there the provider is a gateway setting: pin the model of the entity and the same code is served by Jev, Clef or a model you host.
Request bodies are read the way TypeSafe reads them: plain JSON, compressed with gzip or zstd (the SDKs compress the bodies of 1 KiB or more), or multipart/form-data, the JSON in a part named request and each picture in a part of its own. Up to 32 MiB.
Pictures
A request shows pictures to the decision model as assets, the way of the System One API: each one is declared once under a reference of your choice, and the reference is written where the picture belongs, in the state, in the instructions or in the criteria of a question.
{
"state": {
"photo": "6f1c2a3e-photo",
"ticket": "My screen is broken since this morning"
},
"questions": {
"broken": {
"type": "noul",
"instructions": "Does `photo` show a broken screen?"
}
},
"assets": {
"6f1c2a3e-photo": {
"type": "image",
"data_base64": "iVBORw0KGgoAAAANSUhEUgAA...",
"format": "png"
}
}
}
Every decision model that looks at pictures answers it:
- Jev gets the request exactly as it was written, references in place
- Clef, the OpenAI models and self-hosted servers get the pictures the way their own API takes them, each reference being replaced by the name of its picture (
[image 1]) so the questions still say which picture they are about
The images array of Cloudflare (data URLs, shown before the state) is read too, and reaches Jev as assets. A model that reads text only refuses pictures with a 400: a decision is never made on a part of what it is about.
Model routing
When several decision models are configured in refs, the model field of the request picks one:
{
"model": "providerName/modelName",
"state": "...",
"questions": { }
}
The decision model can be referenced by:
- Entity ID:
decision-model-id###jev-latest— the form to prefer, since it cannot be read any other way - Entity name (slug):
my_jev/jev-latest - Its own model: a request naming the very model an entity is configured with goes to that entity as it is. Model names with a slash of their own, like
typesafe/jev-1.13on OpenRouter ortelnyx/decision-flash, are therefore routed right without any prefix - A
providerfield:"provider": "decision-model-id"picks the entity and leavesmodelfree
If nothing matches, the first configured ref is used, and asked for the model of the request — or for its own when the request names none.
Errors
A client written for the System One API gets the errors it expects, with the status the provider would have answered:
| Status | When |
|---|---|
422 | The request is not a System One request: no state, no questions, a question without type or instructions. Refused by the gateway, with the field at fault |
400 | An asset nothing refers to, a picture that is not a PNG or a JPEG, a multipart part that is the picture of no asset, or pictures for a model that reads text only |
413, 415 | A body over 32 MiB, or compressed with something else than gzip or zstd |
401, 422, 429, 529... | What the provider answered, body included. A Retry-After header is passed along, so SDK retries behave as they do against the provider |
402 | A budget is exhausted |
403 | The model is not one the consumer may use, or it has no known price while the decision model requires one |
504, 502 | The provider did not answer in time, or could not be reached, and no fallback took over |
Errors raised by the gateway use the envelope of the System One API:
{
"detail": {
"error_type": "budget_exceeded",
"message": "budget exceeded"
}
}
Route example
A complete route configuration exposing a decision endpoint:
{
"frontend": {
"domains": ["decisions.my-domain.com"]
},
"backend": {
"targets": [
{
"hostname": "request.otoroshi.io",
"port": 443,
"tls": true
}
]
},
"plugins": [
{
"enabled": true,
"plugin": "cp:otoroshi.next.plugins.OverrideHost",
"config": {}
},
{
"enabled": true,
"plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.DecisionModels",
"config": {
"refs": ["decision-model-entity-id"]
}
}
]
}