Skip to main content

Plugins

The Decision models backend plugin exposes decision models on any Otoroshi route. It speaks the System One API of TypeSafe, on the path of your choice.

The same endpoint is also part of the LLM OpenAI Compatible API plugin, on /systemone, next to chat completions, embeddings and the rest. The OpenAI decisions backend plugin and the /decisions path of the same plugin serve the decision models with the decisions API of OpenAI.

Plugin configuration​

Add the plugin to your route:

{
"enabled": true,
"plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.DecisionModels",
"config": {
"refs": ["decision-model-entity-id"]
}
}
ParameterTypeDefaultDescription
refsarray[]List of Decision Model entity IDs

Usage​

curl --request POST \
--url http://myroute.oto.tools:8080/v1/systemone \
--header 'content-type: application/json' \
--data '{
"state": "The checkout has been failing for every customer for the last hour.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this support request urgent?"
}
}
}'

Response​

{
"model": "jev-1.13.0",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.95
}
},
"usage": {
"input_tokens": 84,
"output_tokens": 0
}
}

With the TypeSafe SDKs​

The official SDKs work against the gateway: give them the URL of your route as base URL and an Otoroshi API key. They call <base url>/v1/systemone.

from typesafe_sdk import TypeSafeClient, Noul, Choice

client = TypeSafeClient(
api_key="your-otoroshi-api-key",
base_url="https://decisions.my-domain.com",
)

result = client.system_one(
"The checkout has been failing for every customer for the last hour.",
{
"urgent": Noul(instructions="Is this support request urgent?"),
"team": Choice(
instructions="Which team should handle this request?",
criteria={"billing": "Invoices and payments", "technical": "Outages and bugs"},
),
},
)

print(result.nouls["urgent"].noul)
print(result.choices["team"].choice)

From there the provider is a gateway setting: pin the model of the entity and the same code is served by Jev, Clef or a model you host.

Request bodies are read the way TypeSafe reads them: plain JSON, compressed with gzip or zstd (the SDKs compress the bodies of 1 KiB or more), or multipart/form-data, the JSON in a part named request and each picture in a part of its own. Up to 32 MiB.

Pictures​

A request shows pictures to the decision model as assets, the way of the System One API: each one is declared once under a reference of your choice, and the reference is written where the picture belongs, in the state, in the instructions or in the criteria of a question.

{
"state": {
"photo": "6f1c2a3e-photo",
"ticket": "My screen is broken since this morning"
},
"questions": {
"broken": {
"type": "noul",
"instructions": "Does `photo` show a broken screen?"
}
},
"assets": {
"6f1c2a3e-photo": {
"type": "image",
"data_base64": "iVBORw0KGgoAAAANSUhEUgAA...",
"format": "png"
}
}
}

Every decision model that looks at pictures answers it:

  • Jev gets the request exactly as it was written, references in place
  • Clef, the OpenAI models and self-hosted servers get the pictures the way their own API takes them, each reference being replaced by the name of its picture ([image 1]) so the questions still say which picture they are about

The images array of Cloudflare (data URLs, shown before the state) is read too, and reaches Jev as assets. A model that reads text only refuses pictures with a 400: a decision is never made on a part of what it is about.

Model routing​

When several decision models are configured in refs, the model field of the request picks one:

{
"model": "providerName/modelName",
"state": "...",
"questions": { }
}

The decision model can be referenced by:

  • Entity ID: decision-model-id###jev-latest — the form to prefer, since it cannot be read any other way
  • Entity name (slug): my_jev/jev-latest
  • Its own model: a request naming the very model an entity is configured with goes to that entity as it is. Model names with a slash of their own, like typesafe/jev-1.13 on OpenRouter or telnyx/decision-flash, are therefore routed right without any prefix
  • A provider field: "provider": "decision-model-id" picks the entity and leaves model free

If nothing matches, the first configured ref is used, and asked for the model of the request — or for its own when the request names none.

Errors​

A client written for the System One API gets the errors it expects, with the status the provider would have answered:

StatusWhen
422The request is not a System One request: no state, no questions, a question without type or instructions. Refused by the gateway, with the field at fault
400An asset nothing refers to, a picture that is not a PNG or a JPEG, a multipart part that is the picture of no asset, or pictures for a model that reads text only
413, 415A body over 32 MiB, or compressed with something else than gzip or zstd
401, 422, 429, 529...What the provider answered, body included. A Retry-After header is passed along, so SDK retries behave as they do against the provider
402A budget is exhausted
403The model is not one the consumer may use, or it has no known price while the decision model requires one
504, 502The provider did not answer in time, or could not be reached, and no fallback took over

Errors raised by the gateway use the envelope of the System One API:

{
"detail": {
"error_type": "budget_exceeded",
"message": "budget exceeded"
}
}

Route example​

A complete route configuration exposing a decision endpoint:

{
"frontend": {
"domains": ["decisions.my-domain.com"]
},
"backend": {
"targets": [
{
"hostname": "request.otoroshi.io",
"port": 443,
"tls": true
}
]
},
"plugins": [
{
"enabled": true,
"plugin": "cp:otoroshi.next.plugins.OverrideHost",
"config": {}
},
{
"enabled": true,
"plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.DecisionModels",
"config": {
"refs": ["decision-model-entity-id"]
}
}
]
}