Behaviour for pluggable model data sources.
Experimental
This behaviour is experimental and may change in a future release.
A source answers one question: what does your catalog say about this provider? It returns the provider's catalog-level data and its models, in one go. Where the data comes from (a bundled snapshot, another package's catalog, a file, a network fetch) is the source's own concern. Where model data comes from covers choosing a source; this page is for writing one.
Omni ships Omni.Sources.ModelsDev (the default, reading a bundled
models.dev snapshot) and Omni.Sources.LLMDB. Select
a source globally or per provider:
config :omni, :models, source: Omni.Sources.LLMDB
config :omni, :providers, openai: [source: Omni.Sources.ModelsDev]A source value is a module or a {module, opts} tuple; opts are passed
to fetch/2.
The fetch/2 contract
fetch/2 receives the provider's id, which is its catalog key in snake
case (:anthropic, :fireworks_ai), and the source's opts. Translating
the id to the catalog's own key is the source's job:
Omni.Sources.ModelsDev hyphenates it, Omni.Sources.LLMDB uses it as
it is.
Return {:ok, provider, models} with raw catalog data only. Omni
layers provider modules and config over it afterwards:
provider- anOmni.Providerstruct withid, and whatever the catalog knows ofname,api_key({:system, var}when the catalog names exactly one environment variable), anddialects: anOmni.Provider.Endpointper dialect the provider's models speak, holding the catalog's whole URL for it. Leave the rest unset.models-Omni.Modelstructs without a provider. Each model'sdialectscomes from the catalog's own data (for example, an npm package mapped to a dialect), one per capability the catalog says the model serves (seeOmni.Model). Omni never fills in a missing dialect, so skip a model the catalog gives no dialect for.
Use endpoints/3 to derive the endpoints from the models' URLs. Keep
templated URLs (https://${ACCOUNT}.example.com/v1) as they are: a
provider's host: config fills the origin, and a request to a URL still
holding a placeholder returns an error.
Skip, with a warning, models the source can't represent faithfully: a model whose dialect hint is unrecognised, say, or one served from a different URL than other models of its dialect.
Return {:error, :unknown_provider} when the catalog has no such provider;
Omni then treats it as a custom provider defined in config. Any other error
is logged and treated the same way.
Use build_model/1 to construct models from string-keyed data maps.
Caching expensive work
fetch/2 is called once per provider in a load pass. A source that
derives models from something costly to produce, such as a large decoded
snapshot or a network response, should wrap that work in memo/2. The
result is computed once per pass and garbage collected when the pass
ends, so it doesn't linger for the VM's lifetime. Omni.Sources.ModelsDev
does this.
Summary
Callbacks
Returns the catalog's provider data and models for a provider id.
Types
Callbacks
@callback fetch(id :: atom(), opts :: keyword()) :: {:ok, Omni.Provider.t(), [Omni.Model.t()]} | {:error, term()}
Returns the catalog's provider data and models for a provider id.
See The fetch/2 contract.
Functions
@spec build_model(map()) :: Omni.Model.t()
Builds a model with no provider from a string-keyed data map.
The shared constructor for model sources. Recognised keys: "id",
"name", "dialects" (a list of dialect string ids, see
Omni.Dialect.get/1, each placed under the capability its module serves:
["openai_responses", "openai_decisions"] gives %{text: ..., decision: ...}), "release_date" (ISO 8601 YYYY-MM-DD, or YYYY-MM which parses
as the first of the month), "reasoning" (boolean), "reasoning_options",
"input_modalities"/"output_modalities" (lists of strings),
"input_cost", "output_cost", "cache_read_cost", "cache_write_cost",
"context_size", "max_output_tokens". Costs and limits default to 0,
modalities to ["text"], reasoning to false, and the name to the id.
"reasoning_options" is a list in models.dev's form, which becomes the
model's :thinking map (see Omni.Model) when "reasoning" is true:
[%{"type" => "effort", "values" => ["low", "medium", "high"]},
%{"type" => "toggle"},
%{"type" => "budget_tokens", "min" => 1024, "max" => 32_000}]Option maps may have string or atom keys. Unknown types are ignored.
Raises ArgumentError for an unknown dialect string, or for two dialects
serving the same capability. Modality strings are converted with
String.to_atom/1, so filter untrusted data to the modalities Omni
supports (:text, :image, :pdf) first.
@spec endpoints( [{Omni.Model.t(), String.t() | nil}], String.t() | nil, module() | nil ) :: {%{required(module()) => Omni.Provider.Endpoint.t()}, [Omni.Model.t()], [{Omni.Model.t(), String.t()}]}
Derives a provider's endpoints from its models' URLs.
Takes {model, url} pairs, where url is the model's own URL in the
catalog or nil, the catalog's provider URL, and the dialect the catalog
names at provider level (or nil). Each dialect's endpoint takes the URL
of its first model by id: the model's own URL, else the provider URL. A
later model of the same dialect at a different URL can't be served, and
is returned in skipped with its URL. The provider-level dialect gets
the provider URL when no model claimed it. A provider URL with no dialect
is dropped.
{endpoints, models, skipped} =
Omni.Source.endpoints(pairs, "https://api.acme.ai/v1", Omni.Dialects.OpenAICompletions)
@spec memo(term(), (-> result)) :: result when result: var
Returns fun's result, computed once per cache scope under key.
The scope is the enclosing with_cache/1. Outside one (a one-off
fetch/2 call, for example) there is nothing to share the result with,
so fun just runs. Name keys as {YourSource, :name} tuples, so sources
don't collide.
@spec with_cache((-> result)) :: result when result: var
Runs fun inside a cache scope for memo/2, on the calling process.
Omni.Provider.load/1 wraps each load pass in a scope, so every memo/2
result is shared across the pass's fetch/2 calls and released when the
pass ends. The scope is then cleared and the process garbage collected,
so data needed only for the pass (a decoded catalog, say) does not linger
on the caller's heap. Nested calls reuse the enclosing scope.