Omni.Source behaviour (Omni v2.0.0)

Copy Markdown View Source

Behaviour for pluggable model data sources.

Experimental

This behaviour is experimental and may change in a future release.

A source answers one question: what does your catalog say about this provider? It returns the provider's catalog-level data and its models, in one go. Where the data comes from (a bundled snapshot, another package's catalog, a file, a network fetch) is the source's own concern. Where model data comes from covers choosing a source; this page is for writing one.

Omni ships Omni.Sources.ModelsDev (the default, reading a bundled models.dev snapshot) and Omni.Sources.LLMDB. Select a source globally or per provider:

config :omni, :models, source: Omni.Sources.LLMDB

config :omni, :providers, openai: [source: Omni.Sources.ModelsDev]

A source value is a module or a {module, opts} tuple; opts are passed to fetch/2.

The fetch/2 contract

fetch/2 receives the provider's id, which is its catalog key in snake case (:anthropic, :fireworks_ai), and the source's opts. Translating the id to the catalog's own key is the source's job: Omni.Sources.ModelsDev hyphenates it, Omni.Sources.LLMDB uses it as it is.

Return {:ok, provider, models} with raw catalog data only. Omni layers provider modules and config over it afterwards:

  • provider - an Omni.Provider struct with id, and whatever the catalog knows of name, api_key ({:system, var} when the catalog names exactly one environment variable), and dialects: an Omni.Provider.Endpoint per dialect the provider's models speak, holding the catalog's whole URL for it. Leave the rest unset.
  • models - Omni.Model structs without a provider. Each model's dialects comes from the catalog's own data (for example, an npm package mapped to a dialect), one per capability the catalog says the model serves (see Omni.Model). Omni never fills in a missing dialect, so skip a model the catalog gives no dialect for.

Use endpoints/3 to derive the endpoints from the models' URLs. Keep templated URLs (https://${ACCOUNT}.example.com/v1) as they are: a provider's host: config fills the origin, and a request to a URL still holding a placeholder returns an error.

Skip, with a warning, models the source can't represent faithfully: a model whose dialect hint is unrecognised, say, or one served from a different URL than other models of its dialect.

Return {:error, :unknown_provider} when the catalog has no such provider; Omni then treats it as a custom provider defined in config. Any other error is logged and treated the same way.

Use build_model/1 to construct models from string-keyed data maps.

Caching expensive work

fetch/2 is called once per provider in a load pass. A source that derives models from something costly to produce, such as a large decoded snapshot or a network response, should wrap that work in memo/2. The result is computed once per pass and garbage collected when the pass ends, so it doesn't linger for the VM's lifetime. Omni.Sources.ModelsDev does this.

Summary

Types

t()

A source in configuration: a module or a {module, opts} tuple.

Callbacks

Returns the catalog's provider data and models for a provider id.

Functions

Builds a model with no provider from a string-keyed data map.

Derives a provider's endpoints from its models' URLs.

Returns fun's result, computed once per cache scope under key.

Runs fun inside a cache scope for memo/2, on the calling process.

Types

t()

@type t() :: module() | {module(), keyword()}

A source in configuration: a module or a {module, opts} tuple.

Callbacks

fetch(id, opts)

@callback fetch(id :: atom(), opts :: keyword()) ::
  {:ok, Omni.Provider.t(), [Omni.Model.t()]} | {:error, term()}

Returns the catalog's provider data and models for a provider id.

See The fetch/2 contract.

Functions

build_model(data)

@spec build_model(map()) :: Omni.Model.t()

Builds a model with no provider from a string-keyed data map.

The shared constructor for model sources. Recognised keys: "id", "name", "dialects" (a list of dialect string ids, see Omni.Dialect.get/1, each placed under the capability its module serves: ["openai_responses", "openai_decisions"] gives %{text: ..., decision: ...}), "release_date" (ISO 8601 YYYY-MM-DD, or YYYY-MM which parses as the first of the month), "reasoning" (boolean), "reasoning_options", "input_modalities"/"output_modalities" (lists of strings), "input_cost", "output_cost", "cache_read_cost", "cache_write_cost", "context_size", "max_output_tokens". Costs and limits default to 0, modalities to ["text"], reasoning to false, and the name to the id.

"reasoning_options" is a list in models.dev's form, which becomes the model's :thinking map (see Omni.Model) when "reasoning" is true:

[%{"type" => "effort", "values" => ["low", "medium", "high"]},
 %{"type" => "toggle"},
 %{"type" => "budget_tokens", "min" => 1024, "max" => 32_000}]

Option maps may have string or atom keys. Unknown types are ignored.

Raises ArgumentError for an unknown dialect string, or for two dialects serving the same capability. Modality strings are converted with String.to_atom/1, so filter untrusted data to the modalities Omni supports (:text, :image, :pdf) first.

endpoints(pairs, provider_url, provider_dialect)

@spec endpoints(
  [{Omni.Model.t(), String.t() | nil}],
  String.t() | nil,
  module() | nil
) ::
  {%{required(module()) => Omni.Provider.Endpoint.t()}, [Omni.Model.t()],
   [{Omni.Model.t(), String.t()}]}

Derives a provider's endpoints from its models' URLs.

Takes {model, url} pairs, where url is the model's own URL in the catalog or nil, the catalog's provider URL, and the dialect the catalog names at provider level (or nil). Each dialect's endpoint takes the URL of its first model by id: the model's own URL, else the provider URL. A later model of the same dialect at a different URL can't be served, and is returned in skipped with its URL. The provider-level dialect gets the provider URL when no model claimed it. A provider URL with no dialect is dropped.

{endpoints, models, skipped} =
  Omni.Source.endpoints(pairs, "https://api.acme.ai/v1", Omni.Dialects.OpenAICompletions)

memo(key, fun)

@spec memo(term(), (-> result)) :: result when result: var

Returns fun's result, computed once per cache scope under key.

The scope is the enclosing with_cache/1. Outside one (a one-off fetch/2 call, for example) there is nothing to share the result with, so fun just runs. Name keys as {YourSource, :name} tuples, so sources don't collide.

with_cache(fun)

@spec with_cache((-> result)) :: result when result: var

Runs fun inside a cache scope for memo/2, on the calling process.

Omni.Provider.load/1 wraps each load pass in a scope, so every memo/2 result is shared across the pass's fetch/2 calls and released when the pass ends. The scope is then cleared and the process garbage collected, so data needed only for the pass (a decoded catalog, say) does not linger on the caller's heap. Nested calls reuse the enclosing scope.