Text generation is a conversation: you send a context, the model writes the next message. This guide covers what goes into the context, how to read and stream what comes back, and the features built on top: thinking, tools and structured output.

Two functions do the work. Omni.generate_text/3 returns the finished response. Omni.stream_text/3 returns a stream of events as the model writes. Both take a model, a context and options.

Contexts and messages

The context is what the model sees: an optional system prompt, the messages so far, and any tools. For a single prompt, a string is enough:

{:ok, response} = Omni.generate_text("openai:gpt-6.1-sol", "Name three Elixir web frameworks.")

For anything more, build a context:

context =
  Omni.context(
    system: "You answer in one sentence.",
    messages: [Omni.message("Name three Elixir web frameworks.")]
  )

{:ok, response} = Omni.generate_text("openai:gpt-6.1-sol", context)

A message has a role, :user or :assistant, and a list of content blocks. Omni.message/1 takes a string for a user message, or role: and content:.

To carry a conversation on, push the response onto the context and add the next user message:

context =
  context
  |> Omni.Context.push(response)
  |> Omni.Context.push(Omni.message("Which is the oldest?"))

{:ok, response} = Omni.generate_text("openai:gpt-6.1-sol", context)

Omni keeps no state between requests. The context is yours to hold, store and trim. For persistence, Omni.Codec turns messages into JSON-safe maps and back without loss.

Images and PDFs

A user message can mix text with attachments. An attachment is a source, {:base64, data} or {:url, url}, and a media type:

message =
  Omni.message(
    role: :user,
    content: [
      "What's in this image?",
      Omni.attachment({:base64, Base.encode64(File.read!("photo.png"))}, "image/png")
    ]
  )

{:ok, response} = Omni.generate_text("google:gemini-3.5-flash", [message])

Each model lists what it accepts in model.input_modalities (:text, :image, :pdf). Sending an attachment the model doesn't accept returns {:error, {:unsupported_modality, :image}} before any request is made.

The response

Omni.generate_text/3 returns an Omni.Response:

  • message is the assistant's message, and messages every message the request produced. They differ when tools ran: messages then holds each tool use and tool result along the way.
  • stop_reason says why the model stopped: :stop, :length (it hit the token limit), :tool_use, :refusal, :error or :cancelled.
  • usage holds token counts and their cost in USD, worked out from the model's pricing.
  • output holds the decoded map when you asked for structured output.

Streaming

Omni.stream_text/3 returns an Omni.StreamingResponse once the provider starts to answer. There are three ways to consume it.

Register handlers for the events you care about, then run the stream to its end:

alias Omni.StreamingResponse

{:ok, stream} = Omni.stream_text("anthropic:claude-sonnet-5-5", "Tell me a story")

{:ok, response} =
  stream
  |> StreamingResponse.on(:text_delta, fn %{delta: text} -> IO.write(text) end)
  |> StreamingResponse.complete()

Take only the text:

stream
|> StreamingResponse.text_stream()
|> Enum.each(&IO.write/1)

Or enumerate every event. Each is {type, data, partial_response}, where the partial response is the response as built so far:

for {type, data, _partial} <- stream do
  case type do
    :text_delta -> IO.write(data.delta)
    :tool_use_start -> IO.puts("Using #{data.name}")
    :done -> IO.puts("\nStopped: #{data.stop_reason}")
    _ -> :ok
  end
end

Text, thinking and tool use blocks each arrive as a _start, any number of _deltas and an _end. A stream finishes with one :done or one :error. Omni.StreamingResponse lists every event and its data.

The stream is lazy and runs in your process, so nothing is read until you consume it. To stop early, call StreamingResponse.cancel/1.

Thinking

Many models can reason before they answer. The :thinking option asks for it:

{:ok, response} =
  Omni.generate_text("anthropic:claude-sonnet-5-5", "How many primes are there below 100?", thinking: :high)
  • An effort level: :low, :medium, :high, :xhigh or :max.
  • A token budget: a positive integer, such as thinking: 20_000.
  • false switches thinking off, on models that can switch it off.

Leave the option out and the model's own default applies: some models think unless told not to, others only when asked.

Models differ in what they accept, and Omni translates for each. A level the model doesn't have is rounded up to the nearest it does. A model that takes a token budget is given one sized to the level (:low 1024, :medium 4096, :high 16384, :xhigh 24576, :max 32768). A budget sent to a model that takes levels becomes the smallest level that covers it, so 20_000 is :xhigh. A model that only switches thinking on and off is switched on by any level or budget. A model that doesn't reason ignores the option. model.thinking shows what a model supports (see Omni.Model).

Reasoning comes back as Omni.Content.Thinking blocks, and streams as :thinking_delta events. Some providers return a summary, some return it encrypted, and some return nothing readable.

Keep those blocks in the conversation: on later turns Omni sends them back, which models need to continue their reasoning across tool use. They only go back to the provider and dialect that wrote them. If you switch model mid-conversation, earlier thinking is left out and the text and tool uses are kept.

Tools

Tools let the model call your code. A tool has a name, a description that tells the model when to use it, a schema for its input, and a handler:

weather =
  Omni.tool(
    name: "get_weather",
    description: "Gets the current weather for a city",
    input_schema: Omni.Schema.object(%{city: Omni.Schema.string()}, required: [:city]),
    handler: fn input -> Weather.current(input.city) end
  )

context =
  Omni.context(
    messages: [Omni.message("Do I need an umbrella in London?")],
    tools: [weather]
  )

{:ok, response} = Omni.generate_text("anthropic:claude-sonnet-5-5", context)

When the model answers with tool uses, Omni validates each input against the schema, runs the handlers in parallel, and sends the results back in a new user message. This loop repeats until the model answers with stop_reason: :stop. The response you get is the final one; response.messages holds every step, and response.usage is the total.

The handler receives input with the schema's own keys. It may return any term. A handler that raises, or a tool name the model made up, becomes an error result the model can read and recover from.

Three options shape the loop:

  • :max_steps caps the number of requests (default :infinity). Set max_steps: 1 to manually handle the tool loop.
  • :tool_timeout is how long each tool may run, in milliseconds (default 30_000).
  • When streaming, a :tool_result event is emitted for each result between steps.

Tool modules

For a tool you reuse, define a module. It keeps the schema and the code together, and can take state when it is built:

defmodule MyApp.Tools.FindOrder do
  use Omni.Tool,
    name: "find_order",
    description: "Looks up an order by its number"

  def schema do
    import Omni.Schema
    object(%{number: string()}, required: [:number])
  end

  def init(repo), do: repo

  def call(input, repo), do: repo.get_by(Order, number: input.number)
end

context =
  Omni.context(
    messages: messages,
    tools: [MyApp.Tools.FindOrder.new(MyApp.Repo)]
  )

Omni.Tool covers stateless tools, descriptions and schemas that depend on state, and other schema validators.

Running tools yourself

A tool with no handler is one the model may ask for but Omni won't run. The loop stops and the response comes back with stop_reason: :tool_use and the Omni.Content.ToolUse blocks in its message. This suits tools that need a person's approval, or that run somewhere else. Answer each tool use with a tool result in the next user message:

results =
  for %Omni.Content.ToolUse{} = tool_use <- response.message.content do
    Omni.tool_result(tool_use_id: tool_use.id, name: tool_use.name, content: run(tool_use))
  end

context =
  context
  |> Omni.Context.push(response)
  |> Omni.Context.push(Omni.message(role: :user, content: results))

To stop the loop for tools that do have handlers, pass max_steps: 1. Omni.Tool.Runner then runs a list of tool uses for you, with the same validation and timeouts the loop applies.

Structured output

Pass a JSON Schema as :output to get data instead of prose:

schema =
  Omni.Schema.object(
    %{
      sentiment: Omni.Schema.enum(["positive", "neutral", "negative"]),
      topics: Omni.Schema.array(Omni.Schema.string())
    },
    required: [:sentiment, :topics]
  )

{:ok, response} = Omni.generate_text("openai:gpt-6.1-sol", review_text, output: schema)

response.output
#=> %{sentiment: "negative", topics: ["delivery", "refunds"]}

The schema goes to the provider, which constrains the model to it where the provider can. Omni then decodes the answer and validates it against the schema. If it doesn't fit, Omni tells the model what was wrong and asks again, up to three times.

Omni.Schema has builders for the common JSON Schema types, and any map in JSON Schema form works. Keys in response.output are the keys you wrote in the schema.

Omni's own validator is meant for schemas of simple shape, and Omni.Schema.validate/2 lists what it checks. For a schema that needs more, such as exact oneOf, recursive $ref or checked date formats, or to validate with a library you already use, such as JSV, pass a {module, state} adapter as :output. Omni.Schema.Adapter shows how to write one.

Prompt caching

A long system prompt or conversation costs less, and answers sooner, when the provider caches it. Some providers cache on their own. Others, Anthropic among them, cache only when asked:

Omni.generate_text("anthropic:claude-sonnet-5-5", context, cache: :short)

:short is the provider's standard cache lifetime, a few minutes. :long asks for its extended one where it has one: an hour on Anthropic, a day on OpenAI. response.usage reports cache_read_tokens and cache_write_tokens.

Options

Every text request takes these:

OptionWhat it does
:max_tokensThe most tokens the model may write
:temperatureSampling temperature
:thinkingReasoning effort (see Thinking)
:outputA schema for structured output
:cache:short or :long prompt caching
:max_stepsThe most requests the tool loop may make
:tool_timeoutMilliseconds each tool may run
:timeoutMilliseconds to wait for each chunk of the response (default 300_000)
:api_keyA key for this request
:hostAnother origin for this request, such as "http://localhost:4000"
:headersExtra request headers
:metadataA map passed to the provider
:rawWhen true, keeps each raw request and response on response.raw

Omni.stream_text/3 documents each in full.

Errors

Both functions return {:ok, result} or {:error, reason}. Problems Omni can see before sending (an unknown model, a bad option, an attachment the model can't take, no API key) are errors with no request made. An HTTP error from the provider is {:error, {:http_error, status, body}}. A stream that ends without the model finishing is {:error, :incomplete_stream}.