Omni.StreamingResponse (Omni v2.0.0)

Copy Markdown View Source

A streaming text response, yielding events as the model writes.

Returned by Omni.stream_text/3. The stream is lazy and runs in the process that consumes it: nothing is read from the provider until you enumerate it. Streaming introduces the three ways to consume one:

{:ok, stream} = Omni.stream_text("anthropic:claude-sonnet-5-5", "Hello!")

{:ok, response} =
  stream
  |> StreamingResponse.on(:text_delta, fn %{delta: text} -> IO.write(text) end)
  |> StreamingResponse.complete()

A stream can be consumed once. To stop early, call cancel/1.

Struct fields

Use the functions in this module; the fields are for building on top of them.

  • :stream - the lazy stream of events
  • :cancel - a function that closes the HTTP response, or nil

Events

Each event is a {type, data, partial_response} tuple. The third element is the Omni.Response as built so far, rebuilt after every event.

Every content block arrives as a _start, any number of _deltas and an _end. :index is the block's position in the message content. Blocks never overlap: one ends before the next starts.

{:text_start,     %{index: 0}, %Response{}}
{:text_delta,     %{index: 0, delta: "Hello"}, %Response{}}
{:text_end,       %{index: 0, content: %Text{}}, %Response{}}

{:thinking_start, %{index: 0}, %Response{}}
{:thinking_delta, %{index: 0, delta: "..."}, %Response{}}
{:thinking_end,   %{index: 0, content: %Thinking{}}, %Response{}}

{:tool_use_start, %{index: 1, id: "call_1", name: "weather"}, %Response{}}
{:tool_use_delta, %{index: 1, delta: "{\"city\":"}, %Response{}}
{:tool_use_end,   %{index: 1, content: %ToolUse{}}, %Response{}}

When Omni runs tools, a :tool_result event follows for each result, before the next request starts. Its data is the result block, and its third element is the finished response of the step that used the tool:

{:tool_result, %ToolResult{}, %Response{}}

Every stream ends with exactly one of these:

{:done,  %{stop_reason: :stop}, %Response{}}
{:error, reason, %Response{}}

:done means the model finished, with the response's stop reason. An :error reason is {:stream_error, reason} when the stream failed part-way, or :incomplete_stream when it ended before the model finished.

Summary

Types

An event: its type, its data and the response so far.

The type of an event (see Events).

t()

A streaming response.

Functions

Cancels the request by closing the HTTP response.

Runs the stream to its end and returns the final response.

Registers a handler for events of one type.

Returns a stream of text delta binaries.

Types

event()

@type event() ::
  {event_type(), map(), Omni.Response.t()} | {:error, term(), Omni.Response.t()}

An event: its type, its data and the response so far.

event_type()

@type event_type() ::
  :text_start
  | :text_delta
  | :text_end
  | :thinking_start
  | :thinking_delta
  | :thinking_end
  | :tool_use_start
  | :tool_use_delta
  | :tool_use_end
  | :tool_result
  | :error
  | :done

The type of an event (see Events).

t()

@type t() :: %Omni.StreamingResponse{cancel: (-> :ok) | nil, stream: Enumerable.t()}

A streaming response.

Functions

cancel(streaming_response)

@spec cancel(t()) :: :ok

Cancels the request by closing the HTTP response.

Call it when you stop consuming a stream before its end, so the connection is not left open.

complete(sr)

@spec complete(t()) :: {:ok, Omni.Response.t()} | {:error, term()}

Runs the stream to its end and returns the final response.

Handlers registered with on/3 fire along the way. Returns {:ok, response}, or {:error, reason} if the stream ended with an error.

on(sr, event_type, callback)

@spec on(t(), event_type(), (map() -> any()) | (map(), Omni.Response.t() -> any())) ::
  t()

Registers a handler for events of one type.

Returns a new streaming response with the handler in its pipeline. The handler is called with the event's data as the stream is consumed (by complete/1 or any Enum function), and its return value is ignored.

Handlers chain, and each fires for its own events:

{:ok, response} =
  stream
  |> StreamingResponse.on(:text_delta, fn %{delta: d} -> IO.write(d) end)
  |> StreamingResponse.on(:thinking_delta, fn %{delta: d} -> IO.write(d) end)
  |> StreamingResponse.on(:done, fn %{stop_reason: r} -> IO.puts("\nStop: #{r}") end)
  |> StreamingResponse.complete()

A two-argument handler also receives the partial response, for example to track token usage as it grows:

StreamingResponse.on(stream, :text_delta, fn _event, partial ->
  send(self(), {:tokens, partial.usage.output_tokens})
end)

text_stream(sr)

@spec text_stream(t()) :: Enumerable.t()

Returns a stream of text delta binaries.

Keeps only the :text_delta events and takes the text from each, for when the text is all you need:

stream
|> StreamingResponse.text_stream()
|> Enum.into("")