Runs a set of live scenarios against one model and reports which pass.
mix omni.verify anthropic:claude-haiku-5-5
mix omni.verify openai:gpt-6-luna --add image,pdf
mix omni.verify fireworks_ai:accounts/fireworks/models/kimi-k3 --only text,tool_use
The model is a provider:model reference (a slash works too), or the
provider id and model id as two arguments.
Use it to check a provider before relying on it: a catalog provider Omni
doesn't verify, a custom provider from your config, or a new model on a
built-in. Requests are real and billed, using the provider's configured API
key, and capped at 8192 output tokens so a runaway generation fails with
:length instead of running on. Every selected scenario runs, even after a
failure. Each result line gives a short reason for a failure, and failures
from an error (an HTTP status, a crash) are followed by the full error. The
task exits non-zero if any scenario fails.
The task starts your application, so providers and models from your
config :omni are available. A catalog provider that isn't loaded is
loaded for the run.
Scenarios
A model runs the default scenarios of each capability it has: a text model the four text defaults, a decision model the two decision scenarios, a model with both all six.
Text, run by default:
text- a short prompt; passes on a:stopresponse with textthinking- a reasoning question withthinking: :low; passes when the response shows thinking (text, a signature, or redacted data)tool_use- a weather tool without a handler; passes on aget_weathertool use for Londonstructured_output- an object schema via:output; passes when the output matches it
Text, run on request:
no_thinking- the reasoning question withthinking: false; passes when no thinking comes back. Only meaningful on a model whose data says it can switch thinking off: on one that can't, Omni sends nothing and the model reasons as usual.image- a PNG logo; passes when the answer names the brandpdf- a one-page PDF; passes when the answer names the authorkitchen_sink- two tools with handlers andthinking: :low, run through the tool loop, then a follow-up turn over the whole history; passes when both turns finish with text after at least one tool roundcache_usage- a long system prompt withcache: :short, then a second turn on top of the first; passes when the second turn reports cache-read tokens
Decision, run by default:
decision- a string state with a predicate, a choice and a score; passes when every answer is well formed and the clear-cut predicate and choice come back as expected (the score is checked for shape only)decision_json- a map state and a question with map instructions; passes on the same rule
A scenario a model doesn't support fails; select the ones that apply.
Options
--only a,b- run exactly these scenarios--add a,b- run these as well as the defaults--skip a,b- leave these out of the defaults--delay ms- pause between scenarios, for rate-limited accounts