# `mix omni.verify`
[🔗](https://github.com/aaronrussell/omni/blob/v2.0.0/lib/mix/tasks/omni.verify.ex#L1)

Runs a set of live scenarios against one model and reports which pass.

```sh
mix omni.verify anthropic:claude-haiku-5-5
mix omni.verify openai:gpt-6-luna --add image,pdf
mix omni.verify fireworks_ai:accounts/fireworks/models/kimi-k3 --only text,tool_use
```

The model is a `provider:model` reference (a slash works too), or the
provider id and model id as two arguments.

Use it to check a provider before relying on it: a catalog provider Omni
doesn't verify, a custom provider from your config, or a new model on a
built-in. Requests are real and billed, using the provider's configured API
key, and capped at 8192 output tokens so a runaway generation fails with
`:length` instead of running on. Every selected scenario runs, even after a
failure. Each result line gives a short reason for a failure, and failures
from an error (an HTTP status, a crash) are followed by the full error. The
task exits non-zero if any scenario fails.

The task starts your application, so providers and models from your
`config :omni` are available. A catalog provider that isn't loaded is
loaded for the run.

## Scenarios

A model runs the default scenarios of each capability it has: a text
model the four text defaults, a decision model the two decision
scenarios, a model with both all six.

Text, run by default:

- `text` - a short prompt; passes on a `:stop` response with text
- `thinking` - a reasoning question with `thinking: :low`; passes when
  the response shows thinking (text, a signature, or redacted data)
- `tool_use` - a weather tool without a handler; passes on a
  `get_weather` tool use for London
- `structured_output` - an object schema via `:output`; passes when the
  output matches it

Text, run on request:

- `no_thinking` - the reasoning question with `thinking: false`; passes
  when no thinking comes back. Only meaningful on a model whose data says
  it can switch thinking off: on one that can't, Omni sends nothing and
  the model reasons as usual.
- `image` - a PNG logo; passes when the answer names the brand
- `pdf` - a one-page PDF; passes when the answer names the author
- `kitchen_sink` - two tools with handlers and `thinking: :low`, run
  through the tool loop, then a follow-up turn over the whole history;
  passes when both turns finish with text after at least one tool round
- `cache_usage` - a long system prompt with `cache: :short`, then a
  second turn on top of the first; passes when the second turn reports
  cache-read tokens

Decision, run by default:

- `decision` - a string state with a predicate, a choice and a score;
  passes when every answer is well formed and the clear-cut predicate
  and choice come back as expected (the score is checked for shape only)
- `decision_json` - a map state and a question with map instructions;
  passes on the same rule

A scenario a model doesn't support fails; select the ones that apply.

## Options

- `--only a,b` - run exactly these scenarios
- `--add a,b` - run these as well as the defaults
- `--skip a,b` - leave these out of the defaults
- `--delay ms` - pause between scenarios, for rate-limited accounts

---

*Consult [api-reference.md](api-reference.md) for complete listing*
