Compare Clef models.

Compare the documented contract first

Choose a model around the input you actually have, the answer types you need, and the mistakes you can tolerate. A context window describes how much a hosted model can receive; it does not tell you how well it will route a difficult ticket. A lower token price also does not establish the lowest cost per correct decision.

Prices cover hosted inference only. No independent benchmark results published yet.

The playground defaults to Clef Flash and can switch to Clef. Clef Omni is included here as a documented option, but cannot run in this site's first release. Trial limits are shared between the two runnable models. Missing live configuration keeps the tool in clearly marked example mode.

For another decision API, read Clef vs Jev. That article compares access, input, outputs, licensing, and prices from primary documentation. It does not declare a performance winner.

Our evaluation method

No independent benchmark results published yet. No live evaluation has been performed for this release. The repository includes an executable operator script and a small corpus so that future measurements can be inspected, rather than reconstructed from a chart.

The corpus contains 24 authored synthetic English cases: eight support tickets, eight agent-routing requests, and eight content-triage examples. Each record stores its state, full question configuration, and proposed reference labels. These labels have not been independently human reviewed. Human label review, including disagreements and ambiguous cases, is required before publishing benchmark quality claims. This corpus is a development fixture, not a representative sample of customer traffic.

What the script measures

For each of Clef Flash and Clef, the script sends one separately recorded warm-up request, then runs each of the 24 cases three times. That is 72 measured attempts per model plus one warm-up, or 146 potential paid calls in all. There are no automatic retries. A failure stays a failure.

Choice and Noul hit rates average repeated answers within each unique sample, then average the eligible samples equally. Repeats do not turn 24 independent examples into 72. Noul uses a disclosed 0.5 threshold for this fixture. Score uses mean absolute error against the proposed reference level and preserves fractional outputs. Every quality metric excludes failed or incomplete responses and reports how many unique samples contributed; the failure rate remains separate.

Latency uses successful end-to-end adapter round trips, including network and validation time, with nearest-rank p50 and p95. It is not pure model execution time. Warm-up timing and cost are kept in separate summaries. The raw records preserve every attempt so readers can inspect the aggregate.

Reproduce an authorized run

From the repository, run bun run evaluate to check the default refusal. The CLI will not load the provider or contact it without the exact opt-in below. Set your real Cloudflare Account ID and token securely, and set EVAL_REGION to where the process is actually running. The command uses Bun with the react-server condition so it can reuse the server-only provider adapter.

# Requires your credentials and explicit authorization for paid requests.
export EVAL_REGION="YOUR_ACTUAL_EXECUTION_REGION"
RUN_LIVE_EVAL=1 bun run evaluate > eval-report.json

The output stores the date, region supplied by the operator, model/provider, corpus version and SHA-256 hash, question-config version and hash, full cases, raw results, successes, failures, latency, usage, and documented unit prices. Keep that JSON alongside any published claim. Do not commit credentials or use real customer inputs in the fixture.

When a successful response omits usage, or a request fails after it may have reached the provider, total cost is unknown, not zero. Known token totals and known costs are reported separately as partial evidence. Warm-ups may also cost money. The report estimates inference charges only and cannot establish a billing ceiling.

What a useful next evaluation needs

Have a human review and freeze the labels before running. Add representative data from the intended workflow with permission, preserve a held-out set, and document selection rules. Inspect errors by case type, not just the mean. Repeat at the deployment location that matters to your users. Evaluate Jev separately only after implementing and verifying an equivalent adapter; this release has no Jev execution path.