Evaluate AI decisions

evaluate answers named Choice, Score, and Boolean questions about shared state. It returns typed values and probabilities for routing, scoring, guardrails, and verification. Omit account to use the platform Vercel AI Gateway credential and the default typesafe-ai/jev model. Jev makes bounded decisions; it doesn’t generate text. Use generate when you need a written or schema-generated response.

Example

automations/triage-message.automation.ts

import {
  automation,
  evaluate,
  onHttpRequest,
  respondToHttpRequest,
  t,
} from "automate.ax"
import { z } from "zod"

export default automation("Triage a message", () => {
  const request = onHttpRequest({
    body: z.object({ message: z.string() }),
  })

  const triage = evaluate({
    state: request.body,
    questions: {
      department: {
        type: "choice",
        instructions: "Which team should handle this message?",
        criteria: {
          billing: "Payments, charges, and refunds",
          support: "Product help and other requests",
        },
      },
      urgent: {
        type: "boolean",
        instructions: "Does this message require immediate attention?",
      },
    },
  })

  respondToHttpRequest({
    body: t`Route: ${triage.answers.department.choice}`,
    requestId: request.requestId,
  })
})

The Choice answer is typed as "billing" | "support". The Boolean answer’s probability is the model-estimated probability that the answer is true.

Inputs

Input	Type	Required	Description
`state`	`string \| JSON object \| JSON array`	Yes	Shared content evaluated by every question. Signals are supported.
`questions`	Named Choice, Score, or Boolean question map	Yes	Static question definitions. Keys and Choice option names type the returned answers.
`model`	`EvaluateModelId`	No	Gateway or provider-native evaluation model ID. Defaults according to `account`.
`account`	Supported provider account reference	No	Customer-managed provider credentials. Omit for the platform gateway.
`maxRetries`	`number`	No	Maximum retries for transient provider failures. Defaults to 2.
`headers`	`Record<string, string>`	No	Additional provider request headers.
`providerOptions`	`Record<string, Record<string, JSONValue>>`	No	Provider-specific options.

Each question requires instructions, which can be a string, JSON object, or JSON array:

choice requires a nonempty criteria map. Each key is an available answer and each value describes that option.
score requires at least two ordered criteria levels. The first level has index 0.
boolean can describe true and false with optional criteria.

Descriptions can be strings, JSON objects, JSON arrays, or null.

Output

answers contains one answer for every question:

Question	Answer
Choice	`choice` and, when supplied by the provider, a complete `probabilities` map.
Score	Fractional `score` from `0` through the last level index and, when supplied, a probability distribution keyed by level index.
Boolean	`probability`, the estimated probability that the answer is true.

Choice and Score probability distributions are optional because language-model adapters may return only a selected choice or score. Jev returns native distributions and adds its separate confidence statistic under providerMetadata.typesafe.confidence. Don’t treat Boolean probability or Jev confidence as interchangeable values. Choose automation thresholds against labeled examples from your task. The result also exposes token usage, warnings, optional rounding precision, provider metadata, and response metadata.

Provider accounts

Customer-managed Vercel AI Gateway, OpenAI, Anthropic, and Google Generative AI accounts support evaluate. OpenAI, Anthropic, and Google adapt structured language-model output to the same question types; their Boolean probabilities are prompted estimates and aren’t guaranteed to be calibrated. Omitting model uses the selected account’s default:

Account	Default model
Platform or Vercel AI Gateway	`typesafe-ai/jev`
OpenAI	`gpt-5.6-luna`
Anthropic	`claude-haiku-4-5-20251001`
Google Generative AI	`gemini-3.5-flash-lite`

Connect customer-managed credentials with bunx automate.ax accounts, then pass the provider’s named account reference. Calls that omit account count toward the organization’s monthly Platform AI allowance.

Built-in triggers

Built-in actions

Runtime utilities

Integrations

Evaluate AI decisions

Example

Inputs

Output

Provider accounts

​Example

​Inputs

​Output

​Provider accounts

Example

Inputs

Output

Provider accounts