We don’t know the exact answer,

but we know the form any valid answer must take.

An LLM agent's reply is non-deterministic. Every call can paraphrase, reorder, rewrite. That's the deal — you traded determinism for language.

But the agent's behavior should not drift. If a user asks Do you ship to Berlin? today and gets a warm yes, they should still get a warm yes tomorrow — even if the wording shifted.

Here's the kind of regression that's easy to ship by accident.

The fix is an hour's work. Keeping the fix — so the next prompt tweak doesn't silently re-break it — takes a test.

You write the behavior down.

One YAML file. Fifteen lines. That is the whole test.

• i.
  One YAML = one test. The filename is the scenario name. Discovery reads the folder: support is the agent, shipping-coverage is the category.
• ii.
  turns is what the user says. One entry for single-turn, many entries for a conversation.
• iii.
  expect.llm.criteria is plain English. No regex, no parsing. Sentences a judge model can read.

A traditional assertion pins the string. An LLM can paraphrase the same thought fifty ways and fail all fifty — even though the behavior is right.

Ours pins the behavior. We assert intent, absence-of-things, warmth. Properties that survive paraphrase.

// traditional unit test — brittle
test('replies with Berlin line', () => {
  const reply = agent.ask('Do you ship to Berlin?');
  expect(reply).toBe(
    'Yes, we ship to Berlin.'
  );
});
// copy change → test breaks. vibe change → still green.

expect:
  llm:
    criteria:
      - "Acknowledges Berlin as a city we ship to"
      - "Does NOT present any service-paused or not-supported message"
      - "Does NOT ask the user to pick a different city"

LLM variance is a feature of the system under test, not a bug in the test. The assertion shape has to match the thing it's measuring.

Three expectation types can appear under expect:. They're not mutually exclusive — a single scenario can combine all three. But they trade speed for expressiveness.

reply is regex against the turn's reply text — instant, deterministic, good for structured copy. extract compares a capture/extract node's output against expected fields — precise, for what the agent captured. llm routes the reply through the in-workspace assert-agent flow with plain-English criteria — for intent, tone, absence. Costs about 1–2 seconds and Flowise credits per criterion. Spend them where regex can't reach.

The atom handles single-turn behavior. But agents live in conversations — a user says Germany, the agent clarifies, they pick Berlin, then the real work starts.

A scenario can list many turns. Criteria bind to a specific turn number, so each step gets its own checkpoint.

A multi-turn scenario is a story with checkpoints. Every turn a beat. Every beat a contract. The story is what you're keeping from drifting.

A suite is an execution plan — which scenarios, how many runs each, which environment. It references scenarios by path and fails loud at load time if a path is wrong.

One command runs it.

Pass-lines collapse. Fails expand — per-criterion check, the judge's reason, the summary. The transcript and per-turn request/response payloads land in runs/<timestamp>-<scenario>/ if you want to dig.

That's the release gate. Before a push to stage, run the suite. If it's green, you ship.

You can do this in three minutes.

1. 1
   Pick a folder. scenarios/<agent>/<category>/ — the folder names become discovery metadata.
2. 2
   Create a YAML. Copy the shape below and give it a name.
3. 3
   Write turns + expect. One turn for single-turn, many for conversation. Criteria in plain English.
4. 4
   (Optional) add to a suite. Drop the path into a suite under suites/ to make it part of a batch.
5. 5
   Run it. bob test --scenario <path> — or, once the scenario is in a suite, bob test --suite <name>.

name: my-first-scenario
label: "My first scenario"
flowPattern: scripted-turns
turns:
  - "Your user message here"
expect:
  llm:
    criteria:
      - "Acknowledges the user's intent"
      - "Does NOT fabricate information"

That's the whole system. The rest is practice.