User reference / Run an experiment

2. Define the case

Write the challenge and identify exactly what the participating models will receive.

Reviewed 2026-09-21Process executor 0.1.5

Describe the case

A task JSON contains the prompt that models must respond to. Give enough detail to expose disagreements: the setting, measurable symptoms, constraints, competing priorities, required deliverable and what a workable response must address.

EXAMPLE / REFERENCE

{
  "about": {
    "title": "Incident management process",
    "domain": "software",
    "kind": "process",
    "setting": "B2B payments platform",
    "summary": "Define an incident process under operational constraints."
  },
  "prompt": "Define an incident management process for a 28-team platform. Include severity, 24x7 ownership, escalation, communications, postmortems, measurable outcomes and an adoption plan. State assumptions and tradeoffs.",
  "input_elements": {}
}

This is a short structural example. The published incident experiment uses a much more detailed challenge; read its full prompt.

Case fields

Documented in E 0.1.5

Field	Meaning and use
`prompt`Reviewed E 0.1.5	Required task statement. Inserted into proposal, voting and analysis prompts. Changing it changes prompt identity.
`about.title`Reviewed E 0.1.5	Human-readable title used by the CLI when registering the prompt. If absent, the task filename stem is used. The catalogue can also supply a title when rendering history.
`about.domain`Reviewed E 0.1.5	Catalogue classification for the subject area and the “By domain” grouping. It is descriptive metadata, not agent behavior.
`about.kind`Reviewed E 0.1.5	Deliverable classification. The catalogue orders `plan`, `process`, `investigation` before other values. It does not select a different executor.
`about.setting`Reviewed E 0.1.5	Short context label shown in the prompt library and prompt history.
`about.summary`Reviewed E 0.1.5	Short library description. The prompt text is the fallback.
`input_elements`Reviewed E 0.1.5	Structured context saved with the prompt and displayed in reports. In executor 0.1.5 it is held in state but is not separately interpolated into the model prompts. Put essential facts in `prompt` itself.
`base`Reviewed E 0.1.5	Optional reusable configuration path, relative to this task file. It selects the inherited council unless overridden by CLI `--base`.

Keep prompt identity stable

Documented in E 0.1.5

A prompt key is derived from the exact statement and input_elements; title and about metadata are excluded. Runs with the same key can be browsed together even when their councils differ. Changes in whitespace in the prompt can produce a different key; this is content identity, not semantic deduplication.

For a comparison of model combinations or round counts, keep the case unchanged. To study a change to the task prompt, preserve both variants and state the difference. See Experimental interpretation.

Use hard cases for observation

Documented in E 0.1.5

The test challenge is a vehicle for investigating collaboration. A confident, unanimous answer may still be infeasible or wrong. Record what would count as improvement before reading the result: a resolved objection, an explicit dependency, a preserved alternative, or a constraint that is finally addressed. The tool's analyst can help inspect these; it is not an external ground-truth evaluator.

Continue with agents and rounds.

Source files used for this reference

Reviewed at 7361ea0b71a5.

← Documentation home Find a report element →