Run Evals Against a Skill

Use the bmad-eval-runner skill to run a skill’s evals in a clean workspace and produce a graded report.

When to Use This

After editing a skill, to confirm nothing regressed
Before publishing a module, to validate every skill you ship
When debugging a description that fires on the wrong queries
When checking that dependency skills are wired correctly

When to Skip This

Quick iteration where you are running the skill manually and reading the output yourself
Skills with no defined evals (the runner halts on missing evals; it does not invent them)

Step 1: Confirm Eval Discovery

The runner looks for evals in this order, taking the first match:

The path you pass via --evals
<skill-path>/evals/
<skill-path>/../../evals/<skill-name>/
<project-root>/evals/<skill-name>/
<project-root>/evals/**/<skill-name>/ (fuzzy)

If discovery fails, the runner halts. It does not invent evals.

Step 2: Know What Isolation You Get

There is nothing to choose here. Every case runs in its own clean working directory with the skill under test staged into it, and the subprocess environment is built from scratch instead of inherited: PATH, a fresh empty HOME inside the case folder, CLAUDE_CONFIG_DIR inside that HOME, and the adapter’s auth variable. Your global CLAUDE.md, your auto-memory, and your host-installed skills are all out of reach, including for trigger evals.

Step 3: Pick Mode

Pass --mode artifact|trigger|both. Default is both if both eval files are found.

Mode	Effect
artifact	Runs `evals.json` only
trigger	Runs `triggers.json` only
both	Runs everything in parallel

Step 4: Run the Skill

Invoke the eval runner from your project. A typical invocation:

bmad-eval-runner ./src/skills/my-skill --workers 8

The runner stages each eval’s workspace, executes claude -p against the prompt, captures the stream-JSON transcript, and rsyncs any files the skill wrote. After all evals complete, it spawns a grader subagent per eval (in parallel) and aggregates the verdicts.

Step 5: Inspect Results

When the run finishes, the runner emits two paths:

The run folder, at ~/bmad-evals/<run-id>/ (or your configured bmad_builder_reports location)
An HTML report at <run-folder>/report.html

Open the report for the summary view. Drop into the run folder for full transcripts, artifacts, and grading details for any eval you want to examine.

What You Get

~/bmad-evals/20260509-172903-my-skill/
├── run.json                     # Run metadata
├── report.html                  # Aggregate HTML report
├── A1/
│   ├── prompt.txt               # The eval's prompt verbatim
│   ├── transcript.jsonl         # Stream-JSON tool calls and messages
│   ├── artifacts/               # Files the skill wrote
│   ├── grading.json             # Per-expectation verdicts
│   └── metrics.json             # Timing and tool-call counts
├── A2/
│   └── ...
└── triggers-result.json         # Trigger eval rates

Run folders are never deleted automatically. Disk management is your call.

Tips

Pass --eval-ids A1,B3 to run only specific evals while iterating
Pass --workers 8 to parallelize aggressively (default is 4)
A specific eval can override the default timeout by setting "timeout": 900 in its evals.json entry

A Worked Example

The bmad-product-brief skill in the BMad Method repository (bmad-code-org/BMAD-METHOD) ships a complete eval suite at evals/bmm-skills/bmad-product-brief/. To run it end-to-end:

bmad-eval-runner ./src/bmm-skills/1-analysis/bmad-product-brief --workers 8

The run produces 17 graded artifact evals (A1-A8 output grading, B1-B8 transcript grading, C1 configuration compliance), 15 trigger eval verdicts, and an aggregated HTML report. Use it as the model when writing evals for your own skills.

Next Steps

For the complete evals.json and triggers.json schema, see Eval Format. For concepts and patterns, see What Are Evals.