Testing & Simulation

Full YAML Schema for Test Cases

# Complete test case schema (.claude/tests/*.yaml)
id: string                    # Required. Unique identifier (e.g., "test-001")
description: string           # Required. Human-readable test intent
context: string               # Required. Input text for activation simulation
expected_skills:              # Required. Skills that MUST activate
  - skill-name.md             # Filename relative to .claude/commands/
expected_absent:              # Optional. Skills that MUST NOT activate
  - other-skill.md
tags:                         # Optional. Labels for filtering test runs
  - regression
  - commit
  - typescript
weight: 1.0                   # Optional. Weight in F1 calculation (default: 1.0)
min_activation_score: 0.75    # Optional. Minimum score for expected skills

Simulation Flags

# Full simulation command options
skill-creator test [options]

Options:
  --simulate          Run activation simulation (default mode)
  --suite <path>      Path to test suite directory (default: .claude/tests/)
  --filter <tag>      Run only tests with matching tag
  --verbose           Show full SimulationResult for each test
  --threshold <n>     Override activation threshold (0.0–1.0)
  --calibrate         Auto-tune threshold to maximize F1
  --export <file>     Write results to JSON file for CI integration

Output Format

// Full SimulationResult structure (--verbose output)
{
  "suite_summary": {
    "total": 12,
    "passed": 11,
    "failed": 1,
    "precision": 0.92,
    "recall": 0.88,
    "f1": 0.90,
    "mcc": 0.85,
    "threshold_used": 0.72
  },
  "results": [
    {
      "test_id": "test-001",
      "passed": true,
      "activated": ["commit-style.md"],
      "precision": 1.0,
      "recall": 1.0,
      "f1": 1.0
    }
    // ... one entry per test case
  ]
}

The Precision/Recall Tradeoff

Every activation threshold creates a tradeoff. A low threshold (e.g., 0.3) means skills activate easily — high recall, but many false positives lower precision. A high threshold (e.g., 0.9) means skills are selective — high precision, but relevant skills may fail to fire, lowering recall.

The F1 score is the harmonic mean of precision and recall. Unlike the arithmetic mean, F1 punishes extreme imbalances — a system with 100% precision and 0% recall gets an F1 of 0, not 50%. This makes F1 the right optimization target for skill activation.

Threshold Calibration Procedure

# Step 1: Run calibration sweep
skill-creator test --calibrate --target-f1 0.90

# The simulator sweeps thresholds from 0.3 to 0.95 in 0.05 increments
# For each threshold, it runs the full test suite and records F1
# Output: threshold recommendation + F1 curve

# Step 2: Review recommendation
# Calibration result:
# Optimal threshold: 0.72 (F1: 0.91 at this setting)
# Current threshold: 0.65 (F1: 0.87)
# Recommended change: +0.07 to threshold

# Step 3: Apply to integration.json
{
  "activation_threshold": 0.72
}

F1 and MCC Metrics

F1 Score: 2 * (P * R) / (P + R) — The standard metric for binary classification quality. Best for imbalanced datasets where negative examples (skills that should not fire) vastly outnumber positive examples.

MCC (Matthews Correlation Coefficient): Takes all four quadrants of the confusion matrix into account (TP, TN, FP, FN). More reliable than F1 when the class distribution is highly skewed. Range: −1 (inverse prediction) to +1 (perfect prediction).

Integration with CAP-025 (Evaluator-Optimizer)

In v1.9, the Evaluator-Optimizer (CAP-025) extended this calibration process with A/B testing: instead of tuning a single skill variant, CAP-025 can run two skill versions head-to-head on the same test suite and select the higher-performing variant automatically. CAP-008 provides the simulation infrastructure; CAP-025 adds the comparative evaluation and automatic promotion logic.