CAP-054: Pipeline Learning

Structured Data

Capability Metadata

FieldValue
IDCAP-054
Versionv1.13
DomainPattern Intelligence
StatusDocumented
RelatedCAP-050, CAP-055
Sourcesrc/patterns/pipeline-learning.ts
AlgorithmJaccard similarity for feedback matching

Configuration Reference

Pipeline learning config:

{
  "pipeline_learning": {
    "jaccard_threshold": 0.70,
    "min_observations_to_learn": 3,
    "library_version_limit": 10,
    "best_match_top_k": 3
  }
}

Technical Specifications

Jaccard similarity calculation:

def jaccard_similarity(list_a: set, list_b: set) -> float:
    """Feedback matching: how similar are two pipeline lists?"""
    intersection = len(list_a & list_b)
    union = len(list_a | list_b)
    return intersection / union if union > 0 else 0.0

Key components: Observation-to-list compiler, Jaccard feedback engine, versioned pipeline list library, best-match retrieval (top-K similarity search)

Overview

Pipeline learning turns pipeline list execution history into an improving library, matching new workflows to the most similar previously-learned list. The system closes the loop on pipeline lists: you execute a workflow, the system observes the tool sequence, compiles it into a candidate list, measures similarity to existing library entries using Jaccard scoring, and either merges it into an existing entry (if similarity exceeds the threshold) or adds it as a new entry.

The feedback engine uses Jaccard similarity because pipeline lists are naturally set-like: a list of tool invocations is an ordered set, and two lists that share the same tools (regardless of precise ordering) are more similar than two lists with no tool overlap. Jaccard intersection-over-union gives a principled 0-to-1 similarity score without requiring an explicit distance metric or embedding model.

The versioned library preserves up to 10 versions of each entry, enabling rollback when a workflow changes and the updated list diverges from earlier versions. Best-match retrieval returns the top-3 most similar library entries for a given incoming workflow, allowing the caller to choose the most appropriate match rather than relying on a single best-guess.

History

v1.13's pipeline learning closed the loop on pipeline lists: execute → observe outcomes → Jaccard match to prior lists → retrieve best match for next similar workflow. Before pipeline learning, pipeline lists were manually authored and statically versioned — a developer would write a list, test it, and commit it. The feedback loop was entirely manual.

The observation-to-list compiler was the enabling piece. It takes a raw session observation (a sequence of tool_use/tool_result pairs) and converts it into a normalized pipeline list format — deduplicating sequential identical calls, identifying loop boundaries, and extracting the stable tool sequence that represents the workflow's structure. This normalization step is what makes Jaccard comparison meaningful: without it, two observations of the "same" workflow with minor variations would compare as very different.

The 70% Jaccard threshold was established empirically. Below 70%, merged entries were too heterogeneous to be useful as templates. Above 70%, the system missed real variations that should be distinct entries. The threshold is configurable because different domains have different natural variation — a highly structured domain (CI pipeline) might use 85%, while a more exploratory domain (research workflows) might use 60%.

Key Concepts

For Students & Teachers

Imagine a recipe book that learns. Every time you cook something, the book watches and writes down what you did. Then it compares your new recipe to ones already in the book: do they share the same ingredients? (That's Jaccard similarity — shared ingredients divided by all ingredients in either recipe.) If the new recipe is 70%+ similar to an existing one, they're merged; otherwise, a new recipe is added.

When you want to cook something similar in the future, the book finds the three most similar recipes to what you describe and shows them to you — you pick the best match. The book doesn't pick for you; it narrows the search space from hundreds of recipes to three good candidates.

Versioning is like keeping old editions of the recipe — if the merged version turns out worse than the original, you can go back to version 3 of the recipe and start over. The system never loses information; it just accumulates it in a structured way.

Usage Examples

Example 1: Compile observation to pipeline list

# Compile the latest session observation into a pipeline list
skill-creator pipeline compile --session latest

# Output:
# Observation: obs-abc123 (47 tool invocations)
# Normalized list: 8 unique tool steps (dedup: 39 repeated calls removed)
# Jaccard match against library: 3 candidates
#   1. "lint-test-commit" (similarity: 0.875) — MATCH (above 0.70 threshold)
#   2. "test-only-workflow" (similarity: 0.542) — no match
#   3. "full-ci-cycle" (similarity: 0.411) — no match
# Action: Merging into "lint-test-commit" (v4 → v5)

Example 2: Retrieve best matches for a new workflow

# Find the top-3 most similar library entries for a workflow description
skill-creator pipeline match \
  --tools "read_file,bash,edit,bash,git_add,git_commit"

# Output:
# Top-3 matches from library (Jaccard similarity):
#   Rank 1: "lint-test-commit" (score: 0.833)
#     Tools: read_file, bash, edit, bash, bash, git_add, git_commit
#   Rank 2: "quick-fix-commit" (score: 0.714)
#     Tools: read_file, edit, git_add, git_commit
#   Rank 3: "full-ci-cycle" (score: 0.500)
#     Tools: read_file, bash, edit, bash, bash, bash, git_add, git_commit, bash

Example 3: View library entry versions

# Show version history for a library entry
skill-creator pipeline versions "lint-test-commit"

# Output:
# Entry: lint-test-commit
# Current: v5 (merged 2026-02-24, 3 observations)
# v4 (merged 2026-02-23, 2 observations)
# v3 (merged 2026-02-22, 1 observation)
# ...
# v1 (created 2026-02-20, initial compile)
# Rollback: skill-creator pipeline rollback "lint-test-commit" --version 4

Related Capabilities