sc:learn
Pipeline Deep Dive

What sc:learn does

sc:learn is the knowledge ingestion command for Skill Creator. You point it at a source — a markdown file, a PDF, a GitHub repo, a zip archive, a URL — and it runs a seven-stage pipeline that extracts structured mathematical primitives from the content, deduplicates them against your existing registry, and generates skills, agents, and teams from what it learns.

Every piece of knowledge that enters the system goes through sanitization, human approval, structural analysis, semantic deduplication, and provenance tracking. Nothing is silently overwritten. Everything is reversible via sc:unlearn.

Function Signature

async function scLearn(
  source: string,
  options: ScLearnOptions = {}
): Promise<ScLearnResult>

The function returns a ScLearnResult containing: a success flag, a unique sessionId (for unlearn/revert), a report with full provenance, a changeset (null in dry-run mode), and an errors array.

The Seven Stages

Every invocation flows through these stages in order. Early exits at stages 1 and 3 prevent bad data from reaching extraction. The pipeline broadcasts progress events at each stage via the onProgress callback.

Acquire

The acquirer is the entry point. It detects what you gave it, downloads or extracts it, normalizes everything to plain text, and stages the results with provenance metadata.

Source Detection

The source string is analyzed to determine both the type and the familiarity — the familiarity tag drives how aggressively the sanitizer checks the content downstream.

Supported Formats

Staging

Every file is normalized to plain text and written to .learn-staging/ as a .staged.txt file. The staging record carries the original filename, byte size, encoding, and source path — this metadata follows the content through every subsequent stage for provenance tracking.

Sanitize

The sanitizer enforces hygiene rules tiered by familiarity. STRANGER content gets the full battery of six check categories. HOME content gets a lighter scan — it still checks for prompt injection and embedded code, but trusts local files enough to skip hidden character and external resource checks.

Check Categories

The sanitizer outputs a HygieneReport with all findings grouped by severity. Any critical finding means the report is marked as "not passed" — the HITL gate will require explicit user approval to continue.

The autoApproved flag is only set when the source is HOME familiarity and there are zero findings of any severity. This is the one path that skips user interaction entirely.

HITL Gate

The Human-In-The-Loop gate implements three approval rules — no exceptions. The gate uses dependency injection (PromptFn) for testability, falling back to @clack/prompts for CLI interaction.

The Three Laws

Both approved and approved-with-warnings allow the pipeline to continue. The full decision audit trail — status, rationale, who decided, timestamp — is recorded in the HITL result for downstream provenance.

Analyze

The analyzer takes each staged file's plain text and produces a structural map plus domain classification. This drives the extractor's heuristic selection in the next stage.

Structure Extraction

Markdown headings (# through ######) and numbered sections (1.1, 2.3.1) are parsed into a hierarchical tree of DocumentSection objects using a stack-based nesting algorithm. Each section carries its title, level, content, children, and character offset in the original document.

If no headings are found, the entire content is wrapped in a single root section titled "Document."

Content Type Classification

The analyzer scores content against five pattern families using weighted regex match counts:

Below a cumulative score of 2, the content is classified as unknown — the extractor will use all strategies simultaneously and cap confidence at 0.4.

Domain Detection & Complex Plane

When domain definitions are provided, keywords are scored against activation patterns using the formula:

score = min(1.0, sqrt(matchCount / totalPatterns) × (1 + 0.15 × matchCount))

Domains exceeding the activation threshold (0.1) are used to compute a weighted centroid position on the Complex Plane. This position follows the primitive through deduplication — it's how the prefilter quickly identifies potential duplicates by proximity.

Extract + Wire

This is where raw text becomes structured mathematical primitives. The extractor uses content-type-specific heuristics to find definitions, theorems, algorithms, techniques, and axioms. Then the dependency wirer connects them.

Content-Type Heuristics

Each Candidate Primitive Gets

A unique ID (domain + slugified name), a formal statement, a computational form (derived per type — e.g., definitions get "Defines: ...", theorems get "Given preconditions, then: ..."), the source section and offset, extraction confidence, applicability patterns (top section keywords), and keywords from the statement text. Dependency arrays start empty.

Dependency Wiring — Four Passes

After extraction, the wirer fills in dependencies, enables, and prerequisites arrays by analyzing relationships between candidates:

Dedup + Merge

The deduplication pipeline runs each candidate against the existing registry in two tiers — a fast prefilter to eliminate obviously-new content cheaply, then deep semantic comparison only for realistic duplicate candidates.

Tier 1 — Pre-filter

Two conditions must both be true for a candidate to be flagged (AND logic):

// Both must pass for a candidate to be flagged as potential duplicate
planeDistance(candidate, existing) <= 0.2   // Euclidean distance on Complex Plane
sharedKeywords(candidate, existing) >= 2     // Case-insensitive keyword intersection

Candidates that don't match anything skip straight to the merge engine as genuinely-new.

Tier 2 — Semantic Comparison

Flagged candidates get deep comparison using Jaccard similarity on tokenized text. Each candidate-existing pair is classified into one of five classes:

Merge Engine Safety

Generalization Merge Strategy

When a candidate generalizes an existing primitive, the merge is mathematically precise:

// The generalized primitive takes the broader scope
formalStatement    ← candidate.formalStatement       // Candidate's broader statement
computationalForm  ← candidate.computationalForm     // Candidate's broader form
applicabilityPatterns ← union(existing, candidate)    // Both sets of patterns
keywords           ← union(existing, candidate)       // Both sets of keywords
prerequisites      ← intersect(existing, candidate)  // Only shared prerequisites (relaxed)

This preserves the existing primitive's ID and metadata while broadening its scope — the union of patterns widens applicability, and the intersection of prerequisites relaxes conditions.

Changeset Recording

Unless dryRun is true, every modification (add, update, remove) is recorded in the changeset manager with before/after states. This enables sc:unlearn to cleanly revert all changes from a session. Revert operations run in reverse order and validate graph integrity before executing — if removing a primitive would create dangling references, the revert is blocked unless force=true.

Generate + Report

The final stage groups merged primitives by domain and generates artifacts when the domain crosses a threshold. Only primitives from add or update actions are included — skipped and conflicted primitives don't generate artifacts.

Generated Artifacts

Importance Scoring

Key primitives are ranked by importance score: the count of downstream enablements plus the count of composition rules. This is a proxy for centrality — primitives that enable many others and participate in compositions are the most valuable teaching material for the skill.

importanceScore(p) = p.enables.length + p.compositionRules.length

Learning Report

Every session produces a full provenance report containing: session ID, source path, timing (started/completed), the entire provenance chain (every merge decision with rationale), counts of primitives added/updated/skipped/conflicted, and lists of generated skills, agents, and teams.

Safety Architecture

Every layer of the pipeline is designed around explicit safety invariants. Here's the complete picture:

Options Reference

Typical Usage

// Learn from a local textbook
const result = await scLearn('linear-algebra.pdf', {
  domain: 'linear-algebra',
  depth: 'deep',
});

// Dry-run against a GitHub repo
const preview = await scLearn('https://github.com/user/docs', {
  scope: ['docs/', 'guides/', 'README.md'],
  dryRun: true,
});

// Learn with progress reporting
const result = await scLearn('notes.zip', {
  onProgress: (stage, detail) => console.log(`[${stage}]`, detail),
});

File Map

src/commands/
  sc-learn.ts              ← Orchestrator (this guide)
  sc-learn.test.ts         ← 20+ test cases covering all paths
  sc-unlearn.ts            ← Revert command using changeset manager

src/learn/
  acquirer.ts              ← Stage 1: source detection, download, normalize
  sanitizer.ts             ← Stage 2: tiered hygiene checks
  hitl-gate.ts             ← Stage 3: human-in-the-loop approval
  analyzer.ts              ← Stage 4: structure + content type + domain
  extractor.ts             ← Stage 5a: heuristic primitive extraction
  dependency-wirer.ts      ← Stage 5b: four-pass dependency wiring
  dedup-prefilter.ts       ← Stage 6a: fast plane+keyword screening
  semantic-comparator.ts   ← Stage 6b: deep Jaccard classification
  merge-engine.ts          ← Stage 6c: merge strategies + conflict mgmt
  changeset-manager.ts     ← Reversible changeset recording
  report-generator.ts      ← Stage 7: provenance report generation
  generators/
    skill-generator.ts     ← Learned skill file output
    agent-generator.ts     ← Agent definition output
    team-generator.ts      ← Team configuration output