BACKPROP: The Gradient That Flows Back Into Governance
In a neural network, backpropagation is the algorithm by which an output error propagates backward through the layers, adjusting each weight so the network performs better on the next pass. The error is real; the adjustment is local; the improvement accumulates across iterations. On May 14, 2026, CANONIC shipped BACKPROP — a governed pattern backpropagation system that does the same thing for the governance tree: takes patterns extracted from 4,668 archived transcripts (Claude JSONL + ChatGPT data blobs, all flowing through the governed CAS at ~/.canonic/LEARNING/IDF/), and propagates them backward into the governance tree as VOCAB additions, LEARNING entries, and CANON updates. The input is 9,947 PROPOSED entries, each content-hash-addressed, each linked back to the source transcript and IDF extract that generated it. The output is a governed APPLIED/REJECTED ledger: 9,186 would-APPLY entries queued for operator-gated review, 761 human-review entries requiring explicit sign-off, and a verify-backprop-mvp gate that fails the build if the hash chain breaks. The governance tree is the model. The transcripts are the training data. BACKPROP is the update rule. The analogy to neural backpropagation is not decorative. It is the literal mechanism: the governance tree learns from its own transcripts the same way a network learns from its own errors.
What Gets Backpropagated
Three categories. First: VOCAB additions — new terms extracted from transcripts that do not exist in any governed VOCAB.md anywhere on the inheritance chain. Second: LEARNING patterns — new entries for LEARNING.md at the relevant service scope, one pattern per distinct signal extracted from the IDF. Third: CANON mutations — proposed edits to CANON.md constraints derived from patterns that reveal a missing required constraint or a violated invariant. CANON mutations require human review; they never auto-apply.
The source is the CAS at ~/.canonic/LEARNING/IDF/. The CAS holds 12,646 IDF entries produced by scan-transcripts Stage-1, which walks 4,668 transcript files and emits deduplicated, content-hash-identified extracts. Every IDF entry is the minimal unit: a pattern signal, its source transcript SHA, and the scope the pattern is attributable to. BACKPROP reads these 12,646 entries and generates PROPOSED rows in the backprop ledger — one row per candidate change, one candidate per IDF signal that either introduces a new VOCAB term, extends a LEARNING table, or implies a CANON constraint the tree does not currently declare.
The three categories are intentionally asymmetric in their review requirements. VOCAB additions are additive and idempotent — a term either exists or it does not; inserting it twice is a no-op at the next deduplication pass. LEARNING entries are additive and ordered — a new pattern row appended to LEARNING.md does not change existing rows. CANON mutations are structural — they change the contract that all downstream consumers validate against, and a wrong CANON change can cascade a build failure across every governed scope that inherits from the mutated node. The asymmetry in operator review burden maps exactly to the asymmetry in blast radius.
The Hash Chain
9,947 PROPOSED entries. Each entry carries five fields: source_sha (the content hash of the IDF entry that generated it), backprop_sha (a hash over the proposed change content — bucket, scope, and body), scope (the governance path where the change should land, e.g. hadleylab-canonic/SERVICES/BLOG), bucket (VOCAB | LEARNING | CANON), and status (PROPOSED | WOULD_APPLY | APPLIED | REJECTED).
The hash chain is the audit trail. verify-backprop-mvp walks every entry and verifies that backprop_sha is reproducible from its declared content fields and that source_sha resolves to a real IDF entry in the CAS. If either check fails — because an entry was hand-edited, because the IDF file was deleted, or because the scope path does not exist — the gate fails and the build refuses to ship. The chain does not prove that the proposed change is correct. It proves that every proposed change is traceable to the transcript that generated it and that no entry has been introduced without a governed source. Traceability is the floor. Correctness is what the operator review provides above it.
The verify gate passed 5/5 on the initial BACKPROP MVP shipment, covering all 9,947 entries across the full hash chain. That result established the baseline: the chain is valid from the IDF inputs through every proposed entry as shipped. Any future build that introduces a new PROPOSED entry without a matching IDF source will halt the gate immediately.
Three-Leg Batch Cleanup
A fresh BACKPROP queue does not clear in a single pass. It requires three legs, in order, and the order matters.
Leg 1: REJECT worktree-scoped entries. These are entries where the scope path resolves to something under .claude/worktrees/ — a transient working-tree artifact from a prior session that was never committed and no longer maps to a real governance node. Approximately 800 entries fell into this category on the initial triage. They are not governance; they are noise from sessions that opened worktrees and closed without cleanup. Rejecting them first clears the queue of entries that would block legitimate Leg-2 processing.
Leg 2: APPLY vocab-add would-APPLY. These are entries in the WOULD_APPLY status whose bucket is VOCAB and whose scope resolves to an existing VOCAB.md on disk. 9,186 entries qualified. The apply operation is idempotent: apply_vocab_add reads the target VOCAB.md, checks whether the term already exists (case-insensitive), and appends it if absent or returns a no-op if present. No-ops do not count as failures. The operation is safe to run without operator review for the same reason that adding a new row to a dictionary is safe: you are not changing any existing entry, you are only extending the namespace.
Leg 3: REJECT unapplied vocab. Any PROPOSED entry where apply_vocab_add returned a no-op — because the target VOCAB.md does not exist yet, or because the scope path is under a service whose VOCAB.md has not been created — stays PROPOSED until the VOCAB.md exists. These entries are not cleared by Leg 2. They require a prior governance step (create the VOCAB.md, declare it in CANON) before BACKPROP can reach them. Running only Legs 1 and 2 leaves these entries as ghost entries in the queue. Leg 3 surfaces them explicitly as REJECTED-pending-vocab-creation so they do not pollute the would-APPLY count.
backprop-batch is a 20-line wrapper over backprop-review --auto. It sequences the three legs with the correct flags and nothing else. Zero duplicate logic between the wrapper and the underlying operator surface.
Operator-Gated Review
The conservative triage default does not auto-apply anything. The initial BACKPROP triage produced 9,186 would-APPLY entries and 761 human-review entries. The 761 human-review entries are primarily CANON mutations and LEARNING patterns whose novelty score exceeded the CANON-implication threshold — they require the operator to read the proposed change and decide whether it accurately characterizes a real constraint or a real pattern before it lands in the governance tree.
backprop-review --auto exposes three independent axes as flags. First: which entries (--scope to restrict to a governance path, --bucket to restrict to VOCAB | LEARNING | CANON, --include-would-apply to include the 9,186 queue). Second: what to do (--auto applies or rejects based on the entry's current routing, --reason attaches a human-readable rejection rationale to the ledger). Third: preview mode (--dry-run prints the operations without executing them). The three axes compose independently. An operator who wants to review only the CANON mutations for a single scope runs --bucket CANON --scope hadleylab-canonic/SERVICES/API --dry-run to preview, then removes --dry-run to execute.
Any operator surface that exposes a --sha <sha> headless path for single-entry review should also expose --auto for batch mode. The two paths share 100% of the mutation logic — the only difference is which entries get routed to it. This is the same separation-of-concerns principle that governs the SESSION_CLOSES_TO_PR pattern: the mechanism that processes one unit and the mechanism that processes a batch of units are the same code, not two different implementations of the same policy.
Novelty Scoring
backprop-novelty assigns a novelty score to each PROPOSED entry against the existing LEARNING corpus before it reaches the review queue. Score < 0.2 = MARGINAL. MARGINAL entries are filtered from the review queue entirely — they are near-duplicates of patterns already in LEARNING, and surfacing them for operator review would obscure the genuinely new signals under a pile of redundant variations on existing rows.
The scoring compares the proposed pattern text against the full LEARNING corpus using TF-IDF similarity. A score of 0.0 means the entry is byte-identical to something already in LEARNING. A score of 1.0 means the entry shares no vocabulary with anything in LEARNING. The 0.2 threshold was calibrated empirically against the initial corpus: patterns below 0.2 were consistently near-duplicates on manual inspection; patterns above 0.2 consistently introduced vocabulary or constraint language not present in the existing corpus.
The novelty filter is not a quality gate. It is a triage gate. A MARGINAL entry is not wrong; it is redundant. An entry that survives the novelty filter is not guaranteed correct; it is guaranteed to be new enough to deserve operator attention. The filter's job is to make the 761 human-review queue tractable. Without it, the queue would contain thousands of near-duplicate LEARNING rows that pad the count without improving the corpus. With it, the operator reviews the entries that are genuinely worth reviewing.
What This Closes
The conventional reading concluded that the metalearning loop was already closed when intake-promote shipped. That is half right: live sessions were covered, but the entire archived corpus had no governed pathway back into the tree until BACKPROP.
BACKPROP closes the final leg of the metalearning loop. The full loop, as the AUTONOMOUS_COMPILER_METALEARNING_LOOP pattern describes it, runs: TRANSCRIPTS to CAS to IDF to BACKPROP to LEARNING.md to CLAUDE.md to next session. Before BACKPROP shipped, the loop had a structural break at the IDF-to-LEARNING leg.
The intake-promote system that shipped in March 2026 handled one portion of the problem: patterns generated in the current session, stored in ~/.claude/projects/*/memory/*.md, were classified by frontmatter type and appended to the relevant LEARNING.md at session-end. That closed the loop for live sessions. It did not address the archived transcript corpus — the 4,668 sessions of Claude JSONL and ChatGPT data blobs that predate the intake-promote system, or that ran in contexts where the intake hook was not active.
Those 4,668 sessions accumulated in the CAS without a governed pathway back into the governance tree. The IDF entries existed. The patterns were extracted. But the step from IDF extract to governed LEARNING row had no automated mechanism; it required the operator to manually read IDF outputs and hand-author LEARNING entries, which did not happen at scale. BACKPROP fills that gap with a governed, hash-chain-auditable, operator-gated mechanism that processes the entire archived corpus in a single batch operation.
After a BACKPROP apply batch, the next build recompiles CLAUDE.md with every accepted pattern from every transcript the platform has ever run. The BFS traversal that generates the CLAUDE.md context window will include patterns from sessions that predated the current governance toolchain. The agent's next session will start with that context. The loop is closed.
The Analogy Holds
Neural backpropagation adjusts weights proportional to the gradient of the loss with respect to those weights. The gradient is large when the weight contributed significantly to the error; the gradient is small when the weight was irrelevant. The update rule is: subtract a fraction of the gradient from the weight. Do this across all layers, from output to input, for enough iterations, and the network converges.
BACKPROP adjusts governance proportional to the novelty of the extracted pattern, weighted by the operator's confidence in the proposed change. Low-novelty patterns below 0.2 are filtered before they reach the update step — the analogue of a near-zero gradient that produces a weight update too small to matter. High-novelty patterns with CANON implications are routed to the 761 human-review queue — the analogue of a large gradient update that requires gradient clipping or learning rate scheduling to prevent the network from overshooting the minimum. The operator review is the learning rate scheduler: it allows the update when the update is safe and holds it when it requires more scrutiny.
The VOCAB additions that clear Leg 2 in batch are the analogue of weight updates in a well-conditioned layer: large batch, safe to apply, convergence is monotonic and provably correct. The CANON mutations that queue for human review are the analogue of weight updates near a saddle point: individually they look like progress, but without care they can destabilize the loss surface.
The vocabulary grows. The constraints sharpen. The model learns. The difference from a neural network is that the model is the governance tree, not a parameter matrix, and the learning is auditable in git: every APPLIED entry carries its source IDF SHA, its backprop SHA, the operator who approved it, and the commit in which it landed. A neural network's weight matrix at step N does not tell you which training example caused which weight to change. BACKPROP's ledger tells you exactly that.
Sources
| Claim | Source | Link |
|---|---|---|
| Backpropagation propagates an output error backward through the layers, adjusting each weight by the gradient of the loss; revived for deep learning by Rumelhart, Hinton, and Williams (1986) | "Backpropagation," Wikipedia | en.wikipedia.org |
| The algorithm efficiently computes the gradient of the loss with respect to network weights by propagating derivatives backward one layer at a time | Nielsen, How the backpropagation algorithm works | neuralnetworksanddeeplearning.com |
| Content-addressable storage verifies integrity because each modified entry generates a new content hash, the property the backprop hash chain relies on | Content-Addressable Storage (CAS), Abilian Innovation Lab | lab.abilian.com |
| 9,947 PROPOSED entries hash-chain valid; verify-backprop-mvp 5/5 PASS; conservative triage 9,186 would-APPLY + 761 human-review | LEARNING: S74 r1/r2 BACKPROP MVP; MEMORY project_session_S74_r2_closure.md | — |
| Three-leg batch cleanup pattern; backprop-review --auto batch mode | LEARNING: BACKPROP_BATCH_THREE_LEGS, BACKPROP_REVIEW_AUTO_MODE (Session S86) | — |
| 4,668 transcripts scanned (Claude JSONL + ChatGPT data); 12,646 IDF entries in CAS | LEARNING: Ghost Worker audit, ~/.canonic/LEARNING/IDF/ | — |
| Novelty filter below 0.2 marks an entry MARGINAL | LEARNING: METALEARNING_OPTIMIZATIONS (Session S46) | — |
BACKPROP: The Gradient That Flows Back Into Governance | ENGINEERING | BLOGS