Group E — Experiments

Structured experiment records. See journal.md for narrative.

Hardware

All Group E runs on Apple M4 Pro (14 cores: 10P + 4E, 48 GB unified memory, macOS Darwin 25.3.0 arm64). Rayon auto-parallelises across cores.

E1: Warm-start (Net2Net) patch insertion across 5 niches

Date: 2026-05-14 Binary: src/bin/group_e_warm_patch.rs Code changes: added add_patch_warm and add_patch_warm_burst to genome/mutation.rs; warm_patch_insertion: bool on MutationConfig; routing in mutate().

Hypothesis: Net2Net-style warm-start insertion (clone an existing patch’s indices+weights with ±2-pixel index perturbation, halve the parent’s outgoing weights and add child→target connections at the halved weight) makes patch additions behavior-preserving at insertion. The host’s fitness no longer drops, so macro-mutants survive selection — closing the loop that culled D2’s cold mutants back to seed count.

Setup: 5 niches (mnist, fashion, kmnist, emnist, mixed). 300K steps each, pop 50, 128 seed patches (50% spatial / 50% random-index), add_patch_prob=0.10, add_patch_burst_prob=0.05 with burst count 4, warm_patch_insertion=true. Otherwise identical to D2 (group_c_niche_growth). Seeded 0xE101 (D2 used 0xD201).

Data caveat: M4 only had MNIST + Fashion locally; KMNIST and EMNIST sourced from HuggingFace parquet mirrors (tanganke/kmnist, claudiogsc/emnist_balanced) and converted to IDX via /tmp/parquet_to_idx.py. EMNIST accuracy lands ~1pp below D1 reference, suggesting minor parquet vs original-IDX drift. The patch-count signal is comparable; the accuracy signal needs a sanity check against original-IDX EMNIST (see E0 in journal).

Patch count, top 3 by fitness:

In D2, only EMNIST’s rank-2 was above seed count. In E1, 4 of 5 niches have above-seed individuals in top-3, and KMNIST has them at rank 1 and rank 2.

Accuracy (top individual, own task, test set):

All within noise of D1/D2. Warm-start grows count but does not (yet) translate to accuracy.

Cycle-breaker firings: 0 across the 5-niche, 1.5M-step run. D2 had 1 in KMNIST. The cycle bug is rare and the warm-start path didn’t trip it (warm insertions add child→target connections at innovation numbers that don’t pre-exist, so the cycle pattern that requires matching innovation in both parents during crossover is no more common than in D2).

Wall time: ~30 minutes total on M4 (10 cores active under rayon). Significantly faster than expected — D2 reference on i9 took comparable time per niche.

Analysis:

1. The cold-mutation problem is unblocked at survival level. D2 culled cold mutants out of the top tier (only 1 of 5 niches had a >128 individual in top-3); E1 keeps them alive in the top tier (4 of 5 niches). KMNIST is the cleanest case: top and rank-2 both at 132 patches, indicating selection consistently preferred the larger architecture across multiple lineages.

2. No accuracy translation in this step budget. The 50/50 head split is behavior-preserving at insertion but pays a “halving tax” each parent lineage must re-earn through SGD. Mutants added in the last ~10 generations don’t have enough training time to differentiate from their parents and contribute independent signal. Predict E2 (longer total steps) closes this.

3. EMNIST is the anomaly. Predicted to grow most (most under-capacitied at 128); actually grew least. Most plausible explanations: parquet/original-IDX drift in the EMNIST distribution (consistent with the −0.8pp accuracy floor); 47-class CE punishes the halving transient more sharply; or noise with N=50.

4. Sanitize stayed clean. 0 dangling drops, 0 cycle breaks, 0 crossover-pruned across all 5 niches. The warm-insertion code path doesn’t introduce new invariant violations.

Key insight: warm-start is a necessary but not sufficient mechanism for count evolution. It removes the selection-culling barrier (necessary), but the 50/50 head split delays accuracy translation by paying a halving tax (insufficient at 300K steps). The split ratio is now the next axis to interrogate — it trades insertion stability against post-insertion training burden.

Conclusion: positive on the immediate hypothesis (cold-mutation problem unblocked) but partial on the downstream claim (accuracy gain from count growth). Sets up two concrete follow-ups: more training steps (E2) and asymmetric splits (E3).

E0: parquet EMNIST data sanity check

Date: 2026-05-14 Binary: src/bin/group_e_emnist_check.rs Goal: Confirm the parquet-derived EMNIST is close enough to D1’s reference IDX that E1’s EMNIST anomaly is not explained by data drift.

Setup: C8/D1 baseline reproduction — 300K steps, pop 50, 128 seed patches, no add_patch mutation. EMNIST niche only.

Result (top 3 on EMNIST test set):

• id=445: 77.72%
• id=456: 77.44%
• id=393: 77.65%

Reference: D1/C8 was 78.3%.

Drift: ~0.6pp below reference. Bounded — orientation/normalization errors would produce >5pp degradation. Most likely cause: different example ordering in the parquet than the canonical IDX, so a train_fraction=5/6 split slices a different test subset.

Implication for E1: E1’s EMNIST top (77.48%) vs E0 baseline (77.72%) — difference of −0.24pp, within run-to-run noise. Warm-start did not damage EMNIST accuracy. The “EMNIST didn’t grow patches” finding in E1 remains a real signal on the count axis but is not the data quality problem the journal flagged.

E2: warm-start with 3× longer training budget (900K steps)

Date: 2026-05-14 Binary: src/bin/group_e_warm_long.rs Hypothesis: E1 grew patch count in 4/5 niches but accuracy stayed flat. The hypothesized cause was insufficient post-insertion training time to repay the Net2Net halving tax. With 3× more steps, warm-start mutants should have time to specialize away from their parents and translate count growth into accuracy.

Setup: identical to E1 except TOTAL_STEPS = 900_000. Same niches, seeds, pop size, hyperparameters. Seed 0xE201 (E1 was 0xE101).

Top-individual accuracy and patch count, E1 → E2:

Population-level average patch count at end of run:

Sanitize stats: 0 dangling drops, 0 cycle breaks across the full 4.5M-step run (5 niches × 900K). Warm-start path doesn’t introduce invariant violations at extended budgets.

Wall time: ~95 minutes total on M4 (~19 minutes per niche).

Analysis:

1. Headroom-driven pattern emerges cleanly. Niches sorted by accuracy headroom show the same rank order on both count growth and accuracy gain:
   • MNIST (97%, saturated) → no count growth, +0.3pp from longer training of existing patches.
   • Fashion (87%, near-saturated) → small count growth, no accuracy gain.
   • KMNIST (90%, mid-room) → clear count growth, modest accuracy gain.
   • Mixed (81%, headroom) → big count growth, big accuracy gain (+1.7pp).
   • EMNIST (77%, most headroom) → biggest count growth (top 151, avg 152.5), biggest accuracy gain (+1.6pp).

2. The E1 EMNIST anomaly was a training-budget artifact, not a structural problem. With 300K steps EMNIST couldn’t sustain macro-mutants in its top tier; with 900K steps it grows count harder than any other niche (19% above seed at the population level). The original D1/D2 prediction that EMNIST should grow most because it has the most under-capacity is restored.

3. Mixed niche is the cleanest two-axis win. +1.74pp accuracy at +10 patches over E1. Mixed carries all four tasks so it has the widest aggregate fitness headroom — warm-start exploits exactly that.

4. MNIST saturation is real. Tripling training time gave +0.3pp with no count growth. That signature — accuracy moves but count doesn’t — is the marker of a niche where extra capacity is selection-neutral. Group B B25 found MNIST gains from depth (not patches); these results align.

5. The Net2Net halving tax is repayable. All non-saturated niches showed accuracy gain proportional to count growth, demonstrating that the warm-start mutants do specialize away from their parents given enough training time. The mechanism works as designed.

Conclusion: E1’s “warm-start unblocks survival but not accuracy” partial result becomes a full positive in E2. The training-time hypothesis is confirmed. Warm-start delivers measurable accuracy gain on non-saturated niches; on saturated niches it correctly does nothing (no fitness signal for extra capacity). The mechanism interacts with ecological speciation in the predicted way — each niche’s own headroom determines the outcome.

E3: split-ratio ablation (0.5, 0.7, 0.9) on Fashion and KMNIST

Date: 2026-05-14 Binary: src/bin/group_e_warm_split_ratio.rs Hypothesis: E1’s halving tax (Net2Net 0.5/0.5 split halves each lineage’s downstream contribution at insertion) might be reducible with asymmetric splits. At warm_parent_ratio = 0.7, parent keeps 70% of its outgoing weight and child gets 30% — child is weaker but parent is less perturbed; child has lower halving tax to repay against a parent that has more to lose if the child fails to specialize. Predict 0.7 outperforms 0.5; 0.9 may underperform because child starts too weak.

Setup: 2 niches (Fashion, KMNIST — the niches with clearest count growth in E1) × 3 split ratios (0.5, 0.7, 0.9) × 300K steps. 6 cells total. Otherwise identical to E1.

Cycle-bug interlude: First run panicked at phenotype.rs:209 in cell 2 (Fashion 0.7/0.3) during gen 1, same signature as the D2 cycle bug that the Phase D sanitize() was supposed to fix. Apparently the mutate-end sanitize misses a class of crossover-induced cycles that the warm-start path can produce. Applied a defensive genome.sanitize() call in Individual::from_genome as a guard. Re-ran with the patch — all 6 cells completed. Total cycle breaks across the run: 1849. The defensive sanitize is firing constantly; the mutate-end sanitize is incomplete.

Top individual per cell (test accuracy on own task):

Patch-count growth at population level (avg across all 50 individuals at end of run):

Analysis:

1. Asymmetric splits did not enable more count growth. Only the 0.5/0.5 cells produced top individuals with >128 patches. Both 0.7 and 0.9 on both niches kept top patches at 128. Population-level averages are essentially flat across all three ratios (129 ± 1 patch).

2. The hypothesis is inverted by the result. I predicted higher parent_ratio would help because the parent has less halving tax to repay. The opposite happened: at parent_ratio > 0.5, the child starts so weak (30% or 10% of original outgoing weight) that it’s effectively a “warm cold start” — most of the original signal is still in the parent, the child contributes almost nothing, and selection treats it as overhead. The 0.5/0.5 ratio is Pareto-optimal in this setup: it gives the child enough downstream influence to be a meaningful lineage that SGD can specialize.

3. Test accuracy is essentially flat across ratios. Fashion ranges 86.27-86.63%; KMNIST ranges 90.18-90.48%. The ~0.3pp variance is within seed noise.

4. Fitness vs test accuracy decouples on Fashion. Fashion 0.7 has the highest fitness (0.8903) but second-lowest test (86.37%). Fashion 0.5 has the lowest fitness (0.8773) but highest test (86.63%). This pattern suggests Fashion 0.7’s population overfits to the rolling-window training distribution slightly more than the 0.5 cell does — possibly because the asymmetric split creates more lineage-internal correlation (parent and child trained on same window with very similar contributions) reducing population diversity.

5. Cycle-break counts are seed-dominated, not ratio-dominated. 0/621/137 on Fashion and 0/0/1091 on KMNIST — no consistent pattern with parent_ratio. The cycle bug fires when the RNG cascade happens to produce a crossover+mutate combination that closes a cycle in the enabled subgraph. The defensive sanitize handles all of them transparently.

Conclusion: The Net2Net 0.5/0.5 split is the right ratio. Asymmetric splits don’t unlock additional count growth or accuracy. The “halving tax can be reduced” intuition was wrong because it ignored that the child’s viability as a lineage drops faster than the parent’s overhead does.

Secondary outcome: discovered and patched (defensively) a cycle bug in the mutate-end sanitize path. The exact crossover+mutation sequence that produces sanitize-resistant cycles is still unidentified; flagged for root-cause investigation.

Updated next-step priority (post-E3):

E3 settles the local design question (use 0.5/0.5). The bigger live questions are:

• E4: capacity ceiling on EMNIST. E2 showed EMNIST hits 79.07% at 151 patches with 900K steps. Where does the asymptote sit? 1.8M steps, seed 128, see how far count climbs and what accuracy plateau it reaches. This bounds the value of warm-start for the most under-capacitied task.
• E5: warm-start + depth. D3 showed depth=32 hidden layer gives +3.3pp on KMNIST. Combine warm-start patches with the depth=32 architecture; does count evolution find a smaller or larger optimum when patches feed a hidden layer instead of outputs directly?
• Cycle-bug root-cause: identify the exact mutate/crossover sequence that produces sanitize-resistant cycles. Worth doing before more experiments because every future run carries the same defensive cost.
• Permuted-MNIST continual learning: the directionally most-valuable experiment per the earlier strategy conversation — tests the “population diversity buffers forgetting” claim against a standard CL benchmark.

E4: EMNIST capacity ceiling at 1.8M steps

Date: 2026-05-14 Binary: src/bin/group_e_emnist_ceiling.rs Wall time: 178 seconds on M4 Pro (1.8M steps, single niche). Hypothesis: E2 showed EMNIST went 128 → 151 patches and 77.48% → 79.07% in 900K steps; the trajectory hadn’t plateaued. Where does it stop?

Setup: EMNIST niche only. 1.8M steps, pop 50, 128 seed patches, warm-start at 0.5/0.5. Identical to E2’s EMNIST cell except for the longer budget. Seed 0xE401.

Top 5 individuals (EMNIST test set):

Average at end of run: 157.5 patches across the population.

Trajectory (EMNIST, increasing budget):

Analysis:

1. Returns are diminishing but not exhausted. Doubling the budget from 900K → 1.8M gave +0.83pp and +10 patches. The trajectory shape suggests the asymptote is around 80-81% at 170-180 patches — call it 4-6 more million steps of budget, plus diminishing gains beyond.
2. Best test accuracy is at rank 3, not rank 1. id=2659 hits 80.09% at 156 patches; the top-by-fit individual (id=2706) is at 79.90% with 161 patches. Suggests the fitness signal (rolling-window training accuracy) is starting to lose discriminative power above 80% — the top of the population is close enough that test-set ranking and training-fit ranking decouple.
3. Zero cycle breaks across the entire 1.8M-step run. This seed simply didn’t hit the cycle bug.

Conclusion: warm-start with extended budgets continues delivering on EMNIST, but with diminishing returns above 150 patches. The asymptote is the 80-81% range. Further gains likely require an architectural change (depth, which we test in E5) rather than more patches.

E5: warm-start + depth=32 hidden layer

Date: 2026-05-14 Binary: src/bin/group_e_warm_depth.rs Wall time: 110 seconds on M4 Pro (2 niches × 900K steps). Hypothesis: D3 showed depth=32 gives +3.3pp on KMNIST. Combining warm-start with depth=32 — does count evolution find a different optimum when patches feed a hidden bottleneck instead of outputs directly? Two competing predictions: hidden layer caps useful patch count, or depth amplifies each patch’s contribution.

Setup: 2 niches (KMNIST, EMNIST). 900K steps, pop 50, 128 seed patches, hidden_size=32, warm 0.5/0.5. Output: 77 classes (4-way label space, single-niche training).

Top individual per niche, three-way comparison:

Analysis:

1. The hidden layer caps count growth on KMNIST. With depth, KMNIST stays at 128 patches (no growth), 6669 connections — identical to D3’s seed structure. The 32-node hidden bottleneck makes extra patches selection-neutral. This is the capacity-ceiling shape predicted by hypothesis 1.

2. EMNIST count grows under depth but accuracy drops. EMNIST goes to 139 patches (some growth) but accuracy is 74.89% — worse than E2’s no-depth 79.07%, and similar to D3’s depth-no-warm 75.56%. This matches Group B B34’s finding that depth hurts EMNIST regardless of training schedule. The combined warm+depth doesn’t rescue it.

3. D3 > E5 on KMNIST is a deeper finding. Adding warm-start to depth=32 makes KMNIST worse than depth-alone (91.91% vs 93.49%). The warm-start mutation overhead (extra connections from child patches, half-strength patch contributions during training) hurts KMNIST when depth is already providing the right inductive bias. Tasks where depth helps may not benefit from additional count evolution.

4. The aggressive sanitize survives heavy cycle activity. This run produced 15,072 cycle breaks total (vs E3’s 1849, E2’s 0). The 32-node hidden layer creates a denser cycle-prone topology when crossover mixes connections across lineages. The new “all inter-cycle edges per pass × 16 cap” sanitize handles all of them with 0 late-cycle breaks. The cycle bug is now fully resolved (see journal for fix details).

Conclusion: depth and count evolution are substitutive, not additive, in this setup. KMNIST gets the capacity it needs from the 32-node hidden layer; extra patches are wasted. EMNIST’s structural problem with depth is robust to warm-start. For best results: use depth where it helps the task (KMNIST), use warm-start where it doesn’t (EMNIST, Mixed). Group B’s per-task architectural conditionality finding is reinforced.

E6: Permuted-MNIST continual learning

Date: 2026-05-14 Binary: src/bin/group_e_permuted_mnist.rs Wall time: 38 seconds on M4 Pro (2 conditions × 3 tasks × 300K steps = 1.8M total). Hypothesis: Population diversity should buffer catastrophic forgetting better than a single network. Continual-learning literature (EWC, replay, generative replay) addresses forgetting via regularization or rehearsal. The Group E informal hypothesis: 50 individuals carrying slightly different solutions naturally preserve more historical knowledge than 1 individual.

Setup: 3 sequential tasks, each is MNIST with a fixed pixel permutation (task 0 = identity). 300K steps per task. Two conditions:

• A: pop=1, no warm-start (single-individual baseline).
• B: pop=50, warm-start enabled (full population dynamics).

After each task, evaluate every individual on every task; report best accuracy per cell.

Accuracy matrix — Condition A (pop=1):

• Final avg accuracy: 50.07%
• Forgetting (max acc – final acc, per task): [79.7pp, 55.8pp, 0pp] → avg 45.18pp

Accuracy matrix — Condition B (pop=50 warm):

• Final avg accuracy: 51.53%
• Forgetting: [72.4pp, 60.3pp, 0pp] → avg 44.22pp

Comparison:

• A → B final accuracy: +1.46pp (50.07% → 51.53%)
• A → B average forgetting: −0.96pp (45.18pp → 44.22pp)
• Task 0 retention after task 2: 16.9% (A) vs 24.6% (B) → +7.7pp
• Task 1 retention after task 2: 39.4% (A) vs 35.1% (B) → −4.3pp (B actually worse)

Analysis:

1. Population dynamics provide weak forgetting protection at best. Both conditions still lose ~45pp on average — the bulk of the catastrophic-forgetting effect remains. Population shows a small task-0 preservation advantage (+7.7pp), partially offset by a task-1 disadvantage (−4.3pp). Net effect is within experimental noise.

2. This is a negative-leaning result for the informal “population buffers forgetting” hypothesis. The current ecological-speciation mechanism, applied to a sequential task stream with strong selection pressure on the current task, does not preserve historical knowledge in any meaningful way. Selection during task 2 drives the population toward task-2 specialists; whatever task-0 knowledge survives is incidental, not preserved by mechanism.

3. The negative result has a clear interpretation. Catastrophic forgetting in deep learning is fundamentally an optimization problem — gradient descent on task 2 overwrites task 1 features. Population dynamics on top of gradient descent doesn’t fix this: each individual still does gradient descent, and selection during task 2 picks for task-2 performance, throwing away task-0-specialist individuals.

4. What would actually buffer forgetting? Three credible mechanisms not currently in the system:

   • Replay buffer: keep a small sample from each prior task, train on a mix.
   • Explicit task-aware speciation: at task boundaries, save the best-of-task-k individuals to a separate niche, protect them from selection by the current task.
   • EWC-style regularization: penalize weight changes proportional to per-weight Fisher information from prior tasks.

   The current “ecological speciation” is not a forgetting-protection mechanism — it’s a task-partitioning mechanism. Different niches handle different tasks because they see different data. A single niche seeing tasks sequentially has no protection.

Conclusion: E6 is a clean negative result for “population dynamics alone buffer catastrophic forgetting.” The full Group E warm-start machinery does not transfer to continual learning without an explicit mechanism for preserving prior-task knowledge. This is an important constraint for the broader research direction — if online/continual learning is the goal, the next step is to design an explicit forgetting-protection mechanism, not to scale up the existing ecological speciation.

Side benefit: E6 ran completely cleanly (0 cycle breaks, 0 dangling drops). The aggressive sanitize plus the simpler genome topology (no patches → output direct) avoided cycle scenarios entirely.

E7: replay-based continual learning

Date: 2026-05-18 Commit: (pending — E7 binary group_e_replay_cl.rs) Binary: cargo run --release --bin group_e_replay_cl Output: notes/group_e/e7_output.txt

Hypothesis

E6 established that population dynamics alone do not buffer catastrophic forgetting on Permuted-MNIST. The CL literature’s textbook fix is a replay buffer: store a small sample of each prior task’s data and mix it into the current task’s training stream. E7 asks two questions:

1. Is the 45pp forgetting gap closable with a tiny replay buffer? If a buffer of ~100 examples per prior task substantially closes the gap, replay is a practical CL mechanism for this system.
2. Does population diversity stack with replay? E6 showed population alone is too weak. Replay + population might combine — population providing extra forgetting protection beyond what raw data rehearsal supplies.

Setup

Same 3 sequential permuted-MNIST tasks as E6 (perm seeds 0xE601+101, 0xE601+102; task 0 = identity). 300K steps per task. Identity LR schedule (LR 0.05 → 0.005 linear within each task). Replay strategy is balanced: at task k, every prior task’s buffer plus the current full training data are sampled with equal probability 1/(k+1). The buffer for task i is the first N examples of its training split (fixed, not resampled).

Four conditions:

Result

Accuracy matrices (rows = “after training task k”, columns = eval task):

A (pop=1 no replay)             B (pop=1 replay=100)
  0.966  0.085  0.108             0.965  0.130  0.152
  0.474  0.946  0.122             0.785  0.947  0.093
  0.236  0.386  0.944             0.678  0.777  0.933

C (pop=1 replay=1000)           D (pop=50 warm replay=100)
  0.966  0.100  0.114             0.970  0.175  0.116
  0.885  0.945  0.095             0.830  0.948  0.141
  0.875  0.853  0.934             0.757  0.752  0.941

Analysis

1. Replay solves catastrophic forgetting. The 42.98pp average forgetting in condition A drops to 15.24pp with just 100 examples per prior task — a 64% reduction with 0.2% of each prior task’s training set retained. With 1000 examples per task (2%), forgetting drops to 6.07pp — essentially closed. Task 0 retention after seeing two more tasks: 23.6% (A) → 67.8% (B) → 87.5% (C). This is the textbook continual-learning result and confirms replay works in this system.

2. Replay dominates population diversity by ~20×. The E6 finding was that population gave +1.46pp final accuracy and −0.96pp forgetting vs pop=1. E7 shows replay=100 gives +27.4pp final accuracy and −27.74pp forgetting — twenty times the effect. The dominant axis for continual learning in this system is data rehearsal, not topology diversity. This is a strong constraint on the architectural research direction: if CL is a goal, mechanism work should target the replay mechanism before the population mechanism.

3. Population stacks weakly with replay. Condition D (pop=50, replay=100) achieves 81.64% / 13.65pp — only +2.05pp final and −1.59pp forgetting over B (pop=1, replay=100). The synergy is real but small. Reading: population diversity does help, but the load-bearing mechanism is replay, and population is a small modulator on top. Consistent with E6’s finding that population’s direct effect on forgetting is weak; population probably adds value here mostly by giving SGD multiple parallel starting points for the replay-influenced loss landscape, not by preserving task-specific knowledge.

4. The buffer-size knee is past 100 but before 1000. Going from 100 → 1000 cuts forgetting by another factor of 2.5× (15.24pp → 6.07pp). The marginal value of more buffer is positive but diminishing. Where exactly the knee is — that’s E7b territory if it’s worth chasing. The minimum-credible CL operating point on this system looks like ~300-500 examples per task.

5. Cycles stayed clean. 0 cycle breaks, 0 dangling drops, 0 late breaks across all four conditions. The aggressive sanitize plus the simple genome topology (no patches → output direct, no inter-patch edges) keeps the population in valid-DAG territory throughout.

What’s next

E7’s positive result revives the CL direction that E6 had soft-deprecated. Productive follow-ups, ranked:

• E8: longer task sequences. 3 tasks is a toy. The CL literature’s hard tests are 5–10 sequential tasks where interference compounds. Replay=100 at 5–10 tasks tells us whether the system holds up at realistic sequence lengths or whether forgetting reasserts itself. Does the +2pp population effect from D widen or stay flat at longer sequences? (Most informative next experiment.)
• E9: buffer composition. Random-first-N vs class-balanced vs uncertainty-based. Probably small effect at this scale but the CL literature has well-known asymmetries here.
• E10: replay + task-aware speciation (was E8 in the original plan). Protected memory niche for best-of-task individuals. With replay solving most of the gap, the remaining headroom for explicit preservation is small — probably only worth running if E8 reveals a regime where replay alone is failing.

E8: long-sequence replay-based CL

Date: 2026-05-18 Commit: (pending — E8 binary group_e_long_cl.rs) Binary: cargo run --release --bin group_e_long_cl Output: notes/group_e/e8_output.txt

Setup

E7’s conditions B (pop=1, replay=100) and D (pop=50 warm, replay=100) extended to N_TASKS ∈ {5, 8}. Replay buffer size held at 100/task to test sequence-length scaling under fixed per-task memory. 300K steps per task, 0.05 → 0.005 LR schedule per task, same perm-seed family as E6/E7 (0xE601 + 100 + t).

Result

Analysis

1. Replay-100 forgetting climbs ~linearly with task count. At N=3 the system loses 15pp average; at N=8 it loses 27pp. Per-task buffer of 100 examples is sufficient at short sequences but becomes a binding constraint as task count grows. The mechanism is straightforward: at task k, the current task is 1/(k+1) of the training mix; each prior task’s buffer also gets 1/(k+1), but stays at 100 examples. Effective signal per prior task drops as task count grows.

2. The 60% floor on early tasks. At N=8, tasks 0–5 settle around 60–68% final accuracy — well above random (10%) but well below converged single-task accuracy (~95%). Suggests the 100-example buffer maintains some task-specific features but not enough to fully preserve the task-k solution against subsequent interference.

3. Population-as-CL-mechanism is twice-rejected. D−B gaps are +2.05pp (N=3), +2.75pp (N=5), +1.28pp (N=8). The effect doesn’t widen with harder regimes — it stays a small stable modulator. The original “population diversity buffers catastrophic forgetting” hypothesis is now dead: short-sequence (E6) and long-sequence (E8) both show population alone or stacked with replay contributes only a 1–3pp modifier.

4. Strong recency bias in the final-row matrix. At N=8 condition D: task 7 = 90.8%, task 6 = 74.1%, …, task 0 = 60.3%. The “shape” of forgetting is a smooth decay from most-recent to most-distant rather than a cliff. Consistent with replay providing a graded preservation effect proportional to total remaining buffer mass.

5. No forward transfer. All off-diagonal entries above the diagonal are in the 10–18% range — the system has no pre-task knowledge of permutations it hasn’t seen. This is structural to permuted-MNIST (random permutations don’t share features) rather than a system limitation.

Conclusion

Replay-100 is a partial CL solution: it converts catastrophic forgetting into graceful degradation. The remaining 27pp forgetting at N=8 is not catastrophic — early tasks retain 60%+ accuracy, ten times above chance — but it’s not solved either. The system has crossed from “CL doesn’t work” to “CL has bounded degradation per task.”

Population diversity provides a stable ~2pp modifier and doesn’t scale with difficulty. The mechanism worth investing in next is buffer scaling, not population mechanisms.

What’s next (E9)

Hold N=8 fixed, sweep buffer sizes. If replay=1000 returns to ~10pp forgetting at N=8, the story is “scale buffer linearly with task count” — easy operating recipe. If forgetting stays elevated even with 10× the buffer, there’s compounding interference that replay alone cannot fix, and structural mechanisms (task-aware speciation, weight regularization) become well-motivated for E10+.

E9: replay buffer size sweep at N=8

Date: 2026-05-18 Commit: (pending — E9 binary group_e_buffer_sweep.rs) Binary: cargo run --release --bin group_e_buffer_sweep Output: notes/group_e/e9_output.txt

Setup

Hold N_TASKS=8 fixed, pop=1, no warm-start. Sweep replay buffer size: 100, 300, 1000, 3000 examples per prior task. 300K steps per task, same perm seeds and LR schedule as E7/E8. Isolates the buffer axis from the (settled-small) population axis.

Result

Analysis

1. Log-linear scaling with no diminishing-returns knee in the sampled range. Each ~3× buffer increase cuts forgetting by roughly half (28 → 20 → 13 → 6). No regime where adding more buffer stops helping; the system stays in the “replay solves it if you pay” regime throughout.

2. No compounding interference. The E8 fork (“buffer or mechanism?”) resolves clearly: buffer. Scaling buffer linearly with task count restores low-forgetting CL at N=8 to the same range as E7-C achieved at N=3 (6pp). There is no structural CL barrier on permuted-MNIST at this scale that replay can’t fix.

3. task0_drop is consistently larger than avg_forget by ~7-15pp. Task 0 forgets more than the average because it has the most subsequent tasks interfering. Implies that future CL work would benefit from asymmetric buffer weighting (more memory for older tasks) rather than uniform allocation.

4. The 3000-buffer point is “store most of the data and rehearse it”, not “small buffer CL”. 3000 × 8 = 24K stored examples — 48% of one task’s training set per task. The “small buffer suffices” narrative holds at N=3 with 100 examples (0.2%), but degrades to “store half the data” at N=8 for ~6pp residual forgetting.

5. The mechanism question closes. Structural CL mechanisms (task-aware speciation, EWC-style regularization, parameter isolation) are unmotivated at this scale and task family. Replay alone is sufficient at any practical operating point. E10 (task-aware speciation) is correspondingly deprioritized.

Operating recipe for CL in this system

If targeting N-task continual learning with bounded forgetting:

• buffer_per_task ≈ K * N for some K ≈ 100-400 depending on tolerable forgetting (10pp ≈ K=400; 20pp ≈ K=150)
• Population size = 1 is sufficient; pop=50 adds 1-3pp at ~50× memory cost
• No warm-start needed (it does nothing meaningful for CL — its mechanism is patch-count growth, which is orthogonal to forgetting protection)

Closing the CL direction

E6 → E9 gives a complete picture:

• E6: population alone doesn’t buffer forgetting (negative).
• E7: replay-100 at N=3 closes 64% of the gap (positive).
• E8: replay-100 degrades with task count (partial).