arXiv:2607.14571 · July 2026

Gate-Zero Growth

A Geometric Framework for Function-Preserving Continual Learning

Dante Lok^1,2

¹Votee AI ²Beever AI

0.0max logit diff at growth
(exactly function-preserving)

Δ_A = 0.04old-domain forgetting, ppl
(Gate-FP + Isolation)

300M→857MTransformer, WikiText-103
→ BookCorpus

bit-identicalseeded, deterministic
reproduction

TL;DR: Add capacity to a trained network through a gate initialised at exactly zero, and the grown model computes the identical function — max logit difference 0.0, not merely small. Under a transversality condition this growth induces rank separation in the functional Jacobian, so new capacity can only enter through the gates. Result: near-zero old-domain forgetting (Δ_A = 0.04) when a 300M model is grown to 857M and adapted across domains, while a non-function-preserving control forgets an order of magnitude more under the same recipe.

Abstract

We introduce gate-zero growth, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces rank separation in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation.

As gates open during continual learning, function drift is O(‖α‖²) and Jacobian leakage O(‖α‖∞), giving a controlled departure from the FP locus. On a 300M→857M Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting (Δ_A < 0.1) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control (G_stack) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in continual learning.

Overview

At the growth point, take the functional Jacobian of the grown model and partition its columns. The zero gate forces a strict trichotomy — one theorem, three subspaces.

Rank separation at the growth point. Grow through a gate initialised at exactly zero and the functional Jacobian splits three ways: the old model is untouched, the new weights are invisible, and the gates alone control how new capacity enters. Non-FP operators collapse this structure — every direction moves the function at once, damaging old behaviour before continual learning even begins.

Method

Gate-zero growth is a single operator plus a choice of how much to constrain the old model during adaptation.

Grow through a zero gate. Insert new residual blocks whose output is scaled by a gate α initialised at exactly zero. Because 0 · r = 0 holds exactly in IEEE-754, the grown network reproduces the original function bit-for-bit — verified as max logit difference 0.0.
Rank separation guarantees the geometry. Under a transversality condition, the functional Jacobian at the growth point separates into unchanged old directions, exactly-flat new-weight directions, and gate directions that are the sole first-order source of new functional variation.
Open gates under control. As continual learning opens the gates, function drift is O(‖α‖²) and Jacobian leakage O(‖α‖∞) — a bounded, controllable departure from the function-preserving locus.
Pick an operating point. Isolation freezes old weights and gates for exact preservation; Freeze-Nothing trades a little preservation for the best joint frontier. Both reach near-zero forgetting; non-FP growth reaches neither.

Results

A 300M base trained on WikiText-103 (domain A) is grown to 857M and adapted to BookCorpus (domain B). Δ_A is the increase in domain-A perplexity after adaptation — forgetting, measured directly.

Growth × recipe	Δ_A (ppl)	FP at growth
Gate-zero + Isolation	0.04	yes — \|Δlogit\| = 0.0
Gate-zero + no protection	340.2	yes
G_stack + no protection	1144.0	no

Stability–plasticity frontier across growth operators and CL recipes. — **Stability–plasticity frontier.** Gate-zero growth traces the frontier between preserving domain A and learning domain B; the non-FP control pays an order-of-magnitude forgetting penalty at matched recipes.

MoE cross-architecture diagnostic. — **Cross-architecture (MoE).** The same gate-zero construction on a Mixture-of-Experts backbone: preservation transfers across architectures; the plasticity bottleneck is mechanistically localised.

Takeaways

Function preservation is verifiable, not aspirational. The zero gate gives max logit difference 0.0 — exact, not approximate.
Rank separation explains why it preserves. New capacity is confined to the gate subspace, so old behaviour is untouched until gates deliberately open.
One geometry, many constructions. LoRA's zero-init B, ReZero's zero residual scalar, and zero-init adapters are all special cases of the same local structure.
Reproducible by construction. Every randomness surface is seeded, kernels are deterministic, and all 43 result files ship in the repo — with a one-command reproduction of the appendix study.

BibTeX

@article{lok2026gatezero,
  title   = {Gate-Zero Growth: A Geometric Framework for
             Function-Preserving Continual Learning},
  author  = {Lok, Dante},
  journal = {arXiv preprint arXiv:2607.14571},
  year    = {2026}
}