Reference library
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Paper summary
Weight normalization parameterizes a weight vector by its direction and a separate scale, w=g v/||v||, and trains those parameters by stochastic gradient descent. It alters gradient scaling/projection and conditioning, with a monotonic norm argument specifically for ordinary gradient descent without momentum. It is equivalent to batch normalization only under restrictive whitened-input/single-layer conditions. The paper supplies a data-dependent one-minibatch initialization for feedforward settings, explicitly noting recursion requires standard initialization instead.
Relevance to the project
Clarifies why parameter normalization, learned scale, preactivation normalization and output gain must be distinguished when discussing F regulation. Fixing or separating a weight norm does not fix temporal activation statistics under correlated feedback. The method is a comparison/proposal source, not an adopted reparameterization of the current uniform F/H baseline.
Targeted methods reading; performance claims not independently audited. Results concern gradient training with specific optimizer assumptions; they do not prove sustained endogenous activity, symmetry of temporal changes, or stability under evolved online updaters. Zero norm handling and changing-input covariance require separate engineering choices.
Source record
- Reference ID
- weight-normalization-2016
- Archived version
- Published NeurIPS 2016 proceedings PDF, 9 pages
- Summary provenance
- reference-library/papers/weight-normalization-2016/metadata.json · summary
- Review status
- Targeted full-text methods review; neither exhaustive full-paper reading nor independent reproduction
- Reading scope
- PDF pages 2–4 read in full: Sections 2, 2.1, 2.2 and 3, equations 1–6 and assumptions/caveats. Front matter checked; experiments, mean-only batch normalization, closing discussion and references not critically reviewed. Text extraction is a search aid; no numerical reproduction or figure audit.