Reference library
Layer Normalization
Paper summary
Layer normalization centers and rescales a layer’s summed inputs using statistics across hidden units for each individual case, followed by per-unit gain and bias before activation. In recurrent networks the statistics are recomputed at each time step and learned gains/biases are shared across time. It supports a single-case online regime and has invariances different from weight or batch normalization: rescaling an entire layer is distinct from scaling one neuron’s incoming weights. The paper frames the method as improving gradient training and stabilizing hidden-state scale.
Relevance to the project
A comparison for local versus stage-wide observable statistics and where gain sits in an activation pipeline. It couples units through current normalization statistics, unlike a rule that observes only each unit’s own saturated output. Stabilized layer scale is not evidence that every initialized recurrent system remains varied under constant input; no normalization change is adopted.
Training/test scale control is a different objective from autonomous exploration, anti-repetition and task computation. Singular/near-zero variance handling, spatial correlations and online-changing adaptive weights need independent analysis. Exact v1 methods read; experiments and supplements excluded.
Source record
- Reference ID
- layer-normalization-2016
- Archived version
- arXiv:1607.06450v1, 21 July 2016, 14-page preprint
- Summary provenance
- reference-library/papers/layer-normalization-2016/metadata.json · summary
- Review status
- Targeted full-text methods review; neither exhaustive full-paper reading nor independent reproduction
- Reading scope
- PDF pages 2–4 read in full: Sections 2, 3, 3.1, 4, 5.1 and opening 5.2, including equations 1–7 and invariance comparison. Front matter checked; remainder of geometry analysis, experimental results and supplementary implementation details not critically read. No experimental reproduction or figure audit.