Reference library
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Paper summary
Analyzes gradient-learning dynamics in deep linear networks. Orthogonal initialization can preserve singular values through linear products, whereas average norm preservation by Gaussian matrices can conceal poorly conditioned directions. The nonlinear discussion adds activation-dependent qualifications.
Relevance to the project
Assistant interpretation: motivates measuring singular-value spread and transient amplification alongside average variance. Rectangular, nonlinear, delayed and plastic dynamics require additional analysis.
Reading is focused, not exhaustive; no independent replication. Focused full-text reading of opening assumptions and selected orthogonal-versus-Gaussian initialization/dynamical-isometry discussion (printed pages 10–11). Not a complete proof, appendix or nonlinear experiment audit.
Source record
- Reference ID
- arxiv-1312-6120
- Archived version
- arXiv:1312.6120v3, 19 February 2014; 22-page preprint
- Summary provenance
- reference-library/papers/arxiv-1312-6120/metadata.json · summary
- Review status
- focused primary full-text methods reading; not exhaustive critical review
- Reading scope
- Focused full-text reading of opening assumptions and selected orthogonal-versus-Gaussian initialization/dynamical-isometry discussion (printed pages 10–11). Not a complete proof, appendix or nonlinear experiment audit.