Reference library
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Paper summary
Uses mean-field and random-matrix analyses of recurrent networks at initialization. Vanilla networks have restrictive conditions for long signal propagation; gating broadens useful regimes. Nonlinear Jacobians depend on activation derivatives, so an orthogonal weight matrix alone does not establish dynamical isometry.
Relevance to the project
Assistant interpretation: recurrent gain depends on inputs, nonlinear operating points and the complete transition Jacobian. Gating is a comparison, not an adopted requirement.
Reading is focused, not exhaustive; no independent replication. Focused full-text reading of sections 2–3 mean-field signal propagation, correlation/Jacobian discussion and selected orthogonal/gated initialization comparisons. No exhaustive supplement, derivation or experiment audit.
Source record
- Reference ID
- pmlr-chen18i
- Archived version
- Published ICML 2018, PMLR 80:873–882
- Summary provenance
- reference-library/papers/pmlr-chen18i/metadata.json · summary
- Review status
- focused primary full-text methods reading; not exhaustive critical review
- Reading scope
- Focused full-text reading of sections 2–3 mean-field signal propagation, correlation/Jacobian discussion and selected orthogonal/gated initialization comparisons. No exhaustive supplement, derivation or experiment audit.