Reference library
Decoupled Weight Decay Regularization
Paper summary
Shows that L2 penalties and weight decay behave differently in adaptive optimizers such as Adam. Decoupling decay from the loss-gradient update improves evaluated generalization and makes decay tuning less dependent on learning-rate choice.
Relevance to the project
A reference if external training is later used for mechanism networks. More generally, separate intended parameter regulation from the numerical update that implements it.
AdamW is not a proposed lifetime plasticity rule for the engine.
Source record
- Reference ID
- adamw
- Archived version
- 1711.05101v3
- Summary provenance
- reference-library/papers/adamw/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.