Reference library
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Paper summary
GaLore projects gradients into low-rank subspaces to reduce optimizer-state memory while permitting updates of full model weights. Pretraining and fine-tuning experiments evaluate accuracy and memory tradeoffs.
Relevance to the project
Relevant if external optimization of mechanism networks is explored. It separates restrictions on transient update state from permanent restrictions on parameter representation.
No optimizer or gradient requirement has been adopted for lifetime F updates.
Source record
- Reference ID
- galore
- Archived version
- 2403.03507v2
- Summary provenance
- reference-library/papers/galore/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.