Reference library
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Paper summary
ZeRO shards optimizer state, gradients, and parameters to remove redundant memory in distributed training. Memory and communication analyses plus large-model experiments show how additional devices can support larger models.
Relevance to the project
A resource-accounting reference if external multi-GPU training is later needed. It does not remove the simulation's ordered observation/action dependency.
Distributed training savings depend on sharding stage, hardware, and communication costs.
Source record
- Reference ID
- zero
- Archived version
- 1910.02054v3
- Summary provenance
- reference-library/papers/zero/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.