Reference library
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Paper summary
Describes intra-layer tensor parallelism for training large Transformers across GPUs. A small set of communication operations distributes matrix computations, with scaling experiments measuring throughput and efficiency.
Relevance to the project
An engineering reference for identifying synchronization and memory bottlenecks. The simulation's environment-to-entity round trip needs its own latency measurements.
Efficient distributed training is different from efficient small batched inference.
Source record
- Reference ID
- megatron
- Archived version
- 1909.08053v4
- Summary provenance
- reference-library/papers/megatron/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.