Reference library
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Paper summary
Switch Transformers simplify sparse expert routing and address communication and training instability. Selected experts increase parameter capacity without activating the entire model for each input, with large-scale training demonstrations.
Relevance to the project
An engineering comparison for specialization and batched work allocation if multiple neuron types are later explored. It is not a selected control-network design.
Active computation, total storage, and routing overhead are separate costs.
Source record
- Reference ID
- switch-transformer
- Archived version
- 2101.03961v3
- Summary provenance
- reference-library/papers/switch-transformer/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.