Reference library
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Paper summary
Grouped-query attention shares key-value heads among query-head groups, balancing cache cost and expressiveness. The paper also adapts existing multi-head checkpoints using a limited additional training budget.
Relevance to the project
A comparison for sharing information-routing parameters and memory. Useful if bounded-history attention is investigated, while per-neuron communication remains a separate problem.
Decoder-cache savings are workload-specific.
Source record
- Reference ID
- gqa
- Archived version
- 2305.13245v3
- Summary provenance
- reference-library/papers/gqa/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.