Reference library
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Paper summary
FlashAttention tiles exact attention to reduce transfers between GPU main memory and on-chip storage. It avoids materializing large intermediate attention arrays, improving memory use and measured runtime without the usual attention approximation.
Relevance to the project
A reference for keeping state resident on GPU and optimizing communication. Profile small per-neuron workloads before assuming an attention kernel is advantageous.
Mathematical exactness does not imply bitwise equality across implementations.
Source record
- Reference ID
- flashattention
- Archived version
- 2205.14135v2
- Summary provenance
- reference-library/papers/flashattention/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.