Reference library
RoFormer: Enhanced Transformer with Rotary Position Embedding
Paper summary
RoFormer introduces rotary position embeddings that encode position through rotations of query and key representations. Their interaction carries relative-position dependence within attention, with evaluations of the resulting architecture.
Relevance to the project
A possible encoding reference for ordered bounded histories if attention is used. Temporal position and physical neuron position are distinct roles.
Rotary encoding does not choose the project's spatial metric or history lengths.
Source record
- Reference ID
- rope
- Archived version
- 2104.09864v5
- Summary provenance
- reference-library/papers/rope/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.