Reference library
Theoretical Limitations of Self-Attention in Neural Sequence Models
Paper summary
Derives limits on fixed-depth self-attention for representing selected periodic and hierarchical formal languages as sequence length grows. The analysis distinguishes mathematical expressiveness under resource constraints from practical approximate language performance.
Relevance to the project
A counterweight to unrestricted universality claims. Compare F architectures at fixed dimensions, memory, precision, and update budgets.
The theorem's architecture and asymptotic assumptions matter; it is not a blanket impossibility result.
Source record
- Reference ID
- arxiv-1906-06755
- Archived version
- arXiv preprint; retrieved version indicated in PDF
- Summary provenance
- reference-library/papers/arxiv-1906-06755/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.