Reference library
Deduplicating Training Data Makes Language Models Better
Paper summary
Measures duplication and train-test overlap in language-model corpora. Exact and near-duplicate removal reduces memorized output and can improve learning efficiency while making evaluation less contaminated.
Relevance to the project
Relevant to optional model training and to fair lineage/novelty analysis: repeated samples should not inflate evidence of new behavior.
Text deduplication metrics do not directly measure behavioral diversity.
Source record
- Reference ID
- deduplication
- Archived version
- 2107.06499v2
- Summary provenance
- reference-library/papers/deduplication/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.