Reference library
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Paper summary
Tests continued pretraining on domain and task text before supervised adaptation. Across several domains and classification tasks, additional relevant unlabeled data improves the evaluated downstream results.
Relevance to the project
A baseline for environment-specific adaptation if pretrained models are later used. Separate retained parameter changes from temporary context effects.
Task-directed continued training differs from unsupervised lifetime adaptation in the simulation.
Source record
- Reference ID
- continued-pretraining
- Archived version
- 2004.10964v3
- Summary provenance
- reference-library/papers/continued-pretraining/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.