Overview
Encoder-only models are trained using masked language modelling, which makes them well-suited for language understanding tasks. They remain widely used for low-resource languages, and are often adapted to these languages through continued pretraining (CPT).
However, most studies evaluate these CPT-adapted models only after training, which provides limited insight into how they adapt. To address this gap, we investigate the learning dynamics of four encoder-only PLMs during monolingual CPT adaptation to isiXhosa and isiZulu, two South African Nguni languages.
Objectives
-
Downstream Performance
Analyse how the downstream task performance of encoder-only models changes during CPT adaptation to isiXhosa and isiZulu.
-
Cross-Lingual Geometry
Analyse how cross-lingual alignment and isotropy change across the layers of encoder-only models during CPT.
-
Prior Exposure
Determine how a model's prior exposure to the target languages affects its learning dynamics during CPT.
Methodology
We study how encoder-only models learn isiXhosa and isiZulu in four steps:
-
Model Selection
We select four encoder-only PLMs with different levels of prior exposure to isiXhosa and isiZulu: RoBERTa (English-only), XLM-R (massively multilingual), AfriBERTa (African-language) and Nguni-XLMR (Nguni-language).
-
Monolingual Continued Pretraining
We continue pretraining each model on isiXhosa and isiZulu text from the WURA corpus for 200,000 steps, saving 20 checkpoints per run. Checkpoints are saved more often early in training, where model behaviour changes most rapidly.
-
Downstream Performance Analysis
We fine-tune each checkpoint on named-entity recognition (NER), part-of-speech (POS) tagging and news topic classification (NTC), and report the mean Macro F1A score that averages performance across all of a task's labels. across five random seeds.
-
Cross-Lingual Geometry Analysis
At five selected checkpoints, we measure cosine similarityHow closely a sentence lines up with its English translation inside the model. and IsoScoreHow evenly the model spreads its representations across the available space. at every layer, using parallel English and target-language sentences from FLORES-200.
Results
Downstream Analysis
We track how NER, POS tagging and NTC performance changes across all CPT checkpoints in isiXhosa and isiZulu.
Downstream performance generally follows three stages. The largest changes happen within the first 40,000 steps, gains slow down between 40,000 and 100,000 steps, and most models largely stabilise after that. NTC generally improves the most, while POS tagging and NER change comparatively little. The exception is Nguni-XLMR on isiZulu, which was already adapted to the language. Its NTC and POS tagging scores become unstable, which suggests that additional CPT may interfere with what the model already knows.
Overall, our results suggest that far fewer CPT steps may be enough to reach most of the eventual downstream performance. They also indicate that CPT benefits tasks requiring language- and domain-specific knowledge, such as NTC, more than grammatical tasks, such as POS tagging.
Cross-Lingual Geometry Analysis
We compare how cross-lingual alignment and isotropy vary across layers at five selected CPT checkpoints.
Alignment and isotropy are generally strongest in the intermediate and upper-middle layers, before representational degenerationWhen representations fail to use the full capacity of the available space. becomes more pronounced in the final layers. Both generally increase during CPT, with the largest changes in the first 40,000 steps, while additional CPT on the already adapted Nguni-XLMR lowers both. RoBERTa and XLM-R start with high cosine similarity but low IsoScore, which suggests that their initial similarity may partly reflect anisotropyWhen representations cluster in a narrow region of the space, which can make unrelated representations look similar. rather than meaningful alignment.
Overall, our results suggest that CPT changes cross-lingual geometry in two ways: it brings isiXhosa and isiZulu closer to English, and it reduces anisotropy, helping models make better use of their representation space. Prior work has linked both greater alignment and lower anisotropy to better downstream performance.
Prior Exposure Analysis
How much a model changes during CPT depends on its prior exposure to the target language. RoBERTa and AfriBERTa, which had not seen isiXhosa or isiZulu before CPT, show some of the clearest early improvements, especially in NTC. XLM-R changes little for isiXhosa, which was in its pretraining data, but improves more for isiZulu, which was not. Nguni-XLMR, already adapted to both languages, starts strong and changes little. These results suggest that direct prior exposure reduces the adaptation needed during CPT, while additional CPT on an already adapted model can disrupt existing knowledge.
Conclusions
-
Key Takeaway
Most downstream gains occur early in CPT, so shorter CPT budgets may achieve similar downstream performance at a lower computational cost, which is valuable in low-resource settings.
-
Our Contribution
We extend learning dynamics research beyond pretraining from scratch to CPT of encoder-only models, showing that CPT does not affect all models, tasks, or layers equally.
-
Future Work
Future work could examine additional low-resource languages and architectures, and apply actionable interpretability, such as model-specific stopping criteria, to improve CPT itself.
Resources
All the code used for our continued pretraining, fine-tuning and cross-lingual geometry analysis is available in our GitHub repository: github.com/CodeWithSiya/cpt-learning-dynamics
Contact
-
Siyabonga Madondo
Author