Overview

Encoder-decoder models use an encoder to represent an input and a decoder to generate the output, which makes them well-suited for generation tasks such as translation. They are often adapted to low-resource languages through continued pretraining (CPT).

However, these CPT-adapted models are usually evaluated only after training, which leaves it unclear how and when their capabilities change. To address this gap, we investigate the learning dynamics of three encoder-decoder PLMs during CPT adaptation to isiXhosa, a South African Nguni language.

Objectives

  1. Downstream Performance

    Analyse whether the downstream generation performance of encoder-decoder models improves consistently during CPT adaptation to isiXhosa.

  2. Grammatical Knowledge

    Analyse how part-of-speech (POS) knowledge is encoded across the encoder layers, and how this changes during CPT.

  3. Bilingual CPT

    Determine whether including English during CPT helps models preserve their existing linguistic knowledge.

Methodology

We study how encoder-decoder models learn isiXhosa in four steps:

  1. Model Selection

    We select three encoder-decoder PLMs with different levels of prior exposure to isiXhosa: T5 (English-only), ByT5 (massively multilingual) and Nguni-ByT5 (Nguni-language).

  2. Monolingual and Bilingual Continued Pretraining

    We continue pretraining each model on isiXhosa text from the WURA corpus for 10,000 steps, saving 20 checkpoints per run. Checkpoints are saved more often early in training. We repeat this with bilingual CPT, adding an equal amount of English text.

  3. Downstream Performance Analysis

    We fine-tune each checkpoint on machine translation (MT) in both directions and data-to-text (D2T) generation, and evaluate it using chrFHow closely the generated text matches a reference text, character by character. and BLEUHow closely the generated text matches a reference text, word by word..

  4. Probing Analysis

    At each checkpoint, we train a linear probeA simple classifier trained on one layer's representations, to test how easily that layer's information can be read out. on every encoder layer to predict the POS tag of each word, using isiXhosa sentences from MasakhaPOS.

Results

Downstream Analysis

We track how MT and D2T performance changes throughout CPT.

chrF and BLEU across CPT steps for English-to-isiXhosa MT, isiXhosa-to-English MT and data-to-text, for T5, ByT5 and Nguni-ByT5
This plot shows each model's chrF and BLEU scores for MT and D2T throughout CPT.

For English-to-isiXhosa MT, Nguni-ByT5 outperforms ByT5, which outperforms T5, throughout CPT, matching their prior isiXhosa exposure. Nguni-ByT5 stays relatively stable, while ByT5 and T5 first drop below their base models before recovering, reaching most of their gains around halfway through training. T5, with no prior isiXhosa exposure, shows the clearest upward trend. For isiXhosa-to-English MT, some checkpoints that are best at generating isiXhosa are worst at generating English, although this trade-off is small. For D2T, none of the models show a sustained improvement during CPT.

Overall, our results suggest that further CPT on isiXhosa produces limited downstream gains. Our ablations show that increasing the amount of task-specific fine-tuning data changes performance more than extending CPT on the same corpus.

Probing Analysis

We first compare how POS probe accuracy varies across the encoder layers of the base models.

POS probe accuracy by encoder layer for the T5, ByT5 and Nguni-ByT5 base models, comparing first, mean and last pooling
This plot shows each base model's POS probe accuracy by encoder layer, comparing first, mean and last pooling.

For all three base models, mean poolingA word can be split into several tokens. Mean pooling averages their representations into one vector per word. gives the highest POS accuracy, which fits isiXhosa, where several parts of a word can carry grammatical information. POS accuracy peaks in the middle encoder layers and drops towards the output.

Overall, our results suggest that grammatical knowledge is most accessible in the middle of the encoder, a pattern that CPT largely preserves.

Monolingual vs. Bilingual CPT

We then track how POS probe accuracy changes during CPT, with and without adding an equal amount of English text.

POS probe accuracy by encoder layer for T5, ByT5 and Nguni-ByT5 at CPT steps 0, 2k, 4k, 6k, 8k and 10k, under monolingual and bilingual CPT
This plot shows each model's POS probe accuracy by encoder layer at selected CPT checkpoints, under monolingual and bilingual CPT.

Under monolingual CPT, ByT5's accuracy declines across the encoder, while Nguni-ByT5 stays stable. Bilingual CPT preserves isiXhosa POS accuracy, keeping adapted checkpoints closer to their base models, and ByT5 shows smaller declines. Fine-tuning results follow similar trends in both settings.

Overall, our results suggest that including English during CPT helps preserve grammatical knowledge in the target language, especially when isiXhosa adaptation would otherwise disrupt a model's existing representations, as with ByT5. Prior Nguni adaptation appears to keep these representations stable either way.

Conclusions

  • Key Takeaway

    Further isiXhosa CPT produces limited downstream gains, but substantial changes inside the model, showing why it is worth studying learning dynamics and not just final performance.

  • Our Contribution

    To our knowledge, we provide the first study of CPT learning dynamics in encoder-decoder models for isiXhosa, and show that including English also preserves grammatical knowledge.

  • Future Work

    Future work could use larger, more varied pretraining data, since repeated passes over a small corpus may add little, and add control baselines to the probing analysis.

Resources

All the code used for our continued pretraining, fine-tuning and probing is available in our GitHub repository: github.com/Zahara2580/learning-dynamics

Contact

← Back to Homepage