MohammadKhosravi/llama3.1-8b-dual-fpmt-ablation-cefr-only-6k

Dual-Controller PrefixMemory-Tuning for CEFR-Controlled Text Generation (E2A)

Overview

This repository contains the Eexperiment 2A architectural ablation of the proposed Dual-Stream Factorized PrefixMemory-Tuning (Dual-FPMT) framework.

Unlike the complete Dual-FPMT architecture, this model does not receive independent linguistic conditioning signals. Instead, it employs two parallel PrefixMemory streams (Lexical and Syntactic) that are both conditioned on the same shared CEFR embedding.

The purpose of this ablation is to determine whether the improvements observed in the full Dual-FPMT architecture arise from:

• Independent linguistic conditioning using separate feature hierarchies.
• Simply increasing controller capacity through two parallel PrefixMemory pathways.

Therefore, this experiment isolates the architectural contribution of the dual-stream design while removing explicit lexical and syntactic feature conditioning.

---

Base Model

Foundation Model

Property	Value
Model	Meta-Llama-3.1-8B-Instruct
Parameters	8 Billion
Backbone	Decoder-only Transformer
Transformer Layers	32
Hidden Dimension	4096
Frozen During Training	Yes

Only the Dual-FPMT controller parameters are optimized during training. All Llama-3.1-8B backbone weights remain frozen.

---

Architecture

The E2A controller consists of two independent PrefixMemory retrieval pathways, referred to as the Lexical Stream and Syntactic Stream. Unlike the complete Dual-FPMT architecture, both streams receive the same CEFR embedding as input.

Each stream maintains its own:

• MLP controller
• Low-rank external key memory
• Low-rank external value memory
• Learnable scaling coefficient

The retrieved memory biases are combined additively before being injected into every self-attention layer of the frozen Llama model.

Figure 1. Overall architecture of the Dual-Controller PrefixMemory-Tuning(E2A).

---

Mathematical Formulation

CEFR Embedding

The shared CEFR embedding is

$$e=\mathrm{E}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}(c)e\; =\; \backslash mathrm\{Embedding\}(c)$$

where

• $c$ is the target CEFR label.
• $e \in \mathbb{R}^{128}$.

---

Independent Controller Networks

Each stream produces its own modulation tensor

$$S_{\mathrm{l}\mathrm{e}\mathrm{x}}={\mathrm{M}\mathrm{L}\mathrm{P}}_{\mathrm{l}\mathrm{e}\mathrm{x}}(e)S\_\{\backslash mathrm\{lex\}\}\; =\; \backslash mathrm\{MLP\}\_\{\backslash mathrm\{lex\}\}(e)$$

$$S_{\mathrm{s}\mathrm{y}\mathrm{n}}={\mathrm{M}\mathrm{L}\mathrm{P}}_{\mathrm{s}\mathrm{y}\mathrm{n}}(e)S\_\{\backslash mathrm\{syn\}\}\; =\; \backslash mathrm\{MLP\}\_\{\backslash mathrm\{syn\}\}(e)$$

where

$$S\in {\mathbb{R}}^{32\times 64}S\; \backslash in\; \backslash mathbb\{R\}^\{32\; \backslash times\; 64\}$$

---

PrefixMemory Retrieval

For each transformer layer,

$$Q_{\mathrm{l}\mathrm{e}\mathrm{x}}=X{W}_{\mathrm{l}\mathrm{e}\mathrm{x}}^{\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}}Q\_\{\backslash mathrm\{lex\}\}\; =\; XW^\{\backslash mathrm\{down\}\}\_\{\backslash mathrm\{lex\}\}$$

$$Q_{\mathrm{s}\mathrm{y}\mathrm{n}}=X{W}_{\mathrm{s}\mathrm{y}\mathrm{n}}^{\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}}Q\_\{\backslash mathrm\{syn\}\}\; =\; XW^\{\backslash mathrm\{down\}\}\_\{\backslash mathrm\{syn\}\}$$

where

• $X$ denotes the hidden states.
• $W^{\mathrm{down}} \in \mathbb{R}^{4096 \times 64}$.

---

Nonlinear Memory Modulation

$${\mathrm{\Phi }}_{\mathrm{l}\mathrm{e}\mathrm{x}}=\mathrm{E}\mathrm{L}\mathrm{U}(Q_{\mathrm{l}\mathrm{e}\mathrm{x}}\odot S_{\mathrm{l}\mathrm{e}\mathrm{x}})\backslash Phi\_\{\backslash mathrm\{lex\}\}\; =\; \backslash mathrm\{ELU\}\; \backslash left(\; Q\_\{\backslash mathrm\{lex\}\}\; \backslash odot\; S\_\{\backslash mathrm\{lex\}\}\; \backslash right)$$

$${\mathrm{\Phi }}_{\mathrm{s}\mathrm{y}\mathrm{n}}=\mathrm{E}\mathrm{L}\mathrm{U}(Q_{\mathrm{s}\mathrm{y}\mathrm{n}}\odot S_{\mathrm{s}\mathrm{y}\mathrm{n}})\backslash Phi\_\{\backslash mathrm\{syn\}\}\; =\; \backslash mathrm\{ELU\}\; \backslash left(\; Q\_\{\backslash mathrm\{syn\}\}\; \backslash odot\; S\_\{\backslash mathrm\{syn\}\}\; \backslash right)$$

---

External Memory Reconstruction

$$B_{\mathrm{l}\mathrm{e}\mathrm{x}}={\mathrm{\Phi }}_{\mathrm{l}\mathrm{e}\mathrm{x}}{W}_{\mathrm{l}\mathrm{e}\mathrm{x}}^{\mathrm{u}\mathrm{p}}B\_\{\backslash mathrm\{lex\}\}\; =\; \backslash Phi\_\{\backslash mathrm\{lex\}\}\; W^\{\backslash mathrm\{up\}\}\_\{\backslash mathrm\{lex\}\}$$

$$B_{\mathrm{s}\mathrm{y}\mathrm{n}}={\mathrm{\Phi }}_{\mathrm{s}\mathrm{y}\mathrm{n}}{W}_{\mathrm{s}\mathrm{y}\mathrm{n}}^{\mathrm{u}\mathrm{p}}B\_\{\backslash mathrm\{syn\}\}\; =\; \backslash Phi\_\{\backslash mathrm\{syn\}\}\; W^\{\backslash mathrm\{up\}\}\_\{\backslash mathrm\{syn\}\}$$

where

$$W^{\mathrm{u}\mathrm{p}}\in {\mathbb{R}}^{64\times 4096}W^\{\backslash mathrm\{up\}\}\; \backslash in\; \backslash mathbb\{R\}^\{64\; \backslash times\; 4096\}$$

---

Composite Memory Injection

The two retrieved PrefixMemory vectors are combined using learnable scaling parameters

$$B={\alpha }_{\mathrm{l}\mathrm{e}\mathrm{x}}{B}_{\mathrm{l}\mathrm{e}\mathrm{x}}+{\alpha }_{\mathrm{s}\mathrm{y}\mathrm{n}}{B}_{\mathrm{s}\mathrm{y}\mathrm{n}}B\; =\; \backslash alpha\_\{\backslash mathrm\{lex\}\}\; B\_\{\backslash mathrm\{lex\}\}\; +\; \backslash alpha\_\{\backslash mathrm\{syn\}\}\; B\_\{\backslash mathrm\{syn\}\}$$

Finally,

$$H^{\mathrm{o}\mathrm{u}\mathrm{t}}=H^{\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{n}}+BH^\{\backslash mathrm\{out\}\}\; =\; H^\{\backslash mathrm\{attn\}\}\; +\; B$$

---

Training Objective

The frozen Llama model is optimized using the standard causal language modeling objective.

Only the Dual-FPMT controller parameters are updated.

Optimization

• Optimizer: AdamW
• Learning Rate: $2 \times 10^{-4}$
• Batch Size: 8
• Epochs: 5
• PrefixMemory Rank: 64
• Gradient Clipping: Maximum norm = 1.0

---

Training Dataset

Training uses a balanced subset derived from the EFCAMDAT learner corpus.

Original dataset:

https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html

Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.

Dataset Columns

Column	Description
topic_title	Writing prompt
clean_text	Human learner response
cefr	Target CEFR level

Distribution

Balanced across all CEFR levels.

CEFR	Samples
A1	1,000
A2	1,000
B1	1,000
B2	1,000
C1	1,000
C2	1,000

Total: 6,000 samples

---

Training Performance

Best checkpoint: Epoch 2

[
  {
    "epoch": 1,
    "train_loss": 2.0800,
    "val_loss": 1.9935,
    "val_ppl": 7.34
  },
  {
    "epoch": 2,
    "train_loss": 1.7559,
    "val_loss": 1.9554,
    "val_ppl": 7.07
  },
  {
    "epoch": 3,
    "train_loss": 1.2714,
    "val_loss": 2.1301,
    "val_ppl": 8.42
  },
  {
    "epoch": 4,
    "train_loss": 0.7539,
    "val_loss": 2.5029,
    "val_ppl": 12.22
  },
  {
    "epoch": 5,
    "train_loss": 0.4018,
    "val_loss": 2.9270,
    "val_ppl": 18.67
  }
]

---

In-Domain Evaluation

Benchmark

A balanced benchmark constructed from held-out EFCAMDAT prompts.

Benchmark Statistics

• Total prompts: 702
• Source: EFCAMDAT
• Evaluation type: Prompt-only generation
• Topic distribution: Balanced across all CEFR proficiency levels

Dataset Columns

Column
topic_id
topic_title
cefr

Distribution

CEFR	Samples
A1	117
A2	117
B1	117
B2	117
C1	117
C2	117

Total: 702 prompts

---

The benchmark demonstrates a substantial improvement over the frozen Meta-Llama-3.1-8B-Instruct baseline, increasing strict CEFR accuracy from 24.64% to 62.54%, while reducing the mean absolute CEFR prediction error (MAE) from 1.33 to 0.72. Adjacent accuracy also improves from 63.19% to 80.20%, indicating significantly stronger controllability across neighboring proficiency levels.

---

Evaluation Results

Dual-FPMT (E2A Ablation) In-Domain Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (DUAL-FPMT E2A ABLATION)
============================================================

Total Processed       : 702
Strict Accuracy       : 62.54%
Adjacent Accuracy     : 80.20%
Mean Abs Error (MAE)  : 0.7194
Avg MDD Score         : 2.03
Avg Reading Ease      : 80.78
Avg Sentence Drift    : 2.37 levels
Avg Perplexity        : (Computed externally)

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift
target_cefr
-------------------------------------------------------------
A1                     69.23     1.52       83.22       1.11
A2                     70.94     1.69       83.54       1.56
B1                     75.21     1.88       80.77       2.18
B2                     66.67     1.82       85.75       3.10
C1                     45.30     1.94       75.34       2.79
C2                     47.86     3.32       76.05       3.47

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.65       0.69      0.67       117
A2               0.61       0.71      0.65       117
B1               0.56       0.75      0.64       117
B2               0.56       0.67      0.61       117
C1               0.72       0.45      0.55       117
C2               0.80       0.48      0.60       117

-------------------------------------------------------------

Accuracy                                 0.63       702
Macro Avg         0.65       0.63      0.62       702
Weighted Avg      0.65       0.63      0.62       702

PMT Confusion Matrix

Figure 2. Confusion Matrix of E2A on In-Domain Evaluation Dataset (Inference).

Improvement over Base Llama

• +37.9 percentage points Strict Accuracy
• +17.1 percentage points Adjacent Accuracy
• Lower MAE
• Lower syntactic deviation
• Much stronger readability control

Base LLM Evaluation Results

Baseline LLM Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================

Total Processed       : 690
Strict Accuracy       : 24.64%
Adjacent Accuracy     : 63.19%
Mean Abs Error (MAE)  : 1.3275
Avg MDD Score         : 2.00
Avg Reading Ease      : 49.47
Avg Sentence Drift    : 3.04 levels
Avg Perplexity        : 4.14

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift  avg_perplexity
target_cefr
-------------------------------------------------------------------------
A1                     16.24     1.73       82.56       3.25            5.16
A2                     23.08     1.64       85.44       3.27            5.12
B1                     17.39     2.03       57.80       3.11            3.86
B2                     41.74     2.15       34.18       2.97            3.32
C1                     33.63     2.24       18.95       2.94            3.59
C2                     15.93     2.25       15.54       2.69            3.74

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.28       0.16      0.20       117
A2               0.38       0.23      0.29       117
B1               0.21       0.17      0.19       115
B2               0.22       0.42      0.29       115
C1               0.24       0.34      0.28       113
C2               0.23       0.16      0.19       113

-------------------------------------------------------------

Accuracy                                 0.25       690
Macro Avg         0.26       0.25      0.24       690
Weighted Avg      0.26       0.25      0.24       690

Base LLM Confusion Matrix

Figure 3. Confusion Matrix of Base LLM on In-Domain Evaluation Dataset (Inference).

---

Out-of-Domain Evaluation

Benchmark

Dataset: MohammadKhosravi/ielts-cefr-benchmark-prompts

A manually curated benchmark containing IELTS-style writing prompts designed to evaluate prompt generalization beyond the EFCAMDAT training domain.

Dataset Repository

https://huggingface.co/datasets/MohammadKhosravi/ielts-cefr-benchmark-prompts

Dataset Columns

Column
prompt
cefr

Distribution

Balanced across all CEFR proficiency levels.

CEFR	Samples
A1	50
A2	50
B1	50
B2	50
C1	50
C2	50

Total: 300 prompts

---

The E2A Dual-FPMT controller generalizes beyond the EFCAMDAT domain, improving strict CEFR accuracy from 18.33% to 44.67% on unseen IELTS-style prompts while substantially reducing the mean absolute error (MAE) from 1.57 to 1.08. These results indicate that the dual-stream PrefixMemory architecture learns transferable proficiency control rather than memorizing in-domain prompt distributions.

---

Evaluation Results

Metric	PrefixMemory	Base Llama
Strict Accuracy	44.67%	18.33%
Adjacent Accuracy	67.00%	54.33%
MAE	1.08	1.5733
Average MDD	1.94	2.07
Reading Ease	72.30	45.18
Sentence Drift	2.57	2.53

Improvement over Base Llama

• +26.34 percentage points Strict Accuracy
• +12.67 percentage points Adjacent Accuracy
• Lower MAE
• Significantly improved readability control
• Strong transfer to an unseen prompt style

---

Intended Use

This model is intended for research on:

• Parameter-Efficient Fine-Tuning (PEFT)
• PrefixMemory-Tuning
• Dual-stream memory architectures
• Architectural ablation studies
• Controllable text generation
• CEFR-controlled writing generation
• Educational NLP
• Readability control
• English proficiency modeling

---

Citation

If you use this model in your research, please cite the following resources:

• PrefixMemory-Tuning (arXiv:2506.13674)
• Meta-Llama-3.1-8B-Instruct
• EFCAMDAT Corpus

---

Repository

This repository contains the E2A architectural ablation of the proposed Dual-Stream Factorized PrefixMemory-Tuning (Dual-FPMT) framework and is intended to facilitate reproducible research on parameter-efficient controllable language generation.