�Advanced Machine Learning Lab�Project Presentation

Using Language and Image generative models to make English language exercises.

1

By: László Kovács, Márton Juhász

2025.12.17.

Project overview

The main goal of the project was creating English language exercises based on CEFR difficulty using natural language models and image generative techniques.

​

Main Goals:

1. Explore architectures for text generation.
2. Explore NLP methods to make comprehension and fill in the blank exercises.
3. Explore architectures for image generation.
4. Explore methods for creating image comprehension exercises.

2

The CEFR levels

• CEFR stands for Common European Framework of Reference for Languages an international  standard describing language ability on a six-level scale. (A1, A2, B1, B2, C1, C2)

3

Image is from The German Language Levels – SmarterGerman

Text generation

4

Text data acquisition

• For text generation we needed CEFR annotated datasets.
• CEFR-SP Dataset: 17k English sentences annotated with CEFR-levels assigned by two English-education professionals.

(CEFR-SP/CEFR-SP at main · yukiar/CEFR-SP)

​

5

Text data pre-processing

• To not fraction learning merged difficulty levels into: A, B, C
• We also limited to dataset to contain the same number of sentences in each difficulty level.
• Final dataset contained 2730 sentences (930/difficulty level)
• Data was divided into training and validation set
• Training dataset: 2511 (90%)
• Validation dataset: 279 (10%)

​

6

Text generation overview

Goal: Try out different architectures for generating texts/sentences based on CEFR difficulty level.

Architectures:

1. LSTM RNN
2. Conditional VAE
3. Decoder only Transformer
4. Fine-tuning a pre-trained model

​

Question: Can low amount or low-quality data be counterbalanced by complex architecture?

7

Long Short-Term Memory RNN

8

LSTM RNN

• Long Short-Term Memory architecture is a variation of simple RNNs, that can capture long-term dependences.
• It involves a memory cell controlled by three gates
• Forget gate: Controls what information is removed from the memory cell
• Input gate: Controls what information is added to the memory cell
• Output gate: Controls what information is output from the memory cell

​

​

9

Image is from What is LSTM - Long Short Term Memory? - GeeksforGeeks

Our LSTM RNN model

• Explored two different embedding:
• Whitespace-based word embedding: Every unique word separated by a space is treated as a distinct "token“
• Dictionary size: 8076
• Glove embedding: Unsupervised learning algorithm designed to generate embeddings
• Has pre-defined dense vectors for around 6 billion English words
• LSTM model:
• Generates text based on previous word + difficulty level condition
• Number of layers: 2
• Hidden size: 32
• Embedding size: 16

​

​

​

10

Difficulties during the LSTM RNN

• Increasing training loss during training
• Cause: To high learning rate
• Solution:
• Schedular: Decreases learning rate if loss starts to increase
• Increasing validation loss
• Possible causes:
• Not enough data (probable)
• High learning rate
• Solution:
• Dropout layer (0.3): Randomly turns of neurons during training
• L2 regularization: penalizes the model for having weights that are to high.
• Early stopping

​

​

​

​

​

11

val

Word Embedding LSTM RNN Results

• Results:
• Good training a validation loss decrease stabilizing at the same value
• Output is seemingly random words following each other
• Probable causes:
• To simple architecture
• Not enough data

​

​

​

​

12

Output Example:

Level A:

• despite partially Norman proto-oncogene explosion Our genome paid supplementary changed.
• This 60 hours vellum apartheid leg Statistics reactive Octopuses nocturnal.

​

Level B:

• acoustical or coats physics 1431 coffee 87 Murids less predation.
• 72 monofloral why 70,000 contractual serf crown Hox type environments.

​

Level C

• unfolding counselor orchestration thermodynamic melting take bracelet cumulative goal TV.
• mid-air led parabola classroom terrestrial vanilla prominent Anti-nuclear addictive Millipedes.�

GloVe Embedding LSTM RNN Results

• Results:
• Validation loss increases while training loss decreases (overfitting)
• Output:
• generates similar sentences to the training data
• Starts to resemble normal grammar

​

​

​

13

Output Example:

Level A:

• I wish I couldn’t know to buy your passport?
• I wish I had wind.

​

Level B:

• The movie received the countryside shadowy.
• The following of a event is leased by the union is a one-year.

​

Level C

• In the late 19th century , the league of pteridosperms is the main S of the corpus spongiosummid.
• The gymnosperms of glaucophytes , and have found the squid , which can be a mutualistic�

Conditional Variational AutoEncoder

14

Conditional Variational AutoEncoder

• Generative model that learns a continuous probabilistic representation of the features. This makes it capable of generating new data.
• Not used as much anymore, simple architecture
• Main parts:
• Encoder: compresses input into a low dimensional representation.
• Latent space: probability distribution of the data defined by the mean and the variance
• Decoder: randomly samples the latent space and tries to reconstruct the original input.
• Total loss: Reconstruction loss + KL divergence

​

​

15

Image is from(Variational AutoEncoders – GeeksforGeeks)

Our CVAE model

• Instead of word tokenization we used SentecePiece tokenization by Google.
• Model: all-MiniLM-L6-v2
• For the encoder: Data split to sentences (SentencePiece embedded) and labels (One-hot encoded)
• For the decoder: Tokenized word sequence (same as in LSTM)
• CVAE model:
• Encoder: Dense Network
• Decoder: LSTM Network
• Latent space dimension: 64
• Vocab Size: 30522
• Schedular to ensure loss decrease
• Dropout layer (rate = 0.3) to help validation loss decrease

​

​

​

​

​

​

16

Difficulties of CVAE

• The main problem of CVAE was the Posterior Collapse

​

​

​

​

17

Posterior Collapse: ��The model ignores important structural features of the input data and fails to learn meaningful representations in the latent space

• Solution ideas:
• KL annealing: gradually increasing the weight of the KL divergence term in the loss function from zero to a target value (0.3) → unsuccessful
• KL Thresholding: optimizes to a max(KL, λ) value instead to 0. (success)

​

​

​

​

​

Results of CVAE

• Training is resource intensive
• Outputs are incoherent sentences (overfitting)
• Level A: we he he the there homo a many if deep
• Level B: fourth most runner visitors the metals all ne meta green
• Level C: football the under enzymes partiallyization article bio meta the

18

• Options for improvement:
• Increase latent space (pricey)
• Increase the weight of KL even more → worst reconstruction loss
• Longer training
• Larger database

​

Decoder only Transformer

19

Decoder only transformer

• Modern method for text generation
• Transformer architecture uses attention mechanism to process an entire sentence at once instead of reading words one by one.
• Positional encoding added to embedding
• CEFR conditioning with start tokens <A>, <B>, <C>

​

​

​

​

20

Our Transformer model

• Multi-head attention model
• Complex model architecture
• Small Model parameters:
• Embedding dimension: 256
• Attention heads: 4
• Layers: 4
• Dropout: 0.3
• Batch size: 32
• Main issue is validation plateauing during training (overfitting)
• Cause: Model can’t learn anymore (too simple)
• Tried scaling up the model

​

​

​

​

​

21

• Large Model parameters:
• Embedding dimension: 512
• Attention heads: 8
• Layers: 6
• Dropout: 0.3
• Batch size: 64

​

​

​

​

​

Transformer results

• The larger model showed similar tendences in validation and training loss.
• Remained at quite high values
• Small model output produced more coherent sentences
• Because with overtraining it started to learn the data sentences
• Large model output wasn’t good, mostly generated function words/grammatical words.
• Probably because these can be found in high number in sentences

​

​

​

22

Transformer output examples

Small model

23

Large model

Level A (Basic):

• The chef was a very young with my is in the so. governors.
• He was his.
• She had to me my wife . a huge .

​

Level B (Intermediate):

• In this and 's that he were the from his middle 's performance and were not to the
• A large or unlockable and is an for one .
• Since she was , at some can be more in the military .

​

Level C (Advanced):

• Other enzymes , and have also common and animals have been a highly of the surface to
• The three major are a common to , with two that are known as a other
• With the treatment are a of other as the highly of a theory .

Level A (Basic):

• I are a world is a same and to the first with you , with the year.
• The be the be the same and the same to be the day.
• The song is i in a be and and the day.

​

Level B (Intermediate):

• He was have the number of a first and a number of a same as a first in.
• A city , the lot is a number is a first to be a first is the same.
• A be a be the first in a number of the same.

​

Level C (Advanced):

• he is the be in the first .
• the main in the same the first.
• the lot , in the first in the day .

Fine-tuning GTP-2

24

Fine-tuning a GTP-2 model

• Fine-tuned a GTP-2 model.
• Expended its tokenizer with special difficulty tokens: <A>, <B>, <C>
• These served as control codes
• Advantages:
• As the model is already trained and therefore speaks fluent English, a small dataset is enough to make it generate desired difficulty data
• Can easily learn any format
• Gives good and coherent results
• Disadvantage:
• Large model, training is slow, but need less epocs

​

​

​

​

25

GTP-2 model output example

26

Level A (Basic):

• I am impatient .
• You should have seen the fire alarm in the mailbox .
• We can't wait to see what she has in store for us .

​

Level B (Intermediate):

• Should have been a movie .
• Weekly subscriptions to the site cost about $100 .
• In the past , Japan was a major trading partner of the United States .

​

Level C (Advanced):

• Negligence of duty is a basic constitutional right .
• The Hypthesis process is similar to the synthesis of proteins , although the two functions are different .
• Consequently , the number of amino acids in the human gut has been estimated from published databases .

Generating exercise based on texts.

27

Generating exercises from texts

• Goal: Creating an AI pipeline to generate multiple choice English language exercises:
• Fill in the blank
• Text comprehension questionary
• The pipeline:
1. Input: A sentence or text of given CEFR level (preferably generated by us)
2. Keyword Extraction: spacy library’s en_core_web_sm model
3. Question generation: T5 model trained by SQuAD dataset (Stanford Question Answering Dataset)
4. Distractor generation:
1. Wordnet: Lexical database designed for NLP, finds synonyms/antonyms
2. Sen2Vec: Uses word vector representation to find words used in similar context (better)
• Same pipeline can be used for both exercise types, but for the fill in the blank question generation is not needed.

​

​

​

28

Example fill in the blanks excersise

​

• If it wasn't a ______ of time , I would be finished by now .
• a) question
• b) Question-
• c) First Question
• d) Initial Question
• Spyros Kyprianou died on 12 ______ 2002 after a long fight with prostate cancer .
• a) March
• b) Early March
• c) July
• d) Early April

​

29

Example for text comprehension

It was a stupid mistake. Before I tell you about my mistake, I have to tell you about my father. My father Joseph Thomas was a big name in the music business. He managed a lot of different rock bands in the '60s. By the end of his career, he had saved up some money and decided to buy himself a cabin in a little town in Oregon. He didn't want to live in Los Angeles, New York City, Chicago, or any other big cities. He wanted to escape the big city life. My father died in that cabin three years ago. I found out about his life in the cabin a week after he died. He never really liked to answer his cell phone. It would always be on low battery, or off. He never bothered to charge it. He simply liked listening to his records and drinking whiskey. During his time in the music industry, my father had collected various objects. He had different items from different artists. He had one of Jimi Hendrix's guitars, and a medallion that Janis Joplin had given him. Did I know this? No, I didn't. My dad always kept to himself. He was out on the road a lot, and he never really cared about keeping in touch with his kids. So, when he died, I figured all of his stuff at the cabin would be garbage. I told the town's Sheriff to throw everything out. I also told him to send me the paperwork for the cabin. I read about the discovery they made in my father's cabin. I couldn't believe it! Hendrix's guitar sold for over a million dollars, and so did the medallion. As I had already asked the Shariff to throw out everything as garbage, I wasn't entitled to anything, which probably made my father happy.

​

Question:

Whose guitar sold for over a million dollars?

a) Hendrix

b) David Gilmour

c) Bowie

d) Frank Zappa

Who gave my father a medallion?

a) Graham

b) Boykin

c) Latimer

d) Janis

​

30

Generating images and questions.

31

Image and question generation overview

Goal: Explore different architectures for image generation, then use the generated images to create questions.

• Image generation
• Pixel-Space Diffusion
• Lightweight UNet Architecture
• Improved UNet Architecture
• Residual blocks
• Latent Diffusion Model (LDM)
• Question generation
• BLIP-2 with LoRA fine-tuning

32

UNet Architecture – Overview

• Main parts:
• Encoder:
• Series of convolutional layers + activation
• Downsamples spatial dimensions via pooling
• Increases feature channels
• Bottleneck:
• The deepest part of the network
• Decoder:
• Upsamples spatial dimensions gradually
• Reduces feature channels

33

Noise image + timestep

Predicted noise

UNet

Lightweight UNet Architecture

• Input: Fashion-MNIST dataset (28×28 grayscale images)
• Encoder:
• 3 convolution layers + ReLU
• Spatial resolution: 28×28 → 7x7
• Channels: 1 → 64
• Bottleneck:
• 1 convolution layer
• Decoder:
• Transposed convolution layer
• Spatial resolution: 7x7 → 28×28
• Channels: 64 → 1
• Time embedding is added directly to the input image

​

34

Improved UNet Architecture

• Input: Fashion-MNIST dataset (28×28 grayscale images)
• Encoder:
• 3 convolution layers + GroupNorm +SiLU
• Spatial resolution: 28×28 → 7x7
• Channels: 1 → 256
• Bottleneck:
• 1 convolution layer
• Self-attention
• Decoder:
• 3 Transposed convolution layer
• Spatial resolution: 7x7 → 28×28
• Channels: 256 → 1
• Time embedding:
• Sinusoidal embedding is �added to bottleneck and decoder

​

35

A. Vaswani et al., Attention is all you need (2017)

https://medium.com/@hirok4/understanding-transformer-sinusoidal-position-embedding-7cbaaf3b9f6a

Results

36

ResBlock Architecture

37

• x: input
• H(x): full function the block is trying �to approximate
• New output: F(x) + x → H(x)= F(x) + x
• F(x) = H(x) – x, residual, small correction
• The block learns F(x), instead of H(x)
• Easier
• Helps train very deep networks
• Learns identyity

​

Extra new features:

• More layers
• Dropouts
• EMA (Exponential Moving Average)

He et al., Deep Residual Learning for Image Recognition (2015)

Results of Pixel-Space Diffusion

38

Model improvement

Latent Diffusion Model - Workflow

• Input image → VAE encoder → μ, logvar
• Reparameterization: z=μ+σ⋅ϵ
• Add noise
• Feed noisy latent to Unet
• UNet predicts the noise → MSE loss

39

Train VAE → freeze it → train LDM

Training VAE

• Input: AFHQ (Animal Faces HQ) 64×64 images
• Encoder: 3 convolutional layers
• Latent space (4 dimensions):
• μ and log variance computed from encoder output
• Reparameterization trick: z=μ+σ⋅ϵ for sampling
• Decoder: ConvTranspose layers → reconstruct images from latent vector
• Loss: L1 + KL
• L1 loss preserves image sharpness better than MSE

40

Reconstructing images - VAE

41

Tried:

• L1 vs MSE
• Higher latent dim

No major improvement

L1 helped a bit

Results

42

Cats

Dogs

Wildlife

Question generator for images

BLIP-2:

• Vision Encoder: A pre-trained image model extracts features from an image
• LLM: Understands text-like embeddings
• Q-Former: Translator

43

arXiv:2301.12597

BLIP-2 without fine-tuning

44

Prompt: Generate three different questions someone could ask about this image.

Result: Generate three different questions someone could ask about this image.

Fine-tuning with LoRa (low-rank adaption)

45

https://towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm/

Visual Question Answering (VQG):

dataset containing open-ended questions about images

Generated questions

• 1. what is the color of this car?madeupword0002
• 2. madeupword0002 what are they doing?EStreamFrame of the driver is showing that his car has an engine cover
• 3. What race is this? guiName_is_one of the five drivers that will be racin
• 1. why is the red bull racing car on a race track?
• 2. What is the type of fuel that he uses?
• 3. what is the name of this car?
• 4.can you hear the sound of cars in this picture?
• 5. what does this racing car look like?

46

ignoring padding tokens when computing loss

Generating exercise based on images.

47

Project conclusions

• Text generation with correct grammar is difficult especially on limited data.
• The importance of large high-quality data can not really be counterbalanced with complex architecture.
• There are widely available tools for NLP making exercises creation easy and simple.
• Training a diffusion model from scratch to generate hiqh quality images on a laptop is impossible

​

48