UNIT – 3

​

​

• Generation of Images: Introduction to Generative Adversarial Networks,
• Adversarial Training Process,
• Nash Equilibrium,
• Variational Autoencoders,
• Encoder-Decoder Architectures,
• Stable Diffusion Models,
• Introduction to Transformer-based Image Generation,
• CLIP, Visual Transformers ViT- Dall-E2 and DallE3
• GPT-4V,
• Issues of Image Generation models like Mode Collapse and Stability.

• Generative Adversarial Networks (GANs) revolutionized AI image generation by creating realistic and high-quality images from random noise.
• In this article, we will train a GAN model on the MNIST dataset to generate handwritten digit images.

​

A GAN consists of two neural networks that compete with each other:

1. Generator (G): Creates fake images from random noise.
2. Discriminator (D): Determines whether an image is real (from the dataset) or fake (created by the generator).

​

Generation of Images: Introduction to Generative Adversarial Networks

2. Working of GAN

Random Noise → Generator → Fake Image → Discriminator → Real/Fake

​

​

The process is:

• Random noise is given to the Generator.
• The Generator produces a new image.
• The Discriminator receives both real images and generated images.
• It predicts whether each image is real or fake.
• The Generator improves its ability to create realistic images.
• The Discriminator improves its ability to detect fake images.
• Through repeated training, the Generator learns to produce highly realistic images.

ADVERSARIAL TRAINING PROCESS IN GENERATIVE AI

Adversarial training is a learning process in which two neural networks compete with each other to improve the quality of generated data. It is most commonly used in Generative Adversarial Networks (GANs).

Main Components

Generator (G)

• Creates synthetic data such as images, text, or audio.
• Takes random noise as input.
• Tries to generate samples that look like real data.

Discriminator (D)

• Acts as a classifier.
• Receives both real samples from the dataset and fake samples produced by the Generator.
• Predicts whether each sample is Real or Fake.

​

Training Process

​

Step 1: Generate Fake Data�Random noise is given to the Generator.

Step 2: Discriminator Evaluation�The Discriminator receives:

• Real data → from the training dataset
• Fake data → from the Generator

Step 3: Calculate Errors�The Discriminator learns to correctly distinguish real samples from generated samples.

Step 4: Update the Generator�The Generator is trained to produce increasingly realistic samples that can fool the Discriminator.

Step 5: Repeat the Competition�Generator and Discriminator are trained repeatedly in an adversarial manner.

Step 6: Convergence�Ideally, the Generator becomes capable of producing highly realistic data, while the Discriminator can no longer reliably distinguish fake data from real data.

NASH EQUILIBRIUM

• Nash equilibrium is an important concept in game theory that provides the optimal outcome in case the player doesn’t deviate from their initial strategy.
• This is done in response to no incentive provided to the players for such deviation.
• This was named after the Mathematician, John Nash, defining the solution of a non-cooperative game involving two or more players.

​

Key Points:

​

• Nash equilibrium provides an optimal solution to encounter the required outcome by not deviating from their initial strategy.
• Since individuals are already aware of each other’s strategy, both the players win as everyone gets the outcome that they thought for.
• One such example is the Prisoner's dilemma.
• Since the players quest to win, they would be performing that strategy that would lead them to such a state. Thus the chosen strategy is the best and the optimal solution that they can use. This is also in conjunction with the dominant strategy.
• Further, as stated there cannot be any Nash equilibrium in a game. So, it is not always true that the strategy chosen is the optimal one.

Nash equilibrium vs Dominant strategy

• Nash equilibrium reveals the optimal solution to be one when the player doesn’t deviate from their initial strategy and when he is aware of other players' strategies as well.
• None of the players will be changing their strategy and will be keeping it the same.

Dominant strategy

• on the other hand, makes sure that the chosen strategy will lead to better results out of all the possible strategies available.
• It doesn’t conform to the strategy the opponent is supposed to use.
• Though both seem quite similar, there is a grave difference between them.
• Nash equilibrium connotes that none of the players will win if one of the users deviates from its strategy, keeping the other players constant, though deviating from its initial strategy, as per dominant strategy, will lead to win of the player, as this being his best solution.

Example of Nash equilibrium

In the given game, we have two players with strategies S1 and S2.

S1 being the optimal strategy considering both the players know each other’s approach or they are made aware of it and both will be moving their initial strategy. They will opt for it because they know deviating from it will not be in the interest of the match.

Hence in case they opt for S1, both will win. In other cases, deviating to S2 will result in defeat of one of them.

Variational Autoencoders

A Variational Autoencoder (VAE) is a Generative AI model that learns the hidden/latent representation of data and uses it to generate new, realistic samples similar to the training data.

• Learns a continuous latent representation
• Enables controlled and meaningful data generation
• Widely used in image synthesis, anomaly detection, and representation learning

1. Encoder (Understanding the Input)

The encoder takes input data like images or text and learns its key features. Instead of outputting one fixed value, it produces two vectors for each feature:

• Mean (μ): A central value representing the data.
• Standard Deviation (σ): It is a measure of how much the values can vary.

These two values define a range of possibilities instead of a single number.

2. Latent Space (Adding Some Randomness)

• Fixed point: Input is not encoded as one fixed value.
• Random point: A random value is selected based on Mean and Standard Deviation.
• Introduces randomness: This creates small variations in the data.
• Generates new data: The model can produce different versions of the original data.
• Realistic samples: Helps generate new and realistic data samples.

3. Decoder (Reconstructing or Creating New Data)

• The decoder takes the random sample from the latent space and tries to reconstruct the original input.
• Since the encoder gives a range, the decoder can produce new data that is similar but not identical to what it has seen.

Encoder-Decoder Architectures

The encoder-decoder model is a neural network used for tasks where both input and output are sequences, often of different lengths. It is commonly applied in areas like translation, summarization and speech processing.

• The encoder processes the input sequence and converts it into a fixed representation (context vector)
• The decoder uses this representation to generate the output sequence step by step
• Works well for tasks where input and output lengths are different

Encoder

The encoder processes the input sequence and converts it into a fixed representation (context vector) using an RNN or LSTM.

• Processes input tokens sequentially and updates hidden states
• Captures relationships between words in the sequence
• Produces final hidden and cell states forming the context vector

Decoder

The decoder uses the context vector from the encoder to generate the output sequence step by step.

• Takes previous output and context to predict the next token
• Generates output sequentially until an end token is reached
• Initializes its states using the encoder’s final states

Working of Encoder Decoder Model

​

The actual working of the encoder decoder model is shown in below diagram. Now we will understand it stepwise

Step 1: Tokenizing the Input Sentence

• The sentence "I am learning AI" is first broken into tokens: ["I", "am", "learning", "AI"].
• Each word (token) is converted into a vector that a machine can understand. This process is called embedding.

Step 2: Encoding the Input

• The encoder processes these embeddings sequentially using an LSTM network.
• At each step, it updates its hidden state based on the current word and previous context. This helps the model understand the sequence order and relationships between words.
• After processing the full sentence, the encoder generates a context vector (final hidden and cell states), which represents the meaning of the entire input sentence.

Step 3: Passing the Context to the Decoder

• The Context Vector is passed to the Decoder as shown in image.
• It acts like a summary of the full input sentence.

Step 4: Decoder Generates Output Step-by-Step

• The Decoder uses the context and starts creating the output one word at a time.
• First it predicts the first word then uses that to predict the second word and so on.

Step 5: Attention Mechanism

• Basic encoder-decoder uses a single context vector, which can limit performance for long sequences.
• Attention mechanism helps the decoder focus on different parts of the input at each step.
• Improves accuracy by not relying only on one fixed representation.

Step 6: Producing the Final Output

• The decoder continues generating until the full translated sentence is produced.
• Each output token depends on the previous ones and the input context. You finally see the output tokens generated on the right side of the diagram completing the translation.

Stable Diffusion is a latent diffusion-based Generative AI model used to generate images from text prompts. It creates an image by starting with random noise and gradually removing the noise until a meaningful image is produced.

Stable Diffusion Models

Introduction to Transformer-based Image Generation

• Transformer-based image generation is a Generative AI technique that uses the Transformer architecture to generate new images from text, image prompts, or learned representations.

​

• Transformers were originally developed for natural language processing, but they are now widely used for image synthesis, image editing, text-to-image generation, and multimodal AI.

Step 1: Input Prompt�The user provides a text description such as “A red car on a mountain road.”

Step 2: Tokenization and Encoding�The text is divided into tokens and converted into numerical representations called embeddings.

Step 3: Transformer Processing�The Transformer uses self-attention to understand relationships between words, image tokens, or visual features.

Step 4: Image Token Generation�The model predicts image tokens that represent different parts of the desired image.

Step 5: Image Reconstruction�The generated tokens are converted into pixels or decoded into a complete image.

Step 6: Final Image�The output is a new image corresponding to the given prompt.

CLIP (Contrastive Language–Image Pre-training)

CLIP (Contrastive Language-Image Pretraining) is a multimodal AI model that learns the relationship between images and text. By training on image–text pairs, it can understand visual content through natural language descriptions and perform image recognition without task-specific training.

• Uses contrastive learning to distinguish matching and non-matching image–text pairs.
• Combines an image encoder and a text encoder to process visual and textual inputs.
• Enables zero-shot image classification using natural language prompts.

Key Features

​

• Multimodal Training: Learns from image–text pairs to build a shared understanding of visual and textual information.
• Zero-Shot Learning: Classifies images into new categories without additional training or fine-tuning.
• Self-Supervised Learning: Learns by distinguishing matching and non-matching image–text pairs instead of relying on manually labelled datasets.

1. Text Encoder

CLIP uses a Transformer-based text encoder to convert text into dense embeddings that capture semantic meaning. The encoder contains 63 million parameters, 12 layers, and 8 attention heads.

2. Image Encoder

CLIP uses an image encoder to convert images into feature embeddings. It experimented with both ResNet and Vision Transformers (ViT), with ViT showing superior performance in image understanding.

3. Dataset

CLIP was trained on about 400 million image–text pairs collected from the web. The dataset, called WebImageText (WIT), covers around 500,000 words by focusing on terms frequently appearing in English Wikipedia, enabling strong generalization across diverse concepts.

​

4. Training Objective

CLIP learns to align image and text embeddings so that matching pairs are similar and non-matching pairs are far apart in a shared space.

• Cosine Similarity: Maximizes similarity for correct pairs and reduces it for incorrect ones.
• Shared Projection Space: Maps image and text embeddings into the same dimensional space for comparison.
• Contrastive Learning: Trains the model to distinguish between matching and non-matching image–text pairs.
• Loss Function: Uses cross-entropy loss to strengthen correct matches and suppress incorrect ones.
• Inference: Computes similarity scores between image and text embeddings to find the best match.

Visual Transformers (ViT), DALL·E 2 and DALL·E 3 in Generative AI

Vision Transformer (ViT) is a Generative AI architecture that applies the Transformer mechanism to image processing. Instead of processing an image using convolution operations, ViT divides the image into small patches, converts them into tokens, and processes them using self-attention.

1. Image Patching and Embedding

This stage converts a 2D image into a sequence of patch embeddings, analogous to tokens in NLP. It forms the input for the Transformer by turning spatial information into a linear sequence.

• Patch Splitting:The image is divided into fixed size and non-overlapping patches each treated as a token and converted into a 1D sequence for the Transformer reducing computation while preserving local spatial information.

2. Positional Encoding

• Need for Positional Encoding: Since Transformers treat tokens as unordered positional encodings are added to retain spatial structure and patch location information.
• Learnable Positional Embeddings: ViT uses learnable positional vectors to capture local and global spatial relationships adapting better than fixed encodings across image resolutions.

3. Adding the Classification Token (CLS Token)

A learnable CLS token is prepended to the patch sequence to aggregate information from all patches, serving as the image-level representation for classification.

• Purpose of the CLS Token: The CLS token is a learnable vector added to patch embeddings that gathers global information and is used for final classification, similar to BERT.
• How the CLS Token Learns Image-Level Representation: The CLS token attends to all patches to learn global image features and its final output alone is used for prediction without CNN style pooling.

4. Transformer Encoder (Pre-LayerNorm Architecture)

8. Classification Head (MLP Head)

Converts the CLS token output into class probabilities using a small feed-forward network.

• MLP Head Structure: The classification head uses one or two fully connected layers on the final CLS token to produce class probabilities, optionally with dropout for regularization. It serves as the ViT’s final decision-making component.
• Softmax for Prediction: Softmax converts logits into normalized probabilities summing to 1, with the highest probability indicating the predicted class. It enables multi-class classification and pairs with cross-entropy loss during training.

DALLE 2 Architecture

DALL-E 2, the successor to its predecessor DALL-E, represents an open-source breakthrough by OpenAI. While DALL-E 1 employed discrete variational autoencoders (dVAE) for image generation, its revolutionary approach had constraints in terms of image resolution, realism, and image refinement. In contrast, DALL-E 2's architecture pivots towards the diffusion model, departing from dVAE, allowing it to directly generate images from CLIP embeddings.

​

The inception of DALL-E 2 is documented in the paper "Hierarchical Text-Conditional Image Generation with CLIP Latents," initially introduced as UnCLIP. The nomenclature UnCLIP was chosen due to its unique approach of image generation by inversely utilizing the CLIP image encoder. Over time, this innovative model earned its more recognized title as DALL-E 2.

Dall-E 2 Architecture

There are three main components of DALLE-2:

1. CLIP: Clip is an open-source model and it tells whether a text embedding matches the image embedding or not.
2. Diffusion Prior (based on transformer): For generating clip image embeddings based on clip text embeddings.
3. Decoder (based on diffusion model): For generating image.

Below is the overall architecture as presented in the original paper. Let us discuss each of the components in detail.

1. Contrastive Language-Image Pre-training (CLIP)

The component shown above the dotted line is the clip component. CLIP is trained first. The objective of CLIP is to align text embeddings and image embeddings. CLIP tells how much a given text relates to an image.Note - It does not tell the caption for an image. It only whether a given description is a good fit or not for a given image

CLIP is trained as per the below process.

1. All image and their true captions or descriptions are encoded. There is a separate encoder for images and a separate encoder for text.
2. The cosine similarity between the encoded vectors of the image and text description is calculated.
3. The training objective is to maximize the similarity between the correct pair of image and text embeddings and minimize the similarity between incorrect image and text embeddings.
4. During inference time the embedding of image and text description is obtained through the trained image and text encoder and a similarity score is calculated which indicates the relevance of text description concern ingo the image

2 . Diffusion Prior

After the training of CLI, P the diffusion prior and image decoder are trained. The objective of diffusion prior is to take text embedding and find its associated CLIP image embeddings. Too so it takes help for the CLIP model trained in step 1.

The diffusion prior is a decoder only transformed with a causal attention mask. The input consists of 4 vectors.

1. Noise clip image embedding
2. Diffusion timestep
3. Encoded text
4. CLIP text embedding
1. For a given correct text image pair we find its CLIP image embedding and CLIP text embedding
2. Noise is added to the CLIP image embedding. The amount of noise added depends on the diffusion timestep. As we proceed further and further more noise is added
3. The purpose of the diffusion prior is to predict the CLIP image embeddings.
4. The MSE loss between the original CLIP embedding and predicted CLIP image embedding is used as the loss function

3. Image Decoder

The purpose of the image decoder is to use the CLIP image embedding predicted by the diffusion prior and generate an image based on this. The diffusion model of the image decoder is based on the paper GLIDE: Towards Photorealistic Image Generation and Editing with text-Guided Diffusion Models. The GLIDE paper combined the diffusion process with UNET architecture.

UNET Architecture

• The left part of the U-NET architecture consists of a series of convolutional and pooling layers, which function as an encoder. This part captures the contextual information in the input image.
• The right part of the U-Nent consists of a series of upsampling and convolutional layers, forming the decoder
• The U-Net architecture consists of skip connections. These connections directly link the corresponding encoder and decoder layers to enable the network to use both high-level spatial information and low-level spatial information.

Diffusion process

�

�

�

• The above unet architecture is utilized to build and train a diffusion model as shown below. The training process for a diffusion model involves two main steps:
• Forward diffusion: The model takes a clear image as input and adds noise to it.
• Reverse diffusion: The model aims to reconstruct the original clear image from the noisy version
• Between each block of the encoder and decoder we have a cross-attention layer. This cross-attention layer takes the clip embeddings of text embeddings and output of the previous layer to generate a context vector which is used in the next layer. This helps in guided text generation.
• The output of the decoder is further up sampled to increase the resolution of the image.

GPT-4V (GPT-4 with Vision) is a multimodal Generative AI model developed by OpenAI. Unlike traditional language models that primarily process text, GPT-4V can understand both text and visual information such as images, charts, diagrams, and screenshots.

GPT-4V

1. Image Understanding:�GPT-4V can identify and interpret objects, scenes, people, and other visual information in images.

2. Text Recognition:�It can understand text appearing inside images, documents, and screenshots.

3. Visual Question Answering:�Users can ask questions about an image, and the model can generate answers based on its visual content.

4. Chart and Graph Analysis:�It can analyze graphs, tables, and other visual representations of data.

5. Multimodal Reasoning:�It can combine information from text and images to produce a more context-aware response.

6. Image-Based Explanation:�It can explain diagrams, educational figures, and technical illustrations in natural language.

Main Features

Applications

​

GPT-4V can be used in:

• Education: Explaining diagrams, mathematical problems, and textbook images.
• Healthcare: Assisting with interpretation of medical images, subject to appropriate professional validation.
• Document Analysis: Understanding scanned documents and forms.
• Data Analysis: Interpreting charts and graphs.
• Software Support: Understanding screenshots and user-interface problems.
• Accessibility: Helping describe visual content for users who cannot easily access it visually.
• Customer Support: Analyzing product images and screenshots.

Example

Suppose a student uploads an image of a bar chart and asks:

​

• "Which category has the highest value?“

​

• GPT-4V can analyze the visual chart, identify the relevant bars, and generate a textual answer.

Issues of Image Generation Models: Mode Collapse and Stability

​

​

Image generation models, especially GANs (Generative Adversarial Networks), can produce realistic images, but they face several important challenges. Two major issues are Mode Collapse and Training Stability.

​

1. Mode Collapse
2. Training Stability

1. Mode Collapse

​

Definition:

�Mode collapse occurs when a generative model produces very similar or nearly identical images, even when different random inputs are provided.

Different Random Inputs

↓

┌────┼────┬────┐

z₁ z₂ z₃ z₄

↓ ↓ ↓ ↓

┌────┴────┴────┴────┐

Generator

↓

Similar Images

🖼️ 🖼️ 🖼️ 🖼️

Instead of learning the complete diversity of the training dataset, the generator discovers a small number of outputs that can successfully fool the discriminator.

Example:�If the training dataset contains cars, buses, trucks, and bikes, a collapsed generator may produce mostly cars.

Causes:

• Poor balance between generator and discriminator.
• Generator finds an easy output that fools the discriminator.
• Insufficient diversity in training data.
• Unstable optimization.

Effects:

• Low diversity in generated images.
• Important data patterns are not learned.
• Different input noise vectors may produce similar outputs.

Ways to reduce Mode Collapse:

• Mini-batch discrimination.
• Wasserstein GAN (WGAN).
• Gradient penalty.
• Better dataset diversity.
• Improved generator/discriminator architectures.

2. Training Stability

​

Definition:

�Training stability refers to the ability of an image generation model to learn smoothly without sudden failures, oscillations, or divergence.

​

The Generator (G) tries to create realistic images, while the Discriminator (D) tries to distinguish real images from generated images.

If one becomes much stronger than the other, training can become unstable.

​

Problems caused by instability:

• Oscillating losses.
• Generator produces poor-quality images.
• Discriminator becomes too powerful.
• Vanishing or exploding gradients.
• Training may fail to converge.
• Generated image quality changes unpredictably.

UNIT -4

• Generation of Painting, Music, and Play:
• Variants of GAN,
• Types of GAN, Cyclic GAN,
• Using Cyclic GAN to Generate Paintings,
• Neural Style Transfer,
• Style Transfer,
• Music Generating RNN,
• Muse GAN, Autonomous agents,
• Deep Q Algorithm,
• Actor-critic Network.