Generative AI

23CSPE413

Textbooks:

1. Generative Deep Learning Teaching Machines to Paint, Write, Compose, and Play – David Foster 2nd Edition.

2. Rajan Gupta, Sanju Tiwari, Poonam Chaudhary by Generative AI: Techniques, Models and Applications (Springer, 2025)

Reference Books:

1. Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
2. Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow by Aurelien Geron
3. Hands-On Large Language Models: Language Understanding and Genera

What is Generative Modeling?

• Generative Modeling is a branch of Machine Learning
• It learns patterns from existing data
• It trains a model to generate new data similar to a given dataset.
• Output is not copied, but new and realistic

Train a generative model on a dataset to learn the underlying patterns and relationships within the data. Once trained, the model can be sampled to generate new, realistic examples that were not present in the original dataset.

• Text – GPT, Claude, Gemini, LLaMA
• Images – Midjourney, DALL·E, Stable Diffusion
• Audio – ElevenLabs, Bark, Music
• GenVideos – Sora, Runway, Hey
• GenCode – GitHub Copilot, Code Llama, CodeWhisperer

Example:

• Train on horse images → Generate new horse images

Suppose we train a model using horse images.

The model learns the structure, shape, and texture of horses. After training, it can generate completely new horse images that were not part of the dataset.

Figure 1-1. A generative model trained to generate realistic photos of horses

• Collect dataset
• Train the model
• Learn patterns
• Generate new samples

• A generative model requires a large dataset of examples Known as training data
• Each individual example is called an observation
• Every observation is made up of multiple features
• Images → pixel values
• Text → words or characters
• Audio → sound signals
• The goal is to learn the underlying patterns and relationships within the data
• The model aims to generate new data that follows the same rules as the training data
• This is challenging because:
• Data is often high-dimensional
• Only a small subset of all possible combinations is meaningful
• The model must learn to produce valid and realistic outputs, not random noise

​

Probabilistic Nature of Generative Models

• Generative models must be probabilistic, not deterministic
• Deterministic models always produce the same output for a given input
• Do not create diverse or new samples
• Generative models are probabilistic. This means they use probability to generate different possible outputs instead of a fixed result.
• Generative models include randomness. It allows creation of multiple unique outputs.
• The model learns an underlying probability distribution of the data
• This distribution explains why certain data appears in the dataset and why other data does not.
• Model tries to Mimic the real data distribution by learning Patterns, Relationships and Structure.
• After training the Model generates data by sampling. These outputs are Realistic and unique.

​

Generative Versus Discriminative Modeling

Discriminative modeling estimates p(y|x)

• Discriminative modeling aims to model the probability of a label y given some observation x.
• Discriminative models answer the question:�Given this data, what is its label?
• It learns the relationship between:
• Input (x) → data (image, text, etc.)
• Output (y) → label or category
• It directly models:� P(y | x) = probability of label y given input x

​

Discriminative models are like judges. They look at something and decide what it is.

Example:

Imagine you have images of animals.

• Input (x) = image
• Output (y) = “cat” or “dog”

The model learns:� “If I see this image, how likely is it a cat?”

Generative modeling estimates p(x)

• Generative modeling aims to model the probability of observing an observation x.
• Sampling from this distribution allows us to generate new observations.
• Generative models answer a different question:� What does real data look like?
• Instead of predicting labels, they learn:�P(x) = probability of the data itself
• They understand:
• Patterns
• Structure
• How data is formed

Generative models are like artists. They don’t just recognize things—they create new ones.

​

Using the same animal dataset:

• The model learns:� What does a real cat image look like?
• Then it can:�Generate a new cat image that never existed before

Generative Versus Discriminative Modeling

Figure 1-2. A discriminative model trained to predict if a given image is painted by Van Gogh

Figure 1-1. A generative model trained to generate realistic photos of horses

The Rise of Generative Modeling

• Recent advancements (last 10 years)
• Rapid progress in generative AI
• Realistic image, text, and media generation
• Shift in industry usage
• Earlier: Mostly discriminative applications
• Now: Generative AI widely used
• Modern Applications
• Blog/content generation
• Image generation for products
• Social media & ad copy writing
• Game design & cinematography
• AI-generated video & music
• Generative AI is transforming creativity from human-only → human + AI collaboration

​

​

Figure 1-3. Face generation using generative modeling has improved significantly over the last decade

• Generative modeling is also helping improve other areas of AI, especially reinforcement learning.

Traditional RL:

• Trial-and-error learning and Task-specific training
• An agent (like a robot or AI) tries different actions
• It makes mistakes
• It learns slowly over time based on rewards and penalties
• Generative models can learn a world model, it can simulate the environment (Imagine different scenarios and Test actions internally)
• It can test different strategies internally without actually performing them in the real world.
• Makes learning faster, more flexible, and adaptable to new tasks.

Our First Generative Model

Task�“Generate one more point that looks like it belongs to this dataset.”

Instead of learning a complicated mathematical function, the model simply says:

“Any point inside this rectangle could be generated.”

Inside the orange box: points are allowed.

Outside the orange box: probability is zero.

• Once the model has been built, generating new data is easy.
• Choose any random point inside the orange rectangle.
• For example,

(2.1,1.4) or (−0.8,0.5)

These new points were not in the training dataset, but they look as though they could have been.

• This process is called sampling from the model.

​

The Generative Modeling Framework

Representation Learning

• Real-world data is high-dimensional and complex.
• Models learn important features instead of raw data.
• Convert data into a simpler, lower-dimensional form (latent space).
• Focus on meaningful patterns, not raw details.

Example:

• Data like images contain a huge number of values, making them difficult to process directly.
• Instead, representation learning helps the model focus on important features like shape, color, or structure.

Real-world example:

• In face recognition (like Face ID on phones)
• The system does NOT store every pixel
• It stores features like distance between eyes, face shape, etc.
• So instead of raw data, the system uses a simplified representation

​

Intuition Behind Representation Learning

• Humans describe objects using key features, not raw details.
• Use high-level attributes instead of low-level data.
• Few features → enough to recognize or reconstruct.
• Similarly, machines learn important patterns instead of raw data.
• Example:�Describe a person, Don’t describe using pixels:
• Tall
• Wearing glasses
• Black hair

Enough for someone to recognize them.

Real-world example:

• Police sketches
• Witness descriptions
• Even without exact details, people can identify someone.
• Similarly, machines learn compact descriptions instead of full data.

​

Figure 1-7. The biscuit tin dataset

Think of it as a 2-dimensional map:

• Horizontal direction → latent variable z1​
• Vertical direction → latent variable z2​

Every location (point) in this space represents a possible biscuit tin.

The model has learned that different locations correspond to different characteristics of the tin.

The cylinders represent biscuit tins generated by the model.

Notice that the tins have different:

• Heights
• Widths
• Shapes

For example:

• A short, narrow cylinder
• A tall, narrow cylinder
• A short, wide cylinder
• A tall, wide cylinder

So, moving around in the latent space changes the characteristics of the generated tin.

​

Solid vs dashed cylinders

• The solid cylinders represent tins that are associated with regions of the learned data/manifold.
• The dashed cylinders represent other locations in the latent space where the model can potentially generate a tin.
• The important idea is that each point in latent space is mapped to an object in the original data space.

​

Figure 1-8. The 2D latent space of biscuit tins and the function f that maps a point inthe latent space back to the original image domain

The blue dot is a particular point selected in the latent space.

We can represent it as:

z=(z1​,z2​)

For example:

z=(0.35,−0.72)

The exact values aren't important here.

It simply means:

"Choose this particular location in the latent space."

• The arrow on the right shows:

​

where:

z = point in latent space

f = generative/decoder function

x = generated object in the original data space

So the function

f(z)=x

takes a latent representation and converts it into an actual object.

​

Latent Space

​

●

│

│ z

▼

f

│

▼

Biscuit Tin

Instead of directly specifying:

"Generate a biscuit tin that is 5 cm tall and 10 cm wide."

we select a point in the latent space.

The learned function f knows how to convert that point into an appropriate object

Connection to an Autoencoder

• This becomes particularly important when studying Autoencoders and Variational Autoencoders (VAEs).
• An autoencoder has two major functions:
• Encoder
• It takes the original data and converts it into a latent representation.
• Decoder
• It takes the latent representation and reconstructs the data.

​

Figure 1-9. The dog manifold in high-dimensional pixel space is mapped to a simpler

latent space that can be sampled from

Why Latent Space is Powerful ?

• One major advantage of latent space is that it simplifies complex data.
• Easier to manipulate than raw data
• Small changes in the latent representation that result in meaningful changes in the final output.
• Controls high-level features.

1. Raw Data (High-Dimensional)

• Imagine we give a computer thousands of face images.�Each image has millions of pixels — too complex to understand directly.”
• Problem:�Computer sees only numbers, not “eyes”, “nose”, or “smile”.

​

2. Representation Learning (Key Idea)

• A generative AI model (such as an Autoencoder or GAN) does not work directly with raw pixel values. Instead, it learns to compress each face image into a smaller, meaningful representation, called a latent vector.
• This process is done using a neural network (encoder), which extracts important features from the image.
• The latent vector captures key characteristics of the face, such as:
• Face shape
• Eye size
• Skin tone
• Facial expression
• So instead of remembering thousands of pixel values, the model learns a compact summary of the face.

�Pixels → Meaningful Features → Latent Vector

​

3. Latent Space

• Once every image is converted into a latent vector, each face can be thought of as a point in a hidden space, called the latent space.
• Each latent vector = one point in this space
• The position of the point depends on the features of the face
• This creates a meaningful structure:
• Similar faces → placed close together
• Different faces → placed far apart
• Example:
• Smiling faces naturally form a cluster
• Neutral faces form another cluster
• Young and old faces create gradual patterns across the space
• This means the model has learned not just individual faces, but also the relationships between faces.
• Image → Latent Vector → Position in Latent Space

​

Generative Capability

• Once the model learns the latent representation of faces, it gains the ability to generate and modify images.
• Because each face is encoded as a set of meaningful features, the model can:
• Generate new faces that do not exist in the dataset
• Modify specific features such as:
• Adding or removing a smile
• Making a face look older or younger
• Changing style (e.g., lighting, hair, or expression)
• This works because the model understands the structure of faces, not just pixels.

​

• Think of the latent space like a set of sliders controlling different features:
• Age slider → young ↔ old
• Smile slider → neutral ↔ happy
• Skin tone or style slider
• When you adjust these sliders:
• The model updates the latent vector
• A new face is generated instantly
• Small changes in the latent vector = small, smooth changes in the image

​

Core Probability Theory

• Generative modeling is closely connected to statistical modeling of probability distributions.

Five key terms

1. Sample space
2. Probability density function
3. Parametric modeling
4. Likelihood
5. Maximum likelihood estimation

4. Likelihood

• Likelihood measures how well a model explains the observed data.

• Sample Space → “What can exist?”
• PDF → “How likely is each one?”
• Parametric Model → “How do we summarize this?”
• Likelihood → “How good is our model?”

5. Maximum Likelihood Estimation (MLE)

Find the model that best explains the data we have seen

Imagine:

• Have many points on a map (your data)
• Draw a rectangle to cover them

The best rectangle (MLE) is:

• Not too big (wastes space)
• Not too small (misses points)
• Just right → covers the data well

Connecting to Neural Networks

In AI models:

• The “rectangle” = the model
• The “points” = your training data (faces)
• The goal = adjust model parameters (θ) so it fits the data best

This is called Maximum Likelihood Estimation (MLE)

Loss Function

• Neural networks are designed to minimize loss, so we take negative:

​

​

This is called: Negative Log-Likelihood (NLL)

​

Generative Model Taxonomy

Goal of Generative Models

• Aim to learn the data distribution pθ​(x)
• Enable generation of new data samples similar to training data
• Different models vary in how they represent and learn this distribution

​

Generative Model Taxonomy

Three Main Approaches

A. Tractable Density Models (Explicit + Exact)

• Explicitly model the probability density function
• Density is directly computable (tractable)
• Achieved by imposing architectural constraints

Key Ideas:

• Probability can be evaluated exactly
• Training uses maximum likelihood estimation (MLE)

Examples:

• Autoregressive Models
• Generate data sequentially (e.g., word-by-word, pixel-by-pixel)
• Normalizing Flow Models
• Use invertible transformations to map simple → complex distributions

​

B. Approximate Density Models (Explicit + Approximate)

• Explicitly model density but use an approximation
• True density is intractable, so optimization is indirect
• Key Ideas:
• Use latent variables or sampling techniques
• Optimize lower bounds or approximations

Examples:

• Variational Autoencoders (VAEs)
• Use latent space + variational inference
• Energy-Based Models
• Learn energy function, use Markov Chain sampling
• Diffusion Models
• Learn to denoise data step-by-step

​

C. Implicit Density Models

• Do not explicitly estimate probability density
• Focus only on generating realistic samples
• Key Ideas:
• No direct likelihood computation
• Learn through adversarial or stochastic processes
• Example:
• Generative Adversarial Networks (GANs)
• Generator vs Discriminator framework
• Produces highly realistic data without

density estimation

​

Characteristics of Generative Models

Generative models are machine learning models that learn the underlying patterns in data so they can create new, similar data. Their main characteristics include:

1. Data Generation�They can generate new content such as text, images, audio, video, or code that resembles the data they were trained on.
2. Learning Data Distribution�Instead of only classifying or predicting labels, generative models learn the probability distribution of the training data.
3. Creativity and Diversity�They can produce multiple different outputs for the same input, enabling creative applications like story writing, image generation, and music composition.

4. Handling Unlabeled Data�Many generative models can be trained using large amounts of unlabeled data, making them useful when labeled datasets are limited
5. Conditional Generation�They can generate outputs based on specific inputs or prompts, such as creating an image from a text description or answering a question.
6. High-Dimensional Data Modeling�They can capture complex relationships in high-dimensional data, allowing them to generate realistic and detailed outputs.
7. Generalization�Rather than memorizing the training data, well-trained generative models learn patterns that allow them to produce new, previously unseen examples.

8. Probabilistic Nature�Most generative models use probability to determine the next token, pixel, or data point, leading to varied outputs.
9. Wide Range of Applications�They are used in:
1. Text generation and chatbots
2. Image creation and editing
3. Speech synthesis
4. Music composition
5. Code generation
6. Drug discovery and scientific research
10. Computational Requirements�Training large generative models often requires significant computational resources, memory, and large datasets.

​

Advantages of Generative Models

1. Content Creation�Generative models can automatically create text, images, videos, music, and code, reducing manual effort.
2. Automation�They automate repetitive tasks such as writing reports, generating summaries, creating designs, and producing software code.
3. Personalization�They can generate customized recommendations, advertisements, learning materials, and responses based on user preferences.
4. Data Augmentation�They create synthetic data to expand training datasets, improving the performance of machine learning models when real data is limited.

5. Improved Creativity�They assist designers, artists, writers, and developers by generating new ideas, concepts, and prototypes.
6. Natural Human Interaction�They enable chatbots and virtual assistants to provide more natural, context-aware, and conversational responses.
7. Cost and Time Savings�They reduce the time and cost involved in creating content, designing products, and developing software.
8. Support for Research and Innovation�They help in scientific research, drug discovery, and engineering by generating new hypotheses, molecular structures, and design alternatives.
9. Versatility�A single generative model can perform multiple tasks such as translation, summarization, question answering, image generation, and code generation.
10. Decision Support�They assist professionals by generating insights, simulations, and possible solutions for complex problems.

Disadvantages of Generative Models

1. High Computational Cost�Training and running large generative models require powerful hardware, large amounts of memory, and significant energy.
2. Large Data Requirements�They often need massive datasets to produce accurate and high-quality outputs.
3. Risk of Inaccurate Outputs�Generative models can produce incorrect, misleading, or fabricated information (sometimes called "hallucinations").
4. Bias and Fairness Issues�If the training data contains biases, the model may generate biased or unfair results.

5. Privacy Concerns�Models trained on sensitive data may unintentionally reveal private or confidential information if not properly designed and managed.
6. Security Risks�They can be misused to create deepfakes, phishing content, fake news, or malicious code.
7. Lack of Explainability�It is often difficult to understand why a model generated a particular output, making debugging and accountability challenging.
8. Dependence on Training Data Quality�Poor-quality or outdated training data can lead to inaccurate or low-quality generated content.
9. Intellectual Property and Copyright Issues�Generated content may raise concerns about ownership, copyright infringement, and originality.
10. Maintenance and Updating�Models require regular updates, monitoring, and retraining to remain accurate, relevant, and secure.

Challenges in Generative Models

1. Data Quality and Availability�Generative models require large, diverse, and high-quality datasets. Poor or insufficient data can lead to inaccurate or biased outputs.
2. High Computational Requirements�Training and deploying large models demand powerful GPUs/TPUs, significant memory, and high energy consumption.
3. Bias and Fairness�Models may learn and reproduce biases present in the training data, leading to unfair or discriminatory outputs.
4. Hallucinations and Accuracy�Generative models can produce false, misleading, or fabricated information while presenting it confidently.

5. Privacy and Security�Protecting sensitive data during training and preventing misuse of generated content remain major challenges.
6. Explainability and Transparency�It is often difficult to understand how or why a model generates a particular response, making interpretation and debugging challenging.
7. Ethical and Legal Issues�Concerns include copyright infringement, intellectual property rights, misinformation, and the creation of deepfakes.
8. Evaluation of Output Quality�Measuring the correctness, creativity, and usefulness of generated content is difficult because quality is often subjective.
9. Scalability and Deployment�Deploying large models efficiently while maintaining speed, reliability, and low cost is a significant challenge.
10. Continuous Updating�Models need regular retraining or updating to keep pace with new information, changing user needs, and evolving technologies.

​

Applications of Generative Models

• Generative models are used in many fields to create new content, automate tasks, and solve complex problems. Some major applications are:
• Text Generation�Generate articles, emails, stories, reports, summaries, and chatbot responses.
• Image Generation and Editing�Create realistic images, artwork, logos, illustrations, and edit or enhance existing images.
• Code Generation�Assist developers by generating code, debugging programs, writing documentation, and suggesting code completions.
• Speech and Audio Generation�Produce natural-sounding speech, voice assistants, dubbing, music composition, and sound effects.

5. Video Generation�Create animations, video clips, visual effects, and educational or marketing videos.
6. Healthcare and Drug Discovery�Generate molecular structures, assist in drug discovery, analyze medical images, and support disease diagnosis.
7. Education�Create personalized learning materials, quizzes, practice questions, and tutoring systems.
8. Entertainment and Gaming�Generate game characters, environments, storylines, dialogue, music, and visual content.
9. Business and Marketing�Generate advertisements, product descriptions, marketing content, customer support responses, and personalized recommendations.
10. Scientific Research and Engineering�Support research by generating simulations, designing new materials, optimizing engineering solutions, and analyzing complex datasets.

​