1 of 16

Enhancing the diffusion model to understand simple-prompt

team4

Hanrui Huang

Cheng Chen

21-02-2024

2 of 16

Timeline

​

  1. Introduction
  2. Motivation
  3. Related work
  4. Task definition
  5. Dataset collection
  6. Methods
  7. Baseline and evaluation

2

3 of 16

Introduction

​

3

image：https://arxiv.org/pdf/2112.10752.pdf High-Resolution Image Synthesis with Latent Diffusion Models

​

Our project is about “Text-to-Image”

4 of 16

Introduction: Diffusion Model

​

4

There are many types of generating model.

​

-Generative Adversarial Networks (GANs)

-Variational Autoencoders (VAEs)

-Transformer-based Models(like DALL*E)

…

​

And

​

-Diffusion models(Autoregressive Models)

​

​

​

5 of 16

Introduction: Stable Diffusion

​

5

A latent text-to-image diffusion model

​

Input : text prompts

Output: image

​

Advantage:

More stable than GANs based (Training is difficult)

High quality images

​

Comparison between Latent diffusion Models and others

6 of 16

Introduction: Stable Diffusion

​

6

Structure

7 of 16

Motivation: Semantic Understanding

​

7

The models struggle with semantic understanding when given concise narrative prompts.

Counting

​

​

​

Color

8 of 16

Motivation:Common-sense reasoning

​

8

Not friendly for daily usage!

9 of 16

Motivation:Common-sense reasoning

​

9

LLM has the knowledge！

​

10 of 16

Motivation:Semantic Understanding and Reasoning

​

10

“If we can transfer the semantic understanding and reasoning abilities of LLMs to pre-trained diffusion models so that diffusion models can produce semantically correct and high-quality images even with simple narrative prompts.”    

11 of 16

Related works

​

11

  1. Text-to-image Diffusion
  2. Large Language Models
  3. Semantic Understanding and Reasoning Dataset
  4. Re-implementation of paper: SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models1

1.SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models: https://arxiv.org/pdf/2305.05189.pdf

12 of 16

Task definition

Our objective is to fine-tune the pre-trained diffusion model by distilling knowledge from large language models (LLMs), thereby enabling the diffusion model to better understand and respond to simple text prompts.

​

12

13 of 16

Dataset collection

13

We decided to build a small dataset for training according to the methods described in reference paper:

​

  1. Collect image-text pairs from the civitai website.
  2. Use BLIP1 to generate simple prompts based on the images.
  3. Ensure semantic consistency between the simple prompts and their corresponding images. （Using CLIP2 model for semantic cleaning）

​

​

​

​

1.BLIP: a pre-trained model designed for vision-language intersection tasks, focusing on generating descriptive text from images to improve the semantic match between images and texts.

​

2.CLIP: a multimodal pre-trained model that can understand and evaluate the semantic association between image content and textual descriptions.

​

​

14 of 16

Method

14

Objective 1: To ensure that the features of simple prompts are enhanced with LLM (Large Language Model) gains.(knowledge distillation )

Objective 2: To make the features of simple prompts resemble those of complex prompts.

The QKV-Attention in the adapter allows the model to notice important semantic relationships in the original features while generating new features.

15 of 16

Baseline and evaluation

  • Baseline: The original Stable Diffusion model
  • evaluation metric:
    • semantic quality evaluation : CLIP Score1
    • image quality evaluation: BRISQUE2, CLIP-IQA3, MUSIQ4

​

15

  1. CLIP Score: A measure that uses the CLIP model to evaluate the semantic similarity between generated images and their textual descriptions.
  2. BRISQUE: A no-reference image quality assessment metric that predicts the naturalness of images without requiring a reference image.
  3. CLIP-IQA: An image quality assessment method leveraging the CLIP model to estimate perceptual image quality by comparing images with textual descriptions.
  4. MUSIQ: A multi-scale deep learning model for no-reference image quality assessment that considers various aspects of human perception to evaluate image quality.

​

​

​

​

​

16 of 16

Next step

Following our plan outlined in the proposal, we will progressively complete the following milestones in the remaining time:

​

  1. Build the dataset
  2. Train the model
  3. Evaluation
  4. Analysis and Reflection

16