1 of 33

Image and 3D Scene Editing

方双康

2024.06.15

2 of 33

Editing images from human instructions

(GPT-3) + (Stable Diffusion)

to generate a large dataset of image editing examples.

InstructPix2Pix, is trained on generated data.

3 of 33

InstructPex2Pix as a supervised learning problem

(a) We ﬁrst use a ﬁnetuned GPT-3 to generate instructions and edited captions.

(b) We then use StableDiffusion in combination with Prompt-to-Prompt to generate pairs of images from pairs of captions. We use this procedure to create a dataset

(c) of over 450,000 training examples.

(d) Finally, InstructPix2Pix diffusion model is trained on our generated data to edit images from instructions. At inference time, our model generalizes to edit real images from human-written instructions.

4 of 33

Sampled 700 input captions from the LAION-Aesthetics V2 6.5+ dataset and manually wrote instructions and output captions.

5 of 33

6 of 33

InstructPix2Pix

Use generated training data to train a conditional diffusion model that edits images from written instructions. base-model: Stable Diffusion

initialize the weights of our model with a pretrained Stable Diffusion checkpoint;
To support image conditioning, we add additional input channels to the ﬁrst convolutional layer,

7 of 33

8 of 33

Results

9 of 33

Results

10 of 33

11 of 33

12 of 33

13 of 33

14 of 33

First discover editing directions that reﬂect desired edits in the text embedding space

Propose cross-attention guidance to retain the cross-attention maps of the input image throughout the diffusion process

does not need additional training for these edits and can directly use the existing pre-trained text-to-image diffusion model.

changing foreground object (cat→ dog),

modifying object (adding glasses to a cat image)

changing style of the input (sketch → oil pastel), -for both real images and synthetic images.

15 of 33

16 of 33

Inverting Real Images

initial noise level

(64, 32, 16, 8)

17 of 33

Discovering Edit Directions

cat + c_edit → dog

18 of 33

Editing via Cross-Attention Guidance

19 of 33

Experiments

20 of 33

21 of 33

22 of 33

The background regions are identiﬁed using the object detector Detic
A lower score on Structure Dist means that the structure of the edited image is more similar to the input image.

23 of 33

24 of 33

25 of 33

26 of 33

Others: Fast Distillation

distilling a slow, text-to-image diffusion model into a fast, feed-forward model

We ﬁrst collect 15,000 pairs of input and edited images generated by our editing method proposed in the main paper.
Next, we automatically ﬁlter out pairs with low segmentation overlap or do not sufﬁciently increase the CLIP similarity with the target description
Given pairs of input and edited images, we train a CoModGAN [67] to perform image translation.