1 of 27

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

Narek Tumanyan∗ Michal Geyer∗ Shai Bagon Tali Dekel

Weizmann Institute of Science

*Indicates equal contribution.

​

​

2 of 27

Abstract

A new framework that takes text-to-image synthesis to the realm of image-to-image translation.

Guidance

3 of 27

Contents

1. Introduction

2. Related Work

3. Preliminary

4. Method

5. Results

6. Conclusion

4 of 27

Introduction

1.

​

5 of 27

Introduction

Provide new insights about internal spatial features formed.

​

Better balance between preserving and deviating.

An framework performing text-guided I2I translation without training or fine-tuning.

A possible approach for achieving control of the generated layout is to design text-to-image foundation models that explicitly incorporate additional guiding signals.

6 of 27

Related Work

2.

​

7 of 27

GAN

  • Encourage the output image to comply with the distribution of the target domain.
  • Require datasets of both source and target
  • require training for each translation task.

​

CLIP

  • Provides a joint image-text embedding space.
  • Various methods combine CLIP with a pre-trained image generator.
  • This work uses CLIP and semantic losses to guide a diffusion process.

personalizing

  • A pre-trained text-to-image diffusion model given a few user-provided images.
  • Aim to influence properties of the generated content.
  • DreamBooth / Textual Inversion

​

2. Related Work

8 of 27

Preliminary

3.

​

9 of 27

Diffusion Model

Operate directly in pixel space

Too slow and expensive!

10 of 27

Latent Diffusion Models

Pixel space

Latent space

11 of 27

LDM Method

Encoder / Decoder

Conditional denoising autoencoder + Attention block

domain specific encoder

CLIP

12 of 27

Attention block

Q

KV

Attention block

=

convolutional feature

feature from

domain specific encoder

feature from

domain specific encoder

13 of 27

Method

4.

​

14 of 27

Overview

Cross attention

Get feature

Generate

15 of 27

Extract & Inject

1

Extract

Extract spatial features from the decoder layers and self-attention.

2

Inject

inject the guidance features at certain layers.

16 of 27

Spatial features

The exact scene layout, the shape of the object and its fine-grained pose often significantly vary across generated images from the same prompt under different initial noise XT.

Semantic part segments can be estimated from spatial features in an unconditional diffusion model.

17 of 27

Self-attention

Self-attention modules compute the affinities Alt between the spatial features after linearly projecting them into queries and keys. These affinities have a tight connection to the established concept of self-similarly.

This motivates us to consider the attention matrices Alt to achieve fine-grained control.

18 of 27

U-Net layer

Cross attention

19 of 27

Algorithm

Control semantic feature

Control fine-grained

20 of 27

Results

5.

21 of 27

Wild-TI2I

22 of 27

ImageNet-R-TI2I

23 of 27

Comparisons

24 of 27

Ablation

25 of 27

Conclusion

6.

​

26 of 27

Contributions & Limitations

Limitations

Fails when there is no semantic association between the guidance content and the target text.

Contributions

A new framework for diverse text-guided I2I translation, founded on new insights about the internal representation of a pre-trained T2I diffusion model.

27 of 27

Thanks

For

Watching