Video segmentation…

Is optical flow algorithm doing all the work?

… in a fully self supervised manner

Outline

1. Method
1. The slot attention module
2. Optical flow
3. Applying slot attention to optical flow
4. Losses
5. Full pipeline

​

• Results
• Quantitative results
• Qualitative results
• Ablations

Method

Slot attention module[1]

[1] Object-Centric Learning with Slot Attention, F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, T. Kipf

Slot attention module[1]

Maps from a set of N input features to a set of K slots

• Soft clustering assignment

​

• Guarantees:
• Permutation invariance with respect to the input features
• Permutation equivariance with respect to the slots

​

• Slots randomly sampled from a Gaussian distribution
• K fixed during training
• Can generalize to higher K during testing

​

• Iterative process (3-5 iterations)

​

[1] Object-Centric Learning with Slot Attention, F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, T. Kipf

Hard clustering

Soft clustering

Slot attention iteration

​

• Input appearance tokens
• Input slot tokens
• Output slot tokens

1 iteration of slot attention

Maps from a set of N input features to a set of K slots

• Input appearance tokens
• Input slot tokens
• Output slot tokens

Slot attention iteration

​

​

​

​

​

​

​

“Cross Attention”

GRU

MLP

̰

Slot update using “Cross Attention”

• Input appearance tokens
• Input slot tokens
• Output slot update

​

• Normalization over all slots

​

• Weighted mean instead of weighted sum

Slot update using “Cross Attention”

• Input appearance tokens
• Input slot tokens
• Output slot update

Recurrent Unit for the output slot

Reconstruction Loss

Decoder:�From each slot/layer i, it reconstructs:

• the RGB image of the slot
• the slot mask

�Final reconstruction image

Broadcasting:

• Inputs the slots
• Outputs the broadcasted slots

Slot attention module results, multi iterations

[1] Object-Centric Learning with Slot Attention, F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, T. Kipf

Challenges

Challenges:

• Slot attention works on synthetic data where objects are primitive shapes with simple textures
• More challenging with real life texture, specifically in the background

​

Idea:

• Don’t use RGB images but optical flow images !
• Objects in images may not be textureless, but their motions typically are.

New Pipeline

• Optical flow image as input
• Only 2 slots/queries/layers, foreground and background
• The slots are not Gaussian-initialized but learnable query vectors

Reconstruction

From each slot/layer i, it reconstructs:

• the layer reconstruction
• the layer mask

Additional losses

Encourage the masks to be binary

​

​

​

Encourage temporal consistency

Final loss

Full pipeline

Results

Ablation on the loss

Ablation on the optical flow

​

​

​

​

Results

Results on MoCA

Limitations

• Only 2 layers, foreground and background
• Can it properly generalize to more moving object, or is it able to cluster well camera motion and object motion
• Follow up paper on slot attention applied to videos, has K > 2

​

• Rely only on motion
• What if optical flow is noisy or low quality?
• Use RGB signal
• Jointly optimize flow and segmentation

​

• Does it properly learn motion centric representations for other applications?

​

Other ablations

[2] Group normalization, Y. Wu, K. He