CS294-192: Visual Scene Understanding

Spring 2022

“Бог создал мир из ничего.

Учись, художник, у него”  

– К. Бальмонт

             "The aim of computer vision is to overfit to our visual world"

-- Antonio Torralba (after his third beer)

Overview:

In this small, advanced computer vision class, we will explore the thesis that knowledge of, and appreciation for, the vision problem specifically (as opposed to treating it as a generic learning machine) should be helpful in the development of a truly general visual AI system, as well as provide insights about human vision and cognition.   Consequently, we will be reading an eclectic mix of papers, from the hot-off-the-arxiv to the classic works in computer vision, human vision, psychophysics, and cognitive science.   We will revisit the ideas of Gibson, Koenderink, et al. and see if they apply to the modern age.   The class aims to give students a broader, more historically-informed view of scene understanding, and will hopefully spark their inspiration and creativity in their own research.  Requirements: summaries of weekly readings, 1-2 presentations, active participation in class discussions, and a final project.  

Prerequisites: CS280 or equivalent (no exceptions!)

Course Requirements:

• Written summaries of assigned reading submitted before each class, plus participation in online discussion – at least one post before each class (25%)
• Prepare and deliver 1-2 high quality paper presentations during semester (25%)
• Final project (25%)  [not required for those taking S/U]
• Class participation (25%)

Schedule

Very Tentative Paper List

Week 1:  Theories of Vision: Bottom-Up vs. Top-Down:

• P. Cavanagh, Vision is Getting Easier Every Day, Perception, 1995
• Churchland, Ramanchandran, Sejnowski, A Critique of Pure Vision, 1994
• Koenderink, Sencience, 2019
• D. Hoffman, The interface theory of perception, Object Categorization: Computer and Human Vision Perspectives, 2009
• Vi Hart, 12 tone video
• Adelson & Bergen, The Plenoptic Function and Elements of Early Vision
• Gopnik, Making AI More Human, Scientific American, 2017
• P. Cavanagh, What’s up in Top-Down Processing, 1991
• Koenkerink, Vision, an Optical User Interface

Week 2: Texture / pre-attentive processing:

• Bela Julesz, A brief outline of the texton theory of human vision, 1984
• Walker & Malik, When is scene identification just texture recognition?, 2004
• Brendel & Bethge, Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet, ICLR’19
• Heeger & Bergen, Pyramid-based texture analysis/synthesis, 1995
• Gatys et al, Texture and Art with Deep Neural Networks, 2017
• Lin, RoyChowdhury, Subhransu Maji, Bilinear CNN Models for Fine-grained Visual Recognition, 2015
• Ruth Rosenholtz, Demystifying visual awareness: Peripheral encoding plus limited decision complexity resolve the paradox of rich visual experience and curious perceptual failures, 2020
• R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, ICLR’19

Week 3: Memory / Memorability:

• Brady, Konkle, Alvarez, Oliva, Visual long-term memory has a massive storage capacity for object details, 2008 
• Reflections of the Environment in Memory
• Isola, P., Xiao, J., Parikh, D., Torralba, A., Oliva, A. What makes a photograph memorable? IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
• Bainbridge et al, Drawings of real-world scenes during free recall reveal detailed object and spatial information in memory, Nature Communications, 2019
• Why do we remember what we remember?
• Zhang et al, Understanding deep learning requires rethinking generalization, ICLR 2017.
• Arpit et al,  A Closer Look at Memorization in Deep Networks, ICML 2018.
• Feldman & Zhang, What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation, NeurIPS 2020

Week 4: Images as attractors:

• Vondrick, Pirsiavash, Oliva, Torralba, Learning visual biases from human imagination, NIPS 2015
• Vincent et al, Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. JMLR 2011.
• Ulyanov et al, Deep Image Prior, 2017
• Ramsauer et al, Hopfield Networks is All You Need, 2020.
  (Nice blog post w/ visualizations: https://ml-jku.github.io/hopfield-layers/ )
• Yang Song. Score Function Models Tutorial.  (Blog 2021)
• Will Grathwohl, Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One, ICLR 2019.
• Ho et al, Denoising Diffusion Probabilistic Models. 2020.
• Saharia et al, Palette: Image-to-Image Diffusion Models, 2021
• Nichol et al, GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models, 2021
• Robin Rombach*, Andreas Blattmann*, Dominik Lorenz, Patrick Esser, Björn Ommer, High-Resolution Image Synthesis with Latent Diffusion Models, 2021
• Meng et al (2022), SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

Week 5: Grouping / Segmentation / Self-attention

• Wertheimer, M. (1923) Laws of Organization in Perceptual Forms
• Shi & Malik (2000), Normalized Cuts and Image Segmentation, PAMI 
• Weiss, Y. (1999) Segmentation using eigenvectors: a unifying view, ICCV
• Learning a Classification Model for Segmentation, Xiaofeng Ren and Jitendra Malik. in ICCV 2003
• Malisiewicz & Efros (2007), Improving Spatial Support for Objects via Multiple Segmentations
• Perona & Malik (1990), Scale-Space and Edge Detection Using Anisotropic

Diffusion

• Tomasi & Manduchi (1998), Bilateral Filtering for Gray and Color Images
• Krähenbühl & Koltun (2013), Parameter Learning and Convergent Inference for Dense Random Fields
• Vaswani et al., Attention is All You Need (2017)
• Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)
• M. Naseer et al, Intriguing properties of visual transformers, NeurIPS 2021.
  Slides: https://neurips.cc/media/neurips-2021/Slides/27802.pdf
• M. Caron et al, Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021.
• Tumanyan et al, Splicing ViT Features for Semantic Appearance Transfer, 2021.
• Shamir et al, Deep ViT Features as Dense Visual Descriptors, 2021.

Week 6: Scenes and Qualitative 3D

• Biederman, I. (1981) On the semantics of a glance at a scene.
• W. Warren, Does This Computational Theory Solve the Right Problem? Marr, Gibson, and the Goal of Vision, Perception 41(9), 2012
• Xiaofeng Ren, Fowlkes and Malik (2006), Figure/Ground Assignment in Natural Images
• Hoiem et al, Recovering Occlusion Boundaries from an Image, IJCV 2011
• Hoiem et al, Closing the Loop on Scene Interpretation, CVPR 2008
• Russell & Torralba (2009), LabelMe3D: Building a database of 3D scenes from user annotations
• Isola & Liu (2013), Scene Collaging: Analysis and Synthesis of Natural Images with Semantic Layers
• Gupta et al, Blocks World Revisited: Image Understanding using Qualitative Geometry and Mechanics, ECCV 2010
• David Lopez-Paz, Robert Nishihara, Soumith Chintalah, Bernhard Schölkopf and Léon Bottou: Discovering Causal Signals in Images, Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
• Oktay, Vondrick, Torralba, Counterfactual Image Networks, rejected from ICLR’18
• Hu, Ravi, Berg, Pathak, Worldsheet: Wrapping the World in a 3D Sheet

for View Synthesis from a Single Image, 2021

Week 7: Self-Supervision by space / time / geometry / physics

• De Sa, Learning classification with unlabeled data, NIPS 1993
• Isola et al, Learning Visual Groups From Co-occurrences in Space and Time, 2016
• Doersch et al, Unsupervised Visual Representation Learning by Context Prediction, 2015
• Pinto & Gupta (2016), Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
• Zhang et al (2017), Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• Noroozi et al (2017), Representation Learning by Learning to Count
• Wiskott & Sejnowski. Slow Feature Analysis, 2002
• T. Malisiewicz, A. Gupta, A. Efros, Ensemble of Exemplar-SVMs for Object Detection and Beyond,, ICCV, 2011
• Dosovitskiy et al, Exemplar-CNN: Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks, 2014
• Bojanowski & Joulin, Unsupervised Learning by Predicting Noise, ICML 2017
• Jabri et al. Space-Time Correspondence as a Contrastive Random Walk. NeurIPS 2020
• Pathak et al (2017), Curiosity-driven Exploration by Self-supervised Prediction
• Zhou et al (2017), Unsupervised Learning of Depth and Ego-Motion from Video
• He et al (2021), Masked Autoencoder are Scalable Vision Learners

Week 8: Emergence of Structure and Disentanglement

• Barrow and J.M. Tenenbaum, Recovering Intrinsic Scene Characteristics from Images, 1978
• Tenenbaum & Freeman, Separating Style and Content with Bilinear Models, 2000
• Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, Deep Convolutional Inverse Graphics Network, 2015
• Bolei Zhou et al, Object Detectors Emerge in Deep Scene CNNs, ICLR 2015
• D Bau, JY Zhu, H Strobelt, A Lapedriza, B Zhou, A Torralba. Understanding the Role of Individual Units in a Deep Neural Network. Proceedings of the National Academy of Sciences. 2020.
• Ramesh et al (2021), DALL-E: Zero-Shot Text-to-Image Generation
• Li et al, Language-driven Semantic Segmentation (2022)
• Ling et al, EditGAN: High-Precision Semantic Image Editing, NeurIPS 2021
• Locatello et al, Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ICML 2019 (Best Paper Award).

Week 9: Categorization and Hierarchies

• Rosch http://commonweb.unifr.ch/artsdean/pub/gestens/f/as/files/4610/9778_083247.pdf
• Murphy, Big Book of Concepts, Ch. 3 
• Dawkins, the Salamander’s Tale
• Witgenstein, https://en.wikipedia.org/wiki/Family_resemblance
• Shirky: Ontology is Overrated
• Brighten Godfrey, graphing English
• Malisiewicz & Efros, Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships, 2009
• Srivastava et al, Constrained Semi-Supervised Learning using Attributes and Comparative Attributes, 2013
• X Chen, A Shrivastava, A Gupta,  Neil: Extracting visual knowledge from web data, 2013
• Affordances:

• Gupta et al, From 3D Scene Geometry to Human Workspace, CVPR 2011
• Delaitre et al, Scene Semantics from Long-term Observation of People, ECCV 2012
• Li et al, Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments, 2019

• Few-shot  / Long tail /  Open world
• Soft labels, continuous labels

• Zipf’s distribution & hierarchies

SKIPPING Week 10: Correspondence (explicit vs. implicit), Context and Analogies

• Ullman 79 “The Correspondence Problem”  (temporal correspondence)  ttps://direct.mit.edu/books/book/3900/chapter-abstract/163713/The-Correspondence-Problem (available online through library)  
  (from https://mitpress.mit.edu/books/interpretation-visual-motion)
• Erik Learned-Miller, (2005) Data driven image models through continuous joint alignment.  PAMI, 2006
• Zhou et al (2015), FlowWeb: Joint Image Set Alignment by Weaving Consistent, Pixel-wise Correspondences
• Peebles et al (2021), GAN-Supervised Dense Visual Alignment
• Malisiewicz & Efros, Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships (2009)
• Choi, Torralba, Willsky, Context Models and Out-of-context Objects, 2012
• …
• AlphaFold  (blog post)

Week 11: Datasets, bias, overfitting

• Torralba and Efros, Unbiased Look at Dataset Bias, 2011
• Recht et al., Do ImageNet Classifiers Generalize to ImageNet?
• Engstrom et al., Identifying Statistical Bias in Dataset Replication
• Beyer et al, Are we done with ImageNet?

• L. Schott, J. von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Locatello, et al., Visual Representation Learning Does Not Generalize Strongly Within the Same Domain, 2021

Week 12: Continuous / lLifelong  Learning / Test-time Training:

• Test-time training/adaptation

• kNN-SVM
• Srivastava et al, Data-driven Visual Similarity for Cross-domain Image Matching, 2011
• Shocher, Cohen, Irani, “"Zero Shot" Super-Resolution using Deep Internal Learning”, 2018
• Mullapudi, Chen, Zhang, Ramanan, Fatahalian, Online Model Distillation for Efficient Video Inference, 2018
• Sun et al, Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, 2019

• Generalization / distillation
• Fini et al, Self-Supervised Models are Continual Learners, 2022
• Kumar et al, "Understanding Self-Training for Gradual Domain Adaptation"
• Toward a Universal Law of Generalization for Psychological Science
• Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu, Using Fast Weights to Attend to the Recent Past, 2016

• Also see: Hinton, G. E. and Plaut, D. C. (1987). Using fast weights to deblur old memories. Proceedings of the 9th Annual Conference of the Cognitive Science Society (pp. 177-186). Hillsdale, NJ: Lawrence Erlbaum Associates.

Week 13:  Active Vision, Robotics, Evolution

• Passive vs. Active Vision

• https://marom.net.technion.ac.il/files/2016/07/Held-1963.pdf
• https://link.springer.com/article/10.3758/BF03337301

• Bretenberg’s Vehicles
• Carl Sims
• Ackley & Litmann, “Interactions Between Learning and Evolution” https://www.gwern.net/docs/reinforcement-learning/1992-ackley.pdf
• Open-ended learning https://www.oreilly.com/radar/open-endedness-the-last-grand-challenge-youve-never-heard-of/ (blog post by Ken Stanley, Joel Lehman, others)

Other misc topics:

• Passive vs. Active Vision

• https://marom.net.technion.ac.il/files/2016/07/Held-1963.pdf
• https://link.springer.com/article/10.3758/BF03337301

• Analogies

• Vinigrad
• Word2vec
• Relational version of correspondence
• Discovering attributes

• Evolution:

• Bretenberg’s Vehicles
• Carl Simms
• Ackley & Littman, Interactions between learning and evolution. https://www.gwern.net/docs/reinforcement-learning/1992-ackley.pdf
• Open-ended learning https://www.oreilly.com/radar/open-endedness-the-last-grand-challenge-youve-never-heard-of/ (blog post by Ken Stanley, Joel Lehman, others)

• Comp. Neuro:

• Nice blog post with overview of recent works: https://xcorr.net/2021/12/31/2021-in-review-unsupervised-brain-models/

Similar Courses:

• Alyosha Efros (CMU, 2009):  http://www.cs.cmu.edu/~efros/courses/LBMV09/
• Deepak Pathak (current generation of my CMU class): https://visual-learning.cs.cmu.edu/
• Lana Lazebnik: http://slazebni.cs.illinois.edu/spring20/
• David Fouhey: https://web.eecs.umich.edu/~fouhey/teaching/EECS542_F21/schedule.html
• Andrew Owens: https://web.eecs.umich.edu/~ahowens/eecs542/w22/#schedule
• Phil Isola: https://phillipi.github.io/6.882/2020/schedule.html
• Derek Hoiem: https://courses.engr.illinois.edu/cs598dwh/fa2021/