1 of 20

STATS / DATA SCI 315

Last Lecture

Course Conclusion

​

2 of 20

Deep Learning Topics

Not Covered in 315

3 of 20

Q: Can you discuss Transformers at a high level?

​

​

  • Attention is a DL mechanism that allows an NN to “pay attention to” the relevant part of the input (like a long sentence)
  • It was introduced in this paper (70,000+ citations!!!) in 2017 and it led to a tsunami of research on attention
  • Transformers are NNs built out of attention mechanisms

4 of 20

Attention has taken over NLP

  • RNNs like LSTM have almost been completely taken over by attention
  • You have probably heard about LLaMa (Meta), LaMDA, PaLM (Google), Chinchilla (Deepmind) and GPT-4 (Open AI) in the news
  • These all use transformers in a fundamental way
  • Attention isn’t just an NLP technique though
    • Attention blocks also figure in AlphaFold (Deepmind) that was called a “a stunning advance on the protein folding problem” by Venki Ramakrishnan, a Nobel laureate
    • There are also something called Graph Attention Networks (GATs)

5 of 20

6 of 20

7 of 20

Graph Neural Networks (GNNs)

  • Graphs are a universal language to describe relationships:
    • Social network
    • World Wide Web as a graph
    • Molecules as graphs
  • Deep learning has also revolutionized machine learning with graphs using a family of architectures collectively called Graph Neural Networks (GNNs)
  • One way to view them is as a generation of CNNs to graph structured data
  • Google team used GNNs to train an NN to predict odors from molecular structure

8 of 20

Deep Generative Models aka Generative AI

  • NNs can be used for generating data that “looks like” your training data
  • Generative models is a course by itself!
  • Several big ideas here:
    • Generative Adversarial Networks (GANs)
    • Variational Auto-Encoders (VAEs)
    • Normalizing Flows
    • Stable Diffusions
  • Powerful but controversial consequence: Deepfakes
  • Of course there are use cases for entirely reasonable aims:

9 of 20

Deep Reinforcement Learning

  • RL is a field that goes back to 1960s but took off in the 1980s
  • Approach to building intelligent agents via reward driven trial-and-error
  • Deep RL took off with AlphaGo, the first program that beat a professional human player in the board game of Go
    • There’s a documentary on AlphaGo
  • Deep RL = combination of RL and deep learning
  • Deep RL has been applied to autonomous driving, navigating high-altitude balloons, and for controlling hot plasma to enable nuclear fusion

10 of 20

Q & A

11 of 20

Q: How do I learn more?

  • If I want to continue learning deep learning/machine learning, which courses do you recommend?
  • Do you have course recommendations from here?

12 of 20

Regarding advanced prep for serious ML research

  • There are two aspects of prep: becoming technically competent and building real-world project experience
  • Academic research might tilt a bit more towards former whereas industry might tilt towards the latter but good ML researchers typically have both
  • For someone who is committed to stay with ML in the long run, a long prep is both required and can be a fun challenge
  • There will always be demand for people who truly understand ML and AI

13 of 20

Books

  • There are three technical areas to master: stats, probability, optimization
  • Michael Jordan has a list of some 12 books here: https://news.ycombinator.com/item?id=1055042

1.) Casella, G. and Berger, R.L. (2001). "Statistical Inference" Duxbury Press.

2.) Ferguson, T. (1996). "A Course in Large Sample Theory" Chapman & Hall/CRC.

3.) Lehmann, E. (2004). "Elements of Large-Sample Theory" Springer.

4.) Gelman, A. et al. (2003). "Bayesian Data Analysis" Chapman & Hall/CRC.

5.) Robert, C. and Casella, G. (2005). "Monte Carlo Statistical Methods" Springer.

6.) Grimmett, G. and Stirzaker, D. (2001). "Probability and Random Processes" Oxford.

7.) Pollard, D. (2001). "A User's Guide to Measure Theoretic Probability" Cambridge. The standard advanced textbook is Durrett, R. (2005). "Probability: Theory and Examples" Duxbury.

8.) Bertsimas, D. and Tsitsiklis, J. (1997). "Introduction to Linear Optimization" Athena.

9.) Boyd, S. and Vandenberghe, L. (2004). "Convex Optimization" Cambridge.

10.) Golub, G., and Van Loan, C. (1996). "Matrix Computations" Johns Hopkins.

11.) Cover, T. and Thomas, J. "Elements of Information Theory" Wiley.

12.) Kreyszig, E. (1989). "Introductory Functional Analysis with Applications" Wiley.

14 of 20

Books

  • UC Berkeley’s PhD Prelim Exam in ML makes these recommendations:
    • P. J. Bickel and K. A. Doksum. Mathematical Statistics: Basic ideas and selected topics.
    • B. Efron and T. Hastie. Computer Age Statistical Inference.
    • R. W. Keener. Theoretical Statistics: Topics for a core class.
    • M. J. Wainwright. High-dimensional statistics: A non-asymptotic view.
    • G.R. Grimmett and D.R. Stirzaker. Probability and Random Processes.
    • R. Vershynin. High-dimensional probability.
    • B. Recht and S. J. Wright. Optimization for Data Analysis.
    • D.P. Bertsekas. Nonlinear programming.
    • S. Boyd and L. Vandenberghe. Convex optimization.

15 of 20

Courses at UM

  • STATS 415: Data Mining and Statistical Learning
    • Broader coverage of topics than 315, not focused on DL, uses R
  • EECS 445: Intro to ML
    • Another general intro to ML not specifically to DL, uses Python
  • BTW, “Data Mining” vs “Statistical Learning” vs “Machine Learning”: what’s the difference?
    • More cultural than intellectual
    • DM comes from database tradition, SL comes from stats, ML comes from AI
    • Boundaries are very blurry
  • Vision, NLP, RL, Generative Models, Robotics courses are also natural next steps

16 of 20

Study of intelligence and learning is a broad one!

  • Mathematics: Smale’s 18th problem
  • Cognitive development: Alison Gopnik says:
    • “Cognitive develop­ment and machine learning belong to the same natural category, along with the philosophy of science, epistemology, and vision science, even if they live in oppo­site corners of the campus”

17 of 20

Q: What is the future of AI going to be like?�How do I build my career?

​

​

  • Do you have suggestions on personal projects that will off of and demonstrate the skills that we developed in this class?
  • In your opinion, is data science/machine learning worth the hype and do you think people with the relevant skills will become significantly more in demand?
  • What would you say are the most important skills to have (coding, math, etc)?
  • How we as students can make ourselves competitive for careers in AI?
  • Could you give an idea of what a day in the life of someone with a data science/ML role would look like?
  • What are your thoughts on the future of generative AI? Do you believe it is here to stay?

18 of 20

Hands-on projects

  • Personal projects are a great way to learn and to showcase your work (website/github)
  • Kaggle.com is a great source of data and projects
  • Take an existing project and tweak it!
  • Find a research group that admits undergraduate researchers
  • For vision, there are two great books with lots of stuff online:
  • For NLP:
  • General applied ML book: Hands-on ML w/ scikit-learn, keras, TF
    • reimplementing classic models (LeNet, AlexNet, ResNet) is a good exercise

19 of 20

Future of AI, DS and related fields

  • “Prediction is very difficult, especially if it’s about the future” – Niels Bohr
  • But I can see a few trends that won’t go away soon
    • Hardware trends will continue to impact AI (recall the role GPUs in DL; pay attention to quantum computing!)
    • All our big problems (climate change, inequality, international peace, energy, sustainability, migration) require interdisciplinary perspective (so collaborate!)
    • Training large models on huge datasets, “emergent” properties
    • Human + AI collaboration, explainability, interpretability (esp. in high stakes situation)
    • Ethical issues, societal impact, energy footprint, privacy, copyright issues

20 of 20

Preparation for the future: general advice

“The future cannot be predicted, but futures can be invented” – Dennis Gabor, Nobel Prize in Physics, 1971

  • Don’t worry too much about predicting the future, think how you will impact it!
  • We already talked about acquiring the right technical background
  • Work on impactful real-world projects
  • Read widely! Talk to people who think very differently from you
  • Know yourself! Find a problem or problem area you truly care about. Read something on intrinsic motivation, e.g.,
    • “Drive: The Surprising Truth About What Motivates Us" by Daniel H. Pink

“There are two kinds of people: Those who think they can, and those who think they can't, and they're both right.” – Henry Ford