Machine Learning, Fall 2021 |
Time/Location: Tue/Thu 11:00am-12:15pm in CAS B12 (section A1); 12:30pm-1:45pm in CAS 313 (section B2)
Course Number: CS 542
Instructors:
Prof. Kate Saenko (saenko@bu.edu) office hours: Mon/Th 3:00-4:00pm in MCS-200
Prof. Bryan Plummer (bplum@bu.edu) office hours: Mon/Th 9:00-10:00am in MCS-206
Teaching Fellows:
Hatice Kubra Eryilmaz (eryilmaz@bu.edu ) office hours: Wed 12:00-1:00pm in the EMA student lounge
Diala Lteif (dlteif@bu.edu) office hours: Wed 5:00-6:00pm in the EMA student lounge
Siqi Wang (siqiwang@bu.edu) office hours: Tue 7:30-8:30pm on zoom (info on Piazza)
Graders: Ximeng Sun, Dina Bashkirova, Stan (Sha) Lai, Peter (Zilu) Tang, Fangrui Huang, Nan Zhou
Labs:
Wed 10:00am on Zoom (info on Piazza) 8:00am (KCB 103); 9:05am (EPC 203); 10:10am (MCS B31); 11:15am (CAS 237); 12:20pm (CAS 235); 1:25pm (CAS 222); 3:25pm (CAS 208)
Piazza: https://piazza.com/bu/fall2021/cs542 Please use Piazza, not email, for all questions, including grading or missing class etc. (use the private message to Instructors for such requests)
Who should take this class? If you are a PhD, Master’s or upper-class undergrad interested in doing research in machine learning, this class is for you! It is a math- and programming-intensive class geared primarily towards graduate students in CS/EE. If you are not comfortable with the prerequisites, please DO NOT take this class and consider these alternatives instead. |
SCHEDULE*
Lectures will be recorded and posted later that day on piazza, slides will be available before class/lab at this link
Topic | Details | Assignments | |
Thu Sep 2 | Introduction | what is machine learning? types of learning; features; hypothesis; cost function; regression, linear hypothesis, SSD cost; course information | ps0 (self-test of prerequisites) on Piazza |
Tue Sep 7 | Preliminaries | review of expected mathematical skills for the course; Useful references on matrix calculus | |
Wed Sep 8 | LAB0: Probability and Math Review | ||
Thu Sep 9 | Supervised Learning I: Regression | gradient descent; normal equations; maximum likelihood; Reading: Bishop 1.2-1.2.4,3.1-3.1.1 | ps0 due (5pm) ps1 out |
Tue Sep 14 | Supervised Learning II: Classification | classification; sigmoid function; logistic regression. Reading: 4.3.1-4.3.2; 4.3.4 overview of logistic regression | |
Wed Sep 15 | LAB1: ps1 | ||
Thu Sep 16 | Supervised Learning III: Regularization | more logistic regression, nonlinear features, regularization; bias-variance Reading: Bishop 3.1, 3.2 | ps1 due (5pm) ps2 out |
Tue Sep 21 | Unsupervised Learning I: Clustering | clustering, k-means, Gaussian mixtures. Reading: Bishop 9.1-9.2 | ps0 self-grading due |
Wed Sep 22 | LAB2: ps2 | ||
Thu Sep 23 | Unsupervised Learning II: PCA | dimensionality reduction, PCA. Reading: Bishop 12.1 | ps2 due (5pm) ps3 out |
Tue Sep 28 | Neural Networks I: Feed-forward Nets | artificial neuron, MLP, sigmoid units; neuroscience inspiration; output vs hidden layers; linear vs nonlinear networks; feed-forward neural networks; Reading: Bishop Ch 5.1-5.3 | ps1 self-grading due |
Wed Sep 29 | LAB3: ps3 | ||
Thu Sep 30 | Neural Networks II: Learning | learning via gradient descent; computation graphs, backpropagation algorithm. Reading: Computing Neural Network Gradients | ps3 due (5pm) ps4 out |
Tue Oct 5 | Neural Networks III: Convolutional Nets | Convolutional networks. Reading: Bishop Ch 5.5; Goodfellow et al Ch. 9. (optional) | ps2 self-grading due |
Wed Oct 6 | LAB4: ps4 | ||
Thu Oct 7 | Midterm Review DROP DATE | ps4 due (5pm) | |
Tue Oct 12 | NO CLASS (Monday schedule) | ps3 self-grading due | |
Wed Oct 13 | LAB5: Midterm review | ||
Thu Oct 14 | Midterm (closed book, no electronics, you may bring a single 8“x11” sheet of paper with typed or handwritten notes on both sides) | covers everything up to and including Neural Networks IV; expect questions on material covered in lectures, problem sets, LABs and assigned reading | |
Tue Oct 19 | Probabilistic Models I: LDA | generalized linear models; generative vs discriminative models; linear discriminant analysis (LDA); Reading: Bishop Ch 4.2 | ps4 self-grading due |
Wed Oct 20 | LAB6: ps5 | ||
Thu Oct 21 | Probabilistic Models II: Bayesian Methods | priors over parameters; Bayesian linear regression; Reading: Bishop Ch 1.2, 2.3, 3.3 | ps5 out |
Tue Oct 26 | Support Vector Machines I | maximum margin methods; support vector machines; primal vs dual SVM formulation; Hinge loss vs. cross-entropy loss; Reading: Bishop Ch 7.1.1-7.1.2 | |
Wed Oct 27 | LAB7: ps5 | ||
Thu Oct 28 | Support Vector Machines II | non-separable data; slack variables;kernels; multiclass SVM; Reading: Bishop Ch 6.0-6.2, Ch 7.1.3 | Mon Nov 1 ps5 due (5pm) ps6 out |
Tue Nov 2 | Reinforcement Learning I | reinforcement learning; Markov Decision Process (MDP); policies, value functions, Q-learning | |
Wed Nov 3 | LAB8: ps6 | ||
Thu Nov 4 | Reinforcement Learning II | Q-learning cont’d; deep Q-learning (DQN) | Mon Nov 8 ps6 due (5pm) ps7 out |
Tue Nov 9 | Ensembles, Decision Trees, Random Forest | ensembling, bagging, decision tree, random forest; intro to the challenge problem | |
Wed Nov 10 | LAB9: ps7 | ||
Thu Nov 11 | Practical Advice for Applying ML I | Machine learning system design; feature engineering; cross-validation; feature pre-processing; learning with large datasets | Mon Nov 15 ps7 due (5pm) ps5 self-grading due challenge starts |
Tue Nov 16 | Practical Advice for Applying ML II | SGD and mini-batch GD; metrics (accuracy, F1 score, RMSE, etc.); dataset bias; data augmentation | |
Wed Nov 17 | LAB10: Challenge | ||
Thu Nov 18 | Unsupervised Learning III: Generative Adversarial Networks (GANs) | Implicit generative models; adversarial methods; Generative Adversarial Nets (GANs); Reading: Goodfellow et al. NIPS 2014 | ps6 self-grading due Mon Nov 22 initial challenge submission due (5pm) |
Tue Nov 23 | Applications I: On Device Machine Learning | network/feature quantization; parameter pruning; knowledge distillation; neural architecture search | |
Wed Nov 24 | NO CLASS- THANKSGIVING | ||
Thu Nov 25 | NO CLASS- THANKSGIVING | ||
Tue Nov 30 | Applications II: Semantic Image Segmentation | guest lecture by Ping Hu | ps7 self-grading due |
Wed Dec 1 | LAB11: Challenge | ||
Thu Dec 2 | Applications III: Reinforcement Learning | guest lecture by Sid Mysore | challenge deadline (5pm) |
Tue Dec 7 | Applications IV: Machine Learning Ethics | Ethics in ML; population bias in machine learning, fairness, transparency, accountability; de-biasing image captioning models | |
Wed Dec 8 | LAB12: Challenge Discussion | Final Review | ||
Thu Dec 9 | Final Review | ||
Wed Dec 15 6-8pm | Final exam (closed book, no electronics, you may bring a single 8“x11” sheet of paper with typed or handwritten notes on both sides) | expect questions on material covered in the entire course in lectures (guest lectures excluded), problem sets, LABs, and assigned reading |
*schedule is tentative and is subject to change.
SYLLABUS
This course is a rigorous introduction to modern machine learning concepts, techniques, and algorithms. Topics include regression, classification, unsupervised and supervised learning, kernels, support vector machines, feature selection, clustering, sequence models, Bayesian methods, and more. Weekly problem sets emphasize taking theory into practice, by gaining a thorough mathematical understanding of the machine learning methods, coding them from scratch, and applying them to data sets.
Course Prerequisites
This is a graduate course (open to upper-level undergraduates) and requires the following:
- Linear algebra (CAS CS 132 or MA 242 or equivalent)
- Multivariate Calculus (eg, CAS MA 225), including partial derivatives
- Probability/Statistics (CAS CS 237 or MA 381 or 581 or equivalent)
- Comfort with programming in Python (eg, CAS CS 111 and 112, or equivalent)
You should be very comfortable with programming in Python and be able to both write and understand code quickly. Note that starting in 2022, the above prerequisites will be replaced by Foundations of Data Science (CS 365). Intro to Optimization (CAS CS 507) is also highly recommended as a precursor to this course.
Textbooks
The required textbook for the course is
- Bishop, C. M. Pattern Recognition and Machine Learning.
Other recommended supplemental textbooks on general machine learning:
- Duda, R.O., Hart, P.E. and Stork, D.G. Pattern Classification. Wiley-Interscience. 2nd Edition. 2001.
- Theodoridis, S. and Koutroumbas, K. Pattern Recognition. Edition 4. Academic Press, 2008.
- Russell, S. and Norvig, N. Artificial Intelligence: A Modern Approach. Prentice Hall Series in Artificial Intelligence. 2003.
- Hastie, T., Tibshirani, R. and Friedman, J. The Elements of Statistical Learning. Springer. 2001.
- Koller, D. and Friedman, N. Probabilistic Graphical Models. MIT Press. 2009.
- Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning.
Recommended background reading on matrix calculus
- Reference reading on matrix calculus and linear algebra can be found here (new link)
- Matrix derivatives cheat sheet
- Useful tool http://www.matrixcalculus.org/
Alternative Machine Learning Courses
- Introduction to Machine Learning (ENG EC 414) is more suitable for undergraduates
- CAS CS 506 Computational Tools for Data Science covers machine learning techniques in a more hands-on way
- https://www.coursera.org/learn/machine-learning/ Andrew Ng’s basic intro to Machine Learning course on Coursera, can be taken as a precursor to this course
Deliverables/Graded Work
The main graded work for the course is the midterm, final, problem sets and final challenge. There will be eight self-graded weekly problem sets (ps0-ps7), each consisting of written and programming problems, which are meant to prepare students for the two exams. Note: there are no team projects in this course. The course grade consists of the following:
- Problem Sets 40% (lowest pset will be dropped)
- Challenge 10%
- Midterm 25%
- Final 25%
Challenge
The challenge is an individual programming project that gives each student a chance to apply their knowledge and develop an ML solution to a real-world problem. The goal is to take what you learned in the weekly problem sets to the next level and spend a few weeks implementing machine learning algorithms from scratch and applying them to a challenging dataset. More information about the challenge will be released later on in the term.
Piazza
We will be using Piazza for class Q&A and posting problem sets. The system is highly catered to getting you help fast and efficiently from classmates and instructors. Rather than emailing questions to the teaching staff, we ask you to please post all your questions on Piazza. For private questions like grading please select just the “Instructors” in the “Post to:” field of the Piazza note and select the appropriate folder.
Software/Compute Resources
Programming assignments will be developed in the Python programming language, using mostly iPython notebooks. No special computing resources are required besides a regular laptop.
Students may also have access to the SCC computing cluster, please request this from the instructors. See SCC Quick Start Guide for further information on getting started with the SCC.
Another resource you may want to use for some problem sets or the challenge is Google’s Colab, see https://colab.research.google.com/notebooks/welcome.ipynb
Help! My laptop broke!
If you find yourself in a situation where you need a back up machine, say, due to a broken laptop, the BU Common at the Mugar Library offers access to Microsoft Windows 10 virtual machines. The lab is co-located with the IT Help Center and the Print Center and hosts a broad selection of software including Python and Jupyter. Also note that the IT Help Center is an authorized repair center for most major laptop brands, and can often perform many common types of repairs within one day. They also offer temporary loaners for certain models of laptops while they are working on your machine.
COURSE POLICIES
Late Policy
Late problem sets will incur the following penalties
- 20% off per day, up to 2 days
The lowest of all the problem set grades will be dropped. Exceptions to these policies can only be made in cases of significant medical or family emergencies, and should be documented.
Copyright Notice
All course materials, including lecture slides/notes, videos/screenshots, homeworks, solutions, and solution code are copyrighted by Prof. Saenko and Prof. Plummer. All rights are reserved. Other than for personal use by registered students, the course materials or any portion thereof may not be reproduced or used in any manner whatsoever (including posting on a public forum or website) without the express written permission of the authors. Acting otherwise would go against the ethical code of conduct expected of students at Boston University and from a legal standpoint it may even constitute a violation of copyright law.
Academic Honesty Policy
The instructors take academic honesty very seriously. Cheating, plagiarism and other misconduct will be subject to grading penalties up to failing the course. Students enrolled in the course are responsible for familiarizing themselves with the detailed BU policy, available here. In particular, plagiarism is defined as follows, and applies to all written materials and software, including material found online. Collaboration on homework only in the form of discussion is allowed, but should be acknowledged and you should always come up with your own solution rather than copy others’ solutions (which is defined as plagiarism).
Plagiarism: Representing the work of another as one’s own. Plagiarism includes but is not limited to the following: copying the answers of another student on an examination, copying or restating the work or ideas of another person or persons in any oral or written work (printed or electronic) without citing the appropriate source, and collaborating with someone else in an academic endeavor without acknowledging his or her contribution. Plagiarism can consist of acts of commission-appropriating the words or ideas of another-or omission failing to acknowledge/document/credit the source or creator of words or ideas (see below for a detailed definition of plagiarism). It also includes colluding with someone else in an academic endeavor without acknowledging his or her contribution, using audio or video footage that comes from another source (including work done by another student) without permission and acknowledgement of that source.
Prohibited behaviors include:
- copying all or part of someone else's work, even if you subsequently modify it; this includes cases in which someone tells you what you should write for your solution
- viewing all or part of someone else's work
- showing all or part of your work to another student
- consulting solutions from past semesters, or those found online or in books
- posting your work or our solutions where others can view it (e.g., online).
Incidents of academic misconduct will be reported to the Academic Conduct Committee (ACC). The ACC may suspend/expel students found guilty of misconduct. At a minimum, students who engage in misconduct will have their final grade reduced by one letter grade (e.g., from a B to a C).
Religious Observance
Students are permitted to be absent from class, including classes involving examinations, labs, excursions, and other special events, for purposes of religious observance. In-class, take-home and lab assignments, and other work shall be made up in consultation with the student’s instructors. More details on BU’s religious observance policy are available here.
COVID-19
There is no remote component of this class, with the exception of a single lab session (created for students unable to attend in person due to visa issues). If you miss class due to COVID-19 or any other health related absence, please check the website for lecture recordings and other posted materials. Masks are required and face coverings must be worn over the mouth and nose at all times when in public spaces on campus, including classrooms. Students should be prepared to show proof that they are compliant with health attestations and testing in order to attend class. All students are expected to follow all university guidelines with respect to daily symptom checks, testing, social distancing, and mask wearing when they leave their dorm or home. For a detailed description of official BU policies regarding COVID, please visit: https://www.bu.edu/back2bu/.