Listed as DS-GA 3001: Special Topics in Data Science
Fall 2021
60 Fifth Ave C10
Weds 11–1:45
Material progress in many areas of applied machine learning has recently been driven more by the effects of scale than by the innovations of specialist AI researchers. In NLP, this is showcased well by the transition from GPT-2 to GPT-3: Even though the two models are nearly identical, a simple increase in parameter count and training time led to dramatic quantitative and qualitative improvements in language and reasoning ability. Should we expect these dynamics to hold for the foreseeable future? If so, what kinds of AI-related research are still likely to produce worthwhile and lasting successes? This seminar aims to be a forum for students looking to better understand these issues and to discuss how they should influence one’s choice of research agenda.
This seminar will focus on topics in the academic literature surrounding scaling laws—attempts to forecast future progress driven by model or dataset scale—and the bitter lesson—the observation that attempts to encode expert knowledge into scalable machine learning models tend to fail. We will use GPT-2 and GPT-3 as a recurring case study, but we will discuss relevant topics from other areas of NLP and other areas of applied machine learning.
The seminar will be centered around student-driven discussions of clusters of related papers. In addition, each student will also be asked to write a survey paper—alone or with a teammate—synthesizing research on a topic related to the course.
Enrollment is limited to PhD students who have substantial experience and a publication record working on an applied ML topic. This includes NLP, computer vision, robotics, or general empirical machine learning research. MS or undergraduate students with equivalent experience may be admitted with permission of instructor if capacity allows.
If you meet the prerequisites but are unable to formally enroll, you are welcome to participate as an auditor. However, auditing in the normal sense is not allowed: Anyone who attends meetings after the drop deadline must be committed to staying with the seminar for the full term and submitting a paper.
Each week, a small group of students will be assigned to briefly present the readings—with no more than fifteen minutes of planned material per reading—and lead discussion. These will generally include one required reading and several additional papers (or demos or blog posts) expanding on a related theme.
The only written work requirement will be a survey paper. This paper should be no more than ten pages (in any common ML/AI paper format), and should synthesize work on some narrow topic related to the course. The paper may include original empirical work, but this should not be the primary focus. Papers may be submitted individually or in teams of up to three. Multi-person teams should include a brief collaboration statement.
Formula:
Unexcused late submissions are subject to a 10 percentage point grade deduction.
Office hours by appointment.
Draft sketch of topics and key readings by week. Full agenda to be shared privately. Each session will include a 15m snack/coffee break.
Work you submit should be your own. Please consult the GSAS academic integrity policy for more information: GSAS Statement on Academic Integrity
Penalties for violations of academic integrity may include failure of the course, suspension from the University, or even expulsion.
As a nonsectarian, inclusive institution, NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations. The policy and principles to be followed by students and faculty may be found here: University Calendar Policy on Religious Holidays - NYC
Academic accommodations are available to any student with a chronic, psychological, visual, mobility, learning disability, or who is deaf or hard of hearing. Students should please register with the Moses Center for Students with Disabilities at 212-998-4980.
NYU's Henry and Lucy Moses Center for Students with Disabilities
726 Broadway, 2nd Floor
New York, NY 10003-6675
Telephone: 212-998-4980
Voice/TTY Fax: 212-995-4114
Web site: http://www.nyu.edu/csd