Machine Learning for Language Understanding

LING-UA 52 - DS-UA 203

Sam Bowman

Assistant Professor of Linguistics, Data Science, and Computer Science

Spring 2022

Building computational models that can understand human language has long been a goal for researchers in computational linguistics and in the area of artificial intelligence called natural language processing. Many of the biggest successes in research toward this goal have relied on machine learning: a family of methods that allow computers to learn to reproduce some human behavior by example, rather than by explicit programming. This course covers widely-used machine learning methods for language understanding—with a special focus on machine learning methods based on artificial neural networks—and culminates in a substantial final project in which students write an original research paper in AI or computational linguistics.

If you take this class, you’ll be exposed only to a fraction of the many approaches that researchers have used to teach language to computers. However, you’ll get training and practice with all the research skills that you’ll need to explore the field further on your own. This includes not only the skills to design and build computational models, but also to design responsible experiments to test those models, to write and present your results, and to read and evaluate results from the scientific literature.

Basic Logistics

Format

Lecture: Available on Brightspace, released Monday afternoon/evening

Lab Session:  Tuesdays 3:30–4:45p, BOBS LL138

Optional Live Discussion: Wednesdays 2–3p, GCASL 383

The main lecture content will take the form of a prerecorded video each week, in the flipped classroom style. The normal class time will be used for semi-structured discussions of course content and readings. The lab session, led by the TAs, will introduce concrete skills and tools that you'll use in your assignments and project.

All lab sessions and live discussion sessions will be recorded, but to participate actively, you’ll have to attend in person. Participating in live discussion within these sessions is an easy way to earn the required participation credit, but you can also earn this credit through online discussion, and you are not required to attend any specific live sessions.

Prerequisites

• At least one class with a substantial Python programming component (i.e., CSCI-UA 2 or an advanced CSCI-UA or other programming course)
• Basic experience with calculus (i.e., MATH-UA 121, 122, 123, or credit for testing out of one or more of these courses) and probability theory (e.g. MATH-UA 233/234/235/238, ECON-UA 18, or AP Statistics), or permission of the instructor.

Note on alternative courses: This course covers largely the same material as the graduate-level DS-GA 1012. Other than that, it should overlap only very slightly with other NYU courses on computational linguistics and natural language processing. Students interested in a survey of methods in language technology should consider enrolling in both this course and the topics course Natural Language Processing, which is occasionally offered through Computer Science. These two courses may be taken in either order.

Registration

You may register through either course code (LING-UA or DS-UA)—both course codes correspond to exactly the same course. If you see open space in either course code but you’re being blocked from registering for any reason, contact the relevant department directly. This course is open to students in any NYU school or program. Only department administrators can edit student enrollment, and you do not need instructor permission if you’re confident that you meet the prerequisites: For registration questions, please reach out to departments directly rather than to me.

Auditing

Auditors from any school or department are welcome, subject to the prerequisites above. Auditors may not submit assignments, and may only participate in project teams with permission of the instructor. If you’d like to audit, make a private post on Campuswire during the first week of class introducing yourself as an auditor and giving your email address. The TAs can then add you to Brightspace.

Course Site

Forum (Campuswire)

• Use for: discussion, assignments/grading questions, private messages with staff.
• Use your real name and your NYU NetID email address (example: mn1234@nyu.edu, not my.name@nyu.edu) for participation credit.
• Use code 5251 here to enroll by the first day of class: https://campuswire.com/p/GF14E42F6 
  

New NYU Classes (brightspace.nyu.edu)

• Use for: assignment submission, grades, lecture videos.
• Enrollment is automatic for enrolled students; auditors should request access on Campuswire.

Instructional Staff and Office Hours

Note: Office hour times may change some weeks. Watch the forum for announcements.

Sam Bowman (Instructor)

• Ask me about projects, course content, and administrative issues. Please ask only the TAs about homework and homework grades.
• Office hours by appointment. Sign up here in a short appointment slot labeled Office Hours, and come in person use the call-in link attached to the appointment slot. Please don’t reserve more than one slot at a time.
  

Eugene Choi (TA)

• Office hours: Fridays 12pm - 1pm, https://nyu.zoom.us/j/6202594617 
• Ask me about projects, homework, and lab content.
  

Arka Talukdar (TA)

• Office hours: Monday 5:00pm - 6:00pm, zoom link
• Ask me about projects, homework, and lab content.

Requirements

• Project (50%)

• The main requirement of this course is to produce an original research project on machine learning for language understanding. As with an actual research paper in this field, the length limit will be very short (4 pages), but the expectations for clarity and thoroughness will be high. You should collaborate in groups of two or three. The project will be graded in five parts (percentages are out of the total course grade):

• Mini-proposals: 5%
• Proposal: 10%
• Partial paper draft: 10%
• Paper: 20%
• In-class presentation: 5%

• Rubrics for these milestones are available here.

• Homework (30%)

• We’ll provide three assignments to make sure that you are able to keep up with the material discussed in class. These assignments will involve implementing or extending machine learning models using Python code and answering questions about the models and their behavior.
• There will be an option for students with prior course experience in natural language processing with neural networks to opt out of these assignments, and be graded on the other three requirements instead. You will have to explicitly apply for this option during the second week of the course.

• Reading Quizzes (12%)

• In the weeks where we discuss readings in the recitation section (noted on the schedule), we will provide very short open-book quizzes to check that you have at least a rough understanding of the papers. You’ll have 24h to take these online through NYU Classes/Brightspace ending at the start of the live class session.

• Participation (8%)

• For full credit, we expect you to either participate regularly in in-class discussion or contribute relevant questions and ideas on Campuswire. (Yes, this does mean we’ll be keeping some notes on who participates in class.)

Schedule

For a preview of some topics, see 2021’s slides.

Readings

Paper readings: You’ll have to read lots of NLP research papers for your project, but we’ll read only four together as a class. The readings are examples of good recent research, and we’ll be reading them with a special attention to how they’re designed and written. For each paper we read together, you’ll have to be ready to take a brief quiz on the paper. After the quiz, you’ll have a chance to discuss the paper—both as a technical idea and as a piece of writing.

Reference readings: This course doesn’t follow any one book, but the readings noted in the ‘Reference’ column above should cover most of the technical content discussed in class, and offer greater depth and additional references on many points. These readings are often long and dense, and you are not required to read them before lecture. However, you will likely have to refer to them at least occasionally when completing the assignments and the project. Abbreviations above correspond to:

1. J&M: Dan Jurafsky and Jim Martin’s Speech and Language Processing (3rd ed., available free online, not yet out in print)
2. G: Yoav Goldberg’s Neural Network Methods for Natural Language Processing (1st ed., available free online through the NYU network)
3. B: Emily Bender’s Linguistic Fundamentals for Natural Language Processing: 100 Essentials from Morphology and Syntax (1st ed., available free online through the NYU network)

• For further reading, take a look at the follow-up book: 100 Essentials from Semantics and Pragmatics

4. E: Jacob Eisenstein’s Natural Language Processing (draft, available free online).
5. C&C: Liz Coppock and Lucas Champollion’s Invitation to Formal Semantics Boot Camp (draft, available free online)

We’re using these books because we think they’re useful, and I encourage you to read them in full.

Paper

The final paper should answer a concrete empirical question involving machine learning for language understanding and should include a literature review that shows that this question hasn’t already been answered decisively by others. A paper could evaluate which of two kinds of model works best on some dataset, evaluate an existing type of model on a new real-world language understanding problem, or offer a quantitative analysis of some existing model using new or existing data.

The paper should include the description of the task, models, experiments and conclusion, and should be no more than 4 pages long, with two optional pages for appendices, and unlimited additional pages for references. The main body of the paper should be complete—you should only use appendices to provide additional clarifying details. It should be prepared in LaTeX. Your paper must follow ACL style closely. Your paper must include a link to a public GitHub repository containing any code used in your experiments.

In addition to the paper itself, we will grade four milestones:

• The mini-proposals, which should consist of three one-sentence descriptions of research questions that you’re interested in. We will give quick feedback on these to let you know which are likely to lead to viable class projects. These should be completed individually before project teams are formed.
• The one-page full proposal, which should describe the problem that you plan to solve, what resources (code, data, etc.) are available to help you solve it, and a very rough plan of work. We will try to offer constructive feedback at this point. Proposals should be submitted by full teams, and students are asked to stay with the same project team for the rest of the term beyond this point.
• The partial draft, with three pages of content (not counting references, appendix, and contribution statement). This should essentially be a near-final draft of your paper (in ACL style) but without any non-baseline results, results analysis, or conclusion. It should contain at least one baseline result, produced using your codebase, for any paper that involves an experimental comparison. It should contain a link to your public GitHub repo, which should show evidence that you have implemented everything you’ll need in order to start your project.
• The presentation. You will be asked to give a very short presentation (likely 5 minutes) during the exam period explaining the key finding of your paper.

Here are some examples of papers from past years that earned ‘A’ grades.

Contribution Statements

The partial draft and the final paper must state (in one or two sentences) the contributions of each team member. This doesn’t count toward the page limit.

Course Policies

Collaboration

Projects should be completed in groups of two or three, which should be formed by the time the full proposal is submitted. We strongly encourage you to discuss the readings with your peers before the reading quizzes open, but while the quizzes are open, you may not discuss the readings or quiz questions with anyone. You may discuss the homework assignments with anyone at any time, but you must ultimately complete and write-up the assigned work for yourself.

Projects Submitted to Multiple Classes

You may submit a project to both this class and another class—we encourage it!—but if you do this, your project must be ambitious and thorough enough to justify the amount of credit you’ll be getting for it. If your course project overlaps with a project that will be submitted to another class, you must inform both instructors by the partial draft deadline, and get confirmation from the instructors of both classes that the combined project is substantial enough.

Due Dates

Please upload completed assignments to NYU Brightspace by 1:30 PM on Wednesdays.

Late Work

We will take 20 percentage points from a grade for each day (full or partial) that the work is late. Exceptions are available only in case of documented emergencies.

Extra Credit

Some lab sessions may include opportunities to earn extra credit, of no more than one point each toward the total grade. This will not be offered as part of the first lab session, or any lab session that is advertised as an optional review session. You will have to attend live to be eligible. We never offer extra credit of any kind on request.

Instructor Contact

Use public forum posts or come to office hours if you have questions about the course. Only use email or private forum messages for issues that are highly sensitive.

Plagiarism

If you use any text or figures at all from an outside source in your submitted work, you must do two things: (i) Use quotation marks, block quotes, or an explicit note to make it clear exactly what part of your paper comes from this source. (ii) Clearly cite this source. Yes, this includes descriptions or depictions of any baseline models and data you use.

You may reuse code by others for the final paper, as long as you make this clear in your paper. However, all the code you submit for individual assignments must be your own.

Violations of this policy will result in a zero grade for the submitted work and a referral to the university for further investigation.

Written submissions will use TurnItIn by default to automatically flag possible plagiarism cases for our review. You may opt out if you wish by contacting us.

Participating Productively

I expect you to be civil, to be respectful of your classmates, and to consider the impacts of your decisions on others, both in your research work and in your interactions with your classmates. I'll try to do the same.

Applicable University Policies

Academic Integrity

Work you submit should be your own. Please consult the CAS academic integrity policy for more information: https://cas.nyu.edu/content/nyu-as/cas/academic-integrity.html – penalties for violations of academic integrity may include failure of the course, suspension from the University, or even expulsion.

Religious Observance

As a nonsectarian, inclusive institution, NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations. The policy and principles to be followed by students and faculty may be found here: The University Calendar Policy on Religious Holidays (http://www.nyu.edu/about/policies-guidelines-compliance/policies-and-guidelines/university-calendar-policy-on-religious-holidays.html)

Accessibility Accommodations

Students requesting academic accommodations are advised to reach out to the The Moses Center for Student Accessibility as early as possible in the semester for assistance.

NYU’s Henry and Lucy The Moses Center for Student Accessibility

212-998-4980

https://www.nyu.edu/csd 

mosescsd@nyu.edu