CSCI-SHU 205: Topics in Computer Science

Human-AI Alignment

Hua Shen

Course Website: https://hua-shen.org/src/course_bialign.html

2025-09-01

Lecture 1

Welcome to BiAlign course👏 !

Hello from Your Instructor!

Hua Shen

Assistant Professor of Computer Science

huashen@nyu.edu | huashen218

https://hua-shen.org/

• Office Hour: 11:00 AM - 12:00 PM, Tue/Wed;
• Office: S749; Zoom link: https://nyu.zoom.us/my/hua.shen (Passcode:enQb9h)
• Resources: Course Website, Syllabus Details
• Contact me: huashen@nyu.edu

Outline

1. What is Human-AI Alignment? (20 min)
2. Why Human-AI Alignment? What if Not? (15 min)
3. Course Overview and Logistics (20 min)
4. Discussion: Know more about You 🧡 ! (20min)

By joining today’s class –

· decide whether this course fits you well;

· learn an overview and motivation of HAI-Alignment;

· share what you want to get from this course.

Outline

• What is Human-AI Alignment? (20 min)
• Why Human-AI Alignment? What if Not? (15min)
• Course Overview and Logistics (20 min)
• Discussion: Know more about You! (20min)

Share Your Thoughts 🙌

What’s your understanding of “Human-AI Alignment”

AI systems are deeply integrated into our lives …

​

AI ethics are NOT fully aligned with human values…

Challenges in state-of-the-art research

Human-centered and AI-centered research & skills are largely divided!

“Find” AI Problems for Humans

“Address” Problems in AI

What is Human-AI Alignment?

Shen, Hua, Tiffany Knearem, Reshmi Ghosh et al. "Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions." arXiv:2406.09264.

What is Human-AI Alignment?

Shen, Hua, Tiffany Knearem, Reshmi Ghosh et al. "Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions." arXiv:2406.09264.

Towards Bidirectional Human-AI Alignment

What is Human-AI Alignment?

Shen, Hua, Tiffany Knearem, Reshmi Ghosh et al. "Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions." arXiv:2406.09264.

Towards Bidirectional Human-AI Alignment

Objective of Human-AI Alignment

Maximizing Capabilities AND Minimizing Risks in Human–AI Co-Evolution

Scope of Human-AI Alignment

Human-AI Alignment involves:

• Capture human values and feedback for LLMs / AI Agents
• Develop LLMs / AI Agents to integrate human values & feedback
• Interpret models to humans for human-AI collaboration
• Social impact of LLMs/AI Agents for trustworthy and safe AI
• ……

Who is the human in “Human-AI Alignment”?

Efforts towards Human–AI Alignment so far…

Efforts towards Human–AI Alignment so far…

2025 Tutorial on Human-AI Alignment

@Dec,2025

Hua Shen

Instructors:

Mitchell Gordon

Adam Tauman Kalai

Panelists:

Yoshua Bengio

Dawn Song

Monojit Choudhury

Hannah Kirk

Eric Gilbert

Outline

• What is Human-AI Alignment? (20 min)
• Why Human-AI Alignment? What if Not? (15min)
• Course Overview and Logistics (20 min)
• Discussion: Know more about You! (20min)

State-of-the-art Research

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. Training language models to follow instructions with human feedback. NeurIPS 2022.

Human feedback is primarily rating and ranking…

State-of-the-art Research

"Constitutional ai: Harmlessness from ai feedback." arXiv:2212.08073.

"Red teaming language models with language models." arXiv:2202.03286.

Responsible AI work commonly involves minimal human participation

Missing Diverse Human Participation…

Ion Stoica, Keynote: Reliability: An AI Challenge. Agentic AI Summit 2025. https://www.youtube.com/watch?v=c39fJ2WAj6A (1:27:07)

Without humans in the loop of AI development & deployment…

How?

This BiAlign course equips you with fundamental knowledge, technical skills, and practical project experience to advance human-AI alignment in future research.

Outline

• What is Human-AI Alignment? (20 min)
• Why Human-AI Alignment? What if Not? (15 min)
• Course Overview and Logistics (20 min)
• Discussion: Know more about You! (20min)

Course Overview

I. Foundations

II. Methods

III. Practice

III. Practice

I. Foundations (Week 1-2)

• Overview:
• Evolving Challenges of AI Alignment and Human's Role
• Values & Morals in LLMs:
• Theories, Evaluation and Social Impacts

Course Overview

I. Foundations

II. Methods

III. Practice

III. Practice

II. Methods (Week 3-9)

• Data is Gold:
• Co-Annotation, Styes and Validation
• Post-Training for Alignment:
• Prompt, SFT, RLHF and Interactive Alignment
• Evaluation and Ecosystem:
• Automatic and Human Evaluation & Platform

Course Overview

I. Foundations

II. Methods

III. Practice

III. Practice

III. Practice (Week 10-15)

• Human-AI Interaction:
• Design, Evaluation & Use Cases
• LLM Interpretability:
• Mechanistic & Human - Centered Interpretability
• Risks, Trust and Safety:
• Benchmarks & Human - Centered Auditing

Course Goals:

• Gain Foundational Knowledge on Human-AI Alignment
• Familiarize Cutting-edge Research
• Harness Hands-on Technical Skills
• Practical Projects

Mapping Goals to Course Activities

Check details at: https://hua-shen.org/src/course_bialign.html

Course Activities

Paper Presentation: 20%

• Earlier presentation dates receive higher scores.

Project: 50%

• Proposal: 10%
• Midway Report: 10%
• Midway Presentation: 5%
• Final Submission: 20%
• Final Presentation: 5%

Assignments: 20%

• Assignment 1: 10%
• Assignment 2: 10%

Participation: 10%

• General participation: 5%
• Question contribution: 5%

Clarification on Several Course Activities

Paper Presentation

• Sign-up Link: Sign-up Google Sheet;
• Team: 1-2 students per group;
• Papers: select at least one paper from reading materials to present;
• Earlier presentation dates potentially receive higher scores.

Paper Title

Name(s)

Clarification on Several Course Activities

Course Assignments

Assignment 1 (10%):

Assignment 2 (10%):

LLM Post-Training Alignment (AI)

• Data: Human value related datasets
• Alignment: Prompting, Finetuning, Instruction-tuning, RLHF/DPO (optional)

Human-LLM Interactive Alignment (HCI)

• Frontend UI Design: collect human value data
• Backend LLM Server: incorporate human value to customize AI

Clarification on Several Course Activities

Projects

• Team: 1-2 students per group;

• Scopes (Beyond below Examples):
• Novel human-LLM co-annotated datasets & benchmark
• Improving human-Agentic AI interaction workflow;
• Novel evaluation & oversight on alignment and interaction
• (Human-centered) interpretability and Agentic AI
• Novel solutions of LLMs’/agents’ risks and safety issues

Course Policy

• Please get familiar with NYU Shanghai’s Statement on Academic Integrity (in the Undergraduate Bulletin).
• Generative AI Policies:
• You could use Generative AI ethically for in this course;
• Generative AI can be used to support your research and coding;
• But - the creativity and novel ideas should be your own.
• Late policy:
• Each student could have one free submission late day;
• Final project papers cannot be turned in late under any circumstances
• You can always let me know if you have any questions or need support.

Computing Resource

— Generative AI Tools and Services in NYU Shanghai

Outline

• What is Human-AI Alignment? (20 min)
• Why Human-AI Alignment? What if Not? (15 min)
• Course Overview and Logistics (20 min)
• Discussion: Know more about You 🧡 ! (20min)

Love to know more about you!

What’s your experience + expectation on this course

Share Your Experience & Insights

1. A quick introduction of yourself
2. Why are you interested in this course?
3. What do you want to gain from this course?
4. What do you need (or want to see) in this course?

Your feedback & discussion is always welcome!

huashen@nyu.edu

Summary

What is Human-AI Alignment? (20 min)

Course Overview and Logistics (15 min)

Why Human-AI Alignment? What if Not? (20min)

Discussion: Know more about You! (20min)

Next class –

Overview: Evolving Challenges of AI Alignment and Human's Role

Assignment #0

Due: 11:59 PM, Sep 7, 2025 (Sun).

(China Standard Time)

• Sign-up Link: Sign-up Google Sheet;
• Reading Materials: Project Website;

Reading Materials

Thank You 💛

See you on Wednesday!