Motivation, Syllabus, and Introductions

1

• Machine Learning in Production

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Slack

• We use Slack for this course, including during lectures
• See signup link on Canvas (you can do this right now!)
• Setup the ability to read/post to Slack during lecture.
• Sometime in the next day: introduce yourself on #social

2

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Waitlist situation

We hope to be able to accommodate everyone.

Consider expressing ability to move to alternative section time/locations, especially for 11am we have more flexibility

Please email the course admin Jenni Cooper (cooperj@andrew.cmu.edu ), not us, to inquire about WL status/registration movement.

If and when you join late:

1. Post an introduction on Slack (#social)
2. See the Canvas front page about automatic extensions

3

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Learning Goals

• Understand how ML components are parts of larger systems
• Illustrate the challenges in engineering an ML-enabled system beyond accuracy
• Explain the role of specifications and their lack in machine learning and the relationship to deductive and inductive reasoning
• Summarize the respective goals and challenges of software engineers vs data scientists
• Explain the concept and relevance of “T-shaped people”
• Understand the basic mechanics of this course.

4

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Case Study: Music Generation

5

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Context: Music Generation Research

Lam, Max WY, et al. "Efficient neural music generation." Advances in Neural Information Processing Systems 36 (2024).

6

Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing 95.7% to 99.6% forward passes in MusicLM, respectively, for sampling 10s to 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also in its state-of-the-art musicality, audio quality, and text correlation. Our samples are available at https://Efficient-MeLoDy.github.io/.

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

The Startup Idea

You just completed a research thesis as part of your Master’s/PhD degree about making deep learning for generative AI more energy efficient.

Your recent project was using music generation as a case study. You showed 30% energy improvements with similar quality of generated music on benchmark prompts.

Two friends are excited about the application of music generation.

Idea: Let’s commercialize the idea and sell it to end users or music producers.

7

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

8

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

9

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Breakout: Likely challenges in building commercial product?

​

As a group, think about challenges that the team will likely face when turning their research into a product:

• One machine-learning challenge
• One engineering challenge in building the product
• One challenge from operating and updating the product
• One team or management challenge
• One business challenge
• One safety or ethics challenge

Post answer to #lecture on Slack and tag all group members (skip if nobody in group has slack set up yet)

10

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Examples for discussion

• What does correctness or accuracy really mean? What accuracy do customers care about?
• How can we see how well we are doing in practice? How much feedback are customers going to give us before they leave?
• Can we estimate how good our generated music is? How are we doing for different customers or different genres?
• How to present results to the customers?
• When customers complain about poor output, how to prioritize and what to do?
• What are unacceptable mistakes and how can they be avoided? Is there a safety risk?
• Can we cope with an influx of customers?
• Will responding to the same prompt twice produce the same result? Does it matter?
• How can we debug and fix problems? How quickly?

11

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Examples for discussion 2

• With more customers, generation is taking longer and longer – what can we do?
• Generation sometimes crashes. What to do?
• How do we achieve high availability?
• How can we see that everything is going fine and page somebody if it is not?
• We improve our model but somehow system behavior degrades… Why?
• Tensorflow update; does our infrastructure still work?
• Once somewhat successful, how to handle large amounts of data per day?
• Buy more machines or move to the cloud?
• Models are continuously improved. When to deploy? Can we roll back?
• Can we offer live music generation as an app? As a web service?
• Can we get better the longer a user interacts with us? Will this benefit the next song they generate as well?

12

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Examples for discussion 3

• How many genres can be supported? Do we have the server capacity?
• How specific should genres be?
• How to make it easy to support new types of music?
• Can we generate different types of vocals? Handle different languages or accents?
• Can and should we learn from customer data?
• How can we debug problems in files we are not allowed to see?
• Any chance we might leak private data?
• Can competitors or bad actors attack our system?

13

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

What qualities are important for a good commercial music generator?

14

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

What non-ML components are needed?

15

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

ML in a Production System

16

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

ML in a Production System

17

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Development Teams

18

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

19

and Data engineers + Domain specialists + Operators + Business team + Project managers + Designers, UI Experts + Safety, security specialists + Lawyers + Social Scientists + …

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Data scientist Software engineer

• Often fixed dataset for training and evaluation (e.g., PBS interviews)
• Focused on accuracy
• Prototyping, often Jupyter notebooks or similar
• Expert in modeling techniques and feature engineering
• Model size, updateability, implementation stability typically does not matter

• Builds a product
• Concerned about cost, performance, stability, release time
• Identify quality through customer satisfaction
• Must scale solution, handle large amounts of data
• Detect and handle mistakes, preferably automatically
• Maintain, evolve, and extend the product over long periods
• Consider requirements for security, safety, fairness

20

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Likely collaboration challenges?

21

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

What might Software Engineers and Data Scientists focus on?

22

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

23

By Steven Geringer, via Ryan Orban. Bridging the Gap Between Data Science & Engineer: Building High-Performance Teams. 2016

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

T-Shaped People

Broad-range generalist + Deep expertise

​

24

Figure: Jason Yip. Why T-shaped people?. 2018

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

T-Shaped People

Broad-range generalist + Deep expertise

​

Example:

• Basic skills of software engineering, business, distributed computing, and communication
• Deep skills in deep neural networks (technique) and medical systems (domain)

​

25

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Latest Buzzword: π-Shaped People

π

26

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Syllabus and Class Structure

17-445/17-645/17-745, Fall 2025, 12 units

Monday/Wednesdays 3:30-4:50pm

Recitation Fridays 9:30am, 11am, 2pm

27

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Communication

Email us or ping us on Slack �(invite link on Canvas)�(ask your chatbot to keep the email short)

All announcements through Slack #announcements

Weekly office hours, starting next week, schedule on Canvas

Post questions on Slack – Please use #general or #assignments and post publicly if possible; your classmates will benefit from your Q&A!

All course materials (slides, assignments, old midterms) available on GitHub and course website: https://mlip-cmu.github.io/f2025/

28

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Office Hours

Posted soon on Canvas

Feel free to drop in with questions or just to chat

29

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Class with software engineering flavor

Focusing on engineering judgment

Arguments, tradeoffs, and justification, rather than single correct answer

Practical engagement, building systems, testing, automation

Strong teamwork component

Both text-based and code-based homework assignments

30

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Prerequisites

Some machine learning experience required

• Basic understanding of data science process, incl. data cleaning, feature engineering, using ML libraries
• High level understanding of machine-learning approaches
• Supervised learning
• Regression, decision trees, neural networks
• Accuracy, recall, precision, ROC curve
• Ideally, some experience with notebooks, sklearn, or other frameworks

Basic programming and command-line skills will be needed

No further software-engineering knowledge required

• Teamwork experience in product team is useful but not required
• No required exposure to requirements, software testing, software design, continuous integration, containers, process management, etc.
• If you are familiar with these, there will be some redundancy – sorry!

31

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

First Homework Assignment I1

“Coding warmup assignment”

Out now, due Monday Sep 8

Enhances simple web application with LLM-powered features

Open-ended coding assignment, change existing code, learn new APIs, handle credentials, solve dependency/versioning issue

32

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Active lecture

Case study driven

Discussions highly encouraged

Regular in-class activities, breakouts

Contribute your own experience!

Discussions over definitions

33

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Recordings and Attendance

Try to attend lecture – discussions are important to learning

Participation is part of your grade

No lecture recordings, but textbook and slides available

Contact us for accommodations (illness, interview travel, unforeseen events), or have your advisor reach out. We try to be flexible.

34

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Participation

Participation != Attendance

Grading:

• 100%: Participates actively at least once in most lectures by (1) asking or responding to questions or (2) contributing to breakout discussions
• 90%: Participates actively at least once in two thirds of the lectures
• 75% Participates actively at least once in over half of the lectures
• 50%: Participates actively at least once in one quarter of the lectures
• 20%: Participates actively at least once in at least 3 lectures

35

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

36

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Reading Assignments & Quizzes

Building Intelligent Systems by Geoff Hulten

https://www.buildingintelligentsystems.com

Many chapters assigned at some point in the semester

Supplemented with research articles, blog posts, videos, podcasts, …

Electronic version in the library

37

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Book based on the Class

Mostly similar coverage to lecture

Not required, use as a supplementary reading

Published online at https://mlip-cmu.github.io/book/, �printed through MIT Press

38

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Reading Quizzes

Short essay questions on readings, due before start of lecture (Canvas quiz)

Planned for: about 30-45 min for reading, 10 min for answering quiz

39

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Assignments

Series of 4 small to medium sized individual assignments

• Engage with practical challenges
• Reason about tradeoffs and justify your decisions
• Integrate models and explanations into end-user products
• Written reports, some modeling, some coding

Large team project with 4 milestones:

• Build and deploy a prediction (movie recommendation) service
• Testing in production, monitoring
• Final presentation

Usually due Monday night; see schedule

40

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Research in this Course

We are conducting academic research in this course.

This research will involve analyzing student work of assignment after the end of the semester.

You will not be asked to do anything above and beyond the normal learning activities and assignments that are part of this course. All data will be analyzed in de-identified form and presented in the aggregate, without any personal identifiers.

You are free not to participate in this research, and your participation will have no influence on your grade for this course or your academic career at CMU. If you do not wish to participate, please send an email to Nadia Nahar (nadian@andrew.cmu.edu); instructors will not know who opts out before assigning final grades.

See syllabus for details.

41

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

17-745 PhD Research Project

Research project instead of individual assignments I3 and I4

Design your own research project and write a report

• A case study, empirical study, literature survey, etc.

Very open ended: Align with own research interests and existing projects

See the project requirements and deadlines and talk to us

42

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Midterms and Final Presentation

2 Midterms, pen&paper during lecture time, not cumulative

(find old midterms as examples on the course website)

No final exam, but project presentations in final exam slot

Date not known until Sep 29

May be as late as Dec 15; do not book flights early

43

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Labs

Please attend the lab for which you are registered �(section G received special instructions).

Introducing various tools, e.g., fastAPI (serving), Kafka (stream processing), Jenkins (continuous integration), MLflow (experiment tracking), Docker & Kubernetis (containers), Prometheus & Grafana (monitoring), SHAP (explainability)...

Hands on exercises, bring a laptop

Often introducing tools useful for assignments

About 1h of work, graded pass/fail, low stakes, show & explain work to TA

First lab this Friday: Calling, securing, and creating APIs

44

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Lab grading and collaboration

We recommend that you start each lab before the recitation, but can be completed during

Graded pass/fail by TA on the spot, can retry

Relaxed collaboration policy: Can work with others before and during recitation, but have to show & explain solution to TA individually

(Think of recitations as mandatory office hours)

45

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Lab online options (new)

If you complete the lab ahead of time, you may be able to show it over Zoom on Thursday evenings (6-7:30pm)

You will show/explain your work to the TA over Zoom

TA will grade but not provide help; if you like help, attend your lab session in person on Friday

Logistics on how and when to queue will be posted on Slack

46

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Grading

• 35% individual assignment
• 30% group project with final presentation
• 15% two in-class exams
• 5% participation
• 5% reading quizzes
• 10% labs
• No final exam (final presentations will take place in that time slot)

Expected grade cutoffs in syllabus (>82% B, >94 A-, >96% A, >99% A+)

47

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Grading Philosophy

Specification grading, based in adult learning theory

Giving you choices in what to work on or how to prioritize your work

We are making every effort to be clear about expectations (specifications), will clarify if you have questions

Assignments broken down into expectations with point values, each graded pass/fail

Opportunities to resubmit work until last day of class

[Example]

48

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Token System for Flexibility

8 individual tokens per student:

• Submit individual assignment 1 day late for 1 token (after running out of tokens, 15% penalty per late day)
• Redo individual assignment for 3 token
• Late/redo lab, quiz, team feedback, … for 1 token
• Remaining tokens count toward participation

8 team tokens per team:

• Submit milestone 1 day late for 1 token
• Redo milestone for 3 token

49

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

How to use tokens

No need to tell us if you plan to submit very late. We will assign 0 and you can resubmit

Instructions and Google forms for resubmission on Canvas (pages)

We will automatically use remaining tokens toward participation at the end

#remaining individual tokens reflected on Canvas, for #remaining team tokens, ask your team mentor.

50

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Group project

Instructor-assigned teams

Teams stay together for project throughout semester.

There will be a Catme Team survey around Sep 10 (3pt)

Some advice in lectures; we’ll help with debugging team issues

TA assigned to each team as mentor; mandatory debriefing with mentor and peer grading on all milestones (based on citizenship on team)

Bonus points for social interaction in project teams

51

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Academic honesty

• See web page; in a nutshell: do not copy from other students, do not lie, do not share or publicly release your solutions
• In group work, be honest about contributions of team members, do not cover for others
• Do not falsely pretend to participate in in-class participation
• Collaboration okay on labs, but not quizzes, individual assignments or exams
• If you feel overwhelmed or stressed, please come and talk to use (see syllabus for other support opportunities)

52

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Thoughts on Generative AI for Homework?

ChatGPT, Copilot, Claude Code, …? Reading quizzes, homework, labs, …?

53

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Our Position on Generative AI for Homework

This is a course on responsible building of ML products. This includes questions of how to build generative AI tools responsibly and discussing what use is ethical.

Feel free to use them and explore whether they are useful. Welcome to share insights/feedback.

Warning: Be aware of hallucinations and AI slop. Requires understanding to check answers. We test them; they often generate bad/wrong answers for reading quizzes.

You are responsible for the correctness of what you submit!

54

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

What makes software with ML challenging?

55

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Lack of Specifications

• Traditional SE: specify what to do & how to test
• MLs are usually black boxes (maybe justified, for their complexity…)
• Even if you put specs in, e.g., LLM prompts, unclear if they will follow
• Agents combine multiple unreliable components

56

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Data Focused and Scalable

• MLs get the “specs” from data; Larger the better
• Deductive reasoning (applying logic rules) to Inductive Reasoning (generalizing from observation)
• Cause scalability issues

57

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

ML Models Make Mistakes

• …Often in unexpected ways
• Hard to foresee and capture because no spec
• What does it mean to be correct? Can only evaluate whether it works well enough (on average) on some test data!

58

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Interaction with the environment

Our system must be able to tolerate some incorrect predictions, and be aware of how it might influence the world…

59

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

It’s not all new

We routinely build:

• Safe software with unreliable components
• Non-ML big data systems, cloud systems
• “Good enough” and “fit for purpose” not “correct”
• Cyberphysical systems

ML intensifies our challenges

60

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Complexity

61

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Complexity

62

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Introductions

Before the next lecture, introduce yourself in Slack channel #social:

• Your (preferred) name
• In 1-2 sentences, your data science background and goals (e.g., coursework, internships, work experience)
• In 1-2 sentences, your software engineering background, if any, and goals (e.g., coursework, internships, work experience)
• The kind of job you are aiming for after graduation
• A hobby or a favorite activity outside school

63

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Come to class for exclusive @count_scratchula content!

64

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025

Summary

Machine learning components are part of larger systems

Data scientists and software engineers have different goals and focuses

• Building systems requires both
• Various qualities are relevant, beyond just accuracy

Machine learning brings new challenges and intensifies old ones

65

Machine Learning in Production • Christian Kaestner & Bogdan Vasilescu, Carnegie Mellon University • Fall 2025