Production Machine Learning Responses
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

View only
 
ABCDEFGHIJKLMNOPQRSTUVWXY
1
Does your company train and use machine learning models?
What types of machine learning problems are worked on at your company?
How do the majority of models get retrained?
How often do the majority of models get retrained?
How do the outputs of the models get served? (check all that apply)
Roughly how many different models are serving outputs?
Which categories of performance do you track?
How is the model performance tracked? (check all that apply)
How often do you check model performance?
How big is your company?
How many people are training and/or deploying ML models at your company?
2
YesRegressionManually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API computes outputs in realtime
2-5
Manual SQL queries / scripts;Automated SQL queries / scripts
As needed/ ad hoc
151-5005-10
3
Yes
Regression;Binary Classification;Multiclass Classification;Recommendation Systems;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Weekly;If there is an escalation
500+100+
4
Yes
Binary Classification;NLP
They don't get retrained
They don't get retrained
Outputs are written to a database and some other system uses them;excel files
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts
Weekly;Monthly500+1-4
5
Yes
Regression;Binary Classification;Multiclass Classification;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Automated SQL queries / scripts
Monthly11-501-4
6
Yes
Binary Classification;Multiclass Classification
Manually
Randomly (ad hoc)
An API computes outputs in realtime
1
Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts;Custom-built monitoring system
Daily;Weekly;Monthly
11-501-4
7
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Computer Vision;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
50+
Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
I don't check the performance
500+100+
8
Yes
Multiclass Classification;Recommendation Systems;Computer Vision;NLP
ManuallyMonthly
An API computes outputs in realtime
5-10
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts
Daily;Weekly51-1501-4
9
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Computer Vision;NLP
I don't know, everybody does their own thing
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Custom-built monitoring system
Monthly;Rarely500+100+
10
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;NLP
ManuallyMonthly
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts;Custom-built monitoring system
Weekly51-15011-24
11
Yes
Regression;Binary Classification;Multiclass Classification;Recommendation Systems;NLP
Automatically (e.g. a batch job)
Monthly
Outputs are written to a database and some other system uses them;An API computes outputs in realtime
5-10
Offline test data metrics (e.g. a classifier's AUC on held out test data)
Manual SQL queries / scripts
Rarely500+1-4
12
Yes
Regression;Binary Classification;Multiclass Classification;Recommendation Systems;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
10-20
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts;Custom-built monitoring system
Daily;Weekly;Monthly
151-5005-10
13
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts
Monthly500+50-100
14
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Simulation;Computer Vision
I don't know, everybody does their own thing
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts;Custom-built monitoring system
Daily;Weekly500+50-100
15
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Simulation;Computer Vision;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
10-20
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts
Depends on how often the model needs to be used
500+5-10
16
Yes
Regression;Binary Classification;Time Series;Recommendation Systems;Simulation;NLP
They don't get retrained
They don't get retrained
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
5-10
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts
Daily;Rarely151-5001-4
17
Yes
Regression;Binary Classification;Simulation
Automatically (e.g. a batch job)
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
5-10
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Automated SQL queries / scripts
Daily;Weekly500+11-24
18
Yes
Regression;Binary Classification
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Post-hoc comparison of predictions to outcomes for select experiments
Manual SQL queries / scripts
Irregular but works out to every couple months; usually tied to specific experiments
51-1505-10
19
YesRegressionManually
They don't get retrained
2-5
Offline test data metrics (e.g. a classifier's AUC on held out test data)
Custom-built monitoring system
Rarely11-501-4
20
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Simulation;NLP
Automatically (e.g. a batch job)
Daily
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime;I imagine we do all 3, but not 100% certain
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts;Custom-built monitoring system
Daily500+50-100
21
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Simulation;NLP
I don't know, everybody does their own thing
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them
5-10
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts;Third-party system
Daily;Weekly;Monthly
151-5001-4
22
Yes
Regression;Binary Classification;Multiclass Classification;Time Series
Automatically (e.g. a batch job)
Monthly
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Manual SQL queries / scripts
Monthly11-501-4
23
Yes
Regression;Multiclass Classification;Recommendation Systems
ManuallyMonthly
An API serves pre-computed outputs;An API computes outputs in realtime
10-20
Training data metrics (e.g. a classifier's AUC on the training data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts;Custom-built monitoring system
Daily500+11-24
24
Yes
Regression;Binary Classification
Automatically (e.g. a batch job)
Weekly
An API serves pre-computed outputs;An API computes outputs in realtime
10-20
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Automated SQL queries / scripts;Custom-built monitoring system
Weekly51-1501-4
25
Yes
Recommendation Systems;Reinforcement learning / bandits
Automatically (e.g. a batch job)
Daily
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
50+
Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Daily;Weekly500+50-100
26
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;NLP
Automatically (e.g. a batch job)
Monthly
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
20-50
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts;Third-party system
Monthly151-50011-24
27
Yes
Binary Classification;Multiclass Classification;Time Series;Computer Vision
Manually
Randomly (ad hoc)
An API computes outputs in realtime
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Automated SQL queries / scripts;Custom-built monitoring system
It's static, so ...51-1501-4
28
Yes
Regression;Binary Classification;Recommendation Systems
Automatically (e.g. a batch job)
Daily
An API computes outputs in realtime
10-20
Training data metrics (e.g. a classifier's AUC on the training data)
Application logs
I don't check the performance
151-5001-4
29
Yes
Binary Classification;Multiclass Classification
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API computes outputs in realtime
2-5
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Manual SQL queries / scripts;Automated SQL queries / scripts
Weekly151-5001-4
30
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Computer Vision;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
2-5
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Weekly11-5011-24
31
Yes
Binary Classification;Multiclass Classification;Computer Vision;NLP
Manually
Randomly (ad hoc)
An API serves pre-computed outputs;An API computes outputs in realtime
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Manual SQL queries / scripts
Rarely1-101-4
32
Yes
Regression;Binary Classification;Time Series;Recommendation Systems;NLP
Automatically (e.g. a batch job)
Monthly
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs;An API computes outputs in realtime
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Weekly;Monthly500+25-49
33
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Simulation
Manually
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
20-50
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system;Trading profits
I don't check the performance
11-505-10
34
Yes
Binary Classification;Multiclass Classification;Time Series;Computer Vision
ManuallyMonthly
An API serves pre-computed outputs;An API computes outputs in realtime
2-5
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data)
Custom-built monitoring system
Weekly11-501-4
35
Yes
Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Computer Vision;NLP
Manually
Randomly (ad hoc)
An API computes outputs in realtime
50+
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Rarely11-501-4
36
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Simulation
Automatically (e.g. a batch job)
Daily
An API serves pre-computed outputs;An API computes outputs in realtime
5-10
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Weekly151-5001-4
37
Yes
Recommendation Systems;Computer Vision;NLP
Automatically (e.g. a batch job)
Daily
Outputs are written to a database and some other system uses them;An API serves pre-computed outputs
1
Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Daily151-5001-4
38
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Recommendation Systems;Simulation;Computer Vision;NLP
Manually
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them;An API computes outputs in realtime
20-50
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Automated SQL queries / scripts;Custom-built monitoring system
Monthly500+11-24
39
Yes
Regression;Binary Classification;Multiclass Classification;Time Series;Computer Vision;NLP
Manually
Randomly (ad hoc)
Outputs are written to a database and some other system uses them
10-20
Offline test data metrics (e.g. a classifier's AUC on held out test data)
Manual SQL queries / scripts;Automated SQL queries / scripts
Monthly500+25-49
40
Yes
Regression;Binary Classification;Computer Vision;NLP
I don't know, everybody does their own thing
Randomly (ad hoc)
Outputs are written to a database and some other system uses them;An API computes outputs in realtime
50+
Training data metrics (e.g. a classifier's AUC on the training data);Offline test data metrics (e.g. a classifier's AUC on held out test data);Online metrics (e.g. comparing predictions at the time of model serving to outcomes)
Custom-built monitoring system
Weekly500+100+
41
Yes
Multiclass Classification;Recommendation Systems;Computer Vision;NLP
I don't know, everybody does their own thing
I don't know, everybody does their own thing
Outputs are written to a database and some other system uses them
5-10
Offline test data metrics (e.g. a classifier's AUC on held out test data)
Custom-built monitoring system
Monthly500+50-100
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...