AI Experience Lab

[Loss functions]

Ue-Hwan, Kim

Where are we now?

• Types of Learning
• Overall Workflow
• Theoretical Perspective
• 1. Data Preparation
• 2. Building Models: Neural Networks
• 3. Training Models
• Loss functions
• Backpropagation
• Optimization
• 4. Evaluating Models
• 5. Improving Performance
• Linear Algebra and Numpy
• Summary

​

2

Contents

• Example problem
• Linear classifier
• Softmax classifier
• More on loss functions
• Regularization
• Summary

​

3

Example problem

4

5

credit: Stanford CS231n

6

credit: Stanford CS231n

7

credit: Stanford CS231n

8

credit: Stanford CS231n

9

credit: Stanford CS231n

10

credit: Stanford CS231n

11

credit: Stanford CS231n

12

credit: Stanford CS231n

13

credit: Stanford CS231n

14

credit: Stanford CS231n

Let’s implement

15

credit: Stanford CS231n

16

credit: Stanford CS231n

17

credit: Stanford CS231n

18

credit: Stanford CS231n

19

credit: Stanford CS231n

Poll

Which of the following is true for image classification

• It is a task of extracting semantics from an array of numbers
• Hard-coded algorithms show satisfactory performance

20

Poll

Which of the following is true for image classification

• It is a task of extracting semantics from an array of numbers
• Hard-coded algorithms show satisfactory performance

21

Linear classifiers

22

23

credit: Stanford CS231n

24

credit: Stanford CS231n

25

credit: Stanford CS231n

26

credit: Stanford CS231n

27

credit: Stanford CS231n

28

credit: Stanford CS231n

29

credit: Stanford CS231n

30

credit: Stanford CS231n

31

credit: Stanford CS231n

32

credit: Stanford CS231n

33

credit: Stanford CS231n

34

credit: Stanford CS231n

Today!

Next class!

35

credit: Stanford CS231n

36

credit: Stanford CS231n

37

credit: Stanford CS231n

38

credit: Stanford CS231n

39

credit: Prof. Dr. Andreas Geiger

40

credit: Prof. Dr. Andreas Geiger

41

credit: Stanford CS231n

42

credit: Stanford CS231n

43

credit: Stanford CS231n

44

credit: Stanford CS231n

45

credit: Stanford CS231n

Calculate!

46

credit: Stanford CS231n

47

credit: Stanford CS231n

48

credit: Stanford CS231n

49

credit: Stanford CS231n

50

credit: Stanford CS231n

51

credit: Stanford CS231n

No change

0 / inf

(C - 1)

52

credit: Stanford CS231n

53

credit: Stanford CS231n

Just adding 1

54

credit: Stanford CS231n

55

credit: Stanford CS231n

Just rescaling

56

credit: Stanford CS231n

57

credit: Stanford CS231n

Squared hinge loss

58

credit: Stanford CS231n

Squared hinge loss

59

credit: Stanford CS231n

Let’s implement!

60

Softmax classifier

61

62

credit: Stanford CS231n

63

credit: Stanford CS231n

64

credit: Stanford CS231n

65

credit: Stanford CS231n

66

credit: Stanford CS231n

67

credit: Stanford CS231n

68

credit: Stanford CS231n

69

credit: Stanford CS231n

70

credit: Stanford CS231n

71

credit: Stanford CS231n

72

credit: Stanford CS231n

73

credit: Stanford CS231n

0 / inf

74

credit: Stanford CS231n

75

credit: Stanford CS231n

76

credit: Stanford CS231n

77

credit: Stanford CS231n

78

credit: Stanford CS231n

79

credit: Stanford CS231n

SVM: nothing much

Softmax: sensitive

Let’s implement!

80

More on loss functions

81

82

credit: Prof. Dr. Andreas Geiger

83

credit: Prof. Dr. Andreas Geiger

84

credit: Prof. Dr. Andreas Geiger

85

credit: Prof. Dr. Andreas Geiger

More on loss functions

• regression -

86

87

credit: Prof. Dr. Andreas Geiger

88

credit: Prof. Dr. Andreas Geiger

89

credit: Prof. Dr. Andreas Geiger

90

credit: Prof. Dr. Andreas Geiger

91

credit: Prof. Dr. Andreas Geiger

92

credit: Prof. Dr. Andreas Geiger

Poll

손실함수(Loss functions)에 대하여 옳은 설명은?

• 손실 함수는 어떤 함수라도 될 수 있습니다.
• 손실 함수를 계산하여 도출하는 대신 우도(likelihood)를 최대화합니다.
• 제곱 손실(L2)은 절대 손실(L1)보다 특이값에 더 강건(robust)합니다.
• 훈련 샘플이 i.i.d일 필요는 없다고 가정합니다. (독립적이고 동일하게 분산됨)

​

93

Poll

손실함수(Loss functions)에 대하여 옳은 설명은?

• 손실 함수는 어떤 함수라도 될 수 있습니다.
• 손실 함수를 계산하여 도출하는 대신 우도(likelihood)를 최대화합니다.
• 제곱 손실(L2)은 절대 손실(L1)보다 특이값에 더 강건(robust)합니다.
• 훈련 샘플이 i.i.d일 필요는 없다고 가정합니다. (독립적이고 동일하게 분산됨)

​

94

Poll

Which is correct about loss functions?

• A loss function can be any functions
• Rather than manually derive loss functions, we maximize the likelihood
• The squared loss (L2) is more robust to outliers than the absolute loss (L1) is
• We assume training samples do not have to be i.i.d. (independent and identically distributed)

95

Poll

Which is correct about loss functions?

• A loss function can be any functions
• Rather than manually derive loss functions, we maximize the likelihood
• The squared loss (L2) is more robust to outliers than the absolute loss (L1) is
• We assume training samples do not have to be i.i.d. (independent and identically distributed)

96

More on loss functions

• classification -

97

98

credit: Prof. Dr. Andreas Geiger

99

credit: Prof. Dr. Andreas Geiger

100

credit: Prof. Dr. Andreas Geiger

101

credit: Prof. Dr. Andreas Geiger

What about the logistic regression?

102

What about the logistic regression?

103

What about the logistic regression?

104

What about the logistic regression?

105

106

107

108

109

credit: Prof. Dr. Andreas Geiger

110

credit: Prof. Dr. Andreas Geiger

111

112

credit: Prof. Dr. Andreas Geiger

113

114

credit: Prof. Dr. Andreas Geiger

115

credit: Prof. Dr. Andreas Geiger

116

117

118

119

120

credit: Prof. Dr. Andreas Geiger

121

credit: Prof. Dr. Andreas Geiger

122

credit: Prof. Dr. Andreas Geiger

Let’s implement

123

Regularization

124

125

credit: Stanford CS231n

126

credit: Stanford CS231n

127

credit: Stanford CS231n

128

credit: Stanford CS231n

129

credit: Stanford CS231n

130

credit: Stanford CS231n

131

credit: Stanford CS231n

132

credit: Stanford CS231n

133

credit: Stanford CS231n

134

credit: Stanford CS231n

135

credit: Stanford CS231n

136

credit: Stanford CS231n

137

credit: Stanford CS231n

138

credit: Stanford CS231n

139

credit: Stanford CS231n

140

credit: Stanford CS231n

141

credit: Stanford CS231n

142

credit: Stanford CS231n

143

credit: Stanford CS231n

Next class!

Summary

Today we

• talked about the image classification task
• discussed linear classifiers
• had a look at the SVM and softmax losses
• introduced a probabilistic view on loss functions
• considered regularization to weigh between weights

144