Word2Vec Implementation
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

Comment only
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACADAEAFAGAHAIAJAKALAMANAOAPAQARASAT
1
2
1. Corpus
3
natural language processing and machine learning is fun and exciting
4
5
2. Sliding Window
derekchia.com
Created by:
Derek Chia
6
#1naturallanguageprocessingandmachinelearningisfunandexciting#1Twitter:
@derekchia
7
Y(c=1)Y(c=2)Email:
derek@derekchia.com
8
#2naturallanguageprocessingandmachinelearningisfunandexciting#2Blog post:https://medium.com/@derekchia/an-implementation-guide-to-word2vec-using-numpy-and-google-sheets-13445eebd281
9
Y(c=1)Y(c=2)
Y(c=3)
10
#3naturallanguageprocessingandmachinelearningisfunandexciting#3
11
Y(c=1)Y(c=2)
Y(c=3)
Y(c=4)
12
#4naturallanguageprocessingandmachinelearningisfunandexciting#4
13
Y(c=1)Y(c=2)Y(c=3)Y(c=4)
14
#5naturallanguageprocessingandmachinelearningisfunandexciting#5
15
Y(c=1)
Y(c=2)
Y(c=3)
Y(c=4)
16
#6naturallanguageprocessingandmachinelearningisfunandexciting#6
17
Y(c=1)
Y(c=2)
Y(c=3)
Y(c=4)
18
#7naturallanguageprocessingandmachinelearningisfunandexciting#7
19
Y(c=1)Y(c=2)Y(c=3)Y(c=4)
20
#8naturallanguageprocessingandmachinelearningisfunandexciting#8
21
Y(c=1)
Y(c=2)
Y(c=3)Y(c=4)
22
#9naturallanguageprocessingandmachinelearningisfunandexciting#9
23
Y(c=1)
Y(c=2)Y(c=3)
24
#10naturallanguageprocessingandmachinelearningisfunandexciting#10
25
Y(c=1)Y(c=2)
26
27
3. One-hot encoding
derekchia.com
28
#Token#1#2#3#4#5
29
0natural1000100010000000000000
30
1language0101000001000100000000
31
2
processing
0010010100000010001000
32
3and0000001000101000000100
33
4machine0000000000010001010000
34
5learning0000000000000000100010
35
6is0000000000000000000001
36
7fun0000000000000000000000
37
8exciting0000000000000000000000
38
Y(c=1)
Y(c=2)
Y(c=1)
Y(c=2)
Y(c=3)Y(c=1)Y(c=2)Y(c=3)Y(c=4)Y(c=1)Y(c=2)Y(c=3)Y(c=4)Y(c=1)Y(c=2)Y(c=3)Y(c=4)
39
40
#Token#6#7#8#9#10
41
0natural0000000000000000000000
42
1language0000000000000000000000
43
2
processing
0000000000000000000000
44
3and0100000000000101000001
45
4machine0010001000000000000000
46
5learning1000000100010000000000
47
6is0001010000001000100000
48
7fun0000100010100000010010
49
8exciting0000000001000010001100
50
Y(c=1)
Y(c=2)
Y(c=3)Y(c=4)Y(c=1)Y(c=2)Y(c=3)Y(c=4)Y(c=1)Y(c=2)Y(c=3)Y(c=4)Y(c=1)Y(c=2)Y(c=3)Y(c=1)Y(c=2)
51
52
53
4. Skip-gram Model Architecture
derekchia.com
54
Forward Pass for #1
55
Parameters - Embedding size
10
56
Random initialisation (Range)
-1 to 1
57
58
Calculate Hidden Layer
59
#Token
Input - w_t
Weight 1 - W1
Hidden Layer - h
60
0natural10.236-0.9620.6860.785-0.454-0.833-0.7440.677-0.427-0.0660.236
61
1language0-0.9070.8940.2250.673-0.579-0.4280.6850.973-0.070-0.811-0.962
62
2
processing
0-0.5760.658-0.582-0.1120.6620.051-0.401-0.921-0.1580.5290.686
63
3and00.5170.4360.092-0.835-0.444-0.9050.8790.3030.332-0.2750.785
64
4machine0np.dot0.859-0.8900.6510.185-0.511-0.4560.377-0.2740.182-0.237=-0.454
65
5learning00.368-0.867-0.301-0.2220.6300.8080.088-0.902-0.450-0.408-0.833
66
6is00.7280.2770.4390.138-0.943-0.4090.687-0.215-0.8070.612-0.744
67
7fun00.593-0.6990.0200.142-0.638-0.6330.3440.8680.9130.4290.677
68
8exciting00.447-0.810-0.061-0.4950.794-0.064-0.817-0.408-0.2860.149-0.427
69
1 x 99 x 10-0.066
70
1 x 10
71
72
Calculate y_pred
73
Hidden Layer - h
Weight 2 - W2
Output Layer
Softmax - y_pred
74
0.236-0.868-0.406-0.288-0.016-0.5600.1790.0990.438-0.5511.2580.218
75
-0.962-0.3950.8900.685-0.3290.218-0.852-0.9190.6650.968-1.3690.016
76
0.686-0.1280.685-0.8280.709-0.4200.057-0.2120.728-0.690-1.8280.010
77
0.7850.8810.2380.0180.6220.936-0.4420.9360.586-0.0201.1960.205
78
-0.454np.dot-0.4780.2400.820-0.7310.260-0.989-0.6260.796-0.599=0.5450.107
79
-0.8330.6790.721-0.1110.083-0.7380.2270.5600.9290.0171.1130.189
80
-0.744-0.6900.9070.464-0.022-0.005-0.004-0.4250.2990.7571.3330.235
81
0.677-0.0540.397-0.017-0.563-0.5510.465-0.596-0.413-0.395-1.5280.013
82
-0.427-0.8380.053-0.160-0.164-0.6710.140-0.1490.7080.425-2.3350.006
83
-0.0660.096-0.995-0.3130.881-0.402-0.631-0.6600.1840.4871 x 91 x 9
84
1 x 1010 x 9
85
86
Error For #1
derekchia.com
87
y_predw_c = 1y_pred
w_c = 2
EI
88
SoftmaxTokenDiff
Softmax
TokenDiff
Sum of Diff
89
0.2180natural0.2180.2180natural0.2180.436
90
0.0161language-0.9840.0160language0.016-0.968
91
0.0100processing0.0100.0101processing-0.990-0.980
92
0.2050and0.2050.2050and0.2050.411
93
0.1070machine0.1070.1070machine0.1070.214
94
0.1890learning0.1890.1890learning0.1890.378
95
0.2350is0.2350.2350is0.2350.471
96
0.0130fun0.0130.0130fun0.0130.027
97
0.0060exciting0.0060.0060exciting0.0060.012
98
99
100
Backpropagation for #1
derekchia.com
Loading...