Word2Vec Implementation

	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
1
2	1. Corpus
3	natural language processing and machine learning is fun and exciting
4
5	2. Sliding Window											derekchia.com		Created by:	Derek Chia
6	#1	natural	language	processing	and	machine	learning	is	fun	and	exciting	#1		Twitter:	@derekchia
7	#1	Xĸ	Y(c=1)	Y(c=2)								#1		Email:	derek@derekchia.com
8	#2	natural	language	processing	and	machine	learning	is	fun	and	exciting	#2		Blog post:	https://medium.com/@derekchia/an-implementation-guide-to-word2vec-using-numpy-and-google-sheets-13445eebd281
9	#2	Y(c=1)	Xĸ	Y(c=2)	Y(c=3)							#2
10	#3	natural	language	processing	and	machine	learning	is	fun	and	exciting	#3
11	#3	Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)						#3
12	#4	natural	language	processing	and	machine	learning	is	fun	and	exciting	#4
13	#4		Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)					#4
14	#5	natural	language	processing	and	machine	learning	is	fun	and	exciting	#5
15	#5			Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)				#5
16	#6	natural	language	processing	and	machine	learning	is	fun	and	exciting	#6
17	#6				Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)			#6
18	#7	natural	language	processing	and	machine	learning	is	fun	and	exciting	#7
19	#7					Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)		#7
20	#8	natural	language	processing	and	machine	learning	is	fun	and	exciting	#8
21	#8						Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	Y(c=4)	#8
22	#9	natural	language	processing	and	machine	learning	is	fun	and	exciting	#9
23	#9							Y(c=1)	Y(c=2)	Xĸ	Y(c=3)	#9
24	#10	natural	language	processing	and	machine	learning	is	fun	and	exciting	#10
25	#10								Y(c=1)	Y(c=2)	Xĸ	#10
26
27	3. One-hot encoding																								derekchia.com
28	#	Token	#1			#2				#3					#4					#5
29	0	natural	1	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0
30	1	language	0	1	0	1	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0
31	2	processing	0	0	1	0	0	1	0	1	0	0	0	0	0	0	1	0	0	0	1	0	0	0
32	3	and	0	0	0	0	0	0	1	0	0	0	1	0	1	0	0	0	0	0	0	1	0	0
33	4	machine	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	1	0	0	0	0
34	5	learning	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0
35	6	is	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
36	7	fun	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
37	8	exciting	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
38			Xĸ	Y(c=1)	Y(c=2)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)
39
40	#	Token	#6					#7					#8					#9				#10
41	0	natural	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
42	1	language	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
43	2	processing	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
44	3	and	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	1
45	4	machine	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
46	5	learning	1	0	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0
47	6	is	0	0	0	1	0	1	0	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0
48	7	fun	0	0	0	0	1	0	0	0	1	0	1	0	0	0	0	0	0	1	0	0	1	0
49	8	exciting	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	0	0	1	1	0	0
50			Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Y(c=4)	Xĸ	Y(c=1)	Y(c=2)	Y(c=3)	Xĸ	Y(c=1)	Y(c=2)
51
52
53	4. Skip-gram Model Architecture																derekchia.com
54	Forward Pass for #1
55	Parameters - Embedding size				10
56	Random initialisation (Range)				-1 to 1
57
58	Calculate Hidden Layer
59	#	Token	Input - w_t		Weight 1 - W1											Hidden Layer - h
60	0	natural	1		0.236	-0.962	0.686	0.785	-0.454	-0.833	-0.744	0.677	-0.427	-0.066		0.236
61	1	language	0		-0.907	0.894	0.225	0.673	-0.579	-0.428	0.685	0.973	-0.070	-0.811		-0.962
62	2	processing	0		-0.576	0.658	-0.582	-0.112	0.662	0.051	-0.401	-0.921	-0.158	0.529		0.686
63	3	and	0		0.517	0.436	0.092	-0.835	-0.444	-0.905	0.879	0.303	0.332	-0.275		0.785
64	4	machine	0	np.dot	0.859	-0.890	0.651	0.185	-0.511	-0.456	0.377	-0.274	0.182	-0.237	=	-0.454
65	5	learning	0		0.368	-0.867	-0.301	-0.222	0.630	0.808	0.088	-0.902	-0.450	-0.408		-0.833
66	6	is	0		0.728	0.277	0.439	0.138	-0.943	-0.409	0.687	-0.215	-0.807	0.612		-0.744
67	7	fun	0		0.593	-0.699	0.020	0.142	-0.638	-0.633	0.344	0.868	0.913	0.429		0.677
68	8	exciting	0		0.447	-0.810	-0.061	-0.495	0.794	-0.064	-0.817	-0.408	-0.286	0.149		-0.427
69			1 x 9		9 x 10											-0.066
70																1 x 10
71
72	Calculate y_pred
73		Hidden Layer - h		Weight 2 - W2										Output Layer		Softmax - y_pred
74		0.236		-0.868	-0.406	-0.288	-0.016	-0.560	0.179	0.099	0.438	-0.551		1.258		0.218
75		-0.962		-0.395	0.890	0.685	-0.329	0.218	-0.852	-0.919	0.665	0.968		-1.369		0.016
76		0.686		-0.128	0.685	-0.828	0.709	-0.420	0.057	-0.212	0.728	-0.690		-1.828		0.010
77		0.785		0.881	0.238	0.018	0.622	0.936	-0.442	0.936	0.586	-0.020		1.196		0.205
78		-0.454	np.dot	-0.478	0.240	0.820	-0.731	0.260	-0.989	-0.626	0.796	-0.599	=	0.545		0.107
79		-0.833		0.679	0.721	-0.111	0.083	-0.738	0.227	0.560	0.929	0.017		1.113		0.189
80		-0.744		-0.690	0.907	0.464	-0.022	-0.005	-0.004	-0.425	0.299	0.757		1.333		0.235
81		0.677		-0.054	0.397	-0.017	-0.563	-0.551	0.465	-0.596	-0.413	-0.395		-1.528		0.013
82		-0.427		-0.838	0.053	-0.160	-0.164	-0.671	0.140	-0.149	0.708	0.425		-2.335		0.006
83		-0.066		0.096	-0.995	-0.313	0.881	-0.402	-0.631	-0.660	0.184	0.487		1 x 9		1 x 9
84		1 x 10		10 x 9
85
86	Error For #1													derekchia.com
87		y_pred	w_c = 1				y_pred	w_c = 2				EI
88		Softmax		Token	Diff		Softmax		Token	Diff		Sum of Diff
89		0.218	0	natural	0.218		0.218	0	natural	0.218		0.436
90		0.016	1	language	-0.984		0.016	0	language	0.016		-0.968
91		0.010	0	processing	0.010		0.010	1	processing	-0.990		-0.980
92		0.205	0	and	0.205		0.205	0	and	0.205		0.411
93		0.107	0	machine	0.107		0.107	0	machine	0.107		0.214
94		0.189	0	learning	0.189		0.189	0	learning	0.189		0.378
95		0.235	0	is	0.235		0.235	0	is	0.235		0.471
96		0.013	0	fun	0.013		0.013	0	fun	0.013		0.027
97		0.006	0	exciting	0.006		0.006	0	exciting	0.006		0.012
98
99
100	Backpropagation for #1															derekchia.com