udp socket stress test

	A	B	C	D	E	F	G
1	Description
2	The intent of this experiment is to give us a baseline of where we stand wrt to the performance of ChromeApps using the chrome.socket API. In this document, we focus on the UDP API, specifically on the ability to handle heavy incoming traffic and measuring packet loss.
3
4	tldr;
5	The socket v2 API is 10x faster than v1, but still 4x slower than node.js.
6
7	Experiment details
8	Given a Win32 c++ apps sending UDP packets to "localhost" as fast as possible (naively, nothing fancy), measure the amount of packets various single instance clients can receive at the application layer. Note that packets are fairly small (10 bytes). First, we run only one instance of the "sender" app, then we run 3 instances in parralel to ensure we can overflow to receiver. There are 4 types of "receiver" apps: 1) A Win32 C++ app is used as the baseline to measure "theoritical" (naive implementation, nothing too fancy) best performance on the "native" platform. 2) A node.js app is used as the baseline of what should be possible to achieve with a V8 based asynchronous JavaScript platform. 3) A chome app based on socket.udp v1 API. 4) A Chrome app based on an experimetal socket.udp v2 API. The main difference from v1 is the use of Chrome App Events instead of explicit read+callback operation to receive udp packets.
9
10	Summary of results
11	1) node.js performance is within a factor (~50% on average) of c++ performance 2) socket v1 is two order of magnitude (100x!) slower than c++, probably unacceptable for "real-world" server applications. 3) socket v2 is still one order of magnitude slower than c++ but one order of magnitude faster than v1 and much close(r) to node.js performance (25% of node performance). 4) Packet size (10 bytes vs 5,000 bytes) doesn't change the picture much. Bigger packets seem to slow down the "sender" process more than the "receiver", thus decreasing the amount of packet loss of the receivers.
12
13	Packet size = 15 bytes
14
15			C++ sender	C++ receiver	node.js receiver	udp socket v2	udp socket v1
16		1 sender instance
17		# send reqs	1,000,000
18		Time (s)	4.2
19
20		# recv reqs		1,000,000	550,000	117,909	10,982
21		% loss		0%	45%	88%	99%
22		# recv / sec		238,095	130,952	28,074	2,615
23		Relative perf vs c++		100%	55%	12%	1%
24		Relative perf vs node		182%	100%	21%	2%
25
26		3 sender instances running in parralel
27		# send reqs	3,000,000
28		Time (s)	5.58
29
30		# recv reqs		2,658,426	669,694	167,502	19,031
31		% loss		11%	78%	94%	99%
32		# recv / sec		476,420	120,017	30,018	3,411
33		Relative perf vs c++		100%	25%	6%	1%
34		Relative perf vs node		397%	100%	25%	3%
35
36
37	Packet size = 5000 bytes
38
39			C++ sender	C++ receiver	node.js receiver	udp socket v2	udp socket v1
40		1 sender instance
41		# send reqs	1,000,000
42		Time (s)	8.99
43
44		# recv reqs		998,223	681,306	157,071	14,951
45		% loss		0%	32%	84%	99%
46		# recv / sec		111,037	75,785	17,472	1,663
47		Relative perf vs c++		100%	68%	16%	1%
48		Relative perf vs node		147%	100%	23%	2%
49
50		3 sender instances running in parralel
51		# send reqs	3,000,000
52		Time (s)	24.35
53
54		# recv reqs		2,977,687	1,406,257	353,721	43,054
55		% loss		1%	53%	88%	99%
56		# recv / sec		122,287	57,752	14,527	1,768
57		Relative perf vs c++		100%	47%	12%	1%
58		Relative perf vs node		212%	100%	25%	3%