Introduction

As the first step toward measuring TEAMMATES scalability, we measure the performance of TM up to 100k users.

First, we prefer not to do scalability testing on the live site as it will affect current users and there are not enough data for testing. Our strategy here is to create multiple clones of TEAMMATES with different number of users (i.e., 1k, 5k, 10k, 50k and 100k users), run performance tests on each of these clones, and analyze results.

Automated data creation

TM already has a REST API called BackDoor which supports importing data sent in JSON format into the system. Therefore, we created a script named DataGenerator.java to create a JSON file that had various numbers of imaginary users with random data.

In the test data created, each coordinator has 0-5 classes of size 30-300 stduents (with median around 100). Team size is from 3-5 students.

The total size of JSON data bundle files for 100k app is more than 200MB.

Bulk importing data

There are some limitations to exporting a large number of data items using the BackDoor API.

1.Eclipse stops working for really big data sets due to lack of memory.

2. BackDoor throws a DeadlineExceededException if the request cannot be completed within 60 seconds.

Therefore, the script used for importing data (ImportData.java) breaks the data set into smaller ones and import them one by one.

Performance profiling

Performance profiling is done using another script called PerformanceProfiler.java. This script uses Java Reflection and Annotations to measure time taken by various methods. The script run the annotated methods and records the time taken for each. The results are then appended (rather than overwritten) to another file for later analysis.

The performance profiling measures performance of UI interactions as well as backend operations. In addition, we also wrote a script (called ParallelProfiler.java) to simulate multiple users accessing the application in parallel.

Evaluation

Here are the average time per request when sending one request at a time. The number of runs for each clone is between 5-10.

Table 1: Performances of clones (in milliseconds)

	1k	5k	10k	50k	100k
BD (BackDoor) delete coordinator	524 ms	355 ms	459 ms	442 ms	448 ms
BD delete course	414	473	476	484	490
BD delete evaluation	390	363	551	460	434
BD create coordinator	1126	683	870	630	822
BD create course	497	471	475	596	564
BD create evaluation	618	587	650	795	678
BD create student	455	502	539	573	540
BD delete student	510	538	404	562	530
BD edit evaluation	361	341	488	377	382
BD edit student	415	378	541	556	579
BD edit submission	405	470	439	532	448
BD get coordinator	341	365	425	398	469
BD get course	336	419	405	478	405
BD get courses by coordinator	377	356	424	444	512
BD get evaluation	473	332	424	461	376
BD get key for student	352	375	420	510	566
BD get student	510	461	414	396	470
BD get submission	356	423	428	474	400
Coordinator (Coord ) add course	1425	1582	1618	1760	1816
Coord add evaluation	4582	4905	4636	4213	4131
Coord course delete student	2387	2054	2275	2258	2241
Coord course enroll page	1699	1904	1933	1747	1801
Coord course enroll student	1311	1117	1403	1209	1589
Coord course page	1721	1341	2569	1546	1287
Coord course student detail page	1227	1179	1379	1040	1255
Coord delete course	1265	1271	2288	1291	1327
Coord delete evaluation	2127	2018	2454	3040	2329
Coord view eval page	2194	2252	2219	2192	2082
Coord view eval results	1887	2030	1343	2074	1814
Coord view help page	1271	1471	2369	1033	1095
Coord log out	2490	2435	5446	2647	2796
Coord login	5061	6049	1754	4535	5161
Coord view student eval	1480	1741	2109	1726	1514
Student view course detail page	1874	1886	1887	1952	2311
Student edit submission	1824	1455	1343	1903	2827
Student edit submission page	1244	1290	1597	1357	1489
Student view eval result	1443	1505	1741	2034	2668
Student view homepage	1533	1461	2053	2453	1856
Student log out	2322	2393	2430	2588	2760
Student login	8173	7850	7470	8507	8709

When profiled using 10 parallel threads (running in 3 computers), the results were similar to the above numbers.

Findings

Based on the above data, TM seems to scale well for up to 100k users. There is not much difference in terms of performance between the clones, usually less than a few hundred of milliseconds for backend interaction requests and less than 1 second for browser actions such as user login, logout, create, edit, delete course, evaluation or submission). All backend interaction requests finish in less than 1 second while all the browser actions too no more that than 10 seconds. When hit with multiple requests in parallel, GAE handles the situation well by automatically creating new instances if needed.

After running these scalability tests for all the clones, we confirmed that the current system is able to handle 100,000 users with reasonable performance.

Future work

Repeat the study after major changes to the system.
Extend this study to measure space usage.

---end of report---