1 of 14

Datasets – Moving Target Selection with Delay

Shengmei Liu

Mark Claypool

Worcester Polytechnic Institute

USA

Ragnhild Eg

Kjetil Raaen

Kristiania University College

Norway

Andy Cockburn

University of Canterbury

New Zealand

Carl Gutwin

University of Saskatchewan

Canada

2 of 14

Delay and Interactivity

Delay as low as 1-2 ms can be perceived by humans
Delay around 50 ms is known to affect performance in mouse-based pointing tasks

2

3 of 14

Why Moving Target Selection?

3

[Call of Duty, Activision, 2003]

[Duck Hunt, Nintendo, 1984]

[League of Legends, Riot Games, 2009]

…

4 of 14

Puck Hunt�The Game of Selecting a Moving Target

Time to select puck with mouse
Number of clicks

5 iterations
1 QoE for each combo

4

132 users
167 trials

To gather data on moving target selection, we built two games that allows for the study with controlled amounts of delay, puck hunt and juke. In the 4 datasets, 3 are obtained from puck hunt and 1 is obtained from Juke.

In Puck Hunt, as indicated in the picture, the objective is to select the moving target, a bouncing black ball that moves with a speed. The user proceeds through a series of short rounds, each time using the mouse or game controller thumbstick to control the red ball on screen and move it over on the target and select it with a click.

With the game, players use a mouse to control the red ball in the 2 datasets - Mouse-Aand Mouse-B, and a game controller in Thumbstick dataset. The target moves at three different speeds as indicated in the table on top. Added delay varies between 11values as indicated in the table at the bottom. Each combination of speed and delay is repeated five times, completely randomized. To assess quality of experience, a single 5-point Likert-scale question “Rate the quality of experience of the last round” is presented once for each combination. Objective measures of performance recorded are the elapsed time to select the target and the number of clicks required to do so.

There are 132 participants on this game in total, and each played 167 trials.

5 of 14

Juke!�The Game of Selecting a Dodging Target

Time to select target
Distance from target

5 iterations each
1 QoE for each combo

(+50 milliseconds base delay)

53 users
223 trials

The fourth dataset is from a custom game called Juke!. It also allows study of target selection with controlled amounts of delay like Puck Hunt. Juke! has the same objective, to select the moving target, a moving red ball. However, Juke!’s target movement is governed by force-based physics rule, turn angle and turn frequency.

The target moves with force-based physics, accelerating in the target’s intended direction and with a set limit on maximum speed. The target turn interval varies between 3 values and the target turn angle between 4 values as indicated in the table. The game adds a fixed amount of delay selected from 4different values from 0 to 250ms. Each combination of jink interval, angle & delay appears 5 times, completely randomized. A single 5-point Likert-scale question (“How much lag did you experience”) is presented once for each combination. Objective measures of recorded performance include elapsed time before clicking the mouse and distance between the mouse and the target when clicked.

For this game, we have 53 users in total, each played 223 trials.

6 of 14

Datasets

6

7 of 14

Demographics

7

Next section is demographics information of our users. This table provides a summary of the main user variables in each dataset, with the columns as follows: “Dataset” denotes the source; “Age” is the mean participant age in years, with the standard deviation inparentheses; “Gender” gives the breakdown of number of malesa nd females, with “?” indicating “other” or “prefer not to say”; and “Skill” provides the self-rated skill, which is access by a 5-point Likert sacle question “rate yourself as a computer gamer”. In total we have 185 users, and 74% are males.

The mean self-rating was all above 3.5, showing a slight skew of “high ability”. According to our pervious work regarding player skill, self-rated score is a good indicator for actual player performance.

The demographic questionnaire included a question on participant age. Graph on the left shows boxplots depicting the age distribution for each dataset. Each box depicts quartiles and median, with the red pluses show the mean value. Points depicted by the dots are the outliers. The whiskers span from the minimum non-outlier to the maximum non-outlier. Most users were relatively young around 20, reflecting the typical ages of undergraduate and graduate students from the universities where the participants were recruited.

The graph on the right shows a stacked bar chart depicting the total number of participants in each dataset, with the blue and pink region having numbers and corresponding percents for males and females,respectively. The gray region are others.

8 of 14

Example (Devices)

8

Selection time increases exponentially with latency
Thumbstick takes 2x mouse for the same delays

The datasets have great potential in using. I will propose 4 examples.

The first example is on devices.The Mouse-A and Thumbstick datasets have identical user study parameters except for the selection device – mouse or game con-roller thumbstick. This allows for a comparison of the effects of the two different pointing devices have on selection times. This two graphs depict time versus delay, with data grouped by target speed. The x-axes are the total input delays and the y-axes are the times to select the moving target. The trendlines are for each pointing device tested. Each point is the mean time for all users for that device, delay & speed combination, shown with a 95% confidence interval. Overall, there is an increase in mean selection time as delay increases. This increase appears exponential over the range of delays tested. Comparing the thumbstick data to the mouse data shows the thumbstick as a selection device takes about twice as long as the mouse for the same delays. Comparison of the two graphs shows this 2x elapsed time holds irrespective of target speed.

9 of 14

Example (Demographics)

9

Self-rated score can be indicator for player performance

Also, Responses to the datasets’ demographics questions can be used to answer queries related to user background experience and attributes in relation to game-specific data. Since each dataset included the question “Rate your ability as a computer gamer (1 - low to 5 - high)”, correlations between answers to this question and in-game performance may indicate if self-rated game skills are reasonable indicators of actual game performance. We grouped participants’ normalized target selection times by their self-ratings of ability. This graph shows boxplots of normalized elapsed time on the y-axis for users clustered by self-rating on the x-axis. The “n=” labels on the x-axis indicate the number of participants in each self-rating group. From the graph, the mean and median normalized elapsed times improve linearly with self-rated skill. However, the spread indicated by the boxes shows that some individuals with lower self-ratings performed better than others with higher self-ratings. An one-way between subjects ANOVA test shows there was a significant effect of self-rated skill on elapsed time at the 0.05 significance level for the five conditions.

10 of 14

Example (Comparisons)

10

High-velocity targets are sensitive to delay

Results from the target selection datasets can also be compared to prior studies.

Long and Gutwin measured selection time for two different sized targets with delays from 49 to 299 ms. They found significant main effects for delay on selection time and that the effects of delay are amplified by fast target speeds

The similarities in test settings from the study and mouse-A mouse-B allows for a direct, visual comparison of results shown in the graph. The x-axis is the delay (in ms), the y-axis is the elapsed time to select the target (in ms). Each point is the average time to select the target at the indicated delay.

Overall, the results across the three studies present a consistent picture of a higher selection time with a higher target speed. Most of the data shows a super-linear relationship with an increase in delay. The super-linear increase occurs at different delay amounts, with faster targets seeing a rise at lower delays than slower targets. This means that high-speed targets are extremely sensitive to delay.

11 of 14

Example (Modeling)

11

Selection times can be modeled as lognormal

The datasets can be used to model the distribution of the times to select moving targets, providing for models to be used in simulations. We illustrate this by modeling the distribution times for the time to select a target with a mouse using the Mouse-A and Mouse-B datasets.

Since the distribution of elapsed times appears log-normal, we use multivariate, multiple regression to model the mean and standard deviations of the logarithm of the elapsed times. This allows generation of normal distributions of the log of selection times which can then be exponentially expanded to generate a distribution of elapsed times, useful for simulations.

This graph shows the model generated data compared to the actualdata. The x-axis is the elapsed time in seconds, and the y-axisis the probability distribution. The blue shape is the the elapsed time for the empirical data, and the red shape is generated from the normal distribution using the modeled mean and standard deviation. The model has an excellent fit for the data with 𝑅2of 0.99 and RMSE of 0.03.

12 of 14

Future work�

12

Navigation

Selection

Simulations

Analysis of Performance in FPS game

FPS Gaming Actions

Data

Models

13 of 14

Acknowledgements

Marco Duran and Matthew Thompson

Conducting the Mouse-B study

Robyn Domanico and Linh Hoang

Conducting the Thumbstick study

Chaiwat Ekkaewnumchai and Bhon Bunnag

Developing Juke!
Conducting the Motion study

Users

Participating

13

14 of 14