Data/Analysis Report
DES308 - Analytics & Data Driven Design
Natan Walentin-Konieczny, 1801270
Contents Page
Market Analysis............................................................................................................................................................................Pg.3
Game Design...............................................................................................................................................................................Pg.3
Hooks and KPI’s...........................................................................................................................................................................Pg.4
Integration of Data Collection.......................................................................................................................................................Pg.5
Outcomes of Playtesting..............................................................................................................................................................Pg.6
Analysis and Interpretation...........................................................................................................................................................Pg.7
Version 1...........................................................................................................................................................................Pg.7
Version 2...........................................................................................................................................................................Pg.10
Version 3...........................................................................................................................................................................Pg.14
Version 4...........................................................................................................................................................................Pg.16
How did this inform my game design?..........................................................................................................................................Pg.17
Market analysis
My EndlessRunnerDes308 would enter - as the name suggests - the highly popular endless runner market. Ideally the product would release on the mobile market due to the higher accessibility, larger player base and overall popularity on said platform. For the purposes of the project, the game will be released on PC - the PC market does contain a small following for the endless runner genre however not notably as close as that of Mobile.
The most notable competitors being:
All these games follow a very similar overall design and model, being F2P and mostly differing by ‘’Theme’’ rather than rules or mechanics. On average play sessions can last around 5-10 minutes along with a wide target audience starting from players as young as 9+ years old (depending on theme).
My design will follow the same approach and aim for similar statistics while once again differing in its theme.
Game Design
A western themed third person, 3D endless runner. The goal is to obtain a high score within a single turn while dodging a various amount of randomly spawned obstacles which have pre-designed layouts, with combinations of jumps and dashes to either left or right on a 3 lane track using their keyboard.
On their way the player can pick up bottles which will give them extra lives (maximum of 3 at a time). Each life will be taken away upon collision. If the player collides with an obstacle with no lives left then their turn will be over.
Hooks and KPI’s
Score (at death) - Will enable me to gauge an initial picture of the difficulty level of the build. Extremely high (recurring and higher than expected) scores will show that the difficulty has been impacted negatively (too easy) and vice versa.
Time (of run) - Enables to gauge how long players are spending per run. I.e if single runs are taking up a large amount of time then it is more likely that later engagement may fall (number of attempts per session may drop) - This is to do with balancing and engagement/retention of the player. The sum amount ‘time’ outputs, will show the total time the players spent playing X version of the game (also in relation to engagement, and later - retention).
Speed level reached (at death) - Parallel with 'score'. Can be used to see if a particular speed level is problematic to players (i.e if the speed increase begins to become too high). A user funnell shows the pass rate of players in the current session. This hook can also be counted to visualise the amount of deaths per speed level which could potentially indicate whether a certain speed level is too high.
Object (Collided with) - Can indicate numerically whether a specific object is a major cause of deaths within the game version. This along with qualitative data can indicate what the problem (if any) is with said obstacle.
Death Position - Can indicate if players are dying at a substantial rate within a single lane. Can impact engagement if players are not moving lanes (could suggest that players are not moving lanes).
Evolution
Unique User ID - By version 2, ID’s were used for separating data coherently and for potential segmentation/cohorts - completely unique code which does not allow for identification of personal information.
Life's Collected (per run) - By version 3, coins were changed to serve as ''an extra life''. This can be used to observe the potential impact of the collectible compared with other KPI’s.
All of the hooks can be compared against each other to visualise correlations of potential discrepancies as required by the design of the application. Qualitative data collected via google forms can also indicate potential issues which can be later compared against above KPI’s.
Each entry of data indicates a single turn/run.
Integration of Data Collection
Telemetry was achieved with a custom manual data collection method, Implemented into my build manually using internal code/scripts.
The use of external plugin data collection software such as ‘’Unity Analytics’’ would’ve been beneficial due to their versatility and extensive data collection features/solutions, along with an in-depth automatic dashboard.
Ultimately I decided to not use any of these due to the time limitations of the project, and the long wait times for data to pass through and update onto the dashboard.
To substitute, I used the supplied ‘’Data Recorder’’ script which creates and writes ‘.csv’ files with any attached data points. All hooks are collected in various parts of the project’s source code (i.e different scripts). Therefore, I modified it to satisfy my design needs with the Data Recorder being called each time the player dies, meaning that all of the above hook (data) points are collected and written into the file at the end of a turn.
To automate and simplify the data collection process, a ‘’send email’’ script was used. The purpose of the script is for the application to send out an email to a specified receiver email address with the ‘Data Recorder’ data file as an attachment.
To not infringe upon any privacy and maintain security a throwaway google mail account was created for this purpose.
This allowed me to easily collect player data and apply it into my custom dashboard created on Google Sheets. A unique user ID system enabled the data to be separated and segmented as required.
Quantitative player data was collected using Google forms which would evolve along with the changing design of the application and would be filled out each time the players finished testing. Adjusted forms would be created for FTU’s.
Outcomes of Playtesting
Test session 1 was conducted by a total of 6 FTU’s. Results indicated that the overall FTUE seemed fair, yet further investigation would be required to observe and improve the overall onboarding, as users mentioned controls as ‘confusing’.
Testers were discouraged and uniformly pinpointed a slow movement speed of the character - an unbalanced large portion of deaths occurred during lane switching.
Users were discouraged by obstacle ambiguity, and were not fully sure how to tackle certain obstacles with many early deaths being caused predominantly by the ‘FallenTree’. Regardless, players were able to reach fairly high speed levels with satisfactory engagement levels.
The UI seemed to be a neutral element to the testers, however, it potentially suggested that collectibles are not appealing enough to collect. Further testing was required.
Test session 2 was conducted with now 6 recurring users from the previous session with an additional 3 new FTU’s.
Feedback has shown that users appreciate the increased movement speed, yet still, engagement levels have dropped evenly across the board. Difficulty also seems to have unintentionally increased, with less users achieving higher speed levels.
The new obstacles also seem to have encountered similar issues as before, with players not immediately knowing how to tackle them - a single obstacle being the culprit to an unbalanced majority of deaths.
FTUE testing confirmed that initial onboarding can be confusing regarding controls, and that initial turns can be heavily impacted by this with early turns ending with deaths prematurely.
Test session 3 was conducted with the same 6 recurring users, with 3 additional FTU’s.
New changes have now decreased the difficulty level dramatically, with players achieving record high scores early on, and afterwards stopping to play. Hard to pinpoint the exact reason behind this - changes should have been more spread out to pinpoint the source of the issue however a speculation was made. Player feedback has also indicated a drop in game difficulty.
FTUE testing has shown that the new ‘how-to-play’ screen and additional controls have improved overall onboarding, with players not reporting issues with control confusion or any similar issues. Initial scores have also improved.
The issue with obstacle ambiguity also has been addressed with the new simple colour coded replacements, along with death distribution being more balanced.
Testing was carried out by 3 recurring users.
Difficulty seemed to have balanced out to a more appropriate level with an improved rate of engagement.
Qualitative data suggested that users had initial issues with the controls, even if Quantitative data indicated an adequate level of onboarding.
** Scale used: 1- Completely Uncomfortable 7- Very Intuitive
As the impact of this could not be tested, then a second FTUE test would be required.
Further qualitative data indicated that users were displeased with the speed of lane-changing.
Although this data indicated that the majority of the deaths were indeed during lane-changing, it would not show for definite that slow speed was the culprit. The same KPI was used to see the distribution at higher speed levels where this would have a higher impact:
The potential impact on the deaths that occurred during lane-change was observed:
** Scale used: 1- Bad 7- Perfect!
At higher speed levels, change-lane speed was an issue with deaths being primarily caused by it.
Further analysis showed that deaths caused by the ‘FallenTree’ mostly occurred in early speed levels, suggesting that players may not be immediately understanding how to tackle the obstacle causing premature failures.
Qualitative data also remarked issues with certain obstacle ambiguity. Investigation indicated that a vast majority of deaths were being caused by the ‘FallenTree’
Player feedback also suggested that even though players did not find the UI as intrusive, their responses would indicate that collectibles that are directly tied into it were not found appealing.
However data indicated an unexpected increase in difficulty with players no longer being able to reach the same speed levels as before.
The increased lane-change speed was overall positively approved by testers, With lane-change death distribution being balanced out improving the general UX.
** Scale used: 1- Too Slow 7- Too Fast
The level of engagement also decreased from the previous version:
Even with a higher churn rate, more deaths occured in early speed levels than before, which was potentially the impact on engagement.
Further investigation suggested that a speculative/potential reason behind this was the new obstacles designs/layouts as both qualitative feedback and data showed little to no improvement with ‘obstacle ambiguity’. Without further testing it was hard to pinpoint as too many factors were changed.
(Yet, since less complaints were made, the new layout approach was considered as well-accepted)
Feedback also suggested that players did not have an incentive to collect the coins, making them a fairly uninteresting/unimpactful feature.
FTUE testing showed similar results, with onboarding being worse than before in terms of initial scores.
FTU feedback also did not improve, with testers having issues with grasping controls.
It did suggest however, that the current obstacles were harder to overcome even with an improved maneuverability hence the spike in difficulty and lowered engagement.
This did suggest however, that the current obstacles were harder to overcome even with an improved maneuverability hence the spike in difficulty and lowered engagement.
Testing shows that although engagement has improved to an extent with a play time increase, Difficulty has dropped dramatically with players reaching much higher speed levels.
Players are no longer taking as many attempts as before which was unexpected. Even with a higher initial engagement, later retention can drop if the game isn't challenging.
Feedback also indicates the same, with players uniformly agreeing that the game has become too easy.
Further investigation indicates that players reach an incredibly high score on their last turn before quitting. This could indicate that potentially the ‘’intensity’’ of the game is not appealing enough to keep going with an end sense of ‘’completion’’.
The 2 additional features that could potentially play into this (as I suspect) would be the new collectibles and increase in distance between obstacles with each level. However, as an error on my behalf it is hard to pinpoint due to the amount of changed factors.
Data shows that on last attempts, players seem to obtain a significant amount of bottles, this could mean that bottles are spawning too frequently, making the ability to dodge failure simple.
Yet such a high pickup rate may indicate that the collectibles are more appealing than before.
Feedback also suggested the same
On the contrary, obstacle ambiguity seems to have been tackled, with obstacle death distribution being more acceptable and feedback indicating that players know how to overcome all of them.
FTUE testing also shows that onboarding has improved with players no longer commenting on control issues.
Although data is limited, the last retest indicates that balancing has indeed improved the difficulty of the game, which also seemed to improve engagement across the board.
More testers would have been beneficial to properly analyse, however the funnel shows that balancing the the rate of bottles and distance between obstacles could improve it further.
How did this inform my game design?
*Bracket timestamps refer to evolution video.
While data indicated a clear issue with Obstacle ambiguity (with ‘FallenTree’ in particular) I decided to redesign the obstacles entirely into a series of pre-design layouts instead of spawning singular random obstacles per lane (randomly) as this approach made it difficult to observe potential issues with any obstacle. The pre design layouts would also be more appealing visually with an attempt to clearly convey necessary feedback on how to tackle them. This could enhance the UX but also would be able to observe any (recurring, or if any) potential issues - the decrease in obstacles was to narrow down a window of error (2:32).
Feedback and data also enabled me to increase the maneuverability of the character, in order to balance the game difficulty and decrease the amount of unnecessary deaths caused by lane switching (2:18).
‘’Obstacle Ambiguity’’ remained a prevalent issue with the newly laid out obstacles. Unexpectedly, game difficulty also appeared to have jumped significantly as compared to the previous version. As such aspects have an impact on the overall UX and can hinder retention, further balancing was required. I decided to potentially re-think the overall visual design of the obstacles and their range (i.e level of difficulty to overcome) as my analysis hinted that they were simply too hard to beat/dodge even with the increased movement speed at early levels. To accurately convey the necessary information on how to overcome them, the obstacles were simplified and colour coded, additional obstacles were also added in to balance out the death distribution and enhance the UX (3:52) - lastly, obstacle distancing was added each time a speed level is reached in order to balance out the difficulty at higher speeds(4:44).
FTUE testing showed that onboarding was also potentially hindered by this along with an also prevalent issue regarding controls. To improve game onboarding, a ‘’How-to-Play’’ Screen was added which also included information regarding obstacles and how to tackle them with the aim of further balancing the difficulty and improving engagement(3:21).
In addition - Coins were replaced with ‘’Bottles’’ which would give the player extra health (total of 3 at a time) in order to make collectibles more appealing and have players take more risks in obtaining them (5:19).
Based on the data that I collected, I was able to perform substantial changes to the design of my game throughout each iteration with the aim of improving engagement levels further, while balancing the game difficulty and enhancing the UX.
While onboarding and engagement improved to an extent, the difficulty dropped dramatically. Data suggested that the potential reasons behind this was the frequency of the newly added bottle pickups and obstacle spacing being too high. Major balancing was implemented, where the frequency of bottle spawns along with distance between obstacles each level was decreased(6:04).
Even with a lack of testers, data still indicated that the assumptions were correct and balancing made the game more appropriately difficult. At this stage, further tuning and extra testers would be required to fully balance out the game with the current feature set.
Due to the time limitations and approach error on my behalf, large changes were made to each iteration of the game. Each change made it difficult to pinpoint exact causations behind discrepancies. Ideally these changes should have been more spread out and analysed more thoroughly which could have improved the way I have informed my game design.
Link to video: https://youtu.be/qd3R5CPSIQk
Timestamps refer to this.