Illum - UX Research Report | Elsa Liu

September 2019

​

​

Illum

Contents

• Study Overview
• Executive Summary
• Detailed Results
• New User Experience
• Controls & Mechanics
• Narrative & Emotions
• Level Design & Puzzle Difficulty
• Pacing & Overall Experience
• Post-Playtest Survey Results
• Gameplay
• Controls
• Mechanics
• Level of Difficulty
• Art Style
• Narrative
• Atmosphere

​

​

​

​

​

​

​

​

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

Game Page Gameplay Walkthrough Behind the Scene

2

Study Overview

​

​

​

​

​

​

​

​

​

Study Method and Scope

​

This user research study focuses on identifying topline issues for the beta build of Illum, an indie game that is being developed in the class CTIN 532. This following report highlights actionable insights backed by behavioral evidence.

​

7 players have been recruited to playtest the game remotely, and their behaviors will be observed through Zoom. Each player will play the beta build from start to finish, approximately 15-20 minutes. During the session, players are encouraged to “think aloud”, keeping a running monologue of what they are thinking and doing.

​

The areas of focus of this study involve: new player experience, controls & mechanics, narrative & emotions, level design & puzzle difficulty, pacing & overall experience.

​

Players will also complete a post-playtest survey for their perceptive of the game, and an exit-interview for additional questions.

​

​

Illum

3

What Needs Work�

1. Scaffolding: The game provides limited guidance when players are confused or struggling, causing them to question their understanding of the mechanics.

​

• Wall Painting Narrative: The wall painting contents are vague and ambiguous to the players, causing a lack of understanding of the world mystery in the overall narrative.�
• World Exploration: The amount of explorative objects is very limited, reducing the player engagement and understanding of the backstory. �
• Negative Feedback: The game has no negative feedback to indicate when the players have done something incorrectly, causing confusion on not being able to perform controls.��Pace of Challenge: The pacing of difficulty between levels shifted quickly, causing confusion to some players in the second level where they receive very limited help in puzzle solving.

Executive Summary - Top Five Pros & Cons

What’s Working Well �

1. New User Experience: The game controls and mechanics are easily understood because players are given enough demonstration and practice when learning them.

​

• Emotional Connection: By interacting with the light friend, wall paintings, and cutscenes, the players establish a strong immersion and emotional connection with the game world.

​

• Immersion: The game create a world that is believable, immersive, and engaging, motivating the players to keep exploring and interacting with the environment.

​

• No Time Limit: The lack of an explicit time limit allows the players to enjoy the world’s details, while also having time to strategize and complete puzzles.

​

• Player Goals: Players understand intermediary and overall goals of the game through the narrative, visual, and audio clues presented to them.

​

​

​

​

​

​

​

​

​

​

1

2

3

4

5

1

2

3

4

5

4

Detailed Results ��Players’ behaviors and feedback of game elements during playtest

5

• Tutorial + Guidance: The players understand basic controls and in-game guidance when they are introduced via fade-in animations and environmental hints.
• Overall Goal: The players understand overall goal of game with the visual and audio clues.
• Limited Text: The lack of text allows the players to absorb the world and focus on gameplay, without having to rely on text for all learning.
• Graphics + Audio: The graphics and audio are visually appealing and immersive to the players.

​

​

​

• Game World Exploration: The amount of explorative objects is very limited, reducing the player engagement and motivation to interact with the world.
• Overall Narrative: The storyline is unclear to some players due to the lack of explanation, reducing the emotional connections between the players, light friend, and the game world.
• Slow Feedback: Due to the slow visual feedback, 2/7 players fail to light the wall paintings up before moving forward to the next wall painting.
• Character Movement: 1/7 players remark that character movement speed is too slow compared to the side of the game world they have to travel.

​

Suggestions:

• Consider providing more storyline through clear explorative wall paintings so players understand their mission of helping a light friend wake up a their sleeping kingdom and save other light balls.
• Consider speeding up the character movement a little bit to reduce the travel time, or adding more explorative objects to fill the empty spaces in-between levels.

​

​

​

New User Experience

"I wish there are more things to explore.” - P7

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

6

• Demonstration + Practice: The mechanics are demonstrated to the players, and the players are given enough chance to practice the mechanics in a risk-free environment before they enter the actual puzzles.
• Knowledge Transfer: The game practices industry standard controls while introducing unique mechanics with easy-to-use user interactions, reducing the burden on memory when learning to play.
• Punishment: The game does not punish players for failing to solve the puzzles, allowing them to change their strategy at any point of play.

​

​

​

​

• Scaffolding: The game provides limited to no guidance when players are stuck or confused in the puzzle, which became a point of frustration for players who want help.
• Accidental Solutions: In some cases, players can solve puzzles by accident, leaving them confused on what they did to reach the solution.
• Checkpoint Saving: One player accidentally turns off the computer while playing, and loses all game progression when comes back to the game due to the lack of save system on checkpoints.

​

​

Suggestions:

• Consider assisting players by highlighting important items when players are stuck on puzzle solving, or allowing them get clues from the light friend if they interact with it for help.
• Consider adding a save system that automatically saves player progression upon checkpoints to prevent accidental shutdown.

​

​

​

Controls & Mechanics

"I don’t know what I did, but the puzzle is solved.” - P4

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

7

• Emotional Connection: Once players understand the story, their gameplay feel more immersive and creates strong a sense of emotional connection with the light friend and the world as they moved through the game.
• Cutscenes: The cutscenes are visually appealing and tell the story clearly to the players, creating a sense of immersion and engagement.
• Light Friend Interaction: The emotions and relationship between the players and the light friend are clear to the players, although some players might not fully understand the overall narrative.

​

​

​

​

​

​

• Skippable Narrative: Because some of the wall paintings are explorative objects and not required for players to light up to finish the game, 1/7 players does not want to explore and thus misses skippable paintings, which causes confusion in the overall narrative.
• Wall Painting Understandability: 3/7 players, even though they light up and read every wall paintings, they are still unclear of the meanings behind each painting due to the ambiguity of the drawn contents, causing a lack of understanding of the mystery.
• Lack of Contents: 3/7 players remark that they want to see more wall paintings that tell story about the game world, such as culture, relationship, the mystery of why the kingdom falls dark.

​

​

Suggestions:

• Consider adding more explorative wall paintings that tell the mystery about the game world for players to explore and understand the story.
• Consider having more clarified wall paintings that do not confuse players.

​

​

​

Narrative and Emotions

"I didn't really understand the drawings on the walls and the relationship between the drawings and the light.“ - P1

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

8

• Indication + Guidance: The game gives clear lighting indication and guidance on interactable objects and possible path, motivating players to engage and explore in the game world.
• Immersion: The game create a world that is believable, immersive, and engaging by its characters, environments, visuals, and audio, motivating the players to keep exploring the in the game.
• Level of Difficulty: The puzzles have a easy to average level of difficulty and most players are able to solve the puzzles within the expected time length, creating a overall peaceful and relaxing gaming experience which is coherent with the game atmosphere.

​

​

​

​

​

• Negative Feedback: 2/7 players, when exploring the world, try to walk and jump onto the buildings that they are not able to reach and received no negative feedback. This caused confusion as players think they are not performing the controls correctly.
• Light Gem Difference: Some players have issue distinguishing static and moving light gems because of the unclear visual differences and the lack of feedback when interacting with the static gems.
• Camera Position: At certain levels, such as the stairs, the camera positions are set static and players are not able to adjust, causing confusion in camera control and create possible dead zones.
• Error Prevention: Players are forced to re-climb the stairs and the path again if they fall down from the upper levels by accident, causing unnecessary repetitive play.

​

Suggestions:

• Consider providing more distinctive visual or audio feedback to communicate when a player has done something incorrectly.
• Consider differentiating static and moving light gems by changing the colors or the shapes to make them distinguishable.
• Consider allowing players to adjust the camera positions within a small range, instead of making the camera fully static.

​

​

​

Level Design & Puzzle Difficulty

"I’m trying to attract this gem but it doesn’t move.” - P3, when trying to interact with static light gem and received no feedback.

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

9

• Slow Gameplay: 6/7 players enjoy the slow pace of the game as this allows them to enjoy the overall atmosphere, and the interaction between the light friends and the environment.
• No Time Limit: The lack of an explicit time limit allowed the players to enjoy the world’s details, while also having time to strategize and complete difficult puzzles.
• Trial and Error: The game allows players to try different styles when interacting with the environment, enabling players to make and learn from the mistakes without severe consequences.

​

​

​

​

​

• Pace of Challenge: The pacing of difficulty between levels shifted quickly, causing confusion to some players in the second level where they receive very limited help in puzzle solving.
• Lack of Progression: The game lacks a sense of progression because of the limited levels it contains, reducing player satisfaction of completing puzzles and the understanding of the story curve.

​

​

​

​

​

Suggestions:

• Consider adding more levels while increasing the difficulty as players progress through the game, making players feel smart and satisfied when successfully solving puzzles. This also helps uncovering the mystery gradually.

​

​

​

Pacing & Overall Experience

"I wish there are more levels.I think the game has a great concept and story, and has the potential to expand.” - P5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

10

Post-Playtest Survey Results ��Players’ expectations and understanding of game elements post-playtest

11

Post-Playtest Survey Results

​

​

​

​

​

​

​

​

​

7 players have been recruited to playtest the game remotely. We designed a short survey that each player must complete after the playtest session.

​

The following data results represent players’ perception and understanding of a variety of game elements post-playtest, including gameplay, controls, mechanics, difficulty, artstyle, story, and overall atmosphere.

​

Most of the rating scores reflect players’ perception of the specific game aspects and can be used directly as the reference for whether they are working well or not. The Level of Difficulty, on the other hand, is determined by whether the ratings satisfied the expectations set by the developer. In this game, the developers expect the game difficulty level to be easy to average according to the overall player experience goal.

​

​

Illum

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

12

• 7/7 players liked the game. Despite any prior confusions with the narrative or the puzzles, 7/7 players enjoyed the explorative and calming gameplay and puzzle that they needed to solve, as well as the visual style, music, and atmosphere.

​

​

​

Gameplay

Overall, there is a positive feedback on the gameplay enjoyment.

Group Average: 4.14/5

Group Median: 4/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

13

Controls

• 7/7 players agreed that the controls are easy to use. This is likely due to the adherence to the industry standard controls, reducing the learning burden for players when they were learning the game.
• Despite that 1/7 players skipped the flashlight tutorial in the beginning, they were still able to return to the tutorial and learn and master the mechanic quickly.

​

​

​

Overall, there is a positive feedback on the ease of use of the controls.

Group Average: 4.71/5

Group Median: 5/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

14

Mechanics - Interaction with Light Friend

3

• 3/7 players expressed neutral feedback on the light friend interaction. These three players were seen not interacting much with the light friend often. This might be because that they were unaware of the ability to interact with the light friends due to the lack of tutorial, or not interested to interact with it.
• 4/7 players agreed that they enjoyed the interaction with the light friend, with 2 expressed that they strongly agree.

​

• Because the developer are intended not to use direct tutorialization to the interaction with the light friend, but to let the players to discover themselves. Possible solution can be to have some animation of light friend moving around near the flashlight and making inquisitive voice, implying that flashlight and the light friend have some relationship for players to discover.

​

​

​

Overall, there is a neutral to positive feedback on the interaction with the light friend.

Group Average: 3.86/5

Group Median: 4/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

15

Mechanics - Connecting

• 4/7 players agreed that they enjoyed the interaction with the light friend, with 2 expressed that they strongly agree. This is likely due to the easy-to-understand controls used in the connecting mechanic, and the visual feedback after successfully connecting everything together.

​

​

​

Overall, there is a positive feedback on the connecting mechanic.

Group Average: 4.57/5

Group Median: 5/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

16

Mechanics - Flashlight

• 7/7 players agreed that they enjoyed the flashlight related game mechanics, with 2 strongly agreed. This might be due to the variety of mechanics players can achieve through using the flashlight, including lighting up the wall paintings, attracting the gems, and interacting with the light friend.

​

​

​

Overall, there is a positive feedback on the flashlight-related mechanics.

Group Average: 4.29/5

Group Median: 4/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

17

Level of Difficulty

• 5/7 players rated the level of difficulty to be average and 2/7 players rated easy. Despite that some players may spend longer time to complete the puzzles or have minor confusion during the play, all of them were able to successfully finishing the game with 15-20 minutes of play.
• This level of difficulty is coherent to the expectation of the developers, who wanted to create a game with easy-to-follow gameplay and focuses on the overall calming and warm-hearted experience.

​

​

​

Overall, players found the gameplay difficult to be easy to average.

Group Average: 2.71/5

Group Median: : 3/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

18

Art Style

• 7/7 players expressed that they liked the visual style of this game, with 5 of them strongly agreed. This showed a very high likeability of the art, and that the visual style is coherent with the overall gameplay experience.

​

​

​

Overall, there is a positive feedback on the art style of the game.

Group Average: 4.71/5

Group Median: 5/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

19

Narrative

• 2/7 players expressed neutral feelings about the story. This might because of the failure to understand the narrative that were hidden in the environment, which could be difficult to discover and interpret.
• 5/7 players understood and liked the story of the game, with 1 strongly agreed. For players who had explored the game world and lighted up all the wall paintings, they were more likely to understand the narrative hidden in the world, and thus liked the story.

​

• Because the developers intended to use the wall paintings to imply the narrative instead of directly tell to the players, possible solution can be to consider adding more wall paintings in the game world, giving more contexts to the story. This can also encourage player exploration in the game, offering a more inquisitive and engaging interaction with the environment.

​

​

​

Overall, there is a neutral to positive feedback on the art style of the game.

Group Average: 3.86/5

Group Median: 4/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

20

Atmosphere

• 7/7 players agreed that they liked the mood of the game. The game provides a calming and warm-hearted environment through the visual, audio, and the mechanics, and was enjoyed by the players.

​

​

​

Overall, there is a positive feedback on the overall mood/atmosphere of the game.

Group Average: 4.57/5

Group Median: 5/5

Insights ⬤ Positive ⬤ Critical ⬤ Neutral 💡 Suggestion

21

Questions?

​

Elsa Liu

zhengzhl@usc.edu

22