League of Legends Research Study

SI 311: UX & Gaming Research Final Report by Adrian Berrigan

Table of Contents

Table of Contents 1

Executive Summary 2

Research Methods & Introduction 3-8

Post-Match Survey Results 9-11

Observational Study Results 12-13

Wrap-Up Interview Results 14

Limitations & Confounding Variables 15

Takeaways & Recommendations 16-17

Appendices 18

Appendix A: Observational Study Control Group Script 19

Appendix B: Observational Study Treatment Group 1 Script 20

Appendix C: Observational Study Treatment Group 2 Script 21

Appendix D: Wrap-Up Interview Control Group Script 22

Appendix E: Wrap-Up Interview Treatment Group 1 Script 23

Appendix F: Wrap-Up Interview Treatment Group 2 Script 24

Executive Summary

What is League?

League of Legends is a multiplayer online battle arena (MOBA) where you fight in teams of five to destroy your opposing team’s base. Everyone chooses a different character with individual strengths and weaknesses. Matches typically last around 30 minutes and the combat is top down isometric style with similar controls to a RTS/MMO hybrid.

Study Description

I conducted a series of ~ 1-hour playtests (two 5v5 matches) to evaluate the impact that “team chat” and “all chat” have on the emotional and satisfaction levels of League’s active players

Intended Impact

To provide a better understanding of the effect All Chat and Team Chat has on players and determine whether they are beneficial for the game. The end goal of this being an improvement in players’ overall satisfaction and team satisfaction.

2

Study Objectives

01

02

03

04

Compare self-reported emotional and satisfaction levels of players who have access to either team or all chat compared to players with chat disabled

Identify negative effects that these chats have on players, such as increased levels observed of toxicity or harassment

Determine the impact having access to chat has on overall engagement and experience with the game

Observe the emotions that players experience during a League of Legends game and how these emotions differ depending on chat access

3

Research Questions

​

• How does access to team chat and all chat affect the overall satisfaction of players after League of Legends matches?

​

• To what extent do players' emotional states differ depending on whether they have access to team chat, all chat, or no chat at all?

​

• Are player interactions in chat different depending on the game state (winning/losing)
• Pings vs Chat
• Perception of Teammates

​

4

Study Setup

03

04

01

02

Control Group

One: Chat Disabled

Sample

Nine active League of Legends players (10 games played with last month minimum)

Treatment Groups

Two: (All Chat + Team Chat) & (Team Chat Only)

Recruitment

Players recruited via word of mouth

5

Study Setup - Recruitment Details

6

• Participants were recruited via word of mouth on Discord and League Chat.

​

• No incentives were offered to participants

• Participants were screened before they were recruited for study (10 games played with last month minimum)

​

• 17 people were messaged in total with 9 agreeing to participate.

​

Study Setup - Sample Details

7

GENDER

12%

88%

Male

Female

Gold 11%

Masters 11%

Diamond 22%

Platinum 55%

Study Design & Implementation

Assign & Setup

Playtest & Survey

Complete random assignment to guarantee 3 participants per group. Inform player of their group and have them enable/disable chat accordingly

Experience Interview

Observe each player play through two games. After each game have them complete a post-match survey

Participants complete short (under 10 minute) interview about their experience

8

Post-Match Survey

​

• On a scale of 1-5, how would you rate your overall satisfaction with your experience last match?

​

• On a scale of 1-5, how would you rate your overall satisfaction with your teammate's last match?

​

• On a scale of 1-5, how often did you feel the following during the last match
• Angry (Scale of 1-5)
• Frustrated (Scale of 1-5)
• Anxious (Scale of 1-5)
• Stressed (Scale of 1-5)
• Confident (Scale of 1-5)
• Happy (Scale of 1-5)

​

9

Key Post-Match Survey Findings

​

1. Winning played the largest role on reported satisfaction and emotions, but treatment effects were statistically significant as well

​

• Certain self-reported emotions (anger, frustration, and happiness) statistically changed due to the effects of treatment while others (anxiety, confidence, and stress) did not

​

• Strong link between All Chat + Team Chat being enabled and Low Game Satisfaction

​

• Link between Team Chat being enabled and Lower Team Satisfaction

​

​

10

Post-Match Survey Data

11

Observational Study

• Players screen shared via discord as they played their assigned 2 matches

​

• Interactions with players during the games were kept to a minimum due to concern over observational bias. For the study to be successful player interaction being natural was vital

​

• Chat interactions were noted, as was use of the ping system

​

• Player KDA, game state, outcomes, specific chat occurrences (extreme toxicity), good/bad macro play, good team rotations, and more were manually collected via observation

​

12

Observational Findings

A

B

Pings

Participants within the control group used ping based communication at a higher rate than their chat enabled counterparts. This seemingly increased their teams usage of pings and based on my personal observations led to better all around macro, play calls, and rotations

Teammates

Players in the treatment groups were VERY vocal about their teammates (likely due to the nature of the experiment). Multiple participants verbally stated things like “I’m not normally this toxic” after typing negative statements towards their teammates. They seemed to be more aware of chat than they otherwise would have been outside of an experiment.

13

Wrap-Up Interview Overview & Findings

• After both matches concluded players were asked to spend 10 minutes answering questions. These included follow-up questions to things noticed during observational study and questions meant to understand why players felt certain ways and how they perceive chat affecting them (see appendices for script).

​

• 8/9 participants completed wrap-up interviews with one participant leaving due to time constraints

​

• Responses more or less reflective of findings noted in survey and observational results.
• Team chat often hurt team dynamics more than it helped (especially when already losing)
• All chat only made things worse regardless of game state

​

• Nearly all participants gave relatively short feedback and responses. Study fatigue may be a contributing factor

14

Study Limitations

Bias

Confounding Variables

Study participants ended up being recruited were people either I knew or my friends knew. This is means there is a high probability of my study results having certain biases

Sample Size

A confounding variable I didn’t think of is game mode. Some people played norms and some played ranked. Player rank was also not well distributed. No “low elo” players were represented in my sample and they might respond differently

A lot of the quantitative analysis I did does not have enough data to be viewed as statistically significant. I would need a far larger sample size

15

Takeaways

​

• It appears that the conventional wisdom among players that you should “mute all” has some basis to it. Lower overall player satisfaction with All Chat enabled and lower team satisfaction with Team Chat enabled

​

• Based on observations, pings as a communication tool proved to be more effective and satisfying than Team Chat (regardless of game state)

​

• Riot has attempted to remove All Chat in the past, but then changed their mind. This research suggests that they might be correct in that initial decision to remove it

​

• Participants seemed acutely aware of how not having or having access to chat affected them. General perception is All Chat was an overall detriment whereas Team Chat amplifies their opinions (both positive and negative) of their teammates.

​

16

Recommendations & Future Research

• Future research is definitely needed. Better sample recruitment (including low elo representation) and larger sample size is a must.

​

• Focus these studies on All Chat since that had the most negative impact. Riot should continue their efforts towards this. Creation of a player base-wide survey for feedback on All Chat would be a goldmine

​

• All Chat should be disabled by default.

​

• Rank + Win Rate should be controlled for in any future analysis. Regression analysis will be ideal (especially with a larger sample).

​

• Qualitative research into undering why players have worse experiences when All Chat is enabled is an area of need. Riot should conduct user interviews to make progress towards understanding this.

​

​

17

Appendices

18

Appendix A: Observational Study Start (Control Group)

1. Thank you for agreeing to do this study today! My name is Adrian, and I’m a User Researcher currently studying Information Science at the University of Michigan.
2. Today, we will be having you play two matches of League. After each match, you will fill out a survey, and at the end of your 2nd match, I will ask you a series of questions about your experience. If at any time you need to stop the test or if something comes up just let me know. Before we get started, do you have any questions?
3. Now before we begin, I’m going to ask you to disable both All Chat and Team Chat. I’ll give you a moment to do that.
4. *Begin*

19

Appendix B: Observational Study Start (Treatment Group: Team Chat Only)

• Thank you for agreeing to do this study today! My name is Adrian, and I’m a User Researcher currently studying Information Science at the University of Michigan.
• Today, we will be having you play two matches of League. After each match, you will fill out a survey, and at the end of your 2nd match, I will ask you a series of questions about your experience. If at any time you need to stop the test or if something comes up just let me know. Before we get started, do you have any questions?
• Now before we begin, I’m going to ask you to disable All Chat, but keep team chat enabled. I’ll give you a moment to do that. Please feel free to interact with your team as you would normally.
• *Begin*

20

Appendix C: Observational Study Start (Treatment Group: Team Chat + All Chat Enabled)

• Thank you for agreeing to do this study today! My name is Adrian, and I’m a User Researcher currently studying Information Science at the University of Michigan.
• Today, we will be having you play two matches of League. After each match, you will fill out a survey, and at the end of your 2nd match, I will ask you a series of questions about your experience. If at any time you need to stop the test or if something comes up just let me know. Before we get started, do you have any questions?
• Now before we begin, I’m going to ask you to check to make sure Team Chat and All Chat are enabled.. I’ll give you a moment to do that. Please feel free to interact with both your team and enemy team as you would normally.
• *Begin*

21

Appendix D: Wrap-Up Interview (Control)

• How did you feel about not having chat access during your matches? Did you feel that not having access to chat limited your ability to communicate?
• According to your post-match surveys, you felt *insert result*. Do you feel this is an accurate representation of your emotional state and satisfaction levels?
• *Follow-up questions from things noticed during observation*
• How did you utilize pings in the absence of chat?
• Is there anything else you would like to share about your experience today?

22

Appendix E: Wrap-Up Interview (Treatment: Team Chat Only)

• How did you feel about not having access to All Chat during your matches? Did you feel that not having access to all chat changed the way you played in any way?
• Do you feel having access to Team Chat affected your gameplay? Team dynamics? Positive or negative?
• According to your post-match surveys, you felt *insert result*. Do you feel this is an accurate representation of your emotional state and satisfaction levels?
• *Follow-up questions from things noticed during observation*
• Is there anything else you would like to share about your experience today?

23

Appendix F: Wrap-Up Interview (Treatment: Team Chat + All Chat Enabled)

• Do you feel having access to All Chat affected your gameplay? Positive or negative?
• How did Team Chat affect your team dynamics? Positive or negative?
• According to your post-match surveys, you felt *insert result*. Do you feel this is an accurate representation of your emotional state and satisfaction levels?
• *Follow-up questions from things noticed during observation*
• Is there anything else you would like to share about your experience today?

24