1 of 54

Can we talk?

Usability testing of Canvas Discussions Redesign �with instructors and students

1

2 of 54

Presented by

Jennifer Englund

University of

Minnesota

Rebecca George-Burrs

University of

Minnesota

Sam Garza

Instructure

2

3 of 54

Session goal

3

4 of 54

Usability evaluation partnership

4

5 of 54

UX research at the University of Minnesota

5

Stakeholders

Academic

Technology community

Learning technology vendors

Usability Services

ATSS,

AT Tools,

Video Conferencing Services

6 of 54

UX research at Instructure

Product and feature discovery and research are crucial for understanding our users and their needs
We gather market research, industry data, and conduct user interviews and tests
Design solutions are explored through Alpha and Beta releases in our research and discovery lifecycle
We use several tools for user testing, including Maze, Userlytics, Zoom, and Fable (for accessibility testing)
These tools help us conduct moderated and unmoderated testing to gather thorough data that drives our design solutions

6

7 of 54

Chime in poll

Access the poll

access code 717-472

7

8 of 54

Usability evaluation process for Canvas Discussion tool

8

9 of 54

Why this tool now?

9

10 of 54

Usability evaluation process overview

10

Develop project plan with Usability Services

Conduct usability evaluation sessions with 4 students and 4 instructors

Debrief and review issues

Image sources: collaborate by Nithinan Tatah, Listen by Soky Art, examination by Basicon - all from Noun Project

11 of 54

Focus question

What prevents people from engaging with one another using the new Discussion tool in a satisfying way?

11

Photo by Saif71.com on Unsplash

12 of 54

Participant recruitment

Instructors from across the UMN system who had created or graded a Canvas discussion were invited to participate in the usability evaluation sessions.

Students from across the UMN system who had participated in a Canvas discussion were invited to participate in the usability evaluation sessions.

12

13 of 54

Student Scenarios

Encompassed the following tasks:

Complete a discussion post in response to an instructor prompt
Complete a reply to another student’s post and tag a student in it
Find a reply that an instructor or student made to your post and respond to it
Find your grade for a discussion
Find feedback from your instructor on your discussion

13

14 of 54

Instructor Scenarios

Encompassed the following tasks:

Create a discussion with multiple due dates
Create a discussion that requires students to post before they can read others’ posts
Find the post(s) from specific students
Review student posts in order to grade them
Create a small group discussion
Allow students to create discussion topics

14

15 of 54

Instructure collaboration

Enable instructors and students to test and provide valuable feedback
Gain insights into participants' challenges beyond simply hearing "it's broken"
Determine the appropriate solutions and prioritize necessary fixes

15

Photo by charlesdeluvio on Unsplash

16 of 54

What UX testing uncovered and how we plan to move forward

16

17 of 54

Feature: Mentions (@...)

17

18 of 54

UX Discoveries: Mention Feature

Instructors were unsure of what it did and how to use it�
Students assumed it worked like other social media platforms�
Both users wondered if/how they would be notified of the mention outside of reading it in the thread

18

19 of 54

Response: Mention Feature

Instructure

Created a resource detailing how new functionality works

UMN

Provide communications to students and instructors about the feature (link to Instructure resource) and how to best use it in a online class discussion

19

With what we noticed when observing both students and instructors use this new feature, we decided that we want to target some communications to both user groups. We want to make sure that both users groups are clear that this does indeed work similarly to how other social media platforms use the tool but that the individual user must have the specific “Discussions - New Mention” notification turned on to receive any notifications. We will point to the article about the feature that Instructure has in their knowledge base to clearly inform everyone how this feature works and how to make sure you get notified of any mentions. Instructure has already made a knowledge article about this feature, detailing how it works and adding a link to the article that instructs users how to turn on this notification, so we will be using that knowledge article in our communications.

20 of 54

Feature: Anonymous Discussion

20

The Discussion redesign also comes with that ability to create an anonymous discussion. There are 3 new settings that users can choose, including turning off this feature (which is the default setting), selecting partial anonymity (where students can choose to reveal their name and profile picture or interact anonymously in the discussion), and thirdly, instructors can create a completely anonymous discussion, where student names and profile pictures will be hidden.

It is important to note that if a discussion is set for either partial or total anonymity, that the discussion cannot be graded or set as a group discussion.

Actual wording of the options shown in the screenshot:

Off: students names and profile pictures will be visible to other members of this course
Partial: students can choose to reveal their name and profile picture
Full: students names and profile pictures will be hidden

21 of 54

UX Discoveries: Anonymous Discussion

Instructors didn’t see the options for an anonymous discussion when first creating a new Discussion.
When instructors tried to go back and edit the discussion, the selection for the discussion to be anonymous were no longer available to be checked (grayed out).

21

22 of 54

Response: Anonymous Discussion

Instructure

Updated the feature to allow the editing of anonymous discussion options until a learner has participated in the discussion

UMN

Including this information in a communication to instructors

22

Other similar feature options that are present in Assignments and Quizzes are still editable after saving the initial activity. Features will lock once a learner has engaged in the activity, but as an instructor first sets up the activity, the options stay available past the initial saving the activity.

The product manager of the Discussion tool saw the inconsistency of those options being unavailable to edit after the instructor first makes the discussion. She went back to her team with this issue, and they are changing the settings to be consistent with how options work with Assignments and Quizzes. All options will be available for editing after the instructor has saved the activity as long as no learners have participated in the activity.

We will be introducing the new feature options for making a discussion anonymous, making sure to note that this type of discussion is not available for graded discussions or group discussions. We will use any knowledge articles that Instructure makes about this feature option to further communicate about it to instructors.

23 of 54

Feature: Instructor Feedback Options

23

24 of 54

UX Discovery: Instructor Feedback Options

Confusion with feedback after selecting “view the full discussion”

24

25 of 54

Response: Instructor Feedback Options

Instructure

Feedback about this confusion has been sent to the team that works on Speedgrader
Prioritized improvements specific to Discussions in Speedgrader

UMN

Draft information for the “Tips and Tricks” section to September 2024 issue of Teaching with Technology newsletter
Add information to the Canvas Essentials course (for new instructors) about grading discussions

25

Our Instructure partner saw how this could be very confusing to instructors, and she has sent the feedback from the UX study to the team who works on Speedgrader.

In the meantime, we have some extra communications to more clearly help instructors know the difference to reply within a discussion vs. give feedback via Speedgrader. Our current plan is to have a Tip and Trick section to September 2024 issue of Teaching with Technology newsletter focus on this issue. We also think it best to take a moment to educate new instructors about the possible confusion when grading Discussions in Speedgrader. We are adding information to the Canvas Essentials course (for new instructors and TAs) about grading discussions to highlight that individual student feedback goes in the Speedgrader column vs. feedback for the class to consider should go in the Discussions thread in the center of the page.

26 of 54

Feature: Canvas Groups

26

27 of 54

UX Discovery: Canvas Groups

Instructors expressed strong feelings about being annoyed with the Groups feature and how it integrates with Discussions

Participant #2: "I've had trouble with setting up groups...peer review in groups was a nightmare"
Participant #3: creating group discussion forums is “slow and ponderous.”

27

28 of 54

Response: Canvas Groups

Instructure

Feedback from UX study has kickstarted a discovery group to improve the Groups tool

UMN

Already have extra support documentation for Groups
We will communicate out any information as Instructure improves upon this tool

28

29 of 54

Takeaways

29

30 of 54

Next Steps: University of Minnesota

Invite Instructure to upcoming UX studies about Canvas
Connect with other product managers
Use evaluation findings to inform communication and training

30

Photo by charlesdeluvio on Unsplash

31 of 54

Next Steps: Instructure

Continue partnering with institutions

Communicate findings with all stakeholders

Incorporate more feedback and discovery into our work going forward

31

32 of 54

Poll results

We welcome participation in future UX studies: complete the form if you would like to be added to the interest list.

32

Photo by Katie Moum on Unsplash

33 of 54

Questions?

33

34 of 54

Try It Out

Available now as a Feature Option on a course by course basis
Unless your college has forced it on or off in your sub-account
Navigate to your course > Settings > Feature Options tab

34

35 of 54

Enforcement Date

You can toggle the Feature Option on and off
July 20, 2024, it's forced on by Instructure
It's not like Classic Quizzes and New Quizzes
You do not need to 'migrate' or 'upgrade' existing Discussion Topics

Instructor Resources for Discussion Redesign

Learn about other new features not covered in our presentation.

35

36 of 54

Section Title

Section Subtitle

36

37 of 54

Title

Body

37

38 of 54

Section Title

Description

38

39 of 54

Title

Body 1

Body 2

39

40 of 54

Title

Body

40

41 of 54

Title

Body

41

42 of 54

Section Title

Section Subtitle

42

43 of 54

Section Title

Description

43

44 of 54

Title

Body 2

Body 1

44

45 of 54

This is a long quote treatment. Add your quote here.

Quote Attribution

45

46 of 54

46

47 of 54

47

48 of 54

48

49 of 54

49

50 of 54

50

51 of 54

Title Here

Subtitle here

51

52 of 54

Title Here

Subtitle here

53 of 54

Title Here

Subtitle here

53

54 of 54

Title Here

Subtitle here

54