ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
INSTRUCTIONS FOR THIS SPREADSHEET (Ideally an online spreadsheet, using Google Sheets or Excel Online)
2
1. Lead Evaluator conducts kickoff meeting, gathers key information on Product Info, Target User Type, and Fundamental Tasks, and inserts this into Product Info tab.
3
2. Lead Evaluator records Happy Path/Typical Path, determins where the step boundaries are, names/describes each step, and enters this into column B of Team Task Evaluation.
4
3. Team assembles, and each rater is given their own tab (Rater 1, Rater 2, etc.) in which to enter data, focusing on cells colored Magenta, Purple, Blue or Light Blue in their tab.
5
4. For each task, raters walk through each step, silently rate in the sheets in their Rater tabs, including their Notes and Rationale for why each step got the rating it did.
6
5. After a task is rated, the team walks through the steps again, reviewing everyone's rating from the Team Task Evaluation tab, then decides on the "Decided on" score, which is what is reported. Group notes and assumptions are documented.
7
6. When all tasks have been rated, the Inter-rater Reliability is calculated using the instructions in the "Determining IRR" tab.
8
9
Colors used in cells of this spreadsheet and their meaning
10
Color NameColor Meaning
11
OrangeRequired input from Lead Evaluator
12
PinkOptional input from Lead Evaluator
13
MagentaInput from PURE Rater 1
14
PurpleInput from PURE Rater 2
15
BlueInput from PURE Rater 3
16
Light BlueInput from PURE Rater 4
17
BlackHeader or section boundary color
18
Dark GreySecondary header color
19
GreyInstructions or information
20
WhiteWhite space or auto-populated data
21
RedPURE SCORE color
22
YellowPURE SCORE color
23
GreenPURE SCORE color
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100